Working with data and reporting for more then 10 years. Started in Financial sector and moved to internet Marketplace in recent years where I make sure business users can easily access data and get actionable insights.
Last year we decided to build an in-house solution for Funnel analysis which should be accessible to our business user through our BI tool. Backend part should run on Ap;ache Spark and since the BI tool can only run SQL queries that implies that the solution is a pure Spark SQL implementation of Funnel analysis. In this talk we will cover various Spark SQL features we have used to optimize query performance and implement various filters which enable end users to get actionable insights. KEY TAKEAWAYS: - single query approach to Funnel analysis (can be applied to any funnel-like problem) - using window functions to ensure ordering of the events in the funnel - examples of higher order functions to calculate funnel metrics