Learn about the Big Data Processing ecosystem at Netflix and how Apache Spark sits in this platform. I talk about typical data flows and data pipeline architectures that are used in Netflix and address how Spark is helping us gain efficiency in our processes. As a bonus – i’ll touch on some unconventional use-cases contrary to typical warehousing / analytics solutions that are being served by Apache Spark.
Rohan Sharma is a Senior Data Engineer at Netflix Inc. building data pipelines in Apache Spark to consume from the Peta-Byte Scale Data Warehouse and convert it into actionable insights. In addition, he is on the advisory council of IBM Spark TC and has helped drive improvements in the Spark Ecosystem for the community. Prior to Netflix he worked for Goldman Sachs Inc. in New York for 9 Years and held Data Engineer, Data Architect and Platform Engineer roles.