Building Sessionization Pipeline at Scale with Databricks Delta

Download Slides

Comcast has made a concerted effort to transform itself from a cable/ISP company to a technology company. Data-driven decision making is at the heart of this transformation, and we use data to understand how customers interact with our products, and we see data as the most truthful representation of the voice of our customer. My team, Product Analytics & behavior science (PABS) team plays the role as interpreter, transforming data into consumable insights.

The X1 entertainment operating system, is one of the largest video streaming platforms in the world, and our customers consume more than a billion hours of content a week on X1. Our team consumes X1 telemetry at a rate of more than 25TBs of data per day and uses this data to inform our product teams members about the performance of and engagement with the platform. We also use this data to research customer behaviors to help better inform our product team members about areas of opportunity in our products, which range from fixing bugs to creating new features.

To power these insights, we need to have a reliable real-time data pipelines to deliver these insights, and we need our data scientists and data engineers to be able to quickly and efficiently be able to develop and commit new code to ensure we can measure new features the product teams are developing. To do this in an environment at this scale, we have been using Databricks, and Databricks delta to gain operational efficiencies, optimization and cost savings.

Some of the features from delta that we took advantage of to achieve the desired levels of efficiencies, optimization and cost savings are:

· Distributed writes to s3 (essentially eliminating 500 errors)

· s3 log with fast reads and ACID transactions (massive increases in s3 scans/reads, and enabling consistent views of the bucket/table)

· Vacuum

· Pptimize (which has allowed us to reduce a 640 node job to 40, and massively increase efficiencies of our clusters as well as our DS/DE’s)


Try Databricks
See More Spark + AI Summit in San Francisco 2019 Videos

« back
About Jim Forsythe

Jim Forsythe leads the Product Analytics & Behavior Science (PABS) team for the Technology, Product and Xperience organization at Comcast where he is responsible for transforming bits of data into consumable, productive insights. Jim's day is challenged with building data pipelines, researching new ideas, developing key metrics and informing data-driven decision making. Prior to Comcast, Jim led data science teams for a fortune 500 management consulting firm. He specialized in large scale product analytics, cloud platforms, user behavior research and retention modeling for new product initiatives.

About Eliot Knudsen

Eliot Knudsen is a Solutions Architect at Databricks, working with strategic clients on technical implementations and cloud deployments. Prior to Databricks Eliot was a Lead Data Scientist at Tamr where he applied machine learning to large-scale data integration in the Fortune 100. Eliot is a graduate of Carnegie Mellon University where he studied computational mathematics, statistics and machine learning.