Antony is currently an engineer on the Personalization Infrastructure team at Netflix. His primary focus is on building various big data infrastructure components using Spark that help our algorithmic engineers to innovate faster and improve personalization for our members. In the past, he has built machine learning pipelines at Salesforce and AgileOne.
Spark, GraphX and MLlib play an important role in several current generation machine learning platforms. This talk outlines how we integrated Spark, Python, R, Docker and other toolkits into a scalable orchestration framework that supports heterogeneous workloads. We will demonstrate how our users design and run machine learning pipelines that seamlessly exchange data and artifacts between Spark jobs and other components. This is deployed on a single multi-tenant cluster that allows engineers to choose between multiple Spark versions with built in capabilities to monitor long running workflows. We will explain our journey building this framework atop Mesos and our special treatment of Spark within it.