In this talk, Aish and Sudeep present techniques for visualizing large scale machine learning systems in Spark. They show the techniques that are employed by Netflix to understand and refine the machine learning models behind Netflix’s famous recommender systems that are used to personalize the Netflix experience for their 80 millions members around the world. They will discuss how visualizing the rich ecosystem of algorithms not only helps to monitor and interpret how different pieces of the recommendation system orchestrate to generate the final experience, but also to detect anomalies, inconsistencies and temporal and spatial patterns that would be very hard to reveal otherwise. These explorations often provide an intuitive guideline to new directions in algorithmic development. They also present a new OSS Scala library, Vegas, that aims to be the “missing MatPlotLib” for Spark/Scala.
Aish is a Research Manager at Netflix. He leads an applied research team working on the core recommender system at Netflix. His work combines cutting edge machine learning along with large scale software engineering. Prior to Netflix, Aish was head of data science at Opentable, and also founded his own startup solving large-scale combinatorial optimization problems.