Ankit Agarwal

Production Engineering Manager, Facebook

Ankit Agarwal is Production Engineering manager at Facebook. His team is responsible for handling the productionizing aspects of Spark at FB.

Past sessions

Summit 2019 Scaling Apache Spark at Facebook

April 23, 2019 05:00 PM PT

Spark started at Facebook as an experiment when the project was still in its early phases. Spark's appeal stemmed from its ease of use and an integrated environment to run SQL, MLlib, and custom applications. At that time the system was used by a handful of people to process small amounts of data. However, we've come a long way since then. Currently, Spark is one of the primary SQL engines at Facebook in addition to being the primary system for writing custom batch applications. This talk will cover the story of how we optimized, tuned and scaled Apache Spark at Facebook to run on 10s of thousands of machines, processing 100s of petabytes of data, and used by 1000s of data scientists, engineers and product analysts every day.

In this talk, we'll focus on three areas:
-Scaling Compute: How Facebook runs Spark efficiently and reliably on tens of thousands of heterogenous machines in disaggregated (shared-storage) clusters.
-Optimizing Core Engine: How we continuously tune, optimize and add features to the core engine in order to maximize the useful work done per second.
-Scaling Users: How we make Spark easy to use, and faster to debug to seamlessly onboard new users.