Lyft is on the mission to improve people’s lives with the world’s best transportation. As part of this mission Lyft invests heavily in open source infrastructure and tooling. At Lyft Kubernetes has emerged as the next generation of cloud native infrastructure to support a wide variety of distributed workloads. Apache Spark at Lyft has evolved to solve both Machine Learning and large scale ETL workloads. By combining the flexibility of Kubernetes with the data processing power of Apache Spark, Lyft is able to drive ETL data processing to a different level. In this talk, Li Gao and Rohit Menon will talk about challenges the Lyft team faced and solutions they developed to support Apache Spark on Kubernetes in production and at scale. Topics Include: – Key traits of Apache Spark on Kubernetes. – Deep dive into Lyft’s multi-cluster setup and operationality to handle petabytes of production data. – How Lyft extends and enhances Apache Spark to support capabilities such as Spark pod life cycle metrics and state management, resource prioritization, and queuing and throttling. – Dynamic job scale estimation and runtime dynamic job configuration. – How Lyft powers internal Data Scientists, Business Analysts, and Data Engineers via a multi-cluster setup.
Li Gao is the tech lead in the cloud native spark compute initiative at Lyft. Prior to Lyft, Li worked at Salesforce, Fitbit, Marin Software, and a few startups etc. on various technical leadership positions on cloud native and hybrid cloud data platforms at scale. Besides Spark, Li has scaled and productionized other open source projects, such as Presto, Apache HBase, Apache Phoenix, Apache Kafka, Apache Airflow, Apache Hive, and Apache Cassandra.
Rohit Menon is a Software Engineer on the Data Platform team at Lyft. Rohit's primary area of focus is building and scaling out the Spark and Hive Infrastructure for ETL and Machine learning use cases. Previously he was one of the early engineers on the Data Platform team at Electronic Arts (EA, Inc) focussing on Data Compute and Security. Before EA, Rohit was a Software Engineer at VMware working on Application Deployment Framework on Virtualized environments.