Spark has made writing big data pipelines much easier than before. But a lot of effort is required to maintain performant and stable data pipelines in production over time. Did I choose the right type of infrastructure for my application? Did I set the Spark configurations correctly? Can my application keep running smoothly as the volume of ingested data grows over time? How to make sure that my pipeline always finishes on time and meets its SLA?
These questions are not easy to answer even for a handful of jobs, and this maintenance work can become a real burden as you scale to dozens, hundreds, or thousands of jobs. This talk will review what we found to be the most useful piece of information and parameters to look at for manual tuning, and the different options available to engineers who want to automate this work, from open-source tools to managed services provided by the data platform or third parties like the Data Mechanics platform.
Jean-Yves is the CEO and Co-Founder of Data Mechanics, an automated performance tuning platform for Apache Spark which works on top of any cloud-based data platform. Prior to that, he was a software engineer and team lead at Databricks where he grew the management of Spark infrastructure from early startup days to hundreds of thousands of nodes launched in the cloud per day. Jean-Yves is passionate about simplifying data engineering operations and making it easy for anyone to operate performant and stable data pipelines at scale. He graduated from Ecole Polytechnique and Stanford University.
Julien is the CTO and Co-Founder of Data Mechanics, a YCombinator-backed startup with the mission to automate the often tedious mechanical work performed manually by data engineers today, starting with Spark performance and stability tuning. He previously worked as a data scientist on optimizing BlaBlaCar’s world-leading carpooling marketplace, and led the data team at the website UX optimization platform ContentSquare. He graduated from Ecole Polytechnique and ETH Zurich.