Lessons Learned from Managing Thousands of Production Apache Spark Clusters Daily

Download Slides

At Databricks, we have a unique view into hundreds different companies using Apache Spark for development and production use-cases, from their support tickets and forum posts. Having seen so many different workflows and applications, some discernible patterns emerge when looking at common manageability, debugging, and visibility issues that our users run into. This talk will first show some representatives of these common issues. Then, we will show you what we have done and have been working on in Databricks to make Spark clusters easier to manage, monitor, and debug.
Session hashtag: #SFexp19