Scaling Data Analytics Workloads on Databricks - Databricks

Scaling Data Analytics Workloads on Databricks

Imagine an organization with thousands of users who want to run data analytics workloads. These users shouldn’t have to worry about provisioning instances from a cloud provider, deploying a runtime processing engine, scaling resources based on utilization, or ensuring their data is secure. Nor should the organization’s system administrators.

In this talk we will highlight some of the exciting problems we’re working on at Databricks in order to meet the demands of organizations that are analyzing data at scale. In particular, data engineers attending this session will walk away with learning how we:

  • Manage a typical query lifetime through the Databricks software stack
  • Dynamically allocate resources to satisfy the elastic demands of a single cluster
  • Isolate the data and the generated state within a large organization with multiple clusters


  • « back
About Bogdan Ghit

Bogdan Ghit is a computer scientist and software engineer at Databricks, where he works on optimizing the SQL performance of Apache Spark. Prior to joining Databricks, Bogdan pursued his PhD at Delft University of Technology where he worked broadly on datacenter scheduling with a focus on data analytics frameworks such as Hadoop and Spark. His thesis has led to a large number of publications in top conferences such as ACM Sigmetrics and ACM HPDC.

About Chris Stevens

Chris Stevens is a software engineer at Databricks with a background in operating systems. Previously, he worked on the Windows NT kernel before going on to develop Minoca OS from scratch.