Jeff Pang is a Principal Engineer at Databricks on the Platform team. He has lead the development of several parts of the Databricks Unified Analytics Platform, including Databricks Notebooks, Databricks Community Edition, and Azure Databricks. Prior to Databricks, he worked on large-scale data analytics at AT&T Labs – Research. He obtained his PhD from Carnegie Mellon University and a BA from U.C. Berkeley.
The cloud has become one of the most attractive ways for enterprises to purchase software, but it requires building products in a very different way from traditional software. I will explain some of these challenges based on my experience at Databricks, a startup that provides a data analytics platform as a service on AWS and Azure. Databricks manages millions of VMs per day to run data engineering and machine learning workloads using Apache Spark, TensorFlow, Python and other software for thousands of customers.