Lessons Learned From Running Spark On Docker - Databricks

Lessons Learned From Running Spark On Docker

Download Slides

Today, most any application can be “Dockerized”. However, there are special challenges when deploying a distributed application such as Spark on containers. This session will describe how to overcome these challenges in deploying Spark on Docker containers, with several practical tips and techniques for running Spark in a container environment. Containers are typically used to run non-distributed applications on a single host. There are significant real-world enterprise requirements that need to be addressed when running a distributed application in a secure multi-host container environment.

There are also some decisions that need to be made about the tools and infrastructure. For example, there are a number of different container managers, orchestration frameworks, and resource schedulers available today including Mesos, Kubernetes, Docker Swarm, and more. Each has its own strengths and weaknesses; each has unique characteristics that may be being suitable, or unsuitable, for Spark. Understanding these differences is critical to the successful deployment of Spark on Docker containers. This session will describe the work done by the BlueData engineering team to run Spark inside containers, on a distributed platform, including the evaluation of various orchestration frameworks and lessons learned. You will learn how to apply practical networking and storage techniques to achieve high performance and agility in a distributed, container environment.

Learn more:

  • Lessons Learned from Dockerizing Spark Workloads
  • Running Spark Inside Docker Containers: From Workload to Cluster


  • « back
    About Tom Phelan

    Tom Phelan is co-founder and chief architect of BlueData. Prior to BlueData, Tom was an early employee at VMware and as senior staff engineer was a key member of the ESX storage architecture team. During his 10-year stint at VMware, he designed and developed the ESX storage I/O load-balancing subsystem and modular “pluggable storage architecture.” He went on to lead teams working on many key storage initiatives, such as the cloud storage gateway and vFlash. Earlier, he was a member of the original team at Silicon Graphics that designed and implemented XFS, the first commercially available 64 bit file system.