Towards the True Elasticity of Spark

Download Slides

How well an analytics engine can respond to changing workload demands and resource availability will greatly determine its usefulness and adoption rate. In this talk, we will present a study of the effectiveness of the elasticity property of Spark when deployed on popular resource managers such as Mesos and YARN. In particular, we investigate how well Spark workloads running on Mesos and YARN clusters behave as nodes are added and removed from the clusters. Key measurements include workload runtime, resource utilization delay, average task waiting time, disk I/O and network bandwidth consumption. We then analyze the impact of changing key scheduling parameters (e.g., locality wait time, locality preference, granularity of locality wait time, speculation, resource re-offer interval, etc.) on the above measurements. Lessons from this work will enable the building of effective auto-scaling infrastructure for Spark in a cloud environment.