Adding nodes at runtime (Upscale) to already running Spark-on-Yarn clusters is fairly easy. But taking away these nodes (Downscale) when the workload is low at some later point of time is a difficult problem. To remove a node from a running cluster, we need to make sure that it is not used for compute as well as storage.
But on production workloads, we see that many of the nodes can’t be taken away because:
In this talk, we will talk about how we can improve downscaling in Spark-on-YARN clusters under the presence of such constraints. We will cover changes in scheduling strategy for container allocation in YARN and Spark task scheduler which together helps us achieve better packing of containers. This makes sure that containers are defragmented on fewer set of nodes and thus some nodes don’t have any compute. In addition to this, we will also cover enhancements to Spark driver and External Shuffle Service (ESS) which helps us to proactively delete shuffle data which we already know has been consumed. This makes sure that nodes are not holding any unnecessary shuffle data – thus freeing them from storage and hence available for reclamation for faster downscaling.
Prakhar Jain is a member of the technical staff at Qubole, where he works on Spark. Prakhar holds a bachelor of computer science engineering from the Indian Institute of Technology, Bombay, India.
I work as a Software Engineer at Qubole. At Qubole, I worked mostly on Spark and on parts of YARN. Worked on Spark SQL optimizer, core, autoscaling in cloud, optimizing S3 access in Spark etc. Previously worked at MapR on similar big data systems after completing Masters at Arizona state university.