Michael Feiman - Databricks

Michael Feiman

Principal Product Architect, IBM Spectrum Computing

Michael has been working on distributed computing, grid, cloud and big data for the past 25 years focusing on HPC, Hadoop and Spark workload scheduling and resource management architecture. Michael works currently as principal product architect at IBM Spectrum Computing, he holds an MS in Electrical Engineering.


Dynamic Priorities for Apache Spark Application’s Resource AllocationsSummit 2018

This talk presents a use case of dynamic service-level agreement (SLA) control for Spark applications that are implemented through resource allocation priorities; which can be assigned to each Spark application at submission and then dynamically modified during the whole lifecycle of the application. The Spark application priority is translated by the scheduler to resource allocation adjustments for executors; and include either preemptive or non-preemptive release of resources. The priorities translation algorithm is determined by a scheduling policy to either define the order of executor allocations between applications or calculate the priority-weighed allocation shares among all submitted applications from the entire resource pool. Driver's resource allocation adjustments are always expected to follow the first-in, first-out (FIFO) time sequence. The drivers are expected to be non-preemptive, so that the dynamic SLA for the drivers can control only the driver's start time, then the application priority is used to adjust the order in the driver's pending queue. This presentation describes the extensions to Spark RESTful APIs to dynamically modify previously submitted Spark application's priorities, monitor current priority, and record the updated priorities in the application's events log. We also discuss user permissions for authorizing Spark application priority assignment and its modification and feature extensions. In conclusion, testing results for SLA control of real word Spark applications by using the applications priorities are provided. Session hashtag: #Dev6SAIS