Accelerated Spark on Azure: Seamless and Scalable Hardware Offloads in the Cloud – Databricks

Accelerated Spark on Azure: Seamless and Scalable Hardware Offloads in the Cloud

Download Slides

Microsoft Azure’s advanced compute and network infrastructure allows Spark to run in the cloud without compromising on performance. With the growing arsenal of hardware offloads available on cloud VMs, owning and maintaining bleeding edge hardware is no longer a prerequisite for accelerated compute.

In this talk, we will demonstrate how hardware accelerations in Azure can be utilized to speed-up Spark jobs seamlessly, with the aid of RDMA (Remote Direct Memory Access) support in the VM. We will demonstrate use cases of benchmarks and real-world applications, that achieve impressive performance improvements with minimal configuration.

Session hashtag: #HWCSAIS18

« back
About Yuval Degani

Yuval has recently joined LinkedIn’s Data Infrastructure team as a Staff Software Engineer, where he is focused on scaling and developing new features for Hadoop and Spark. Before that, Yuval was a Senior Manager of Engineering at Mellanox Technologies, leading a team working on introducing new network acceleration technologies to Big Data and Machine Learning frameworks. Prior to his work in the Big Data and AI fields, Yuval was a developer, an architect, and later a team leader in the areas of low-level kernel development for cutting-edge high-performance network devices. Yuval holds a BSc in Computer Science from the Technion Institute of Technology, Israel.

About Evan Burness

Evan Burness is a Principal Program Manager on the HPC & Big Compute team at Microsoft Azure. There, Evan helps set the technical and strategic direction for Azure's HPC products delivered to thousands of users from public and private sector research communities. Evan also helps support the unique and exclusive partnership between Microsoft and Cray around cloud-based and world-class supercomputing. Prior to joining the Azure team, Evan served as Director of HPC at Cycle Computing, and as the Program Manager for the Private Sector at the National Center for Supercomputing Applications (NCSA) at the University of Illinois at Urbana-Champaign.