Yuval Degani is a Senior Manager of Engineering in Mellanox Technologies, leading a team of Software Engineers based in the San Francisco Bay Area. His team’s focus is introducing new network acceleration technologies to Big Data and Machine Learning frameworks. Before that, Yuval was a developer, an architect and later a team lead in the areas of low-level kernel development for cutting edge high-performance network devices. Yuval holds a BSc in Computer Science from the Technion Institute of Technology, Israel.
Microsoft Azure's advanced compute and network infrastructure allows Spark to run in the cloud without compromising on performance. With the growing arsenal of hardware offloads available on cloud VMs, owning and maintaining bleeding edge hardware is no longer a prerequisite for accelerated compute. In this talk, we will demonstrate how hardware accelerations in Azure can be utilized to speed-up Spark jobs seamlessly, with the aid of RDMA (Remote Direct Memory Access) support in the VM. We will demonstrate use cases of benchmarks and real-world applications, that achieve impressive performance improvements with minimal configuration. Session hashtag: #HWCSAIS18
The opportunity in accelerating Spark by improving its network data transfer facilities has been under much debate in the last few years. RDMA (remote direct memory access) is a network acceleration technology that is very prominent in the HPC (high-performance computing) world, but has not yet made its way to mainstream Apache Spark. Proper implementation of RDMA in network-oriented applications can improve scalability, throughput, latency and CPU utilization. In this talk we are going to present a new RDMA solution for Apache Spark that shows amazing improvements in multiple Spark use cases. The solution is under development in our labs, and is going to be released to the public as an open-source plug-in. Session hashtag: #EUres3