Dr. Elephant for Monitoring and Tuning Apache Spark Jobs on Hadoop – Databricks

Dr. Elephant for Monitoring and Tuning Apache Spark Jobs on Hadoop

Download Slides

Dr. Elephant helps improve Spark and Hadoop developer productivity and increase cluster efficiency by making clear recommendations on how to tune workloads and configurations. Originally developed by LinkedIn, Dr. Elephant is now in use at multiple sites.
This session will explore how Dr. Elephant works, the data it collects from Spark environments and the customizable heuristics that generate tuning recommendations. Learn how Dr. Elephant can be used to improve production cluster operations, help developers avoid common issues, and green light applications for use on production clusters.

Session hashtag: #SFdev18

« back
About Carl Steinbach

Carl Steinbach is a Senior Staff Software Engineer at LinkedIn where he leads the Grid Platform Team. In 2014 he launched the Dr. Elephant project with the goal of improving user productivity and the overall efficiency of LinkedIn's cluster infrastructure.

About Simon King

Simon King is a software engineer at Pepperdata where he is the engineering lead on Pepperdata work with both the open source Dr. Elephant project and integration of Dr. Elephant technology with the Pepperdata Application Profiler. Prior to Pepperdata, Simon worked at Microsoft and Yahoo on a variety of technology projects.