Fire in the Sky: An Introduction to Monitoring Apache Spark in the Cloud - Databricks

Fire in the Sky: An Introduction to Monitoring Apache Spark in the Cloud

Download Slides

Writing intelligent cloud native applications is hard enough when things go well, but what happens when there are performance and debugging issues that arise during production? Inspecting the logs is a good start, but what if the logs don’t show the whole picture? Now you have to go deeper, examining the live performance metrics that are generated by Spark, or even deploying specialized microservices to monitor and act upon that data. Spark provides several built-in sinks for exposing metrics data about the internal state of its executors and drivers, but getting at that information when your cluster is in the cloud can be a time consuming and arduous process. In this presentation, Michael McCune will walk through the options available for gaining access to the metrics data even when a Spark cluster lives in a cloud native containerized environment. Attendees will see demonstrations of techniques that will help them to integrate a full-fledged metrics story into their deployments. Michael will also discuss the pain points and challenges around publishing this data outside of the cloud and explain how to overcome them. In this talk you will learn about: Deploying metrics sinks as microservices, Common configuration options, and Accessing metrics data through a variety of mechanisms.
Session hashtag: #EUdev11

About Michael McCune

Michael McCune is a software developer in Red Hat's emerging technology group. He is an active contributor to several radanalytics.io projects, as well as being a core reviewer for the OpenStack API Working Group. Since joining Red Hat three years ago, he has been developing and deploying applications for cloud platforms. Prior to his career at Red Hat, Michael developed Linux based software for embedded global positioning systems.