Dongjoon Hyun

Software Engineer, Apple

I’m a software engineer and currently work for Apple. My main focus area is a fast and efficient data processing. At Apple, as an Apache Spark PMC member and committer and an Apache ORC PMC member and committer, I developed and maintained the internal distributions powered by Apache Spark and Apache ORC.

UPCOMING SESSIONS

PAST SESSIONS

Native Support of Prometheus Monitoring in Apache Spark 3.0Summit 2020

All production environment requires monitoring and alerting. Apache Spark also has a configurable metrics system in order to allow users to report Spark metrics to a variety of sinks. Prometheus is one of the popular open-source monitoring and alerting toolkits which is used with Apache Spark together. Previously, users can use

  1. a combination of Prometheus JMX exporter and Apache Spark JMXSink
  2. 3rd party libraries
  3. implement a custom Sink for more complex metrics like GPU resource usage

Apache Spark 3.0.0 will add another easy way to support Prometheus for general use cases. In this talk, we will talk about the followings and show a demo.

  1. How to enable new Prometheus features.
  2. What kind of metrics are available.
  3. General tips for monitoring and alerting on structured streaming jobs. (Spark side / Prometheus side)

Currently, Apache Spark exposes metrics at Master/Worker/Driver/Executor to integrate with the existing Prometheus server easily with a less effort. This is already available with Apache Spark 3.0.0-preview and preview2. You can try it right now.

Organized by Databricks
If you have questions, or would like information on sponsoring a Spark + AI Summit, please contact organizers@spark-summit.org.

Apache, Apache Spark, Spark, and the Spark logo are trademarks of the Apache Software Foundation. The Apache Software Foundation has no affiliation with and does not endorse the materials provided at this event.