Native Support of Prometheus Monitoring in Apache Spark 3.0

Download Slides

All production environment requires monitoring and alerting. Apache Spark also has a configurable metrics system in order to allow users to report Spark metrics to a variety of sinks. Prometheus is one of the popular open-source monitoring and alerting toolkits which is used with Apache Spark together. Previously, users can use

  1. a combination of Prometheus JMX exporter and Apache Spark JMXSink
  2. 3rd party libraries
  3. implement a custom Sink for more complex metrics like GPU resource usage

Apache Spark 3.0.0 will add another easy way to support Prometheus for general use cases. In this talk, we will talk about the followings and show a demo.

  1. How to enable new Prometheus features.
  2. What kind of metrics are available.
  3. General tips for monitoring and alerting on structured streaming jobs. (Spark side / Prometheus side)

Currently, Apache Spark exposes metrics at Master/Worker/Driver/Executor to integrate with the existing Prometheus server easily with a less effort. This is already available with Apache Spark 3.0.0-preview and preview2. You can try it right now.

Try Databricks
« back
About Dongjoon Hyun


I'm a software engineer and currently work for Apple. My main focus area is a fast and efficient data processing. At Apple, as an Apache Spark PMC member and committer and an Apache ORC PMC member and committer, I developed and maintained the internal distributions powered by Apache Spark and Apache ORC.

About DB Tsai


DB Tsai is an Apache Spark PMC / Committer and an open source and big data engineer at Apple. He implemented several algorithms including linear models with Elastici-Net (L1/L2) regularization using LBFGS/OWL-QN optimizers in Apache Spark. Prior to joining Apple, DB worked on Personalized Recommendation ML Algorithms at Netflix. DB was a Ph.D. candidate in Applied Physics at Stanford University. He holds a Master's degree in Electrical Engineering from Stanford.