I’m a software engineer and currently work for Apple. My main focus area is a fast and efficient data processing. At Apple, as an Apache Spark PMC member and committer and an Apache ORC PMC member and committer, I developed and maintained the internal distributions powered by Apache Spark and Apache ORC.
All production environment requires monitoring and alerting. Apache Spark also has a configurable metrics system in order to allow users to report Spark metrics to a variety of sinks. Prometheus is one of the popular open-source monitoring and alerting toolkits which is used with Apache Spark together. Previously, users can use
Apache Spark 3.0.0 will add another easy way to support Prometheus for general use cases. In this talk, we will talk about the followings and show a demo.
Currently, Apache Spark exposes metrics at Master/Worker/Driver/Executor to integrate with the existing Prometheus server easily with a less effort. This is already available with Apache Spark 3.0.0-preview and preview2. You can try it right now.