Real-Time Anomaly Detection with Spark ML and Akka - Databricks

Real-Time Anomaly Detection with Spark ML and Akka

Download Slides

We present a solution for streaming anomaly detection, named “Coral”, based on Spark, Akka and Cassandra. In the system presented, we run Spark to run the data analytics pipeline for anomaly detection. By running Spark on the latest events and data, we make sure that the model is always up-to-date and that the amount of false positives is kept low, even under changing trends and conditions. Our machine learning pipeline uses Spark decision tree ensembles and k-means clustering. Once the model is trained by Spark, the model’s parameters are pushed to the Streaming Event Processing Layer, implemented in Akka. The Akka layer will then score 1000s of event per seconds according to the last model provided by Spark. Spark and Akka communicate which each other using Cassandra as a low-latency data store. By doing so, we make sure that every element of this solution is resilient and distributed. Spark performs micro-batches to keep the model up-to-date while Akka detects the new anomalies by using the latest Spark-generated data model. The project is currently hosted on Github. Have a look at :

Learn more:

  • Streaming Analytics with Spark, Kafka, Cassandra, and Akka
  • About Natalino Busa

    Natalino is currently data architect at Ing Retail in the Netherlands, where he leads the definition, design and implementation of big/fast data solutions for data-driven financial applications such as personalized marketing and predictive analytics. All-round Software Architect, Data Technologist, Innovator, with 15+ years experience in research, development and management of distributed architectures and scalable services and applications. Previously served as senior researcher at Philips Research Laboratories in the Netherlands, on the topics of system-on-a-chip architectures, distributed computing and parallelizing compilers. Blogs regularly about data analytics, data science and scala reactive programming at