What is Machine Learning Library?

How Spark MLlib provides scalable ML algorithms and utilities so teams can train, evaluate, and deploy models on large datasets with ease

Summary

Understand how Apache Spark’s MLlib focuses on simplicity, scalability and integration so you can concentrate on data and models instead of distributed systems details
Explore the core algorithms and utilities in MLlib, from classification and regression to clustering, collaborative filtering and dimensionality reduction
See how MLlib integrates with Spark SQL, Streaming and DataFrames and supports multiple languages to power end-to-end machine learning workflows

Apache Spark’s Machine Learning Library (MLlib) is designed for simplicity, scalability, and easy integration with other tools. With the scalability, language compatibility, and speed of Spark, data scientists can focus on their data problems and models instead of solving the complexities surrounding distributed data (such as infrastructure, configurations, and so on). Built on top of Spark, MLlib is a scalable machine learning library consisting of common learning algorithms and utilities, including classification, regression, clustering, collaborative filtering, dimensionality reduction, and underlying optimization primitives. Spark MLLib seamlessly integrates with other Spark components such as Spark SQL, Spark Streaming, and DataFrames and is installed in the Databricks runtime. The library is usable in Java, Scala, and Python as part of Spark applications, so that you can include it in complete workflows. MLlib allows for preprocessing, munging, training of models, and making predictions at scale on data. You can even use models trained in MLlib to make predictions in Structured Streaming. Spark provides a sophisticated machine learning API for performing a variety of machine learning tasks, from classification to regression, clustering to deep learning.