Intel’s BigDL on Databricks
Intel recently released its BigDL project for distributed deep learning on Apache Spark. BigDL has native Spark integration, allowing it to leverage Spark during model training, prediction, and tuning. This blog post gives highlights of BigDL and a tutorial showing how to get started with BigDL on Databricks. Intel’s BigDL project BigDL is an open...
Deep Learning on Databricks
We are excited to announce the general availability of Graphic Processing Unit (GPU) and deep learning support on Databricks! This blog post will help users get started via a tutorial with helpful tips and resources, aimed at data scientists and engineers who need to run deep learning applications at scale. What’s new? Databricks now offers...
On Demand Webinar and FAQ: Apache Spark MLlib 2.x: Migrating ML Workloads to DataFrames
Last week, we held a live webinar, Apache Spark MLlib 2.x: Migrating ML Workloads to DataFrames, to demonstrate the ease with which you can migrate your MLlib RDD-based workloads to Spark 2.x MLlib DataFrame-based APIs, gaining all the benefits of simpler APIs, performance and persistence. Mixing presentation and demonstration, we covered migrating workloads from RDDs...
GPU Acceleration in Databricks
Databricks is adding support for Apache Spark clusters with Graphics Processing Units (GPUs), ready to accelerate Deep Learning workloads. With Spark deployments tuned for GPUs, plus pre-installed libraries and examples, Databricks offers a simple way to leverage GPUs to power image processing, text analysis, and other Machine Learning tasks. Users will benefit from 10x speedups...
Apache Spark 2.0 Preview: Machine Learning Model Persistence
Introduction Consider these Machine Learning (ML) use cases: A data scientist produces an ML model and hands it over to an engineering team for deployment in a production environment. A data engineer integrates a model training workflow in Python with a model serving workflow in Java. A data scientist creates jobs to train many ML...
Approximate Algorithms in Apache Spark: HyperLogLog and Quantiles
Introduction Apache Spark is fast, but applications such as preliminary data exploration need to be even faster and are willing to sacrifice some accuracy for a faster result. Since version 1.6, Spark implements approximate algorithms for some common tasks: counting the number of distinct elements in a set, finding if an element belongs to a...
On-Time Flight Performance with GraphFrames for Apache Spark
Introduction Graph structures are a more intuitive approach to many classes of data problems. Whether traversing social networks, restaurant recommendations, or flight paths, it is easier to understand these data problems within the context of graph structures: vertices, edges, and properties. For example, the analysis of flight data is a classic graph problem as airports...
Introducing GraphFrames
We would like to thank Ankur Dave from UC Berkeley AMPLab for his contribution to this blog post. Databricks is excited to announce the release of GraphFrames, a graph processing library for Apache Spark. Collaborating with UC Berkeley and MIT, we have built a graph library based on DataFrames. GraphFrames benefit from the scalability and high...
Auto-scaling scikit-learn with Apache Spark
Data scientists often spend hours or days tuning models to get the highest accuracy. This tuning typically involves running a large number of independent Machine Learning (ML) tasks coded in Python or R. Following some work presented at Spark Summit Europe 2015, we are excited to release scikit-learn integration package for Apache Spark that dramatically simplifies...
MLlib Highlights in Apache Spark 1.6
To learn more about Apache Spark, attend Spark Summit East in New York in Feb 2016. With the latest release, Apache Spark’s Machine Learning library (MLlib) includes many improvements and new features. Users can now save and load ML Pipelines, use extended R and Python APIs, and run new ML algorithms. This blog post highlights...
