Diving into Apache Spark Streaming's Execution Model
With so many distributed stream processing engines available, people often ask us about the unique benefits of Apache Spark Streaming . From early...
New Features in Machine Learning Pipelines in Apache Spark 1.4
Apache Spark 1.2 introduced Machine Learning (ML) Pipelines to facilitate the creation, tuning, and inspection of practical ML workflows. Spark’s latest release, Spark...
Using 3rd Party Libraries in Databricks: Apache Spark Packages and Maven Libraries
In an earlier post, we described how you can easily integrate your favorite IDE with Databricks to speed up your application development. In...
Joint Blog Post: Bringing ORC Support into Apache Spark
This is a joint blog post with our partner Hortonworks. Zhan Zhang is a member of technical staff at Hortonworks, where he collaborated...
Introducing Window Functions in Spark SQL
Get an early preview of O'Reilly's new ebook for the step-by-step guidance you need to start using Delta Lake. In this blog post...
Introducing R Notebooks in Databricks
Apache Spark 1.4 was released on June 11 and one of the exciting new features was SparkR . I am happy to announce...
Announcing SparkHub: A Community Site for Apache Spark
Today, we are happy to announce SparkHub , a service for the Apache Spark community to easily find the most relevant Spark resources...
New Visualizations for Understanding Apache Spark Streaming Applications
Earlier, we presented new visualizations introduced in Apache Spark 1.4.0 to understand the behavior of Spark applications. Continuing the theme, this blog highlights...
Guest blog: PMML Support in Apache Spark's MLlib
This is a guest blog from our friend Vincenzo Selvaggio who contributed this feature. He is a Senior Java Technical Architect and Project...
MyFitnessPal Delivers New Feature, Speeds up Pipeline, and Boosts Team Productivity with Databricks
To learn more about how Databricks helped MyFitnessPal with analytics, check out an earlier article in Wall Street Journal (log-in required). We are...