Technical Preview of Apache Spark 2.0 Now on Databricks
For the past few months, we have been busy contributing to the next major release of the big data open source software we...
New Content in Databricks Community Edition
At the Spark Summit New York , we announced Databricks Community Edition (CE) beta. CE is a free version of the Databricks service...
The Unreasonable Effectiveness of Deep Learning on Apache Spark
Update: this post is an April Fools joke. It is not an actual project we're working on. For the past three years, our...
Apache Spark Trending in the Stack Overflow Survey
Last week, Stack Overflow released the result of their 2016 developer survey . This is one of the most significant surveys in the...
On-Time Flight Performance with GraphFrames for Apache Spark
Introduction Graph structures are a more intuitive approach to many classes of data problems. Whether traversing social networks, restaurant recommendations, or flight paths...
Introducing GraphFrames
We would like to thank Ankur Dave from UC Berkeley AMPLab for his contribution to this blog post. Databricks is excited to announce...
Reshaping Data with Pivot in Apache Spark
Spark Summit East is just around the corner! If you haven’t registered yet, you can get tickets and here’s a promo code for...
Auto-scaling scikit-learn with Apache Spark
Data scientists often spend hours or days tuning models to get the highest accuracy. This tuning typically involves running a large number of...
Faster Stateful Stream Processing in Apache Spark Streaming
Many complex stream processing pipelines must maintain state across a period of time. For example, if you are interested in understanding user behavior...
MLlib Highlights in Apache Spark 1.6
To learn more about Apache Spark, attend Spark Summit East in New York in Feb 2016 . With the latest release, Apache Spark’s...