Databricks Runtime 3.0 Beta Delivers Cloud Optimized Apache Spark
A major value Databricks provides is the automatic provisioning, configuration, and tuning of clusters of machines that process data. Running on these machines...
Processing a Trillion Rows Per Second on a Single Machine: How Can Nested Loop Joins be this Fast?
This blog post describes our experience debugging a failing test case caused by a cross join query running “too fast.” Because the root...
Databricks and Apache Spark 2016 Year in Review
Spark Summit will be held in Boston on Feb 7-9, 2017. Check out the full agenda and get your ticket before it sells...
Introducing Apache Spark 2.1
Spark Summit will be held in Boston on Feb 7-9, 2017. Check out the full agenda and get your ticket before it sells...
$1.44 per terabyte: setting a new world record with Apache Spark
We are excited to share with you that a joint effort by Nanjing University, Alibaba Group, and Databricks set a new world record...
Spark Structured Streaming
Apache Spark 2.0 adds the first version of a new higher-level API, Structured Streaming, for building continuous applications . The main goal is...
Introducing Apache Spark 2.0
Today, we're excited to announce the general availability of Apache Spark 2.0 on Databricks. This release builds on what the community has learned...
Apache Spark as a Compiler: Joining a Billion Rows per Second on a Laptop
When our team at Databricks planned our contributions to the upcoming Apache Spark 2.0 release, we set out with an ambitious goal by...
Technical Preview of Apache Spark 2.0 Now on Databricks
For the past few months, we have been busy contributing to the next major release of the big data open source software we...
The Unreasonable Effectiveness of Deep Learning on Apache Spark
Update: this post is an April Fools joke. It is not an actual project we're working on. For the past three years, our...