Highlights from first expanded Spark + AI Summit
Keynotes show how Unified Analytics (Data + AI) is accelerating innovation Databricks hosted the first expanded Data + AI Summit (formerly Spark Summit)...
Benchmarking Apache Spark on a Single Node Machine
Apache Spark has become the de facto unified analytics engine for big data processing in a distributed environment. Yet we are seeing more...
Introducing Low-latency Continuous Processing Mode in Structured Streaming in Apache Spark 2.3
Import this notebook on Databricks Structured Streaming in Apache Spark 2.0 decoupled micro-batch processing from its high-level APIs for a couple of reasons...
Introducing Stream-Stream Joins in Apache Spark 2.3
Since we introduced Structured Streaming in Apache Spark 2.0 , it has supported joins (inner join and some type of outer joins) between...
Announcing Machine Learning Model Export in Databricks
In recent years, machine learning has become ubiquitous in industry and production environments. Both academic and industry institutions had previously focused on training...
Apache Spark 2.3 with Native Kubernetes Support
This is a community blog from Anirudh Ramanathan and Palak Bhatia , software engineer and product manager respectively at Google, working in the...
Introducing Apache Spark 2.3
Today we are happy to announce the availability of Apache Spark 2.3.0 on Databricks as part of its Databricks Runtime 4.0. We want...
Meltdown and Spectre: Exploits and Mitigation Strategies
In an earlier blog post , we analyzed the performance impact of Meltdown and Spectre on big data workloads in the cloud. In...
Meltdown and Spectre's Performance Impact on Big Data Workloads in the Cloud
Last week, the details of two industry-wide security vulnerabilities, known as Meltdown and Spectre , were released. These exploits enable cross-VM and cross-process...
Databricks Cache Boosts Apache Spark Performance
We are excited to announce the general availability of Databricks Cache, a Databricks Runtime feature as part of the Unified Analytics Platform that...