How to Avoid Drowning in GDPR Data Subject Requests in a Data Lake
With GDPR enforcement rapidly approaching (May 25, 2018), many companies are still trying to figure out how to comply. A big pain point,…
Introducing Low-latency Continuous Processing Mode in Structured Streaming in Apache Spark 2.3
Structured Streaming in Apache Spark 2.0 decoupled micro-batch processing from its high-level APIs for a couple of reasons. First, it made developer’s experience…
Databricks Delta: A Unified Data Management System for Real-time Big Data
Combining the best of data warehouses, data lakes and streaming For an in-depth look and demo, join the webinar. Today we are proud…
Introducing Apache Spark 2.2
Today we are happy to announce the availability of Apache Spark 2.2.0 on Databricks as part of the Databricks Runtime 3.0. This release…
Making Apache Spark the Fastest Open Source Streaming Engine
We started building Structured Streaming in Apache Spark one year ago as a new, simpler way to develop continuous applications. Not only does…
Taking Apache Spark’s Structured Streaming to Production
This is the fifth post in a multi-part series about how you can perform complex streaming analytics using Apache Spark. At Databricks, we’ve…
Processing Data in Apache Kafka with Structured Streaming in Apache Spark 2.2
This is the third post in a multi-part series about how you can perform complex streaming analytics using Apache Spark. In this blog,…
Working with Complex Data Formats with Structured Streaming in Apache Spark 2.1
In part 1 of this series on Structured Streaming blog posts, we demonstrated how easy it is to write an end-to-end streaming ETL…
Real-time Streaming ETL with Structured Streaming in Apache Spark 2.1
We are well into the Big Data era, with organizations collecting massive amounts of data on a continual basis. Yet, the value of…
Spark Structured Streaming
Apache Spark 2.0 adds the first version of a new higher-level API, Structured Streaming, for building continuous applications. The main goal is to…
