Making Apache Spark the Fastest Open Source Streaming Engine
We started building Structured Streaming in Apache Spark one year ago as a new, simpler way to develop continuous applications . Not only...
Running Streaming Jobs Once a Day For 10x Cost Savings
This is the sixth post in a multi-part series about how you can perform complex streaming analytics using Apache Spark. Traditionally, when people...
Taking Apache Spark’s Structured Streaming to Production
This is the fifth post in a multi-part series about how you can perform complex streaming analytics using Apache Spark. At Databricks, we’ve...
Real-Time End-to-End Integration with Apache Kafka in Apache Spark’s Structured Streaming
View the Notebook in Databricks Community Edition Structured Streaming APIs enable building end-to-end streaming applications called continuous applications in a consistent, fault-tolerant manner...
Real-time Streaming ETL with Structured Streaming in Apache Spark 2.1
Explore why lakehouses are the data architecture of the future with the father of the data warehouse, Bill Inmon. Try this notebook in...
Spark Structured Streaming
Apache Spark 2.0 adds the first version of a new higher-level API, Structured Streaming, for building continuous applications . The main goal is...
Continuous Applications: Evolving Streaming in Apache Spark 2.0
Since its release, Spark Streaming has become one of the most widely used distributed streaming engines, thanks to its high-level API and exactly-once...
Faster Stateful Stream Processing in Apache Spark Streaming
Many complex stream processing pipelines must maintain state across a period of time. For example, if you are interested in understanding user behavior...
Guest Blog: Streamliner - An Open Source Apache Spark Streaming Application
This is a guest blog from Ankur Goyal, VP of Engineering at MemSQL Our always-on interconnected world constantly shuttles data between devices and...
Diving into Apache Spark Streaming's Execution Model
With so many distributed stream processing engines available, people often ask us about the unique benefits of Apache Spark Streaming . From early...