Processing Data in Apache Kafka with Structured Streaming in Apache Spark 2.2
This is the third post in a multi-part series about how you can perform complex streaming analytics using Apache Spark. In this blog...
Real-Time End-to-End Integration with Apache Kafka in Apache Spark’s Structured Streaming
View the Notebook in Databricks Community Edition Structured Streaming APIs enable building end-to-end streaming applications called continuous applications in a consistent, fault-tolerant manner...
Working with Complex Data Formats with Structured Streaming in Apache Spark 2.1
In part 1 of this series on Structured Streaming blog posts, we demonstrated how easy it is to write an end-to-end streaming ETL...
Real-time Streaming ETL with Structured Streaming in Apache Spark 2.1
Explore why lakehouses are the data architecture of the future with the father of the data warehouse, Bill Inmon. Try this notebook in...
How to use SparkSession in Apache Spark 2.0
Generally, a session is an interaction between two or more entities. In computer parlance, its usage is prominent in the realm of networked...
Databricks Bi-Weekly Digest: 8/8/16
Continuing with our bi-weekly digest series, here’s our recap of what’s transpired over the last two weeks with Apache Spark since our previous...
Approximate Algorithms in Apache Spark: HyperLogLog and Quantiles
Introduction Apache Spark is fast, but applications such as preliminary data exploration need to be even faster and are willing to sacrifice some...
Faster Stateful Stream Processing in Apache Spark Streaming
Many complex stream processing pipelines must maintain state across a period of time. For example, if you are interested in understanding user behavior...
Diving into Apache Spark Streaming's Execution Model
With so many distributed stream processing engines available, people often ask us about the unique benefits of Apache Spark Streaming . From early...
New Visualizations for Understanding Apache Spark Streaming Applications
Earlier, we presented new visualizations introduced in Apache Spark 1.4.0 to understand the behavior of Spark applications. Continuing the theme, this blog highlights...