The Ubiquity of Delta Standalone: Java, Scala, Hive, Presto, Trino, Power BI, and More!
The Delta Standalone library is a single-node Java library that can be used to read from and write to Delta tables. Specifically, this…
Extending Delta Sharing for Azure
We are excited for the release of Delta Sharing 0.3.0, which introduces several key improvements and bug fixes, including the following features: Delta…
Native Support of Session Window in Spark Structured Streaming
Apache Spark™ Structured Streaming allowed users to do aggregations on windows over event-time. Before Apache Spark 3.2™, Spark supported tumbling windows and sliding…
What’s New in Apache Spark™ 3.1 Release for Structured Streaming
Along with providing the ability for streaming processing based on Spark Core and SQL API, Structured Streaming is one of the most important…
Natively Query Your Delta Lake With Scala, Java, and Python
Today, we’re happy to announce that you can natively query your Delta Lake with Scala and Java (via the Delta Standalone Reader) and…
A look at the new Structured Streaming UI in Apache Spark 3.0
This is a guest community post from Genmao Yu, a software engineer at Alibaba. Structured Streaming was initially introduced in Apache Spark 2.0.…
Introducing Low-latency Continuous Processing Mode in Structured Streaming in Apache Spark 2.3
Structured Streaming in Apache Spark 2.0 decoupled micro-batch processing from its high-level APIs for a couple of reasons. First, it made developer’s experience…
Faster Stateful Stream Processing in Apache Spark Streaming
Many complex stream processing pipelines must maintain state across a period of time. For example, if you are interested in understanding user behavior…
New Visualizations for Understanding Apache Spark Streaming Applications
Earlier, we presented new visualizations introduced in Apache Spark 1.4.0 to understand the behavior of Spark applications. Continuing the theme, this blog highlights…
