Skip to main content
Page 1
Engineering blog

Introducing Apache Spark™ 3.5

September 15, 2023 by Yuanjian Li, Daniel Tenedorio, Martin Grund, Allan Folting, Hyukjin Kwon, Herman van Hövell, Wenchen Fan, Weichen Xu, Gengliang Wang, Allison Wang, Jungtaek Lim, Xiao Li and Reynold Xin in Engineering Blog
Today, we are happy to announce the availability of Apache Spark™ 3.5 on Databricks as part of Databricks Runtime 14.0. We extend our...
Engineering blog

Native Support of Session Window in Spark Structured Streaming

October 12, 2021 by Jungtaek Lim, Yuanjian Li and Shixiong Zhu in Engineering Blog
Apache Spark™ Structured Streaming allowed users to do aggregations on windows over event-time . Before Apache Spark 3.2™, Spark supported tumbling windows and...
Engineering blog

What’s New in Apache Spark™ 3.1 Release for Structured Streaming

April 27, 2021 by Yuanjian Li, Shixiong Zhu and Bo Zhang in Engineering Blog
Along with providing the ability for streaming processing based on Spark Core and SQL API, Structured Streaming is one of the most important...
Engineering blog

A look at the new Structured Streaming UI in Apache Spark 3.0

July 29, 2020 by Genmao Yu, Yuanjian Li and Shixiong Zhu in Engineering Blog
This is a guest community post from Genmao Yu, a software engineer at Alibaba. Structured Streaming was initially introduced in Apache Spark 2.0...