Named Arguments for SQL FunctionsNovember 13, 2023 by Daniel Tenedorio, Xinyi Yu, Allison Wang, Wenchen Fan, Serge Rielau and Richard Yu in Engineering Blog Today, we introduce the new availability of named arguments for SQL functions. With this feature, you can invoke functions in more flexible ways...
Introducing the Support of Lateral Column AliasSeptember 19, 2023 by Xinyi Yu, Wenchen Fan and Gengliang Wang in Engineering Blog We are thrilled to introduce the support of a new SQL feature in Apache Spark and Databricks: Lateral Column Alias (LCA). This feature...
Introducing Apache Spark™ 3.5September 15, 2023 by Yuanjian Li, Daniel Tenedorio, Martin Grund, Allan Folting, Hyukjin Kwon, Herman van Hövell, Wenchen Fan, Weichen Xu, Gengliang Wang, Allison Wang, Jungtaek Lim, Xiao Li and Reynold Xin in Engineering Blog Today, we are happy to announce the availability of Apache Spark™ 3.5 on Databricks as part of Databricks Runtime 14.0. We extend our...
Introducing Apache Spark™ 3.4 for Databricks Runtime 13.0April 14, 2023 by Xinrong Meng, Daniel Tenedorio, Martin Grund, Allan Folting, Hyukjin Kwon, Herman van Hövell, Wenchen Fan, Ying Xiong, Jungtaek Lim, Xiao Li and Reynold Xin in Engineering Blog Today, we are happy to announce the availability of Apache Spark™ 3.4 on Databricks as part of Databricks Runtime 13.0 . We extend...
Introducing Apache Spark™ 3.3 for Databricks Runtime 11.0June 15, 2022 by Maxim Gekk, Wenchen Fan, Hyukjin Kwon, Serge Rielau, Yingyi Bu, Xiao Li and Reynold Xin in Engineering Blog Today we are happy to announce the availability of Apache Spark™ 3.3 on Databricks as part of Databricks Runtime 11.0 . We want...
Introducing Apache Spark™ 3.2October 19, 2021 by Gengliang Wang, Wenchen Fan, Hyukjin Kwon, Xiao Li and Reynold Xin in Engineering Blog We are excited to announce the availability of Apache Spark™ 3.2 on Databricks as part of Databricks Runtime 10.0 . We want to...
Introducing Apache Spark™ 3.1March 2, 2021 by Hyukjin Kwon, Wenchen Fan, Xiao Li and Reynold Xin in Engineering Blog We are excited to announce the availability of Apache Spark 3.1 on Databricks as part of Databricks Runtime 8.0 . We want to...
A Comprehensive Look at Dates and Timestamps in Apache Spark™ 3.0July 22, 2020 by Maxim Gekk, Wenchen Fan and Hyukjin Kwon in Engineering Blog Apache Spark is a very popular tool for processing structured and unstructured data. When it comes to processing structured data, it supports many...
Introducing Apache Spark 3.0June 18, 2020 by Matei Zaharia, Reynold Xin, Xiao Li, Wenchen Fan and Yin Huai in Product We’re excited to announce that the Apache Spark TM 3.0.0 release is available on Databricks as part of our new Databricks Runtime 7.0...
Adaptive Query Execution: Speeding Up Spark SQL at RuntimeMay 29, 2020 by Wenchen Fan, Herman van Hövell and MaryAnn Xue in Engineering Blog Read Rise of the Data Lakehouse to explore why lakehouses are the data architecture of the future with the father of the data...