Adaptive Query Execution: Speeding Up Spark SQL at Runtime
This is a joint engineering effort between the Databricks Apache Spark engineering team — Wenchen Fan, Herman van Hovell and MaryAnn Xue —…
Introducing New Built-in and Higher-Order Functions for Complex Data Types in Apache Spark 2.4
Try this notebook in Databricks Apache Spark 2.4 introduces 29 new built-in functions for manipulating complex types (for example, array type), including higher-order…
Working with Nested Data Using Higher Order Functions in SQL on Databricks
Nested data types offer Databricks customers and Apache Spark users powerful ways to manipulate structured data. In particular, they allow you to put…
SQL Subqueries in Apache Spark 2.0
In the upcoming Apache Spark 2.0 release, we have substantially expanded the SQL standard capabilities. In this brief blog post, we will introduce…
