Simplify PySpark testing with DataFrame equality functions
The DataFrame equality test functions were introduced in Apache Spark™ 3.5 and Databricks Runtime 14.2 to simplify PySpark unit testing. The full set...
A Deep Dive into the Latest Performance Improvements of Stateful Pipelines in Apache Spark Structured Streaming
This post is the second part of our two-part series on the latest performance improvements of stateful pipelines. The first part of this...
Performance Improvements for Stateful Pipelines in Apache Spark Structured Streaming
Introduction Apache Spark™ Structured Streaming is a popular open-source stream processing platform that provides scalability and fault tolerance, built on top of the...
Lakehouse Monitoring: A Unified Solution for Quality of Data and AI
Introduction Databricks Lakehouse Monitoring allows you to monitor all your data pipelines – from data to features to ML models – without additional...
Python Dependency Management in Spark Connect
Managing the environment of an application in a distributed computing environment can be challenging. Ensuring that all nodes have the necessary environment to...
Named Arguments for SQL Functions
Today, we introduce the new availability of named arguments for SQL functions. With this feature, you can invoke functions in more flexible ways...
Introducing Python User-Defined Table Functions (UDTFs)
Apache Spark™ 3.5 and Databricks Runtime 14.0 have brought an exciting feature to the table: Python user-defined table functions (UDTFs). In this blog...
Arrow-optimized Python UDFs in Apache Spark™ 3.5
In Apache Spark™, Python User-Defined Functions (UDFs) are among the most popular features. They empower users to craft custom code tailored to their...
Apache Spark ❤️ Apache DataSketches: New Sketch-Based Approximate Distinct Counting
Introduction In this blog post, we'll explore a set of advanced SQL functions available within Apache Spark that leverage the HyperLogLog algorithm, enabling...
Multiple Stateful Operators in Structured Streaming
In the world of data engineering, there are operations that have been used since the birth of ETL. You filter. You join. You...