Introducing Apache Spark 4.2

Introduction

Apache Spark 4.2 moves more of the modern data and AI stack into the engine itself. Building on Spark 4.x, the release adds governed metrics, vector and top-K primitives, a more Arrow-first Python path, first-class change data capture, and stronger streaming and operational foundations.

This makes Spark more useful on both sides of an AI application. It improves the quality and freshness of the data supplied to AI agents, and it makes Spark easier for applications and agents to invoke as a remote execution service. The AI story is concrete: trusted semantics, native retrieval primitives, fresh change data, and open interfaces to Spark-scale computation.

Spark 4.2 can be understood through four benefits:

• Define truth once: Metric views put governed business metrics in Spark so SQL, BI tools, applications, and AI systems can use the same definitions.
• Reach Spark from everywhere: Spark Connect, PySpark, Arrow, and Python Data Source improvements make Spark easier to call from services and Python ecosystems.
• Run AI-native analytics in SQL: Vector functions, NEAREST BY, sketches, ranking, and geospatial types bring more analytical building blocks directly into Spark SQL.
• Move changing data safely: Auto CDC, the CHANGES surface, Data Source V2, and Real-Time Streaming make continuously changing data easier to process correctly.

Together, these changes help organizations use one open engine to prepare data, define business meaning, retrieve relevant context, and keep analytical and AI applications current.

Metrics and Semantic Modeling: Define Truth Once

Spark 4.2 introduces metric views, bringing a native semantic layer to Spark SQL. Teams can define business metrics once and use them consistently across dashboards, reports, applications, and AI tools.

This matters because many important metrics are not safely additive. Ratios, distinct counts, retention, and similar measures can produce incorrect results when every consumer rewrites the formula at a different grain. Metric views make dimensions and measures first-class objects that Spark understands, allowing the engine to preserve the intended aggregation semantics.

Once a metric view is defined, users can query the same governed measures by different dimensions:

For AI applications, this is especially important. An agent should not calculate revenue differently from a dashboard or return a different answer when a user changes the requested grouping. A governed metric view gives SQL, BI, and AI one source of truth, with Spark analysis, catalog resolution, and permissions applied consistently.

Spark Connect and PySpark: Reach Spark from Everywhere

Spark as a service API

Spark Connect separates the client from the Spark server through a protocol based on gRPC and Arrow. A client builds a logical plan, the server analyzes and executes it, and results return as Arrow batches. The client does not need a full Spark runtime or a colocated JVM.

This makes Spark easier to embed in notebooks, services, developer tools, and AI applications. An agent or application can call Spark from its own runtime while Spark keeps analysis, optimization, execution, and governance on the server.

Spark 4.2 continues closing the compatibility gap with Spark Classic. Improvements include better RDD API compatibility, DataFrame inputs to spark.read.* and SparkSession.emptyDataFrame, improved debuggability, error propagation, status reporting, and YARN cluster-mode support. Together, these changes make PySpark and Spark Connect faster, more compatible, and easier to operate at scale and remote.

A more Arrow-first Python path

Python remains one of the primary ways users build data and AI workloads with Spark. In Spark 4.2, Arrow-optimized Python UDF execution is enabled by default, so existing UDFs can use the faster columnar path without a code rewrite. Pandas 3 support also makes it easier to upgrade Python environments alongside Spark.

For code that needs more control, Arrow UDFs keep data in PyArrow arrays and avoid an unnecessary Pandas conversion. Spark also expands profiling and debugging for Python execution, including time and memory profiling for Python Data Sources, improved worker diagnostics, and logging that can be queried as data.

Spark 4.2 also improves interoperability through the Arrow C Data Interface and the PyCapsule protocol. When both sides support it, Spark DataFrames can move into Arrow-native tools such as Polars or DuckDB without copying or serializing the underlying data. This reduces glue between Spark-scale processing and the broader Python and AI ecosystem.

Python Data Sources further reduce integration friction. Teams can build batch or streaming readers and writers in Python, register them once, and use them through the standard Spark data source interface. In 4.2, profiling makes these connectors easier to tune and operate rather than treating them as black boxes.

Spark SQL: AI-Native Analytics in the Engine

Vector scoring and top-K retrieval

Spark 4.2 adds new SQL primitives for vector similarity search, ranking, and time-series analysis. The release introduces vector distance and similarity functions, vector normalization, vector aggregation, and NEAREST BY, a top-K ranking join for distance-based matching. These primitives enable retrieval, recommendations, entity resolution, and candidate generation at scale.

Native geospatial analytics

Built-in GEOMETRY and GEOGRAPHY types and ST_* functions enable location-aware analytics without external spatial extensions. Spark 4.2 also adds Parquet, WKT/WKB, SRID preservation, and Python conversion support.

Fully qualified built-in functions and temporary views

With Spark 4.2 you can unambiguously invoke Spark provided functions by qualifying them with SYSTEM.BUILTIN. Following the precedent of session variables you can also fully qualify temporary views with SYSTEM.SESSION. This is useful to disambiguate from user defined functions or persisted relations and prevent injection.

SQL search path

Spark 4.2 adds SQL search path support with SET PATH, making it easier to resolve tables, functions, and variables across namespaces, and to libraries of objects simply by adding schemas to the path.

Spark persists the SQL path in views and SQL functions for predictable name resolution.

Starting with Spark 4.2 SQL scripts can DECLARE, OPEN, FETCH, and CLOSE cursors. This allows for more control over row-by-row processing of results sets, which in the past required stepping outside of SQL to use DataFrames.

Spark SQL also adds Tuple sketches, time_bucket for time-series analysis, broader TIME type support across file formats, QUALIFY for filtering window results, Top-K max_by and min_by, and IGNORE NULLS and RESPECT NULLS support for common aggregation functions.

Together, these additions make Spark SQL more expressive for modern analytical applications.

Spark Declarative Pipelines and Auto CDC: Move Changing Data Safely

Spark 4.2 introduces Auto CDC support in Spark Declarative Pipelines (SDP), bringing first-class SCD (Slow Changing Dimensions) Type 1 processing into Spark. Before Auto CDC, consuming a change feed and applying it to a target table required hand-written merge logic that could easily become complex and error-prone, due to handling deletions and out-of-order change events. With Auto CDC, users can simply configure how CDC events should update a target table and let Spark manage the complexities.

Auto CDC provides a Python API for applying CDC changes to an SCD Type 1 target table. It is designed for common ingestion and replication workloads where the latest version of each record must be maintained reliably, such as customer profiles, product catalogs, account records, and operational reference data.

For example, an Auto CDC flow can now be expressed declaratively:

In addition to Auto CDC, Spark Declarative Pipelines also receives important platform hardening, including safer server-side handling for eager analysis and structured identifiers for flows. Together, these changes make declarative pipeline development more reliable and give Spark a foundation for higher-level data engineering patterns.

Real-Time Mode in Structured Streaming: Fresher Operational Data

Real-Time Mode (RTM) in Structured Streaming lets streaming queries process data with millisecond end-to-end latency. This has helped Spark unlock whole new classes of use cases, and is becoming the foundation for operational data applications such as fraud detection, personalization, observability, and real-time feature engineering.

In Spark 4.2, we extended RTM to PySpark: you can now run stateless streaming queries (without Python UDFs) in Real-Time Mode. Python is a popular choice among data scientists and engineers for its ease of use, and this brings RTM's low-latency processing to a much wider audience.

Looking ahead to the upcoming Spark 4.x release, we're bringing stateful support to RTM — and the work is already underway. The effort is tracked in SPARK-54699 with three major components:

• A new streaming shuffle (SPARK-56664) that forwards data from upstream stages to downstream as soon as it's ready, rather than waiting for a stage to complete
• Concurrent stage scheduling (SPARK-57000), allowing multiple stages to run at the same time
• Stateful operator support (SPARK-57228), starting with transformWithState

Beyond stateful support, we're also working to enable Python UDFs (SPARK-57237) in RTM.

Stay tuned — and we'd welcome your feedback and contributions!

Data Source V2: One Surface for Evolving Data Sources

Spark 4.2 marks another major step forward for Data Source V2. DSv2 is becoming the standard foundation for connectors that expose reads, writes, row-level operations, schema evolution, change data, operation metrics, and transactions through Spark.

CDC in DSv2

Spark 4.2 adds first-class change data capture support to DSv2. Connectors can expose change streams through a standard API, and users can query them with the new CHANGES SQL clause, DataFrame APIs, and PySpark bindings. Spark also handles common post-processing in the engine — dropping copy-on-write carry-overs, detecting updates, and computing net changes per row. The same query behaves consistently across any DSv2 connector that supports CDC.

Row-Level Operations, Schema Evolution, and Transactions

Spark 4.2 further enhances support for row-level DML operations in Data Source V2 (DSv2) connectors. MERGE INTO receives additional performance improvements, including whole-stage code generation, along with further enhancements to the schema evolution capabilities introduced in Spark 4.1.

Schema evolution is now also supported for INSERT INTO operations, for both name-based and position-based column resolution, reducing friction when writing to evolving tables. In addition, operation summaries are now available for UPDATE and DELETE, complementing the MERGE INTO summaries added in Spark 4.1. MERGE INTO metrics have also been expanded and refined.

Spark 4.2 introduces additional building blocks for production-grade DSv2 connectors and lakehouse table formats. Key additions include the foundations of a transaction API, enhanced partition-statistics filtering, improvements to storage-partitioned joins, and closer alignment between DSv1 and DSv2 commands and behaviors. Together, these enhancements make DSv2 a more complete platform for implementing lakehouse connectors, transactional table formats, and other large-scale data systems.

Notable Improvements and Acknowledgements

Spark 4.2 includes several platform improvements that make Spark easier to operate, debug, secure, and scale. The Spark Web UI receives a major modernization with Bootstrap 5, dark mode, better SQL plan visualization, query timeline improvements, and server-side pagination. Kubernetes support improves with heterogeneous executor management, stable resource manager APIs, and reduced control-plane overhead. Spark 4.2 also adds JDK 25 support, improves web security, scales the Spark History Server, and upgrades key dependencies including Scala, Parquet, ORC, Arrow, Netty, and Hadoop.

Spark 4.2 reflects the strength of the Apache Spark community, with more than 1,900 commits from over 260 contributors. We thank everyone who contributed code, reviews, testing, documentation, and feedback to make this release possible.

Get Started with Spark 4.2

Download Apache Spark 4.2 from spark.apache.org/downloads and see the full Apache Spark 4.2 release notes for the complete list of changes. Apache Spark 4.2 will also be available in Databricks Runtime 19 Beta.