Deep Dive into the New Features of Apache Spark 3.5
OVERVIEW
EXPERIENCE | In Person |
---|---|
TYPE | Breakout |
TRACK | Data Engineering and Streaming |
INDUSTRY | Enterprise Technology |
TECHNOLOGIES | Apache Spark |
SKILL LEVEL | Intermediate |
DURATION | 40 |
This session will delve into the latest advancements in Apache Spark™ 3.5, highlighting its pivotal role in pushing the boundaries of big data processing and AI. We'll discuss Spark Connect's role in enhancing accessibility, DeepSpeed's integration for AI efficiency, and performance optimizations. Additionally, we'll delve into the new PySpark and SQL features, including built-in functions for array manipulation, SQL IDENTIFIER clause enhancements, expanded API support, and Arrow-optimized Python UDFs, highlighting their impact on building scalable, efficient, and robust data-driven applications.
SESSION SPEAKERS
IMAGE COMING SOON
Daniel Tenedorio
/Sr. Staff Software Engineer
Databricks
IMAGE COMING SOON
Xiao Li
/Engineering Director
Databricks