Databricks Performance Optimization

In this course, you’ll learn how to optimize workloads and physical layout with Spark and Delta Lake and and analyze the Spark UI to assess performance and debug applications. We’ll cover topics like streaming, liquid clustering, data skipping, caching, photons, and more.

Note: This course is part of the 'Advanced Data Engineering with Databricks' course series.

Languages Available: English | 日本語 | Português BR | 한국어

Skill Level

Professional

Duration

Prerequisites

The content was developed for participants with these skills/knowledge/abilities:

Ability to perform basic code development tasks using Databricks (create clusters, run code in notebooks, use basic notebook operations, import repos from git, etc.)
Intermediate programming experience with PySpark

Extract data from a variety of file formats and data sources
- Apply a number of common transformations to clean data
Reshape and manipulate complex data using advanced built-in functions

Intermediate programming experience with Delta Lake (create tables, perform complete and incremental updates, compact files, restore previous versions, etc.)

Outline

Spark Architecture

Spark UI Introduction

Designing the Foundation File Explosion

Introduction to Designing Foundation

File Explosion

Data Skipping and Liquid Clustering

Lab: Data Skipping and Liquid Clustering

Code Optimization

Skew

Shuffle

Spill

Lab: Exploding Join

Serialization

User-Defined Functions

Fine-Tuning - Choosing the Right Cluster

Fine-Tuning: Choosing the Right Cluster

Pick the Best Instance Types

Upcoming Public Classes

Date	Time	Language	Price
Date	Time	Language	Price	May 12	01 PM - 05 PM (Asia/Kolkata)	English	$750.00
May 13	01 PM - 05 PM (Europe/London)	English	$750.00
May 15	01 PM - 05 PM (America/New_York)	English	$750.00
Jun 16	09 AM - 01 PM (America/New_York)	English	$750.00
Jun 17	09 AM - 01 PM (Europe/London)	English	$750.00
Jun 19	09 AM - 01 PM (Asia/Kolkata)	English	$750.00
Jul 14	01 PM - 05 PM (Asia/Kolkata)	English	$750.00
Jul 15	01 PM - 05 PM (Europe/London)	English	$750.00
Jul 16	01 PM - 05 PM (America/New_York)	English	$750.00

Public Class Registration

If your company has purchased success credits or has a learning subscription, please fill out the Training Request form. Otherwise, you can register below.

Private Class Request

If your company is interested in private training, please submit a request.

Request Private Training

See all our registration options

Registration options

Databricks has a delivery method for wherever you are on your learning journey

Self-Paced

Custom-fit learning paths for data, analytics, and AI roles and career paths through on-demand videos

Instructor-Led

Public and private courses taught by expert instructors across half-day to two-day courses

Blended Learning

Self-paced and weekly instructor-led sessions for every style of learner to optimize course completion and knowledge retention. Go to Subscriptions Catalog tab to purchase

Purchase now

Skills@Scale

Comprehensive training offering for large scale customers that includes learning elements for every style of learning. Inquire with your account executive for details

Upcoming Public Classes

Apache Spark Developer

Developing Applications with Apache Spark™

Master scalable data processing with Apache Spark in this hands-on course. Learn to build efficient ETL pipelines, perform advanced analytics, and optimize distributed data transformations using Spark’s DataFrame API. Explore grouping, aggregation, joins, set operations, and window functions. Work with complex data types like arrays, maps, and structs while applying best practices for performance optimization.

Apache Spark Developer

Stream Processing and Analysis with Apache Spark™

Learn the essentials of stream processing and analysis with Apache Spark in this course. Gain a solid understanding of stream processing fundamentals and develop applications using the Spark Structured Streaming API. Explore advanced techniques such as stream aggregation and window analysis to process real-time data efficiently. This course equips you with the skills to create scalable and fault-tolerant streaming applications for dynamic data environments.

SQL Analytics on Databricks

In this course, you'll learn how to effectively use Databricks for data analytics, with a specific focus on Databricks SQL. As a Databricks Data Analyst, your responsibilities will include finding relevant data, analyzing it for potential applications, and transforming it into formats that provide valuable business insights.

You will also understand your role in managing data objects and how to manipulate them within the Databricks Data Intelligence Platform, using tools such as Notebooks, the SQL Editor, and Databricks SQL.

Additionally, you will learn about the importance of Unity Catalog in managing data assets and the overall platform. Finally, the course will provide an overview of how Databricks facilitates performance optimization and teach you how to access Query Insights to understand the processes occurring behind the scenes when executing SQL analytics on Databricks.