Session

Selectively Overwrite Data With Delta Lake’s Dynamic Insert Overwrite

Overview

ExperienceIn Person
TypeLightning Talk
TrackData Lakehouse Architecture and Implementation
IndustryEnterprise Technology, Health and Life Sciences, Financial Services
TechnologiesApache Spark, Delta Lake, Databricks SQL
Skill LevelBeginner
Duration20 min

Dynamic Insert Overwrite is an important Delta Lake feature that allows fine-grained updates by selectively overwriting specific rows, eliminating the need for full-table rewrites.

 

For examples, this capability is essential for:

  • DBT-Databricks' incremental models/workloads, enabling efficient data transformations by processing only new or updated records
  • ETL Slowly Changing Dimension (SCD) Type 2

 

In this lightning talk, we will:

  • Introduce Dynamic Insert Overwrite: Understand its functionality and how it works
  • Explore key use cases: Learn how it optimizes performance and reduces costs
  • Share best practices: Discover practical tips for leveraging this feature on Databricks, including on the cutting-edge Serverless SQL Warehouses
Return to all sessions

Session Speakers

IMAGE COMING SOON

Thang Long Vu

/Software Engineer
Databricks