SESSION

Let's Do Some Data Engineering With Rust and Delta Lake!

Accept Cookies to Play Video

OVERVIEW

EXPERIENCEIn Person
TYPEBreakout
TRACKData Lakehouse Architecture
INDUSTRYEnterprise Technology, Media and Entertainment, Financial Services
TECHNOLOGIESDelta Lake, Developer Experience, ETL
SKILL LEVELIntermediate
DURATION40 min
DOWNLOAD SESSION SLIDES

The future of data engineering is looking increasingly Rusty. By adopting the foundational crates of Delta Lake, data fusion, and arrow, developers can write high-performance and low-cost ingestion pipelines, transformation jobs, and data query applications. Attendees don't need to know Rust ahead of time; we will review some fundamental concepts of the language as they pertain to the data engineering domain. The main goal of this session is to provide attendees with a starting point to learn Rust by applying it to the real-world data problems they're already familiar with:

 

  • Ingesting semi-structured data into Delta tables (e.g. CSV, JSON, etc.)
  • Enriching data from multiple tables to create new silver/gold tables
  • Performing table management (e.g. OPTIMIZE, VACUUM)
  • Exporting data from Delta tables to external systems (e.g. Elastic/OpenSearch)

SESSION SPEAKERS

R Tyler Croy

/Developer
Buoyant Data