Let's Do Some Data Engineering With Rust and Delta Lake!

The future of data engineering is looking increasingly Rusty. By adopting the foundational crates of Delta Lake, data fusion, and arrow, developers can write high-performance and low-cost ingestion pipelines, transformation jobs, and data query applications. Attendees don't need to know Rust ahead of time; we will review some fundamental concepts of the language as they pertain to the data engineering domain. The main goal of this session is to provide attendees with a starting point to learn Rust by applying it to the real-world data problems they're already familiar with:

• Ingesting semi-structured data into Delta tables (e.g. CSV, JSON, etc.)
• Enriching data from multiple tables to create new silver/gold tables
• Performing table management (e.g. OPTIMIZE, VACUUM)
• Exporting data from Delta tables to external systems (e.g. Elastic/OpenSearch)