The Hitchhiker's Guide to Delta Lake Streaming in an Agentic Universe

As data engineering continues to evolve the shift from batch-oriented to streaming-first has become standard across the enterprise. The reality is these changes have been taking shape for the past decade — we just now also happen to be standing on the precipice of true disruption through automation, the likes of which we could only dream about before.

Yes, AI Agents and LLMs are already a large part of our daily lives, but we (as data engineers) are ultimately on the frontlines ensuring that the future of AI is powered by consistent, just-in-time data — and Delta Lake is critical to help us get there.

This session will provide you with best practices learned the hard way by one of the authors of The Delta Lake Definitive Guide including:

• Guide to writing generic applications as components
• Workflow automation tips and tricks
• Tips and tricks for Delta clustering (liquid, z-order, and classic)
• Future facing: Leveraging metadata for agentic pipelines and workflow automation