Lakeflow Connect: Smarter, Simpler File Ingestion With the Next Generation of Auto Loader

Auto Loader is the definitive tool for ingesting data from cloud storage into your lakehouse.

In this session, we’ll unveil new features and best practices that simplify every aspect of cloud storage ingestion. We’ll demo out-of-the-box observability for pipeline health and data quality, walk through improvements for schema management, introduce a series of new data formats and unveil recent strides in Auto Loader performance. Along the way, we’ll provide examples and best practices for optimizing cost and performance.

Finally, we’ll introduce a preview of what’s coming next — including a REST API for pushing files directly to Delta, a UI for creating cloud storage pipelines and more.

Join us to help shape the future of file ingestion on Databricks.