Pranav Anand is a Software Engineer at Databricks. He has been developing Auto Loader and Delta Lake to simplify the lives of Data Engineers. Pranav received his BS in Computer Science at University of Waterloo.
May 27, 2021 12:10 PM PT
Continuously and incrementally ingesting data as it arrives in cloud storage has become a common workflow in our customers' ETL pipelines. However, managing this workflow is rife with challenges, such as scalable and efficient file discovery, schema inference and evolution, and fault tolerance with exactly-once guarantees. Auto Loader is a new Structured Streaming source in Databricks as our all-in-one solution to tackle these challenges.
In this talk, we’ll discuss how Auto Loader: