SESSION

Fast, Cheap and Easy Data Ingestion with AWS Lambda and Delta Lake

Accept Cookies to Play Video

OVERVIEW

EXPERIENCEIn Person
TYPEBreakout
TRACKData Lakehouse Architecture
INDUSTRYProfessional Services
TECHNOLOGIESETL
SKILL LEVELBeginner
DURATION40 min
DOWNLOAD SESSION SLIDES

In this session we will dive into examples of how to work with Delta tables from AWS Lambdas written in Python and Rust. For many ingestion, or lightweight data processing workloads AWS Lambda provides a fast, easy, and cheap execution environment. Lambda can be easily triggered from Kinesis, SQS, Kafka, S3 Event Notifications, and more, making it a powerful tool to consider when moving from the "nightly batch" to a more event-driven data platform. Using the native Python or Rust libraries for Delta Lake we will explore the transaction log, write updates, perform table maintenance, and even query Delta tables in milliseconds from AWS Lambda. After this session attendees will be ready to incorporate Lambda into their existing AWS-based data platforms.

SESSION SPEAKERS

R Tyler Croy

/Developer
Buoyant Data