Building End-to-End Delta Pipelines on GCP

Delta has been powering many production pipelines at scale in the Data and AI space since it has been introduced for the past few years.

Built on open standards, Delta provides data reliability, enhances storage and query performance to support big data use cases (both batch and streaming), fast interactive queries for BI and enabling machine learning. Delta has matured over the past couple of years in both AWS and AZURE and has become the de-facto standard for organizations building their Data and AI pipelines.

In today’s talk, we will explore building end-to-end pipelines on the Google Cloud Platform (GCP). Through presentation, code examples and notebooks, we will build the Delta Pipeline from ingest to consumption using our Delta Bronze-Silver-Gold architecture pattern and show examples of Consuming the delta files using the Big Query Connector.

About Himanish Kushary

Himanish Kushary is a Practice leader with the Resident Solutions Architect team at Databricks. He helps customers across multiple domains with building scalable big data analytics solutions and products on the Databricks Lakehouse platform. He has been involved with big data technologies since 2010 and joined Databricks in 2017.

About Molly Nagamuthu

Molly Nagamuthu is a Senior Resident Solutions Architect at Databricks. She has been working their top-tier strategic customers solving some of the toughest Big Data problems at scale in both Healthcare and Media verticals. Molly has 20 plus years of experience in product development, Engineering and Professional Services. She has proven track record of working for tech startups in the Data Engineering, Analytics and AI space. She has a bachelors and masters in Computer Science and Engineering. She is a passionate facilitator of Girls who code Clubs in NJ and WIT events.