Himanish Kushary is a Practice leader with the Resident Solutions Architect team at Databricks. He helps customers across multiple domains with building scalable big data analytics solutions and products on the Databricks Lakehouse platform. He has been involved with big data technologies since 2010 and joined Databricks in 2017.
May 26, 2021 03:15 PM PT
Delta has been powering many production pipelines at scale in the Data and AI space since it has been introduced for the past few years.
Built on open standards, Delta provides data reliability, enhances storage and query performance to support big data use cases (both batch and streaming), fast interactive queries for BI and enabling machine learning. Delta has matured over the past couple of years in both AWS and AZURE and has become the de-facto standard for organizations building their Data and AI pipelines.
In today’s talk, we will explore building end-to-end pipelines on the Google Cloud Platform (GCP). Through presentation, code examples and notebooks, we will build the Delta Pipeline from ingest to consumption using our Delta Bronze-Silver-Gold architecture pattern and show examples of Consuming the delta files using the Big Query Connector.