Automating Real-Time Data Pipelines into Databricks Delta - Databricks

Automating Real-Time Data Pipelines into Databricks Delta

Data engineering teams have struggled to keep up with the demand for continuously updated data sets for machine learning applications. Such integration can be a manually intensive and complex endeavor, challenging to assemble and often resulting in outdated data when finally ready for data scientists. This session will demonstrate how to overcome these challenges by efficiently transferring changed data at scale and automating the generation of data transformations in Spark to accelerate data pipelines – from the generation of source system data streams right through to the creation of analytics-ready data sets into Databricks Delta.



« back
About Dan Potter

Data engineering teams have struggled to keep up with the demand for continuously updated data sets for machine learning applications.  Such integration can be a manually intensive and complex endeavor, challenging to assemble and often resulting in outdated data when finally ready for data scientists. This session will demonstrate how to overcome these challenges by efficiently transferring changed data at scale and automating the generation of data transformations in Spark to accelerate data pipelines – from the generation of source system data streams right through to the creation of analytics-ready data sets into Databricks Delta.