As Technical Lead Data Products at wetter.com, Carsten is responsible for the IT of their b2b data products. This includes architecture, development, and operations for the cloud infrastructure and analytical data pipelines.
Over the last few years Carsten has built and run analytical data platforms and applications with a strong focucs on technologies like Hadoop, Kafka, and Spark – both on-premise and on AWS. Further, he has a long track record in data warehousing and BI.
November 18, 2020 04:00 PM PT
At wetter.com we build analytical B2B data products and heavily use Spark and AWS technologies for data processing and analytics. I explain why we moved from AWS EMR to Databricks and Delta and share our experiences from different angles like architecture, application logic and user experience. We will look how security, cluster configuration, resource consumption and workflow changed by using Databricks clusters as well as how using Delta tables simplified our application logic and data operations.
From a data scientist/engineers perspective I show how our daily analytical and development work has improved. Many of the points addressed here also apply when moving from some other Spark platform like Hadoop to Databricks.
Speaker: Carsten Herbe