Metadata-Driven Streaming Ingestion Using DLT, Azure Event Hubs and a Schema Registry

At Plexure, we ingest hundreds of millions of customer activities and transactions into our data platform every day, fuelling our personalisation engine and providing insights into the effectiveness of marketing campaigns.

We're on a journey to transition from infrequent batch ingestion to near real-time streaming using Azure Event Hubs and DLT. This transformation will allow us to react to customer behaviour as it happens, rather than hours or even days later.

It also enables us to move faster in other ways. By leveraging a Schema Registry, we've created a metadata-driven framework that allows data producers to:

• Evolve schemas with confidence, ensuring downstream processes continue running smoothly.
• Seamlessly publish new datasets into the data platform without requiring Data Engineering assistance.  

Join us to learn more about our journey and see how we're implementing this with DLT meta-programming - including a live demo of the end-to-end process!