Alexandre is a Data Analyst & Solution Architect indepedent—MCSE, Cosmos DB & Delta lover, working for different clients, including SNCF. He developed his skills through various clients’ projects, teaching at the University and personal proof of concepts. He’s also the Co-Founder of DataRedkite, a product which can quickly give to its user a good management of data in Microsoft Azure DataLake.
November 18, 2020 04:00 PM PT
A major cause of dissatisfaction among passengers is the irregularity of train schedules.
SNCF (French National Railway Company) has distributed a network of beacons over its 32,000 km of train tracks, triggering a flow of events at each train passage. In this talk, we will present how we built a real-time data processing on these data, to monitor traffic and map the propagation of train delays.
During the presentation we will demonstrate how to build an end to end solution, from ingestion to exposure.
The presentation will take place as follows:
-Data Pipeline: how we set up a data transformation pipeline using Spark 3 and Delta with Azure Databricks and how Delta Lake makes dynamically updated data reliable
-Exposure: how we expose our output in the best way depending on the consumer Power BI or REST API.
-Production-ready: finally, we will demonstrate how we have structured our development process to make it reliable and aligned with SNCF best practices.
Speakers: Alexandre Bergere and Kaoula Ghribi