Mathieu Léger is a software engineer at PAYBACK working on its realtime decisioning framework using Spark Streaming.
November 17, 2020 04:00 PM PT
PAYBACK (part of the American Express Group) is a multi-partner loyalty program, rewarding customers for shopping at PAYBACK's partner stores. One of PAYBACK's key challenges is to find the most relevant offers for each customer in real-time. These offers include the assignment of coupons, which entitle customers to additional savings on their purchases. To achieve this goal, the data engineering team uses Spark Streaming to apply complex decision logic. Due to constantly evolving business requirements, this decision logic needs to be configurable at runtime and without any manual deployment effort. With this use case as an example Alexander Boyce and Mathieu Lager will explain how to implement a stateful streaming application with Spark. In particular they will focus on how to read external sources to update internal state, how to broadcast internal state to all executors and how to restore internal state after a restart or failure.
Speakers: Alexander Boyce and Mathieu Leger