We hear a lot about lambda architectures and how Spark can help us crunch our data both in batch and real-time. After a year and a half in the trenches, I’ll share how we at The Weather Company built a general purpose, weather-scale event processing pipeline to make sense of billions of events each day. If you want to avoid much of the pain learning how to get it right, this talk is for you.
Robbie has been involved in the big data community for the last seven years, and he was an early Spark adopter back in 2014. He has contributed to a number of projects, including Apache Cassandra and the Cassandra Spark connector, and is the author of Cassandra High Availability. At IBM, Robbie leads a group that includes the Spark Technology Center, as well as Big Insights and other data processing technologies that power the Watson Data Platform.