At SpotX, we have built and maintained a portfolio of Spark Streaming applications — all of which process records in the millions per minute. From pure data ingestion, to ETL, to real-time reporting, to live customer-facing products and features, continuous applications are in our DNA. Come along with us as we outline our journey from square one to present in the world of Spark Streaming. We’ll detail what we’ve learned about efficient processing and monitoring, reliability and stability, and long term support of a streaming app. Come learn from our mistakes, and leave with some handy settings and designs you can implement in your own streaming apps.
Jack Chapa is a Senior Big Data Engineer at SpotX, playing a critical role in the big data pipelines that power the company. He's contributed to reliable 24/7 streaming applications, ETL pipelines, and infrastructure designs. Also, he loves Scala and Spark!
Landon is a Senior Big Data Engineer at SpotX, a video ad serving platform that reaches 100s of millions of people a day. He engineers pipelines that process dozens of terabytes daily in Spark, and enjoys building solutions to difficult-to-scale problems.