Accelerating Spark Machine Learning with Redis Modules - Databricks

Accelerating Spark Machine Learning with Redis Modules

Download Slides

With the new ability to extend Redis with native modules, we explore the benefits of using them in conjunction with Spark: By storing Machine Learning algorithms’ models in Redis, and using Redis modules, we can offload the processing of these models directly to Redis. This allows fast classification and other processing, without the costs of loading the model data into Spark first. This will cover: * An overview of Redis modules. * How we implemented the technique for selected algorithms. * How we integrated this with Spark. * Benchmark showing the performance gains.

About Dvir Volk

Dvir is a veteran engineer, entrepreneur, open source advocate and even a former newspaper editor. He began his tech career developing his own massively scalable web search engine which is now integrated into many of Israel's top websites. He later went on to co-found the social networking start-up ILCU, and served as Director of Engineering at Giraffic, a P2P video distribution provider. In his last role as Chief Architect at mobile start-up EverythingMe, Dvir designed the company's entire infrastructure around Redis, becoming one of the earliest members and active evangelists of the Redis community.

About Shay Nativ

Shay is an experienced software developer, architect, and entrepreneur. He was the founder and VP R&D of Peak-Dynamics—an energy saving solution for water utilities and CTO at Utab, a web platform for musicians. Shay loves solving complex problems and writing performant code.