Shay is an experienced software developer, architect, and entrepreneur. He was the founder and VP R&D of Peak-Dynamics—an energy saving solution for water utilities and CTO at Utab, a web platform for musicians. Shay loves solving complex problems and writing performant code.
With the new ability to extend Redis with native modules, we explore the benefits of using them in conjunction with Spark: By storing Machine Learning algorithms' models in Redis, and using Redis modules, we can offload the processing of these models directly to Redis. This allows fast classification and other processing, without the costs of loading the model data into Spark first. This will cover: * An overview of Redis modules. * How we implemented the technique for selected algorithms. * How we integrated this with Spark. * Benchmark showing the performance gains.
Redis-ML is a Redis module for high performance, real-time serving of Spark-ML models. It allows users to train large complex models in Spark, and then store and query the models directly on Redis clusters. The high throughput and low latency of Redis-ML allows users to perform heavy classification operations in real time while using a minimal number of servers. This unique architecture enables significant savings in resources compared to current commonly used methods, without loss in precision or server performance. This session will demonstrate how to build a production-level recommendation system from the ground up using Spark-ML and Redis-ML. It will also describe performance and accuracy benchmarks, comparing the results with current standard methods. Session hashtag: #SFeco13