Dvir is a veteran engineer, entrepreneur, open source advocate and even a former newspaper editor. He began his tech career developing his own massively scalable web search engine which is now integrated into many of Israel’s top websites. He later went on to co-found the social networking start-up ILCU, and served as Director of Engineering at Giraffic, a P2P video distribution provider. In his last role as Chief Architect at mobile start-up EverythingMe, Dvir designed the company’s entire infrastructure around Redis, becoming one of the earliest members and active evangelists of the Redis community.
With the new ability to extend Redis with native modules, we explore the benefits of using them in conjunction with Spark: By storing Machine Learning algorithms' models in Redis, and using Redis modules, we can offload the processing of these models directly to Redis. This allows fast classification and other processing, without the costs of loading the model data into Spark first. This will cover: * An overview of Redis modules. * How we implemented the technique for selected algorithms. * How we integrated this with Spark. * Benchmark showing the performance gains.
Getting Ready to use Redis with Apache Spark is a technical tutorial designed to address integrating Redis with an Apache Spark deployment to increase the performance of serving complex decision models. To set the context for the session, we start with a quick introduction to Redis and the capabilities Redis provides. We cover the basic data types provided by Redis and cover the module system. Using an ad serving use-case, we look at how Redis can improve the performance and reduce the cost of using complex ML-models in production. Attendees will be guided through the key steps of setting up and integrating Redis with Spark, including how to train a model using Spark then load and serve it using Redis, as well as how to work with the Spark Redis module. The capabilities of the Redis Machine Learning Module (redis-ml) will be discussed focusing primarily on decision trees and regression (linear and logistic) with code examples to demonstrate how to use these feature. At the end of the session, developers should feel confident building a prototype/proof-of-concept application using Redis and Spark. Attendees will understand how Redis complements Spark and how to use Redis to serve complex, ML-models with high performance. Session hashtag: #EUai4