Building a Recommendation Engine Using Diverse Features - Databricks

Building a Recommendation Engine Using Diverse Features

Download Slides

Building a simple recommendation system is extremely easy using standard tools available through many open source packages. For example, Spark MLlib provides an implementation of state of the art matrix factorization algorithms that could be used to build a recommendation system that can digest large amounts of data. However, such a system only uses a small number of features (e.g. purchase or view history) to build a recommendation system and throws out a large number of features that could potentially be used to improve the quality of a recommendation engine. In this talk, we will discuss how Sailthru leverages diverse features about users and items to build a recommendation system that will be used by several media and ecommerce companies. In particular, we will discuss the following three topics: (a) How we make use of Spark SQL to extract user and item level features, (b) How we run Spark code in production and best practices for building effective Spark application, (c) How we make use GBMs to use diverse features and various algorithms to make final recommendations for each user, and (d) How we make use of Spark MLlib to make recommendations for millions of users by scoring over ten thousand items per user.



« back