Explain Yourself: Why You Get the Recommendations You Do - Databricks

Explain Yourself: Why You Get the Recommendations You Do

Download Slides

Machine learning recommender systems have supercharged the online retail environment by directly targeting what the customer wants. While customers are getting better product recommendations than ever before, in the age of GDPR there is growing concern about customer privacy and transparency with ML models. Many are asking, just why am I receiving these recommendations? While the current Implicit Collaborative Filtering (CF) algorithm in spark.ml is great for generating recommendations at scale, its currently lacks any method to explain why a particular customer is getting the recommendations they are getting. In this talk, we demonstrate a way to expand collaborative filtering so that the viewing history of a customer can be directly related to their recommendations. Why were you recommended footwear? Well, 40% of this recommendation came from browsing runners and 20% came from the shorts you recently purchased. Turns out, rethinking of the linear algebra in the current spark.ml CF implementation makes this possible. We show how this is done and demonstrate its implemented as a new feature to spark.ml, expanding the API to allow everyone to explain recommendations at scale and create a more transparent ML future.



« back
About Niels Hanson

I am currently a Lead Specialist Data Scientist within KPMG's Lighthouse Data & Analytics Center of Excellence. My focus here is on the development of large-scale machine learning pipelines for data-driven decision making in a number of industries, including financial services, retail, and health care. I also like to use my diverse experience with a wide variety of data types to take a novel perspective on classical analytical problems. Some of my specializations include machine learning, software development, graph analytics, mathematical modeling, clustering, database design, information visualization, high-dimensional data mining, and distributed computing.

About Kishori Konwar

Kishori M. Konwar is a Computational Biologist at the Broad Institute of MIT and Harvard where he develops distributed algorithms for large-scale biological problems. His past appointments have been at the MIT Computer Science & Artificial Intelligence Lab (CSAIL) and at the Department of Microbiology at the University of British Columbia. He has many interests including computational biology, marketing, engineering, business intelligence, finance, or any fields that are being transformed by the advent of large-scale data. In his free time, Kishori enjoys playing Tennis and Badminton.