Moment-Based Estimation for Hierarchical Models in Apache Spark - Databricks

Moment-Based Estimation for Hierarchical Models in Apache Spark

Download Slides

Summary: At Stitch Fix, Hierarchical models are one of the core machine learning frameworks used in our recommender systems technology. Hierarchical models allow for estimation on clustered data, when classical assumptions of identically distributed random variables break down. Traditional likelihood-based methods for fitting hierarchical models often struggle with the scale of data found in industry, which has prompted recent research into moment-based procedures for parameter estimation.

Spark doesn’t have a native library for fitting these models, and to our knowledge, no moment-based estimation software has been developed previously utilizing a distributed computational system. This talk will review our development of spark software utilizing these new estimation methods, detail the theory behind the approach, and compare our software to similar open source packages in Spark and other popular languages.

Session hashtag: #DSSAIS17

About Kyle Schmaus

Kyle Schmaus is a Data Scientist at Stitch Fix, where he leads the Styling Recommendations group. He and his team are responsible for the core recommender systems capabilities used by Stitch Fix stylists to select appropriate clothing for clients. He has a master degree in Mathematics from San Francisco State University.