How We Used Databricks, MLeap, and Kubernetes to Productionize Spark ML Faster – Databricks

How We Used Databricks, MLeap, and Kubernetes to Productionize Spark ML Faster

Download Slides

AutoTrader is the UK’s leading digital automotive marketplace. We receive 60 million cross-platform visits each month, while our ML-powered car valuations provide 5.5 million valuations a month to both consumers and dealers.

At AutoTrader, our development culture aims to combine a data-driven approach with continuous delivery. As such we are keen to ensure that any new machine learning models we develop fit this way of working; our Data Scientists should be able to directly build, train, re-train and deploy their models with minimal outside input. Reducing the time to live allows for more experimentation and reduces the cost of getting machine learning models into production.

This talk will describe how we launched our new “Days to Sell” ML-powered metric, using Databricks notebooks to experiment with and tune our model, MLeap to serve out predictions in real time, and Kubernetes to automate the deployment of new models. We will show how we went from model experimentation and design, right through to deploying our model to a production environment. We will also share our thoughts on how we expect this approach to evolve in the future, including how we expect to integrate with MLflow.

Session hashtag: #SAISEnt9

« back
About Edward Kent

Edward is a Senior Developer at AutoTrader. Working in the Data Engineering team, he uses technologies including Spark, Kafka, Scala, Python and Java on a daily basis. Edward has a strong interest in transitioning models from prototype to production-ready and deploying them at scale. He received his PhD in Chemical Engineering and Analytical Science from the University of Manchester in 2013.