Sumanth Venkatasubbaiah is a Senior Engineering Manager at Intuit, where he is responsible for building scalable AI services and capabilities aimed at helping Intuit to become an AI-driven expert platform. He has previously built Big data and ML systems at Verizon and Apple. Sumanth holds a Masters in Computer Science from the University of North Carolina at Charlotte.
May 26, 2021 04:25 PM PT
Intuit products increasingly rely on AI solutions to drive in-product experiences and customer outcomes (a realization of Intuit’s AI-driven expert platform strategy). In order to provide complete confidence to Intuit customers through reliable and predictable experiences, we need to ensure the health of all AI solutions by continuously monitoring, managing and understanding them within Intuit products.
At Intuit, we have deployed 100's of Machine Learning models in production to solve various problems as below:
With so many models in production, it becomes very important to monitor and manage these models in a centralized manner. With very few open source tools available to monitor and manage ML models, data scientists find it very difficult to properly track their models. Moreover, different personas in the organization are looking for different information from the models. For example, the DevOps team is interested in operational metrics. Financial analysts are interested in determining the operational cost of a model and the legal and compliance teams might want to find if the models are explainable and privacy compliant.
At Intuit, we have designed and developed a system that tracks and monitors ML Models across the various Model development lifecycle stages. In this Summit, we will be presenting the various challenges in building such a central system. We would also share the overall architecture and the internals of this system.
[daisna21-sessions-od]
June 25, 2020 05:00 PM PT
The presentation introduces Intuit AI Model Monitoring Service (MMS). MMS is an in-house Spark-based solution developed by Intuit AI to provide ongoing monitoring for both data (statistics of model input/output etc.) and model metrics (precision, recall, AUC etc.) of in-production ML models. The project is soon to be open-source. MMS aims to tackle multiple challenges of in-production ML model monitoring: