Dillon Bostwick has spent the last 2 years as a Solutions Architect at Databricks, where he provides support and architectural guidance for customers using the Databricks’ managed Spark, MLFlow, and Delta Lake platform for their machine learning and big data projects. He is also active in developing new projects in the field that apply devops best practices to accelerate the deployment of Databricks’ infrastructure. Previously, Dillon was a software engineer in the software defined networking space and security space. He was a technical lead for an application intended to automate the orchestration of virtualized network elements at scale.
Continuous Integration and Continuous Delivery are common patterns in software development methodologies over the last decade. CI and CD have evolved and converged but the core message remains: build repeatable processes, automate and version everything. These learnings are based off of decades of experience in traditional software engineering. In parallel, we observe a number of distinct but similar concerns in machine learning: governance, reproducibility, observability, and concept drift, not to mention concerns brought to light in the Google NIPS classic 'Hidden Technical Debt in Machine Learning Systems,' such as entangled data dependencies. For example, a deployed model must be monitored and updated regularly to prevent decay due to concept and data drift. As a result, new paradigms such as MLOps and Martin Fowler's 'CD4ML' (Fowler, 2019) seek to acknowledge the potential application of CI/CD principles to data science, while recognizing the fundamental differences between traditional software and machine learning. We cannot brute force the learnings and wisdom of traditional software development on data science without deep understanding of the cultural and technical differences, which often extend beyond modeling into data preparation and monitoring.