SciCLOps: Databricks Quick Start for Machine Learning, Powered by DABs

Early in our journey, Databricks users at our company implemented workflows individually, leading to inconsistency and slow onboarding for teams. We have since built a tool (that we call “SciCLOps”) powered by Databricks Asset Bundles (DABs) to allow for better workflow reuse and standardization across data engineering, ML, and data science. DABs combined with Terraform, Github Actions, and a service catalog has enabled:

• Users to request Databricks environments with no manual intervention by platform teams
• Much simpler adoption of changes to workflow standards
• Day two support for projects (such as automatic PAT rotation)
• Faster promotion of models and science to production due to consistent workflows

SciCLOps has made significant impact to our onboarding speed and time to production. We will give a demo of our SciCLOps tool, as well as explain our journey in building it and the observations we have made along the way.