We're thrilled to announce the General Availability (GA) of Databricks Asset Bundles (DABs). With DABs you can easily bundle resources like jobs, pipelines, and notebooks so you can version, test, deploy, and collaborate on your project as a unit. DABs help you adopt software engineering best practices for your data and AI projects on the Databricks Platform. DABs facilitate source control, code review, testing, and continuous integration and delivery (CI/CD) for all your data assets as code. With hundreds of customers already using DABs in production today, we are excited to offer this capability to all our customers.
DABs provide a simple, declarative format for describing data and AI projects. This format lets data engineers, data scientists, and AI developers express data and AI projects as source files - serving as end-to-end definition of how these projects are laid out, tested, and deployed. This makes it easier to collaborate on projects during active development and to manage them with best practices such as organizational templates, Git, and CI/CD (such as GitHub Actions, Jenkins, Azure DevOps, etc.).
DABs are defined and managed through configuration you create and maintain alongside source code, helping you define your entire project as source code. With custom DAB templates you can set organizational standards for new projects that include default permissions, service principals, and CI/CD configurations.
Let’s say you have a project with a job and a notebook and you want to test your updates in a development environment so it doesn’t impact your production deployment. With DABs, you can define a dev target that will isolate your changes not just from production but also from development copies your colleagues may be working on. Once you are satisfied with the changes you can deploy to production manually or using an automated CI/CD system.
Using bundles you can maintain a versioned history of your Databricks assets (like jobs, ML serving endpoints, pipelines etc.), and control changes to your environment in a consistent and testable manner. This is especially important for regulated industries assisting in change management governance that require that compliance standards are consistently met.
DABs are created manually or based on a template. The Databricks CLI provides default templates for simple use cases, but for more specific or complex scenarios, you can create custom bundle templates to implement your team’s best practices and keep common configurations consistent.
Going forward, we are working on some exciting capabilities related to DABs, including authoring DABs in the workspace, authoring DABs fully in Python (PyDABs), DABs IDE support, and adding support for all Databricks assets (including Lakeview dashboards).
We invite you to start using DABs for building your pipelines, experiments, and projects. For more information, please visit our documentation.
We look forward to seeing the creative and effective ways you'll use Databricks Asset Bundles to manage and automate your data, analytics, and AI projects.
brew tap databricks/tap; brew install databricks
databricks configure
databricks bundle init
databricks bundle validate; databricks bundle deploy
