Easy ways to optimize your costs

Databricks was built as an open and unified platform to handle huge data workloads at a fraction of the cost of other solutions. As innovators in the data and AI space, we continually push technology forward to find new ways to increase performance, efficiency and, most importantly, ROI for our customers.

We’re a data company. We want our customers to have visibility into how their organization is using our platform and where there are opportunities to maximize the value of their data and AI projects on Databricks.

Understand and manage your Databricks usage

Let’s walk through the process of understanding the billing and cost data in your systems tables using our pre-built AI/BI Cost Dashboard, organizing your consumption data using our best practices for tagging, and creating budgets to manage your spending at an organization, business unit or project level.

System Tables - A robust source of data

It is probably no surprise that the first step to understanding your usage is accessing data about it. Databricks provides usage data in the form of system tables, which are a Databricks-hosted analytical store of your account’s operational data. In this post, we’ll focus on using data in the billing system tables.

 Tagging - The best way to organize costs

As your company’s usage of Databricks grows, you’ll want to break the costs up by team, project or department to make it easier to tie costs to the value they produce. To do so, we recommend using tags, which allow you to create flexible key:value pairings. You can apply tags on clusters, SQL warehouses, jobs and serverless workloads. Tags are the best way to track what’s happening in your Databricks environment.

It’s usually best to start with general keys that track usage to a business unit and a project, then move to more specific tags as needed to support optimizing your data and AI initiatives. Predefine the values for each key. Use policies to enforce usage of the tag key:value pair and control the tag values with the predefined list. Then you can use your dashboard to see usage by tag value and identify which workloads haven’t been tagged yet.

Tags can be enforced using policies, which allow you to require that new objects have a tag on them. This will help you maintain completeness and accuracy.

 Budget Alerts - Get proactive about anomalies

Finally, we understand that you may not want to spend all your time in cost management dashboards and Genie. That’s why we created budgets. You can create budgets for your whole organization, specific workspaces, or use your tags to assign budgets to specific business units or projects.

Once you’ve created your budgets, you can set email alerts to notify the people and teams in your organization when their usage exceeds its planned level. Then, you can use your dashboards to identify the project, workload and/or user who triggered the extra usage.

AI/BI Cost Dashboard - Visualize and understand trends

To help you quickly understand your costs, we’ve included a pre-built AI/BI dashboard that allows you to view your usage over time and drill into your workspaces and SKUs. This can help you quickly identify areas that need your attention, such as a sudden spike in project costs.

AI/BI Genie - Explore Further

Sometimes dashboards don’t have the answers you’re looking for. AI/BI Genie can provide insights on spending trends, anomalies, and cost-saving recommendations through a natural language interface.

Start saving today

If you’re not using your billing data to optimize your Databricks usage, there’s no better time to start than right now. Get oriented with your usage patterns with our built-in dashboards, then start developing your cost management strategy using tagging and budgets. If you need help, ask your account team to set up a deep dive session on cost management strategies.