Announcing the General Availability of Databricks Assistant and AI-Generated Comments

Today, we are thrilled to announce the general availability of Databricks Assistant and AI-Generated Comments on all cloud platforms. Our mission at Databricks is to democratize data and AI by making our platform as simple to use as possible. To that end, we are making these features available at no additional cost for all customers​​​​​​.

Last November, we shared our vision for a new generation of data platforms called Data Intelligence Platforms, which use the power of AI to revolutionize how you manage, govern, and derive insights from data. With the availability of Databricks Assistant and AI-Generated Comments, we have taken a step closer to this vision. These features are powered by compound AI systems that leverage AI models, retrieval, ranking, and personalization systems to understand the semantics of your organization’s data and usage patterns.

Databricks Assistant

With Databricks Assistant, we set out to build the best AI-powered productivity tool for enterprise data. The adoption since the preview has been overwhelming. Assistant is one of the fastest-growing Databricks features ever, with over 150,000 users leveraging it every month to automatically generate code, troubleshoot errors, and create visualizations and dashboards.   

I was able to code 200+ lines of robust code in a language I’ve never coded before. 

— Josue A. Bogran, Solutions Architect Manager, Kythera Labs

In the last several months, we’ve been working hard on enhancing the user experience and quality of responses. Here are some of the top new features:

• Data-aware context for more accurate answers: AI tools like the Assistant need access to high-quality metadata in order to produce the best results. Understanding datasets and their usage patterns is necessary to develop deep semantic understanding and is the main reason we believe richer AI functionality will reside in the data platform. Over the last year, we’ve found that incorporating relevant workspace context has had the biggest impact on the accuracy of Assistant responses. Adding additional context from stack traces, lineage, popular and favorite tables, dataframe schemas, nearby cells, and relevant documentation has greatly improved Assistant accuracy.

• Assistant on every page: We’ve moved the Assistant out of the notebook and SQL Editor and added it to every page in Databricks. This helps you carry conversations across all Databricks product pages. For example, in a single conversation Databricks Assistant could help you diagnose a job failure, recommend ways to fix the code, and help you restart the job. We’ve also added the ability for multiple conversation threads that let you create new Assistant chats or resume previous tasks.

• Inline chat for code and query refinement:  We’ve also been working on a number of UX enhancements to the Assistant. Select a line of code and use the keyboard shortcut (Ctrl+i on Windows, Cmd+i on Mac) for inline chat to quickly iterate on code without leaving the editor.

• Slash commands for easy shortcuts: Leverage semantic search to find tables with /findTables or add comments with /doc. Built-in slash commands make it easy to perform tasks without typing lengthy prompts.

• Assistant autocomplete (preview): We’ve recently introduced Databricks Assistant Autocomplete, a new AI-powered feature that offers real-time code suggestions as you type. We leverage Unity Catalog metadata and other context from your code to provide highly relevant code completions for SQL and Python.  

• Generate AI/BI Dashboard charts (preview): The Assistant can help you quickly create and iterate on visualizations in the Databricks AI/BI Dashboards editor.
  
  
  

We’re thrilled to see how you use the Assistant to improve your productivity in Databricks. We have a number of exciting Assistant features planned for the next few months and can’t wait to unveil those soon. 

“Databricks Assistant plays a pivotal role in boosting productivity. Through pair coding and leveraging Databricks Assistant’s expertise, teams are about 50% faster in their development and analytical tasks.”

— Pedro Antonio Boareto, Data Engineer, Banco Bradesco

AI-Generated Comments in Unity Catalog

Another feature launching to GA today is AI-Generated Comments in Unity Catalog. This feature leverages generative AI to provide relevant table descriptions and column comments. Since launching the feature, more than 80% of the table metadata updates on Databricks are AI-assisted.

Using AI-Generated Comments across your catalog is one of the best ways to improve Assistant answers. Providing descriptive comments for tables and columns greatly improved the accuracy of responses in our benchmarks. In our 5 tips to get the most out of your Databricks Assistant we took it one step further by manually adding examples of row-level data to our column comments. Building descriptive metadata takes time but it is one of the highest-value tasks you can undertake to improve Assistant accuracy. 

A big boost in user productivity

Since launch, we’ve been busy sending surveys to our top Assistant users to better understand how it impacts their productivity and efficiency across various tasks. Based on responses we’ve received so far, 72% of users reported time savings, and users reported a productivity boost of up to 50%. 

Here is a selection of the top comments we’ve received:

"At SiriusXM, we're using Databricks Assistant to create notebooks, author complex queries, identify ways to join tables within data lakes, and resolve coding issues, saving our data teams development time."

 — Bernie Graham, VP Data Engineering, Sirius XM

"The introduction of the Databricks Assistant has made it easier for our user base to improve their skill set. By seamlessly integrating into our development cycles, it has significantly enhanced productivity for our expert data scientists, engineers, and analysts. Users can comprehend, generate, optimize, and troubleshoot code faster than ever before. With the continuous growth of the Databricks ecosystem and the ongoing advancements of the Databricks Assistant, next-level tools are more accessible."

 — Nicholas Heier, Senior Manager, General Motors

"Assistant drives improvements in productivity across our team through the automation of repetitive coding tasks. It's also great for debugging."

 — Stephen Reid, Data Scientist, SSE Business Energy

Databricks Assistant is built for your enterprise needs

As a data and AI company focused on helping enterprise customers extract value from their data to solve the world’s toughest problems, we firmly believe that both the companies developing the technology and the companies and organizations using it need to act responsibly in how AI is deployed.

We designed Databricks Assistant from day one to meet the demands of enterprise workloads. Assistant respects Unity Catalog governance and meets compliance standards for highly regulated industries. Your data is never shared across customers and is never used to train models. For more detailed information, see Databricks Trust and Safety.

Available today

Databricks Assistant is now available for all users across all clouds. Databricks Assistant, AI Generated Comments, and Assistant Autocomplete will be available at no additional cost for all customers. 

To see Databricks Assistant in action check out our demo video to see how you can use Assistant to build data pipelines, SQL queries, and data visualizations.

If you don’t see the Assistant or AI-Generated Comments icons your administrator can follow the instructions documented here to enable Databricks Assistant in your Databricks Account and workspaces.