Adding Intelligence to Databricks Search

We are thrilled to announce major improvements to the search capabilities in your Databricks workspace. These enhancements build on DatabricksIQ, the Data Intelligence Engine within the Databricks Platform, to provide a more intelligent AI-powered search experience, enabling you to leverage natural language and semantic search to find the most relevant content and discover new insights from your enterprise data and AI assets.

Search that uses AI-generated metadata

One of the main benefits of Databricks Search is its utilization of AI-generated table and column comments for your data managed in Unity Catalog. These comments enable the search engine to understand the meaning and semantics of your data, providing the necessary context to generate much more relevant, accurate, and actionable results.

AI-generated comments are powered by DatabricksIQ. For search use cases, DatabricksIQ employs a large language model (LLM) specifically tuned for enterprise data, drawing from example schemas across a variety of  industries. DatabricksIQ not only facilitates context awareness in search but also enhances other AI-powered tools such as the Databricks Assistant for Notebooks, SQL editor, and Lakeview Dashboards.

What’s new in Databricks Search?

Let's take a closer look at the full list of the enhancements we’re bringing to Databricks Search in this release.

Full page search

To start, we’ve added a new full page experience to give you more space to see results, more metadata about the results, and more filters to narrow down your results.

Semantic search

In addition to matching keywords in the search query and assets, search also interprets the meaning of words and allows natural language queries. This means it will retrieve assets with semantic similarity to the most critical parts of the search query and then combine results with keyword search to deliver the best possible outcome. 

Example: The search query “What should I use for geographies” will focus on “geographies” and find related terms containing geographic attributes such as cities, countries, territories, geo-locations, etc.

Search query understanding

Search can now understand the patterns in your search queries by separating what may be a search term from a filter, which means that natural language queries become even more powerful. 

Example: The search query “Show me tables about inspections” will be broken down so that “inspections” is the key term and “table” is the type of object the user is looking for.

Improved relevance with popularity 

Search now uses popularity signals based on how often other users in your workspace are interacting with specific assets to improve how objects are ranked. 

Example: Without popularity boosting, if I’m looking for the food inspections table and I search for “food_inspections”, I can get several tables with “food_inspections” that are returned as results and I won’t know which is the authoritative table until I try querying it. With popularity boosting, the most popular one will get ranked higher so I don’t have to guess and check to see which is the correct one.

As you can see above, we also added visual popularity indicators so that you can quickly see the popularity of each result.

You can also change how search results are ranked with the new “sort by popularity” option. 

Knowledge cards

The top result will turn into a knowledge card when we can identify with high confidence what you’re looking for. This makes it easier to see the best result as well as provides you with additional assset metadata. Currently, knowledge cards are only supported for tables, but we will expand to other objects in the future.

Quick filters

If you know exactly what type of object you’re looking for, quick filters allow you to get to filter for that object type with one click.

Search filter syntax

In addition to specifying filters in the UI, you are also able to specify filters through your search query in the search bar. To learn more about how to specify your filters via syntax, try applied filters in the UI and see how the query in the search bar automatically updates.

Search URL 

If you have common searches that you’re finding yourself going back to often or if you want to share search results with another user, the URL for the search page contains your full query so you can bookmark and share the URL to reproduce the same search results.

Bonus features 

• Recents: When you click on the search bar and before you type any search terms, the dropdown will show your recently viewed objects. This is an easy way to quickly navigate to your recently viewed objects without having to go to the “Recents” or “Home” pages.
• Keyboard shortcut: Use the “CMD+P” (for Mac) / “CTRL+P” (for Windows) keyboard shortcut to get to the search bar without taking your hands off the keyboard. This makes it even easier to search and navigate it to recents (see above).
• Open the search page in a new tab: Instead of pressing “Enter” to search for the terms you’ve typed in the search bar, press “CMD+Enter” (for Mac) / “CTRL+Enter” (for Window) so that the search results will open in a new tab. This will help you preserve the current page you’re working on and let you get back to your original work after you’re done with search without having to refresh the page. 
• Empty search: If you want to see all objects that match a certain filter criteria (e.g., all notebooks owned by me), try performing an “empty” search with only filters and no terms (i.e., “type:notebook owner:me)”. One scenario where this comes in handy is if you want to see all dashboards created by a subject matter expert.
• Search for code: You not only can search for notebooks and queries by their names, but you can also search for the content inside them. This means that you can search for things like function names or table names to see how they're referenced. Two added bonuses are that 1) If all you need is a preview of the matching content inside a notebook/query then you'll be able to see a snippet in the search results, and 2) If you need more than what's shown in the snippet, the notebook/query will automatically scroll to where the search term matched when you click on the search result to open up the object.

What are the use cases for a more intelligent search? 

Intelligent search is critical for working effectively with the data and AI assets in your Databricks Platform, and there are two primary use cases that we are solving; navigation and discovery.

Navigation is all about helping users find what they are looking for quickly and efficiently. This usually implies the user already knows specifically what they want to find. For example, you’re looking for a table called food_inspections, and you just want to get to it as fast as possible. You type “food inspections”, “food_inspections”, or maybe just “food table” into the search box to quickly see all the tables that match your search term.

Discovery is different in that users might have a general notion of what they want but don’t know what specific things to search for.  Building on the previous example, maybe you type “show me everything related to Chicago restaurant grades” into the search box to discover anything that is possibly relevant, including tables, notebooks, jobs, SQL queries, dashboards, ML models, and so on. With discovery use cases, all relevant items are returned, regardless if they match the specific words typed into the search box or not.

We feel that effectively supporting both navigation and discovery use cases is essential for our customers which is why we have invested heavily in adding intelligence to our existing search capabilities. 

Don’t forget to send your feedback

We have come a long way in our journey to intelligent search at Databricks, but we are just getting started, and many more improvements are on the way! As you use these new features, please let us know what you like, don’t like, and what you want to see in future enhancements. To help you do this, we even built a “Send Feedback” link at the top of the search page for you to add your comments and suggestions.

Try out the new Databricks Search today!

Using these new intelligent search is easy, just log into your Databricks Workspace and try it out. It's available in the latest version of Databricks and does not require any additional licenses. To learn more about all of the Databricks Search capabilities, please also read your product documentation. 

Happy searching!