Onboarding your new AI/BI Genie

At Databricks, we want to make data and AI accessible to everyone on the planet. This is why we're building solutions like AI/BI that make it possible for business users, even those who don't speak SQL or write code, to get trusted insights quickly and accurately.

At this point, dashboards are part of most people's daily lives. On the other hand, AI/BI Genie is a newer experience and hence one that many people might not be familiar with yet. With Genie, users can self-serve and obtain answers to questions not addressed in their dashboards without having to learn how to use BI tools or rely on expert practitioners to create insights for them. With this backdrop, we figured it might be worthwhile to take a look at a day in the life of a Genie so that you can get to know how to use it and what to expect.

New hire orientation

Some folks have made the analogy that the process of creating a Genie is akin to hiring a new analyst for your data team. This new hire is smart and hungry to learn, but is a tad robotic, and has zero experience with any of your internal terminology, metrics, definitions, and assumptions. Your new Genie is well-read in public knowledge but very ignorant of company-internal and team-internal knowledge.

So, the first step is to have someone give your new hire some much-needed orientation! In this blog post, we're going to assume that's you, the experienced analyst who has been fielding all of the team's questions, perhaps drowning in them and looking for some help.

The Genie setup page is your guide to what this new hire needs to know, in order to start attempting to answer any questions at all:

Here, the goal is to provide the absolute basics to get Genie going:

• Title and Description: What is the scope and domain of questions they're expected to know how to answer? This is visible to anyone browsing and searching across all of the Genie spaces they can access, so it's important to make this clear and succinct.
• Default warehouse: What compute will this Genie use to run SQL queries for you?
• Tables: What tables are in scope for this Genie to use to answer questions? Genie will read up on all of the comments in each table and in each of its columns and use these to inform its answers.
• Sample questions: Optionally, you can specify some example questions you want to present to people who are talking to this Genie if they're not sure what to ask.

Once you've chosen sensible values for these, hit "Save" and your new Genie is ready for action!

…well, not quite. If you were training a new hire, you'd probably want to check their knowledge first before letting them go off to answer your team's hardest questions.

To ensure your new hire has the best possible start in their new career you'll want to bootstrap their knowledge with some basic information. If you've invested in some basic data governance best practices you'll ensure they get a flying start. This is where Unity Catalog can help. Make sure you have created some useful comments in your tables and columns, and take the time to create primary/foreign key relationships between the tables you want to give to Genie. This basic information will ensure Genie has a lot of basic knowledge on its very first day!

The next step is to ask Genie a few things that you'd expect your team to ask in daily practice.

In this first example, Genie easily gets the question correct.

In Catalog Explorer (above) if we look at the table metadata, we can see the column comments provide enough information for Genie to get the answer right. Yay!

In this second example, Genie guessed wrong because the most obvious definition it infers from the column name for average price turns out to be wrong.

When we view the underlying SQL we can see that Genie picked the wrong column for price. Genie should have picked the ListPrice column, not UnitPrice.

We can fix this by giving the desired column a useful clarifying comment in Catalog Explorer:

With this update, Genie will now produce the right answer.

Above we can see that the user has asked Genie the exact same question as before.

Above we can see that Genie is now using the correct column ListPrice in its query.

Next is another example where Genie won't even try to guess because it knows that it lacks an important context—how a particular team-specific metric is defined. Without this context, Genie has no way of deducing the right answer.

So let's teach Genie this metric! We can just directly provide the metric to Genie and see what happens.

We simply tell Genie the definition of customer churn directly in the UI and run the prompt.

Above we can see that Genie has understood the prompt and is able to query the data correctly.

Now that we've confirmed that Genie learned how to calculate this new metric correctly, we can tell it to remember this for everyone talking to it going forward by clicking "Add as instruction."

This will result in the instruction being saved into the Instructions section, which are all of the things you want Genie to keep in mind as it's doing its job. This one can be found in "Example SQL Queries", but you can manually update any of the instructions as well.

Before releasing your Genie space into the wild, it is recommended that you add as many of these types of instructions as possible for the known/anticipated questions and business semantics you know your users will use. You can do this by saving instructions as you chat or by adding them directly to the Instructions section in your space. The more you train your Genie, the smarter it gets!

Once you've tested Genie enough to be confident that it's providing useful and correct answers to the questions you anticipate your team asking, it's time to let your team talk to your new hire.

Hit "Share" to tell Genie who can talk to it. Depending on who this Genie is supposed to answer questions for, you can choose individual people, a group representing your whole team, or the whole company.

You should also tell your team about what to expect from Genie, since they've probably never worked with such a robotic teammate before. Remember that they probably can't speak SQL, so they might not have a way to tell whether Genie is actually giving them the right answer or just making up a really good guess that's actually wrong. You can tell your team about what questions you expect Genie to answer correctly, and also generally if they have a mission-critical question but aren't sure whether Genie is answering correctly, they should check with you before using it.

Getting to work

So now that you've unleashed your new Genie teammate on your team, let's see how a member of your team interacts with it.

Just like what you did in testing Genie, anyone on the team can just start asking questions and getting answers back.

Your users can give feedback by clicking on the thumbs-up/thumbs-down buttons after seeing Genie's response. These votes show up in the Monitoring section, where you can see the questions your users are asking, and identify new things you need to teach Genie to improve it.

Here, you can view all of the questions that have been asked and any thumbs-up/thumbs-down feedback from your team.

What's most interesting to look for might be questions that you hadn't anticipated earlier, where you're not yet sure whether Genie might answer correctly. For each question, you can look at how Genie answered to check whether it was correct. If it wasn't correct, that's an opportunity to hop into a chat session of your own to teach Genie the right answer.

You can also filter for "thumbs down" interactions because often one of two things happens. First, Genie might've answered incorrectly, so you'd need to teach Genie something it doesn't know. Second, you might think that Genie answered correctly, but your teammate talking to Genie might have different assumptions than you do, leading them to conclude that Genie is not answering correctly or usefully. It'll be helpful to clear up those differences in assumptions.

Learn more about Genie

As you can see, much of the goal of Genie is to make it possible for anyone on any team to be able to get the insights they're looking for, without overtaxing or waiting for the more technical members of their team to get around to answering those questions. We hope you've found these examples helpful, and hope that you'll give Genie a try. For a deeper look at Genie, check out the Databricks documentation around Genie, especially the best practices we've summarized. Please let us know if you have feedback as you do because we're looking forward to learning about your experiences with Genie!