Databricks on Databricks - Transforming the Sales Experience using GenAI Agents

At Databricks, our automation vision is to automate all aspects of the business, making it better, faster, and cheaper.  For the sales teams, we are digitally transforming our seller experience by providing genAI agents that assist the seller across the sales lifecycle.  Our goal is to augment the seller experience with AI capabilities by seamlessly integrating them into their day-to-day tasks and providing a simpler, more effective way for sellers to retrieve information assets as well as orchestrate actions by automating repetitive manual administrative tasks.  

Our “Field AI Assistant” is built on the Databricks Mosaic AI agentic framework and provides a way for sellers to query and interact with data across multiple data sources.  It integrates with several key platforms including:

1. Our Internal Databricks Lakehouse for account intelligence, sales enablement content, and sales playbooks
2. Our Customer Relationship Management Platform (CRM) system
3. Our Collaboration Platform collates and indexes most of our nonstructured data

The AI application is used to:

• Conversationally interact with data across multiple data sources using natural language (starting with English)
• Ability to download and create documents based on the information gathered 
• Take actions based on the data insights (update fields in our CRM, draft a personalized outbound prospecting email, create a tailor-made customer proposal, prep for a customer meeting, etc. 

The field assistant responds to seeded prompts based on user and page context and also provides a chat-like interface for open-ended queries on the above-mentioned datasets.

Business Impact 

Sellers are typically overwhelmed with the volume of information thrown at them.  They need access to data residing in various siloed applications, as part of their normal day-to-day routine.  They require easy access to account, opportunity, and use case data that resides in our CRM, as well as customer market insights and account intelligence, including account consumption data that resides in our lakehouse.  In addition, they also need access to sales content - enablement playbooks, competitive sales collateral as well as product knowledge base articles and product roadmap documents.  It isn’t just limited to data retrieval, but the true efficiency gains occur when the repetitive manual tasks they perform on a daily basis based on the data insights they retrieve can be fully automated.  That is exactly what the role of the field AI assistant is - help the sellers in the day-to-day tasks including information retrieval, distilling the insights from the information, and performing actions based on those insights. 

Solution Overview

Using the Databricks Mosaic AI agent framework, we built a field AI assistant by integrating both structured and unstructured data from multiple data sources. The solution provides a comprehensive approach personalized and tailored for our sellers, available on-demand in our CRM. Some of the capabilities offered include: 

Customer insights provide a 360-degree customer account view with: 

• Financial news/insights about the account
• Competitive data landscape 
• Product consumption by product line and cloud 
• Customer support cases 
• Top Use Cases Driving Revenue
• Seller Recommendations on other use cases that are offered to similar customers 

Data hygiene alerts 

• Use cases that are going live in the next week/month/quarter
• Top use case blockers 
• Use cases that lack key information (ie exec business sponsor etc.)

Sales collateral 

• Sales playbooks 
• Competitive collateral
• Meeting summarization 
• Pitch decks 

Orchestrate action

• Update CRM with the next steps on specific opportunities or use cases 
• Draft a prospecting email to a new customer contact 
• Create a customer-facing proposal 

Our field AI assistant solution is built entirely on our Databricks tech stack. It allows integration into multiple and diverse data sources and provides a scalable infrastructure framework for data retrieval, prompting, and LLM management. It is built using the compound AI agentic framework and supports the addition of multiple tools (SQL queries, Python functions) that are all governed through our Unity Catalog governance layer.

Agent / Tool Framework

Human inputs are inherently ambiguous; LLMs have now given us the ability to use context to interpret the intent of a request and convert this into something more deterministic. To service the request, it might be necessary to retrieve specific facts, execute code, and apply a reasoning framework based on previously learned transformation. All of this information must be reassembled into a coherent output that is formatted correctly for whomever (or whatever) will consume it.  

That is exactly what the field AI assistant does to respond to the queries from the sellers.  The field AI assistant has 1 driver agent and multiple tools and functions that perform the deterministic processing.  

• Data foundation:  This is the set of data sources that the agent interacts with.  In our solution, this data foundation includes data in our Lakehouse, sales collateral, Google docs as well as data that resides in our CRM (Salesforce). 
• Deterministic processing: The set of functions and tools required to produce correct, high-quality responses. The LLM can extract fields from a query and pass these to a standard function call to do deterministic processing. Within the Databricks Platform, the Mosaic AI Tools and Functions capabilities enable this and user-defined functions can perform most activities inside Databricks. These can be typically Python functions or simple SQL queries or APIs that integrate with external apps such as Glean, Perplexity, Aha etc. and these can be invoked using natural language.
• LLM models: We leverage Azure OpenAI, GPT 4 as the foundational model for the field AI assistant solution. That said, the framework supports a multi-model approach where the specific capabilities of each model is evaluated with respect to how it deals with specific use cases. For e.g. we have evaluated our solution with various open source models and we chose Azure Open AI - GPT 4 as the model for our solution based on the groundedness of the model, its ability to generate factual and relevant content, its ability to pick the right user-defined function / tool for processing each prompt, and its ability to adhere to the content output formatting prompting provided to the model.  

That said, our solution architecture is designed to allow for flexibility in adopting new models as they become available in our Mosaic AI agent framework. 

At Databricks, we have leveraged the Mosaic AI Agent Framework which makes it easy to build a genAI application like the field AI assistant. Using this framework, we have defined evaluation criteria and we leverage LLM-as-a-judge capability to score the application responses. The Mosaic AI Gateway provides access controls, rate limiting, payload logging, and guardrails (filtering for system inputs and outputs). The gateway gives the user constant monitoring of running systems to monitor for safety, bias, and quality. 

The components that we leveraged for our field AI assistant are:

• Embedding models and vector databases (Mosaic AI Vector Search)
• Function calling and tool usage interface (Mosaic AI Function Calling)
• Customization of LLMs by prompting or fine-tuning (MLFlow LLMOps, DSPy on Databricks, Mosaic AI Fine-tuning)
• Monitoring, rate limiting and guardrails (Mosaic AI Gateway)
• Unity Catalog for discoverability, governance, and cataloging of the underlying data set, agent and tools

Solution Architecture

Our Learnings

Data is messy - Leveraged Lakehouse,  iterative expansion of datasets, focused on data-engineered pipelines and building clean, GOLD Single Source of Truth datasets 

Measuring ROI is difficult - Be prepared to experiment with small focus groups in the pilot. Building evaluation datasets for measuring model effectiveness is hard and requires focused effort and a strategy that supports rapid experimentation

Data and AI Governance is a MUST - Engage early with Enterprise Security, Privacy, and Legal teams. Build a strong governance model on Unity Catalog for the data as well as the agents and tools 

Conclusion

Through this post, we hope you learned about our Databricks on Databrick's GenAI journey and how we leverage technology like this to help our sellers be more effective.  Utilizing GenAI for this use case has helped to showcase how AI agents can significantly transform and assist every aspect of the seller journey, from prospecting and customer insights retrieval, driving better data hygiene by automating repetitive manual tasks and actioning those data insights to driving opportunities and improving sales velocity. 

Stay tuned for our upcoming posts, where we’ll continue to share our experiences on how AI is reshaping the seller experience at Databricks.