Introducing New Governance Capabilities to Scale AI Agents with Confidence

As we mentioned in our blog earlier this week, AI agents require enterprise data integration and output governance to achieve production quality. Today we're launching updates to Mosaic AI Gateway, Unity Catalog tools, and AI/BI Genie that enable organizations to build production-ready AI agents with robust governance and data integration capabilities.

Here's why this matters: Imagine your developers build an AI agent that summarizes high-priority customer complaints and alerts departments via Slack. Without high quality performance, departments could be flooded with noisy alerts, causing them to miss truly urgent matters. Even worse, if developers receive direct access to Slack API credentials instead of using secure enterprise integration, malicious actors could potentially hijack these credentials and distribute phishing links company-wide.

 At their core, AI agents rely on three essential components: models, tools, and data. Let's explore how today's updates enable you to build well-governed, high quality AI agents across the board:

• Models – Leverage the Mosaic AI Gateway (Public Preview) to govern Foundation Model access, monitor quality, and track usage across providers. Now, enable automatic traffic fallbacks for improved model reliability and integrate custom provider LLMs as External Models.
• Tools – Say goodbye to risky credential management. Unity Catalog Connections and Functions securely integrate enterprise and external APIs as powerful tools for AI agents.
• Data – Securely unleash AI agents on your enterprise data securely. The  AI/BI Genie Conversation APIs (Public Preview) and Vector Search retrieval tool integrate directly with Agent Framework, creating a secure pipeline to your Unity Catalog data while delivering valuable insights.

Databricks provides a unified, end-to-end governance framework for all of your AI agents, eliminating the need for fragmented solutions and driving quality and security across the board. Now, let’s explore the latest updates for each of these key governance components.

Power AI Agents Securely with State-of-the-Art Foundation Models 

At Databricks, we know that using the right Foundation Model – whether open-source or proprietary – is fundamental to building high-quality AI agents. 

The Mosaic AI Gateway is your central control hub for enterprise-grade AI, ensuring both governance and quality across all foundation models and AI agents. With AI Gateway, you can:

• Enforce secure controls across top models (OpenAI GPT, Claude, Llama, Gemini) used within AI agents through centralized permissions, guardrails, and rate limits
• Monitor AI agent quality in production and audit for compliance through automatic payload logging 
• Unify usage tracking across all providers, streamlining cost attribution and usage auditing

Mosaic AI Gateway accelerates innovation and delivers high-quality AI agents by combining governance and flexibility. Developers gain unified access to the most optimal models for their AI agents, all within a governed, centralized framework.

We’ve seen how much value this has driven for our customers, so we’re excited to announce two new capabilities available via AI Gateway for building reliable, high-quality AI agents:

1. Introducing Custom Provider Support in Mosaic AI Gateway (Public Preview)

Many companies have developed custom proxies for their bespoke needs but want to build and deploy their AI agents on Databricks end-to-end. Others need to integrate self-hosted or third-party models into AI Gateway to leverage the best models for their AI agents.

Starting today, AI Gateway supports any OpenAI schema-compatible Foundation Model as an External Model, whether hosted on your custom proxy or coming from an alternate provider. This enables centralized model access management, ensuring security and capturing valuable data to monitor quality, regardless of where your Foundation Model is hosted.

To securely incorporate all of your Foundation Models into your AI agents today, register an LLM from a custom provider to govern access and monitor quality using the Mosaic AI Gateway.

2. Powering Reliable AI Agent with AI Model Fallbacks (Public Preview)

Teams that need to productionize reliable, high traffic AI agents often run into availability issues from third-party AI model providers. Third-party services go down unexpectedly or usage spikes result in hitting quota limits, rendering these Foundation Models (and consequently the AI agents relying on them) unreachable. 

Mosaic AI Gateway’s new traffic fallbacks ensure seamless failovers across multiple providers, regions, and resources, preventing any client-facing disruptions. With automatic fallback mechanisms, enterprises keep AI agents systems running smoothly, even during spikes or outages.

“At Erste Group, ensuring reliable AI-driven operations is critical to our success. Mosaic AI Gateway’s fallbacks feature has strengthened our system’s resilience by automatically redirecting traffic when primary models encounter issues."
— Jürgen Neulinger, Sr. Solutions Manager, Erste Group

Boost AI Agent Capabilities with Secure API Access and Tool Integration

AI agents become more powerful when they integrate with external services (i.e. Teams, Slack) and enterprise back-ends (i.e. internal APIs) to make decisions and take actions. However, distributing these sensitive API credentials to developers, especially at scale, can pose a serious security challenge.  

With Unity Catalog (UC) Connections and Functions (Public Preview), your developers can incorporate fully governed API integrations into their AI agents without the security risks that come with them.  

Notably, IT teams can use UC Connections to centrally and securely manage all API credentials. Based on access permissions, developers can integrate pre-approved connections into their AI agents as tools, and optionally create (and share) UC Functions to call them. This ensures that developers and their code never have access to remote API tokens, and all API access through UC Connections is fully audited.

Here’s an example of how a developer can define a “send Slack message” tool in Python that authenticates to the Slack REST API via a UC Connection.

UC Connections are secure, reusable, and discoverable. Now, developers can build agile AI agents that interact with real-world data and actions— all while maintaining air-tight API governance.

Securely Integrate Enterprise Data with Your Agent Systems

AI agents rely on proprietary data to drive differentiated quality and performance. Therefore, it is paramount that they have a secure and efficient way to interact with the enterprise data fueling high-quality insights and actions. 

To ensure secure, high-performance AI agents, we’re introducing two new capabilities that enable effortless integration of both structured and unstructured enterprise data into your AI agents.

AI/BI Genie Conversation APIs

Developers have been asking for a way to integrate Genie—our powerful AI/BI tool that enables business teams to interact with data using natural language—into their AI agents. We’re excited to announce that Genie Conversation APIs now make this possible!

With Unity Catalog governing all your Genie data, business users from different departments can query  “What are my costs?” and only see results relevant to their department. Integrating Genie Conversation APIs into your Agent Framework makes it easier than ever to build secure multi-agent systems that interact with your Unity Catalog data.

 “Having Genie integrated with Teams has been a huge step forward for data democratization. It makes data insights accessible to everyone, no matter their technical background.”
— Cezar Steinz, Data Operations Manager, Grupo Casas Bahia

To get started, check out our detailed blog announcing Genie Conversation APIs here.

Vector Search Retrieval Tool

Many AI agents leverage Databricks Vector Search to securely talk to unstructured data. To simplify integrating it as a tool into your AI agent system while maintaining Unity Catalog’s data governance features, we’ve introduced Vector Search Retrieval Tool APIs. 

These APIs allow you to smoothly integrate vector search retrievers as tools within your AI agents, ensuring sensitive information stays protected while delivering high-quality insights with confidence.

Securely empowering your AI agents with enterprise data is now as easy as a few lines of code. 

Start Building and Governing Your AI Agents Today

The best enterprises leverage governance to drive secure and quality AI agents in production. When you use Databricks, you can scale high-quality AI agents confidently with every model, tool, and dataset.

Get started governing your agent systems end-to-end in a unified manner today:

• Set up the Mosaic AI Gateway with the Foundation Models your AI agents need.
• Use Unity Catalog Connections to connect AI agent tools to external services securely.
• Integrate the Genie Conversation APIs into your Mosaic AI Agent Framework.
• use the Vector Search Retrieval Tool in your agent systems to securely access unstructured data.
• Watch the demo video.

And make sure you check out the Compact Guide to AI Agents to learn how to get the greatest return on your investment in GenAI.