Unlocking the Potential of Private Data Sharing using Databricks Private Exchanges

We are thrilled to announce an exciting new feature on the Databricks Marketplace that simplifies the process of setting up private exchanges for all Databricks customers. With this feature, becoming a Private Exchange provider is easier than ever.

This blog post will explore the private exchange features of the Databricks Marketplace. We'll compare various exchange mechanisms—public marketplace and private exchanges—and examine the newly introduced feature that simplifies becoming a private exchange provider.

Comparing Private Exchange and Public Marketplace

In the evolving data sharing and monetization landscape, companies have multiple avenues to distribute their data and AI models. Each method offers distinct advantages and caters to different strategic needs. Below, I'll outline two scenarios—Public Marketplace, and Private exchange—and demonstrate how a customer might choose between these options based on its specific requirements and goals.

Public Marketplace:

Imagine I am DataVendorCo, and I would like to monetize my data or AI models by sharing or licensing them to a broad, public audience of practitioners. I will become a public marketplace provider and share my assets on a public Marketplace like Databricks Marketplace, where my data products can be discovered and used by 10,000+ of Databricks customers.

Private Exchange:

Private Exchanges are useful in a couple of use-cases.

First, imagine I am the same DataVendorCo that offers data sets to a wide audience. I often have to deliver custom-made data sets to specific customers, which are tailored to their needs and requirements. To do this, I can use Private Exchanges to deliver custom data sets, using a private listing tailored to that specific customer.

Separately, imagine I am RetailCo, a multinational conglomerate with multiple subsidiaries. I do not want to license or monetize my data with the broader world, but I do want to share data securely with a private network of external partners, such as partner retailers, suppliers, and distributors. I may also want to share data between multiple subsidiaries through a Marketplace-type user interface, with the ability for consumers to seamlessly discover content based on their needs. For these purposes, I would establish a Private Exchange.

So, let's delve deeper into Private Exchanges in the next section.

Using Private Exchanges to Create an Enterprise Data and Analytics Hub

In the previous section, we compared the public marketplace and private exchange. One example of a public marketplace is Databricks Marketplace. It is an open marketplace for all your data, analytics, and AI needs, powered by the open Delta Sharing protocol, and governed by Unity Catalog. The Databricks Marketplace brings together a vast ecosystem of data consumers and data providers to share and collaborate across a wide array of data assets, including datasets, AI models, and Solutions Accelerators.

In Q4 2023, we launched private exchange capability within Databricks Marketplace. A Private Exchange within Databricks Marketplace allows a data provider to make certain data and AI products discoverable only to a specified group of consumers, outside your organization. It also allows Databricks customers to securely exchange data with its external partners or across its subsidiaries. Through a storefront interface for requesting and accessing data products, Private Exchanges can control visibility (who can see) and access (who can use) their products.

Here are two examples of how Databricks customers could benefit from Private Exchange.

Peer-to-peer Sharing: An e-commerce company has developed an AI model to predict online sales trends. They created a private exchange to share the model with a brick-and-mortar retail chain. The retail chain utilizes this AI model to forecast in-store sales, helping them optimize inventory and staffing levels effectively.

Industry collaboration: For example, a manufacturer could set up a private exchange and share production data with Tier 1 and Tier 2 suppliers. Private exchanges provide a Marketplace type UI for discoverability and listing management, enabling any new suppliers to easily discover available data.

Each exchange can be associated with a set of listings that represent the underlying data and AI products being shared. These listings contain rich metadata, including associated notebooks that provide code samples and relevant visualizations. Listings can be made 'instantly available' or 'gated,' giving owners full autonomy in determining who has access to what products. Product fulfillment happens through Delta Sharing and Unity Catalog for governance.

John Snow Labs delivers a comprehensive suite of software, models, and data resources designed to empower healthcare and life sciences organizations in the development, deployment, and management of AI-driven projects. John Snow Labs is a public provider with over 60 AI models available on the Databricks Marketplace. They have utilized the private exchange feature to distribute custom AI models to a specific group of data consumers. Here's what John Snow Labs says about the benefits of private exchange.

"At John Snow Labs, Databricks Marketplace aligns perfectly with our mission to make world-class datasets and AI models accessible to all. The private exchange capability takes things to a whole new level. We've already seen firsthand how it facilitates the secure peer-to-peer exchange of AI models between organizations, and we can't wait to see the transformative impact it will have as more of our customers share valuable AI capabilities while maintaining strict control and privacy"

— David Talby, CTO, John Snow Labs.

Check out this demo video detailing how private exchanges work and how they can be used for intra and inter-org collaboration.

Understanding In-product Private Exchange provider onboarding capability

Becoming a public exchange provider on Databricks Marketplace involves a careful approval process. This ensures the data shared with customers is accurate and high-quality. Strict quality controls are used to only allow vendors that meet high standards.

For data providers who don't want to be public, we aim to make the process easier. Our goal is to enable data providers to share datasets and AI models with a specific group of users, reducing entry barriers for private exchanges.

With the new onboarding capability, companies can now become Private Exchange providers without having to go through the Marketplace public provider onboarding process. They can now agree to the Private Exchange provider terms and complete their setup directly within the product in a self-serve manner. This improvement brings several advantages, including speed, ease of use, accessibility, and lower barriers to entry.

Check out this demo video that walks through the process of signing up to be a private exchange provider.

Getting started as a Private Exchange Provider from within the product

To explore this new feature as an Account Admin, visit the Databricks Marketplace section within our product and follow the guided onboarding process for Private Exchange Providers.

https://docs.databricks.com/en/marketplace/get-started-provider.html#sign-up-as-a-private-exchange-provider

The new in-product Private Exchange provider onboarding capability is a testament to that commitment.

The Databricks Marketplace team is excited about the new feature. It dramatically simplifies the process for our customers to become Private Exchange providers without going through the Marketplace public provider onboarding process. It allows them to complete the onboarding process themselves without needing manual intervention from the Databricks team.

We look forward to helping our customers set up private exchanges and start the journey of controlled and secure data sharing within and across organizations.