What’s New with Data Sharing and Collaboration on the Lakehouse

Databricks provides the first open source approach to data sharing and collaboration across data, analytics, and AI. Customers can share live data sets - as well as AI models, applications, and notebooks - across platforms, clouds, and regions without vendor lock-in. Providers can help accelerate innovation and generate new revenue streams with expanded opportunities for delivering transformative data products. This flexibility is achieved with strong security and governance by integrating Delta Sharing with Databricks Unity Catalog to govern, track, and audit access to shared data sets. We are excited to announce new capabilities to advance the Lakehouse Collaboration Platform. We are excited to announce that Databricks is expanding our Delta Sharing ecosystem with new partners, the general availability of Databricks Marketplace, and Databricks Clean Rooms in Private Preview on AWS.

Accelerate Data Sharing with Delta Sharing

Two years ago we started the open source Delta Sharing project. The industry's first open protocol for secure data sharing. Since launch, many organizations are adopting Delta Sharing. Over 6,000 organizations are consuming data using Delta Sharing. Over 300 Petabytes of data are being shared daily using Delta Sharing on Databricks. This is data that is read and processed across organizations. While many other vendor solutions lock you in, we have seen incredible momentum with customers with Delta Sharing. Enterprises can share and consume data from any platform or vendor that supports the open Delta Sharing protocol. With an open approach powered by Delta Sharing, organizations can put their data to work more quickly and discover insights faster.

We are excited to announce that Databricks is expanding the Delta Sharing ecosystem with new partners, including Cloudflare, Dell, Oracle and Twilio, to seamlessly share data between their platforms, Databricks, Apache Spark™, pandas, PowerBI, Excel and any other system that supports the open protocol.

Databricks Marketplace is now Generally Available!

Built on Delta Sharing are other collaboration capabilities including the Databricks Marketplace. We recently announced the Public Preview of Databricks Marketplace and today we are excited to announce general availability of Databricks Marketplace.

Many cloud providers offer data marketplaces but when we asked data providers, they shared they run into limitations:

• Marketplaces are closed for its specific cloud or a specific data warehouse.
• Restricted to data only, or maybe simple applications.
• Delivery of data either requires ETL or the use of a proprietary delivery mechanism.

Databricks Marketplace is an open marketplace for data solutions so any client that can read delta shares can access the marketplace. This has great benefits for providers to reach more users. For consumers they can easily discover more than just datasets, they can evaluate data products faster because the assets include notebooks and visualizations and they don't have to be a Databricks customer to take advantage of the marketplace. It's truly open.

Available today, the Databricks Marketplace enables you to share and exchange data assets, including data sets and notebooks, in the public marketplace or private exchanges. Databricks Marketplace is open to non-Databricks users as well.

New Data Providers on Databricks Marketplace

Over the past months we've continued to increase partners across industries and we have hundreds of listings on our marketplace that you can explore in our Open Marketplace. Databricks Marketplace also welcomes additional launch data providers from various industries:

• Financial Services: S&P Global, Experian, London Stock Exchange Group, Corelogic, YipitData, Morningstar, and Bloomberg Government
• Healthcare and Life Sciences: IQVIA, Definitive Healthcare, Datavant, and Ontada
• Geospatial: Foursquare, Divirod, Accuweather Veraset, Safegraph, and Carto
• Advertising and Marketing: LiveRamp, Acxiom
• Business Information Services: LexisNexis and ZoomInfo
• Retail: Circana (fka IRI)

“As the adoption of AI and advanced analytics grows, we are excited to partner with Databricks to help customers drive faster time to insights. This collaboration aims to empower businesses with real-time, unified data access, fostering collaboration and driving enhanced analytics and quantitative decision-making capabilities in the financial industry and beyond.”

— Stuart Brown, Group Head of Enterprise Data Solutions at London Stock Exchange

Coming soon: AI Model-Sharing Capabilities

Databricks will also offer AI model sharing in the Databricks Marketplace, enabling data consumers and providers to discover and monetize AI models and integrate AI into all their data solutions. With AI model sharing, Databricks customers will have access to best-in-class models, which can be quickly and securely applied on top of their data. Databricks itself will curate and publish open source models across common use cases, such as instruction-following and text summarization, and optimize tuning or deployment of these models on Databricks. AI model sharing in Databricks Marketplace is expected in preview in the coming year.

Coming soon: Lakehouse Apps in Databricks Marketplace

Data and AI applications are among the fastest-growing software categories, and the growth in generative AI and large language models (LLMs) has accelerated that trend. For customers, Lakehouse Apps will be the most secure way to run applications that unlock the full value of data in their Lakehouse, leverage Databricks-native services, and extend Databricks with new capabilities. Lakehouse Apps will give users safe and easy access to a wide range of innovative new applications and reduce time and effort to adopt, integrate, and manage data and AI applications.

Lakehouse apps are a new way to build, deploy and manage applications for the Databricks platform. Developers will benefit from easier distribution by listing their Lakehouse Apps in the Databricks Marketplace, enabling customers to quickly discover and deploy their software. Lakehouse Apps in Databricks Marketplace is expected in preview in the coming year.

Privacy-safe collaboration with Databricks Clean Rooms

Today we are excited to announce Databricks Clean Rooms in Private Preview on AWS. A data clean room enables secure and isolated collaboration between organizations by allowing the joining and processing of respective datasets while ensuring that no party has direct access or visibility to each other's raw data or, in some cases, proprietary packages or AI models.

Databricks Clean Rooms provides a secure environment for businesses to collaborate with customers and partners on any cloud, prioritizing flexibility, scalability, and interoperability. Unlike other solutions, Databricks Clean Rooms supports multiple languages (SQL, R, Scala, Java, Python) enabling both simple analytical workloads such as joins as well as complex computations such as machine learning. It offers scalability for collaboration with multiple collaborators, reducing the need for managing multiple vendors. Built atop open Delta Sharing, participants can securely share data without data replication or platform lock-in, evolving from simple data sharing to secure joint compute.

Get started with Data Sharing and Collaboration in Databricks

Databricks enables open sharing and collaboration and we can't wait to see how you use Delta Sharing, Databricks Marketplace, Lakehouse Apps and Databricks Clean Rooms to collaborate with ease and trust.

Don't miss more exciting updates for data sharing and collaboration and our sessions at Data and AI Summit. Sessions will be made available on-demand shortly after the conclusion of the conference.

Sign-up to become a data provider (Please choose "Marketplace" as the program of interest).

Submitting your interest to join our Databricks Clean Rooms private preview, or wait to enroll in our public preview that will be available later this year.