Introducing AI Model Sharing with Databricks

Today, we're excited to announce that AI model sharing is available in both Databricks Delta Sharing and on the Databricks Marketplace. With Delta Sharing you can now easily share and serve AI models securely within your organization or externally across clouds, platforms, and regions. In addition, Databricks Marketplace just released 60 new industry-specific AI models from John Snow Labs to help support medical professionals, adding to the existing set of curated Foundation Models already listed. AI model sharing is in Public Preview and available now on Delta Sharing and Marketplace.

The Databricks Data Intelligence Platform supports this new capability to find and share models with end-to-end machine learning capabilities, including model serving, AI training, and model monitoring.

Meeting the Growing Demand for Sharing with Generative AI

In the past few months, Databricks has delivered data and AI solutions supported by the Databricks Data Intelligence Platform. In October, we announced Llama 2 Foundation Models were available in the Databricks Marketplace, an open marketplace for all your data, analytics and AI. Then, in December, we released Foundation Model APIs that provide instant access to popular LLMs, such as Llama 2 and MPT models, directly from within Databricks and also available on the Marketplace.

However, while there is a growing interest to dive into generative AI, the AI space is rapidly evolving with new models being introduced and existing providers continually improving model performance and quality. This makes it difficult to experiment with various models, because they can be on different clouds and can have different UIs, APIs, and costs. Security and governance can also vary widely creating inconsistent ways to manage AI models, with the model serving and sharing environment typically separate from the data infrastructure creating additional burdens, complexity and cost.

Still, organizations are rapidly developing new ways to maximize their data and build innovative business solutions and products. A recent survey by MIT Technology Review of global CIOs found that 88% of these organizations are using generative AI, with 58% of them taking a hybrid approach with these capabilities. They use vendors' large language models (LLMs) for some use cases and build their own models when IP ownership, privacy, security, and accuracy requirements are tighter.¹

Collaborate With AI Models Using Delta Sharing

Delta Sharing now supports AI model sharing so you can train your models in one place and deploy them anywhere. This new capability has game-changing advantages for sharing models, including no redundancy, seamless distribution across clouds and multiple regions to optimize for network latency. In addition, AI model sharing with external parties now allows for increased collaboration using generative AI.

One key advantage of model sharing with Delta Sharing is no redundancy, which gives you a single source of truth, with no need to move your model at any time through the process – including initial deployment, upgrades, or maintenance. Delta Sharing also helps facilitate seamless distribution across different clouds and regions. You can easily deploy your AI models, regardless of where the model was trained or fine-tuned and with minimum effort required when upgrading or updating your model. For example, you can train your model on AWS, and deploy your model in Azure with no need for replication. Recently, Databricks developed an AI-generated documentation feature with LLMs to automatically generate documentation for tables and columns in Unity Catalog. As part of the production pipeline, Delta Sharing helped decouple the training and serving infrastructure to train this model in one region and then distribute this model across several regions around the world for faster serving.

Delta Sharing now allows sharing AI models with external parties to add innovative ways to collaborate. Data recipients can now use open source frameworks (e.g., HuggingFace, Pytorch, etc.) and load these shared models for fine-tuning between the organizations without concern for disclosing the underlying proprietary training data. This helps support increased collaboration using generative AI to further shared business goals.

Watch the demo below to learn more about AI model sharing and serving with Delta Sharing.

New AI Models From John Snow Labs On Databricks Marketplace

Traditional data marketplaces are restricted and only offer tabular data or simple applications - so the value to data collaborators is limited. They also don't offer tools to evaluate the data sets. Databricks Marketplace is an open marketplace that enables you to share and exchange datasets, notebooks, and now AI models across clouds, regions, and platforms. Since launching in June, Databricks Marketplace has over 1,600 listings from more than 140 providers.

Within the healthcare and life sciences industry, there are many use cases for implementing generative AI, from streamlining operations to optimizing patient care. It is estimated that healthcare generates 30% of the world's data volume, growing faster annually than any other industry². For example, 1.7 billion clinical notes were written by over 160,000 outpatient physicians and medical providers across the Epic electronic health record (EHR) network between May 2020 and April 2023. While healthcare data regulations such as HIPAA or GDPR may restrict direct access to this extensive EHR data, the application of AI-driven de-identification techniques allows for legal usage of these records to help benefit patients. The potential of further applying generative AI also extends to extracting pertinent insights from clinical notes composed in natural language. Utilizing advanced natural language processing (NLP) techniques, you can accurately identify and contextualize clinical events, and any correlations and nuances. This approach also significantly streamlines the process of medical coding, transforming unstructured text into structured, codified data. As a result, you can enhance the precision of data analysis and also contributes to more efficient and effective healthcare delivery. Leveraging generative AI to analyze the vast array of annual clinical notes, healthcare organizations are empowered to distill clinical trends, assess the efficacy of medical treatments, or pinpoint medical risk factors influenced by social determinants of health (SDOH) through sophisticated data analysis.

Renowned for their expertise in healthcare AI and Natural Language Processing (NLP), John Snow Labs delivers a comprehensive suite of software, models, and data resources designed to empower healthcare and life sciences organizations in the development, deployment, and management of AI-driven projects. John Snow Labs is already a data provider on Databricks Marketplace with more than 200 data listings.

Today, we are excited to introduce 60 new state-of-the-art and ready-to-use AI models tuned for the Healthcare domain from industry-leading provider John Snow Labs. See some of the featured listings below. View all AI models on Marketplace.

Featured AI Models from John Snow Labs

Data consumers searching for AI models on Marketplace can now access and query OSS and proprietary AI models faster and easier. Pre-built notebooks are included and provide specific use case guidance and information. Consumers can also leverage Databricks model deployment and inference tooling to easily make high-quality predictions with AI models. For example, a user can query models directly from SQL through AI functions, simplifying AI integration into their analytics workflows. A standard interface allows for easy experimentation and comparison.

With built-in governance and security, users can also benefit from managing their AI models in Unity Catalog along with data and features in one catalog. This ensures full visibility and fine-grained control through the AI workflow. Learn more about how AI governance works here.

Marketplace users can also use Foundation Models as-is, or fine-tune with their own data. Whether using Foundation Models or models from data providers, they all work out-of-the-box with Databricks AI capabilities for real-time or batch inference.

Watch the demo below on how to install a new model from John Snow Labs from Databricks Marketplace.

Foundation Models On Databricks Marketplace

In addition to these new AI models, we previously announced last October that Meta AI's Llama 2 foundation chat models were available on Databricks Marketplace. We also released an additional list of curated foundation models in December including Whisper V3, Mistral, BERT, and more AI models to help automate various tasks such as speech recognition, text generation and more. Each comes with a pre-built sample notebook including code and instructions for deployment and usage.

Foundation Models are some of the most popular listings on Databricks Marketplace. Check out the demo below to see how to install a Foundation Model from Databricks Marketplace.

Getting Started with AI Models on Databricks

Today's announcement introduces new AI model sharing capabilities for Delta Sharing and new AI models on Databricks Marketplace. Both of these are currently in public preview, so test drive these AI models and model serving features in Delta Sharing and provide feedback to our team at [email protected]

Watch demos above or read the related eBooks and blog posts below.

• Visit Delta Sharing product page
• Go to Databricks Marketplace and explore all AI model listings, including curated Foundation Models and new AI models from John Snow Labs
• Read A New Approach to Data Sharing, Second Edition eBook
• Read Accelerate Industry Innovation with Data Sharing eBook

¹ MIT Technology Review, "The great acceleration: CIO perspectives on generative AI" https://www.databricks.com/resources/ebook/mit-cio-generative-ai-report

² "The Convergence of Healthcare and Technology," RBC Capital Market