Best-in-class open source generative AI models for free commercial use

Databricks works with thousands of customers to build generative AI applications. While you can use Databricks to work with any generative AI model, including commercial and research, the table below lists our current model recommendations* for popular use cases. Note that the table only lists open source models that are for free commercial use. In addition to the models below, any HuggingFace model can be registered to Unity Catalog and deployed. See docs [ AWS | Azure ] for details.

This page will be updated frequently — check back soon!

Last updated: May 23, 2024

Use case	Quality-optimized	Balanced	Speed-optimized	Notes
Text generation following instructions	dbrx-instruct Meta Llama 3 70B Mixtral-8x7B-Instruct-v0.1 MPT-30B-Instruct †	Meta Llama 3 8B Mistral-7B-Instruct-v0.2 MPT-7B-Instruct † MPT-7b-8k-instruct	phi-2	Thanks to its MoE architecture, DBRX Instruct is highly efficient for inference and achieves best-in-class inference speed among quality-optimized models. See more details in the technical blog. † Supervised fine-tuning using databricks-dolly-15k dataset
Text embeddings (English only)	e5-mistral-7b-instruct (7B)	Bge-large-en-v1.5 (0.3B) e5-large-v2 (0.3B)	bge-small-en-v1.5 (0.1B) bge-small-en-v1.5 (0.1B) e5-base-v2 (0.1B)
Transcription (speech to text)		whisper-large-v3 (1.6B)	distil-large-v2 (0.7B)
Image generation		stable-diffusion-xl
Code generation	CodeLlama-34b-Python-hf (Python optimized) CodeLlama-34b-Python-hf (Python optimized) CodeLlama-34b-Python-hf (Python optimized) CodeLlama-34b-Python-hf (Python optimized) CodeLlama-34b-Python-hf (Python optimized) CodeLlama-34b-Python-hf (Python optimized)	CodeLlama-13b-Python-hf (Python optimized) CodeLlama-13b-Python-hf (Python optimized) CodeLlama-13b-Python-hf (Python optimized) CodeLlama-7b-Python-hf (Python optimized) CodeLlama-7b-Python-hf (Python optimized) CodeLlama-7b-Python-hf (Python optimized)		Code LLMs usually need fine-tuning to follow instructions and work on application-specific code

Note: All models are subject to their applicable licenses and any other
attendant restrictions. This list is solely for informational purposes.

* Evaluating LLMs is difficult and remains an area of active research. Many LLMs are not on this list, and the LLMs on this list may not be perfect for all use cases. These models are ones we have seen work well in the listed scenarios. Also, there are some other models that are best in class but not commercially licensed, such as vicuna-13b and LLaMA.

In general, the space is evolving rapidly. Our recommendation is to design your architecture in a way that would enable you to quickly experiment and swap out specific models easily.

Explore more: Using Large Language Models on Databricks

Explore more examples for different LLM models: Databricks ML example repo