Deploy Private LLMs using Databricks Model Serving
We are excited to announce public preview of GPU and LLM optimization support for Databricks Model Serving! With this launch, you can deploy...
Making Spark Accessible: My Databricks Summer Internship
My summer internship on the PySpark team was a whirlwind of exciting events. The PySpark team develops the Python APIs of the open...
Apache Spark ❤️ Apache DataSketches: New Sketch-Based Approximate Distinct Counting
Introduction In this blog post, we'll explore a set of advanced SQL functions available within Apache Spark that leverage the HyperLogLog algorithm, enabling...
Introducing the Support of Lateral Column Alias
We are thrilled to introduce the support of a new SQL feature in Apache Spark and Databricks: Lateral Column Alias (LCA). This feature...
Introducing Apache Spark™ 3.5
Today, we are happy to announce the availability of Apache Spark™ 3.5 on Databricks as part of Databricks Runtime 14.0. We extend our...
Introducing MLflow 2.7 with new LLMOps capabilities
As part of MLflow 2’s support for LLMOps, we are excited to introduce the latest updates to support prompt engineering in MLflow 2.7...
Best Practices for LLM Evaluation of RAG Applications
Chatbots are the most widely adopted use case for leveraging the powerful chat and reasoning capabilities of large language models (LLM). The retrieval...
Shared Clusters in Unity Catalog for the win: Introducing Cluster Libraries, Python UDFs, Scala, Machine Learning and more
We are thrilled to announce that you can run even more workloads on Databricks’ highly efficient multi-user clusters thanks to new security and...
Efficient Fine-Tuning with LoRA: A Guide to Optimal Parameter Selection for Large Language Models
With the rapid advancement of neural network-based techniques and Large Language Model (LLM) research, businesses are increasingly interested in AI applications for value...
Using MLflow AI Gateway and Llama 2 to Build Generative AI Apps
To build customer support bots, internal knowledge graphs, or Q&A systems, customers often use Retrieval Augmented Generation (RAG) applications which leverage pre-trained models...