Creating a bespoke LLM for AI-generated documentation
We recently announced our AI-generated documentation feature , which uses large language models (LLMs) to automatically generate documentation for tables and columns in...
Python Dependency Management in Spark Connect
Managing the environment of an application in a distributed computing environment can be challenging. Ensuring that all nodes have the necessary environment to...
Named Arguments for SQL Functions
Today, we introduce the new availability of named arguments for SQL functions. With this feature, you can invoke functions in more flexible ways...
Introducing Python User-Defined Table Functions (UDTFs)
Apache Spark™ 3.5 and Databricks Runtime 14.0 have brought an exciting feature to the table: Python user-defined table functions (UDTFs). In this blog...
Arrow-optimized Python UDFs in Apache Spark™ 3.5
In Apache Spark™, Python User-Defined Functions (UDFs) are among the most popular features. They empower users to craft custom code tailored to their...
Announcing MLflow 2.8 LLM-as-a-judge metrics and Best Practices for LLM Evaluation of RAG Applications, Part 2
Today we're excited to announce MLflow 2.8 supports our LLM-as-a-judge metrics which can help save time and costs while providing an approximation of...
Big Book of MLOps Updated for Generative AI
Last year, we published the Big Book of MLOps, outlining guiding principles, design considerations, and reference architectures for Machine Learning Operations (MLOps). Since...
Llama 2 Foundation Models Available in Databricks Lakehouse AI
We’re excited to announce that Meta AI’s Llama 2 foundation chat models are available in the Databricks Marketplace for you to fine-tune and...
A Pattern for the Lightweight Deployment of Distributed XGBoost and LightGBM Models
A common challenge data scientists encounter when developing machine learning solutions is training a model on a dataset that is too large to...
Announcing the Public Preview of AI/BI Dashboards!
We are excited to announce the public preview of the next generation of Databricks SQL dashboards, dubbed AI/BI Dashboards . Available today, this...