Best Practices for LLM Evaluation of RAG Applications
Chatbots are the most widely adopted use case for leveraging the powerful chat and reasoning capabilities of large language models (LLM). The retrieval...
Shared Clusters in Unity Catalog for the win: Introducing Cluster Libraries, Python UDFs, Scala, Machine Learning and more
We are thrilled to announce that you can run even more workloads on Databricks’ highly efficient multi-user clusters thanks to new security and...
Efficient Fine-Tuning with LoRA: A Guide to Optimal Parameter Selection for Large Language Models
With the rapid advancement of neural network-based techniques and Large Language Model (LLM) research, businesses are increasingly interested in AI applications for value...
Using MLflow AI Gateway and Llama 2 to Build Generative AI Apps
To build customer support bots, internal knowledge graphs, or Q&A systems, customers often use Retrieval Augmented Generation (RAG) applications which leverage pre-trained models...
Delta UniForm: a universal format for lakehouse interoperability
Update: BigQuery now offers native support for Delta Lake through BigLake. Check out the documentation for more information. One of the key challenges...
Multiple Stateful Operators in Structured Streaming
In the world of data engineering, there are operations that have been used since the birth of ETL. You filter. You join. You...
Introducing English as the New Programming Language for Apache Spark
Introduction We are thrilled to unveil the English SDK for Apache Spark, a transformative tool designed to enrich your Spark experience. Apache Spark™...
Announcing Delta Lake 3.0 with New Universal Format and Liquid Clustering
We are excited to announce Delta Lake 3.0, the next major release of the Linux Foundation open source Delta Lake Project, available in...
Project Lightspeed Update - Advancing Apache Spark Structured Streaming
In this blog post, we will review the advancements in Spark Structured Streaming since we announced Project Lightspeed a year ago, from performance...
Introducing Lakehouse Federation Capabilities in Unity Catalog
Lakehouse Federation is now in public preview! Data teams face many challenges to quickly access the right data primarily due to data fragmentation...