Announcing the State Reader API: The New "Statestore" Data Source
Databricks Runtime 14.3 includes a new capability that allows users to access and analyze Structured Streaming 's internal state data: the State Reader...
PySpark in 2023: A Year in Review
With the releases of Apache Spark 3.4 and 3.5 in 2023, we focused heavily on improving PySpark performance, flexibility, and ease of use...
GGML GGUF File Format Vulnerabilities
The GGUF file format is a binary file format used for storing and loading model weights for the GGML library. The library documentation...
Data Exfiltration Protection with Azure Databricks
Last updated on: August 23, 2024 In the previous blog , we discussed how to securely access Azure Data Services from Azure Databricks...
Implementing LLM Guardrails for Safe and Responsible Generative AI Deployment on Databricks
Introduction Let’s explore a common scenario – your team is eager to leverage open source LLMs to build chatbots for customer support interactions...
Announcing the General Availability of Databricks Feature Serving
Today, we are excited to announce the general availability of Feature Serving. Features play a pivotal role in AI Applications, typically requiring considerable...
Databricks Expands Brickbuilder Program to Include Unity Catalog Accelerators
Today, we're excited to announce the launch of Brickbuilder Unity Catalog Accelerators. This is an expansion to the Brickbuilder Accelerator program , which...
Simplify PySpark testing with DataFrame equality functions
The DataFrame equality test functions were introduced in Apache Spark™ 3.5 and Databricks Runtime 14.2 to simplify PySpark unit testing. The full set...
A Deep Dive into the Latest Performance Improvements of Stateful Pipelines in Apache Spark Structured Streaming
This post is the second part of our two-part series on the latest performance improvements of stateful pipelines. The first part of this...
Performance Improvements for Stateful Pipelines in Apache Spark Structured Streaming
Introduction Apache Spark™ Structured Streaming is a popular open-source stream processing platform that provides scalability and fault tolerance, built on top of the...