Introducing Python User-Defined Table Functions (UDTFs)
Apache Spark™ 3.5 and Databricks Runtime 14.0 have brought an exciting feature to the table: Python user-defined table functions (UDTFs). In this blog...
Modernizing Prior Authorization with Advanced Analytics
This blog was written in collaboration with Dan Newingham, Solution Delivery Manager, ZS and Aaron Zavora, Technical Director, HLS, Databricks Mandates for electronic...
Arrow-optimized Python UDFs in Apache Spark™ 3.5
In Apache Spark™, Python User-Defined Functions (UDFs) are among the most popular features. They empower users to craft custom code tailored to their...
Databricks + Arcion: Real-time enterprise data replication to the Lakehouse
We are excited to announce that we have completed our acquisition of Arcion , a leading provider for real-time data replication technologies. Arcion’s...
Cybersecurity Lakehouse Best Practices Part 1: Event Timestamp Extraction
In this four-part blog series "Lessons learned from building Cybersecurity Lakehouses," we will discuss a number of challenges organizations face with data engineering...
How Lakehouse AI improves model accuracy with real-time computations
The predictive quality of a machine learning model is a direct reflection of the quality of data used to train and serve the...
How to upgrade your Hive tables to Unity Catalog
In this blog we will demonstrate with examples, how you can seamlessly upgrade your Hive metastore (HMS)* tables to Unity Catalog (UC) using...
Announcing GA of Predictive I/O for Updates: Faster DML Queries, Right Out of the Box
Announcing GA of Predictive I/O for Updates, which harnesses Photon and AI atop Deletion Vectors in order to significantly speed up MERGE, UPDATE and DELETE operations. We've prevented over 15 trillion unnecessary row writes, and we’re just getting started.
To Tailgate or Not? How Databricks + AccuWeather used ML to answer every football fan's burning question
Whether you’re an NFL fanatic, an alumnus rooting for your alma mater or a super fan just trying to catch a glimpse of...
Patronus AI: Using LLMs to Detect Business-Sensitive Information
EnterprisePII is a first-of-its-kind large language model (LLM) data set aimed at detecting business-sensitive information. The challenge of detecting and redacting sensitive business...