Introducing SQL User-Defined Functions
A user-defined function (UDF) is a means for a user to extend the native capabilities of Apache Spark™ SQL. SQL on Databricks has...
Introducing Apache Spark™ 3.2
We are excited to announce the availability of Apache Spark™ 3.2 on Databricks as part of Databricks Runtime 10.0 . We want to...
MLflow for Bayesian Experiment Tracking
This post is the third in a series on Bayesian inference ( [1] , [2] ). Here we will illustrate how to use...
Creating an IP Lookup Table of Activities in a SIEM Architecture
When working with cyber security data, one thing is for sure: there is no shortage of available data sources. If anything, there are...
Developing Databricks' Runbot CI Solution
Runbot is a bespoke continuous integration (CI) solution developed specifically for Databricks' needs. Originally developed in 2019, Runbot incrementally replaces our aging Jenkins...
Native Support of Session Window in Spark Structured Streaming
Apache Spark™ Structured Streaming allowed users to do aggregations on windows over event-time . Before Apache Spark 3.2™, Spark supported tumbling windows and...
Efficient Point in Polygon Joins via PySpark and BNG Geospatial Indexing
This is a collaborative post by Ordnance Survey, Microsoft and Databricks. We thank Charis Doidge, Senior Data Engineer, and Steve Kingston, Senior Data...
5 Steps to Get Started With Databricks on Google Cloud
Since we launched Databricks on Google Cloud earlier this year, we’ve been thrilled to see stories about the value this joint solution has...
Databricks Repos Is Now Generally Available - New ‘Files’ Feature in Public Preview
Thousands of Databricks customers have adopted Databricks Repos since its public preview and have standardized on it for their development and production workflows...
Bringing Lakehouse to the Citizen Data Scientist: Announcing the Acquisition of 8080 Labs
Transforming into a data-driven organization – which means data has permeated into every facet of your company – is critical for driving meaningful...