MLflow for Bayesian Experiment Tracking
This post is the third in a series on Bayesian inference ( [1] , [2] ). Here we will illustrate how to use...
Creating an IP Lookup Table of Activities in a SIEM Architecture
When working with cyber security data, one thing is for sure: there is no shortage of available data sources. If anything, there are...
Developing Databricks' Runbot CI Solution
Runbot is a bespoke continuous integration (CI) solution developed specifically for Databricks' needs. Originally developed in 2019, Runbot incrementally replaces our aging Jenkins...
Native Support of Session Window in Spark Structured Streaming
Apache Spark™ Structured Streaming allowed users to do aggregations on windows over event-time . Before Apache Spark 3.2™, Spark supported tumbling windows and...
Efficient Point in Polygon Joins via PySpark and BNG Geospatial Indexing
This is a collaborative post by Ordnance Survey, Microsoft and Databricks. We thank Charis Doidge, Senior Data Engineer, and Steve Kingston, Senior Data...
Pandas API on Upcoming Apache Spark™ 3.2
We're thrilled to announce that the pandas API will be part of the upcoming Apache Spark™ 3.2 release. pandas is a powerful, flexible...
Databricks SQL: Delivering a Production SQL Development Experience on the Data Lake
Databricks SQL is now generally available on AWS and Azure. Databricks SQL (DB SQL) is a simple and powerful SQL analytics platform for...
Shiny and Environments for R Notebooks
At Databricks, we want the Lakehouse ecosystem widely accessible to all data practitioners, and R is a great interface language for this purpose...
Catalog and Discover Your Databricks Notebooks Faster
This is a collaborative post from Databricks and Elsevier. We thank Darin McBeath, Director Disruptive Technologies -- Elsevier, for his contributions. As a...
Extracting Oncology Insights From Real-World Clinical Data With NLP
Preview the solution accelerator notebooks referenced in this blog online or get started right away by downloading and importing the notebooks into your...