Building the Next Generation Visualization Tools at Databricks
Databricks SQL is now generally available on AWS and Azure. This post is a part of our blog series on our frontend work...
Turning 2 Trillion Data Points of Traffic Intelligence into Critical Business Insights
This is a guest authored post by Stephanie Mak , Senior Data Engineer, formerly at Intelematics. This blog post offers my experience of...
Databricks Sets Official Data Warehousing Performance Record
Today, we are proud to announce that Databricks SQL has set a new world record in 100TB TPC-DS , the gold standard performance...
Moneyball 2.0: Real-time Decision Making With MLB’s Statcast Data
The Oakland Athletics baseball team in 2002 used data analysis and quantitative modeling to identify undervalued players and create a competitive lineup on...
GPU-accelerated Sentiment Analysis Using Pytorch and Huggingface on Databricks
Sentiment analysis is commonly used to analyze the sentiment present within a body of text, which could range from a review, an email...
Introducing Apache Spark™ 3.2
We are excited to announce the availability of Apache Spark™ 3.2 on Databricks as part of Databricks Runtime 10.0 . We want to...
MLflow for Bayesian Experiment Tracking
This post is the third in a series on Bayesian inference ( [1] , [2] ). Here we will illustrate how to use...
Creating an IP Lookup Table of Activities in a SIEM Architecture
When working with cyber security data, one thing is for sure: there is no shortage of available data sources. If anything, there are...
Developing Databricks' Runbot CI Solution
Runbot is a bespoke continuous integration (CI) solution developed specifically for Databricks' needs. Originally developed in 2019, Runbot incrementally replaces our aging Jenkins...
Native Support of Session Window in Spark Structured Streaming
Apache Spark™ Structured Streaming allowed users to do aggregations on windows over event-time . Before Apache Spark 3.2™, Spark supported tumbling windows and...