Skip to main content
Page 1

Announcing GA of Predictive I/O for Updates: Faster DML Queries, Right Out of the Box

November 2, 2023 by Sirui Sun, Ala Luszczak, Sabir Akhadov, Andreas Chatzistergiou, Carmen Kwan, Ole Sasse, Tom van Bussel, Lars Kroll, Pengfei Xu, Polo-François Poli, Bart Samwel, Himanshu Raja, Christos Stavrakakis, Venki Korukanti, Tathagata Das and Nicolas Poggi in Platform & Products & Announcements
Announcing GA of Predictive I/O for Updates, which harnesses Photon and AI atop Deletion Vectors in order to significantly speed up MERGE, UPDATE and DELETE operations. We've prevented over 15 trillion unnecessary row writes, and we’re just getting started.

Announcing Delta Lake 3.0 with New Universal Format and Liquid Clustering

June 28, 2023 by Ryan Johnson, Michael Armbrust, Reynold Xin, Denny Lee, Tathagata Das, Bart Samwel, Terry Kim, Sirui Sun, Himanshu Raja, Rahul Potharaju, Juan Yu and Susan Pierce in Engineering Blog
We are excited to announce Delta Lake 3.0, the next major release of the Linux Foundation open source Delta Lake Project, available in...

How Databricks improved query performance by up to 2.2x by automatically optimizing file sizes

May 22, 2023 by Sirui Sun, Himanshu Raja, Vijayan Prabhakaran, Terry Kim, Bart Samwel, Rahul Mahadev, Rajesh Parangi Sharabhalingappa, Rahul Potharaju and Kam Cheung Ting in Platform Blog
Optimizing tables has long been a necessary but complicated task for data engineers. One particularly thorny area has been getting to the optimal...

Announcing the Public Preview of Predictive I/O for Updates

April 25, 2023 by Piyush Revuri, Bart Samwel, Ala Luszczak, Lars Kroll, Polo-François Poli, Frank Munz and Himanshu Raja in Platform Blog
Previously, we’ve shown you how a new technology called Predictive I/O could improve selective reads by up to 35x for CDW customers without...

Introducing Ingestion Time Clustering with Databricks SQL and Databricks Runtime 11.2

November 17, 2022 by Piyush Revuri, Tom van Bussel, Joe Widen, Bart Samwel, Sabir Akhadov and Carmen Kwan in Engineering Blog
Databricks customers are processing over an exabyte of data every day on the Databricks Lakehouse platform using Delta Lake , a significant amount...

Faster MERGE Performance With Low-Shuffle MERGE and Photon

October 16, 2022 by Bart Samwel, Piyush Revuri, Himanshu Raja, Justin Breese, Tom van Bussel, Lars Kroll, Sabir Akhadov, Tathagata Das and Prakhar Jain in Platform Blog
At Databricks, one of our key goals is to provide our customers with an industry-best price/performance experience out of the box. From ETL...

Introducing Brickchain: Planet-scale Unified Analytics

March 31, 2019 by Bart Samwel and Reynold Xin in Engineering Blog
Today we are excited to announce Brickchain , the next generation technology for zettabyte-scale analytics, by harnessing all the compute power on the...