I'm pleased to announce the release of Databricks Runtime 4.3, powered by Apache Spark. We've packed this release with an assortment of new features, performance improvements, and quality improvements to the platform. We recommend moving to Databricks Runtime 4.3 in order to take advantage of these improvements.
In our obsession to continually improve our platform’s performance, the Databricks Runtime 4.3 release benefits from substantial performance gains over previous versions of the Databricks Runtime. We have seen over 15% improvement in AWS with the performance improvements in 4.3. With improvements from DBIO caching to skip data more efficiently we’re experiencing over 55% performance improvements with TPC-DS at 1 Terabyte scale on Azure:
Find out more about the DBIO Caching capabilities in Azure Databricks premium SKU with Ls series VMs and how to enable caching with other VM types in Azure in the technical documentation.
In addition to the performance improvements, we've also added new functionality to Databricks Delta:
We’ve added great improvements to Structured Streaming that I’d also like to highlight:
To read more about the above new features and to see the full list of improvements included in Databricks Runtime 4.3, please refer to the release notes in the following locations: