Introducing Databricks Unity Catalog: Fine-grained Governance for Data and AI on the Lakehouse
Data lake systems such as S3, ADLS, and GCS store the majority of data in today’s enterprises thanks to their scalability, low cost, and open interfaces. Over time, these systems have also become an attractive place to process data thanks to lakehouse technologies such as Delta Lake that enable ACID transactions and fast queries. However,...
Introducing Delta Sharing: An Open Protocol for Secure Data Sharing
Data sharing has become critical in the modern economy as enterprises look to securely exchange data with their customers, suppliers and partners. For example, a retailer may want to publish sales data to its suppliers in real time, or a supplier may want to share real-time inventory. But so far, data sharing has been severely...
Simplify Python environment management on Databricks Runtime for Machine Learning using %pip and %conda
Today we announce the release of %pip and %conda notebook magic commands to significantly simplify python environment management in Databricks Runtime for Machine Learning. With the new magic commands, you can manage Python package dependencies within a notebook scope using familiar pip and conda syntax. For example, you can run %pip install -U koalas in...
YipitData Example Highlights Benefits of Databricks integration with AWS Glue
At Databricks, we have partnered with the team at Amazon Web Services (AWS) to provide a seamless integration with the AWS Glue Metastore. Databricks can easily use Glue as the metastore, even across multiple workspaces. YipitData, a longtime Databricks customer, has taken full advantage of this feature, storing all their metadata in AWS Glue. ...
Announcing Databricks Runtime 5.4
Databricks is pleased to announce the release of Databricks Runtime 5.4. This release includes Apache Spark 2.4.3 along with several important improvements and bug fixes . We recommend all users upgrade to take advantage of this new runtime release. This blog post gives a brief overview of some of the new high value features that...
Introducing Databricks Library Utilities for Notebooks
Databricks has introduced a new feature, Library Utilities for Notebooks, as part of Databricks Runtime version 5.1. It allows you to install and manage Python dependencies from within a notebook. This provides several important benefits: Install libraries when and where they're needed, from within a notebook. This eliminates the need to globally install libraries on...
Announcing Databricks Runtime 5.0
We’re excited to announce the general availability of Databricks Runtime 5.0. Included in this release is Spark 2.4. This release offers substantial performance increases within key areas of the platform. Benchmarking workloads have shown a 16% improvement in total execution time and Databricks Delta benefits from substantial improvements to metadata caching, improving query latency by...
Announcing Databricks Runtime 4.3
I'm pleased to announce the release of Databricks Runtime 4.3, powered by Apache Spark. We've packed this release with an assortment of new features, performance improvements, and quality improvements to the platform. We recommend moving to Databricks Runtime 4.3 in order to take advantage of these improvements. In our obsession to continually improve our platform’s...
Announcing Databricks Runtime 4.2!
We’re excited to announce Databricks Runtime 4.2, powered by Apache Spark™. Version 4.2 includes updated Spark internals, new features, and major performance upgrades to Databricks Delta, as well as general quality improvements to the platform. We are moving quickly toward the Databricks Delta general availability (GA) release and we recommend you upgrade to Databricks Runtime...
