Announcing the General Availability of Azure Databricks support for Azure confidential computing (ACC)

Today we are excited to announce the general availability of Azure Databricks support for Azure confidential computing (ACC)!

With support for Azure confidential computing, customers can build an end-to-end data platform with increased confidentiality and privacy on Databricks by protecting data in use, or in memory, with AMD-based Azure confidential virtual machines (VMs). This type of data protection complements the protection of your confidential data using existing Azure Databricks controls such as customer-managed keys for data at rest and private link with TLS encryption for data in transit. As a result, Azure Databricks clusters running on Azure confidential VMs are protected by a comprehensive end-to-end encryption solution, safeguarding data throughout its entire lifecycle. It's easy to get started as you simply select which ACC VM you would like to run your workload on. Refer to our documentation for instructions on how to begin.

In our public preview blog announcement, we gave an overview of confidential computing use cases as well as our partnership with Microsoft. In this blog, we will explore the advantages of our integration with Azure confidential computing and AMD, some of the key features in this release, and resources to learn more.

Bring together data value, security, and power with Azure Databricks, Azure Confidential Computing, and AMD

The Databricks Data Intelligence Platform is built on a lakehouse architecture that unites the reliability, governance, and performance of a data warehouse with the openness, flexibility, and machine learning support of data lakes. Our integration with Azure confidential computing allows you to run your Azure Databricks workloads on Azure confidential virtual machines (VMs) featuring AMD EPYC™ CPUs provided by AMD Infinity Guard SEV-SNP technology. AMD confidential VMs provide this protection via full VM encryption while minimizing the performance impact at the same time. By harnessing the power of AMD EPYC processors, these VMs provide the horsepower needed to handle even the most demanding data processing workloads.

Check out the below video to learn more.

"Azure Databricks on confidential computing VMs is our first choice for the robust protection of confidential customer data across multiple industries. Our successful collaboration with Microsoft and Databricks enables our customers not only to unlock significant value from their data, but it also emphasizes data privacy and ownership throughout the large-scale data analysis of sensitive information."

— Lasse Jenzen, Senior Consultant, ORAYLIS GmbH

Key features available as part of our GA release

You can adopt the protections offered by Azure confidential computing by simply selecting one of the confidential VMs for your Azure Databricks cluster. You can also configure existing Azure Databricks controls such as customer-managed keys, Azure Private Link and Unity Catalog for a comprehensive solution protecting data throughout its entire lifecycle.

With GA, we are adding two additional features:

• VM Grouping: A new confidential VM subgroup will be added to the VM selection drop-down UI so that you can easily see which confidential VMs are available to select to run your workloads.
  
  
• Expansion of confidential VM region availability: Following the initial public preview, Azure confidential computing has expanded significantly, with the inclusion of 5 new confidential VM regions. This brings the total count to 9 available regions: East US, West US, North Europe, West Europe, Southeast Asia, Central India, East Asia, Switzerland North, and Japan East. Be on the lookout for even more regions set to be integrated over the upcoming year. For the most up-to-date information, visit the Azure Global infrastructure site.

Getting Started with Azure Databricks on Azure confidential computing

Visit our Azure Databricks documentation or watch our demo video to learn more about how you can use Azure confidential computing for your Databricks workloads.

To further ignite the art of the possible with Azure Databricks, Azure confidential computing, and AMD, check out our new landing page and eBook: "The Guide to Data and AI Transformation at Scale."

Tune into Microsoft Ignite this week to learn more about the recent innovations with Azure confidential computing. We also hope to see you at one of our Data and AI World Tour locations. Register today!