New Microsoft Azure Data Warehouse Service and Azure Databricks Combine Analytics, BI, and Data Science

Read Rise of the Data Lakehouse to explore why lakehouses are the data architecture of the future with the father of the data warehouse, Bill Inmon.

In the last two years since it first became available, thousands of companies have adopted Azure Databricks, making it one of the fastest growing data and AI services on Microsoft Azure. Customers now process over 2 exabytes per month with millions of server-hours spinning up every day. All of this is driven by organizations like Electrolux, Shell, and renewables.AI that are using Azure Databricks to process data at massive scale for data science and analytics.

Within this amazing adoption is a specific solution architecture to highlight called the Modern Data Warehouse (MDW). Earlier this year we wrote about the performance and scale benefits of this solution, and part of the pattern’s success has been our close integration to Azure SQL Data Warehouse with a high-performance connector that was jointly engineered to make it fast and easy to move data between the two services.

Three ways Azure Databricks works with Azure Synapse Analytics

Today, Microsoft announced the next evolution of their data warehouse service: Azure Synapse Analytics. This is exciting news and we continue to work closely with Microsoft to integrate with Azure Synapse and bring analytics, business intelligence (BI), and data science together in one solution architecture. Here are three key ways Azure Databricks works with Azure Synapse:

1. 1. The high-performance connector between Azure Databricks and Azure Synapse will enable fast data transfer between the services, including support for streaming data. This means customers can continue to use Azure Databricks (up to 50x faster than open source Apache Spark) for extract, transform, and load (ETL) workloads to prep and shape data at scale for Azure Synapse.
   2. Azure Data Factory (ADF) supports Azure Databricks in the Mapping Data Flows feature. This offers code-free visual ETL for data preparation and transformation at scale, and now that ADF is part of the Azure Synapse workspace it provides another avenue to access these capabilities.
   3. Azure Synapse and Azure Databricks can run analytics on the same data in Azure Data Lake Storage. This opens even greater opportunities to combine analytics, BI, and data science solutions with a shared data lake across services.

We would love to hear your feedback as you begin using Azure Databricks and Azure Synapse in the next evolution of the Modern Data Warehouse solution architecture.