Improved Performance and Value With Databricks Photon and Azure Lasv3 Instances Using AMD 3rd Gen EPYC™ 7763v Processors

Databricks has partnered with AMD to support a new chip that lets you run your queries faster, saving you time and money. Combining the latest technologies from Azure Databricks and AMD, users can now take advantage of the new Lasv3-series VMs with the Databricks Runtimes to reduce the total cost of ownership (TCO) and achieve lower time to insights relative to the prior generation Lsv2-series VMs. The new Lasv3-series Azure VMs utilize AMD's 3rd Gen EPYC™ processors with 128-160 lanes of PCIe Gen4 I/O which deliver ample I/O bandwidth for the most demanding data warehouse workloads; up to 256MB of L3 cache per socket speeds up memory intensive analytical operations such as joins, aggregations and sorting.

Read on for details on the improvements we've measured and how you can start running your analyses faster with the click of a button.

Storage optimized Azure VM comparison

The Lasv3 and Lsv2-series are ideal for many data warehouse workloads, offering high-throughput, low latency, local NVMe storage and AMD processors. The Lsv2-series was launched in February of 2019, and has been a common choice for many workload patterns that benefit from the high local instance storage the series provided. In June 2022, the Lsv2-series was refreshed with the Lasv3-series, with processor upgrades to the AMD 3rd Gen EPYC™ 7763v processors and enhanced network bandwidth.

¹ On-demand VM pricing from East US 2 region as of 2022-09-13

Benchmarking workloads

In the last decade or so, TPC-DS has become the de facto standard data warehousing benchmark, adopted by virtually all vendors. TPC-DS performance measurements provide a great signal on system read performance for simple to complex analytical queries. We've supplemented this industry standard benchmark with an internally developed set of ETL-focused benchmarks. Where the TPC-DS workload adequately captures usage patterns similar to that of scheduled reporting or adhoc analytical exploration, the ETL benchmark is more aligned with data transformation patterns common in scheduled ETL type jobs, which include insert into partitioned and unpartitioned tables, denormalization and merge into operations.

¹ Derived from the power test consisting of all 99 TPC-DS queries. These results are not comparable to an official, audited TPC benchmark. Databricks' official TPC-DS results can be found here.
² Calculated with Premium All-Purpose Compute pricing
³ Calculated with Premium Jobs Compute pricing

The performance and value differences between the Lsv2-series and Lasv3-series was measured using a test matrix composed of these two workloads, executed on 20 worker cluster configurations, using the 11.2 Databricks Runtime with and without Photon enabled.

Measuring performance and value: ETL

The ETL benchmark was run during August 2022 at 1TB scale factor. The baseline used is the Lsv2-series configuration without Photon enabled. Improvements to performance and value are quantified relative to this baseline as we upgrade to the Lasv3-series and then enable Photon acceleration.

When upgrading a non-Photon runtime's instance from Lsv2 to Lasv3 we see both a performance and value improvement of 1.7x relative to our baseline. These gains can be directly attributed to the Lasv3-series' AMD 3rd Gen EPYC™ 7763v processors and increased network bandwidth.

Improvements to performance and value are even more substantial when comparing the non-Photon Lsv2 baseline to Photon enabled Lasv3 VMs, in which case we observed a 4.4x improvement to performance and a 3.6x improvement to value.

Measuring performance and value: TPC-DS inspired

The TPC-DS inspired benchmark was run during August 2022 at both 1TB and 10TB dataset volumes. The baseline for benchmark was also set using the Lsv2-series instances without Photon running on a cluster with 20 workers.

When upgrading the cluster configuration to use the new Lasv3-series instances a 1.4x performance improvement was observed relative to the baseline due to the new AMD's 3rd Gen EPYC processors. To further improve the performance Photon was then enabled on the cluster which resulted in a 5.3x speed-up and 2.5x improvement in the price-performance value.

Photon is able to lower the TCO of a workload, delivering price-performance improvements by reducing the necessary Azure instance uptime due to significantly accelerated query processing.

How to enable Photon and L8as_v3 instances

Taking advantage of these improvements in value and performance is simple for Databricks users and no change to your actual Spark code is required. When creating or modifying a Databricks cluster check the "Use Photon Acceleration" option and choose Azure L8as_v3 instances for the worker type.

Summary of performance results

For a variety of benchmarks, pairing Databricks Photon and Azure's Lasv3-series VMs with AMD 3rd Gen EPYC™ 7763v processors resulted in significant performance and value improvements relative to the Lsv2-series predecessor. For analytical read workloads, like TPC-DS, this was measured at up to a 5.3x performance improvement and 2.5x value improvement. For ETL workloads, this was measured at up to a 4.4x performance improvement and 3.6x value improvement. Follow the links below for additional information.

Learn more at

databricks.com/lakehouse
databricks.com/photon
azure.microsoft.com/en-us/pricing/details/virtual-machines/linux/#pricing
amd.com/en/processors/epyc-server-cpu-family
amd.com/en/products/cpu/amd-epyc-7763
docs.microsoft.com/en-us/azure/virtual-machines/lasv3-series
docs.microsoft.com/en-us/azure/virtual-machines/lsv2-series