Apache Spark™ Clusters in Autopilot Mode
Apache Spark™ is a unified analytics engine that helps users use a single distributed computing framework for various use cases. With the advent...
Databricks clusters on AWS now support gp3 volumes, the latest generation of Amazon Elastic Block Storage (EBS) general purpose SSDs. gp3 volumes offer consistent performance, cost savings and the ability to configure the volume’s iops, throughput and volume size separately. Databricks on AWS customers can now easily switch to gp3 for the better price/performance storage of up to 20% per AWS.
If you add additional SSDs to your Databricks cluster instances then we recommend that you switch to gp3 (from gp2).
Follow these instructions to enable gp3.
For advanced use cases when you require additional control over the SSD volume’s iops and throughput, after enabling gp3, you can use the Databricks Clusters API. By default, the Databricks configuration will set the gp3 volume’s IOPs and throughput IOPS to match the maximum performance of a gp2 volume with the same volume size.
gp3 support will roll out to all Databricks on AWS regions by mid-August, 2021.
Try Databricks for free
