Introducing Databricks Fleet Clusters for AWS

We're excited to announce the general availability of Databricks Fleet clusters on AWS.

What are Fleet clusters?

Databricks Fleet clusters unlock the potential of Spot pricing without the hassle of manual instance selection by allowing Databricks to intelligently optimize and automate the process for you. You can use AWS Spot instances at the lowest cost possible, all while ensuring availability, thanks to Databricks' ability to flex across multiple instances in a single cluster. Avoid the complexities of managing your cloud infrastructure and spend your time on what truly matters: your data-driven insights.

Databricks Fleet clusters introduce a set of new fleet instance families for our AWS customers: m-fleet, md-fleet, r-fleet and rd-fleet in sizes xlarge, 2xlarge, 4xlarge and 8xlarge. m instances have 4 GB memory/vCPU, r instances have 8 GB memory/vCPU, and instances with d have local NVME SSD attached.

Under the hood, when a cluster uses one of these fleet instance types, Databricks will select the matching physical AWS instance types with the best price and availability to use in your cluster. Databricks has restricted the set of possible instance combinations to ensure that you get maximum stability and performance out of your cluster. We'll continue to update which instances we use behind the scenes as AWS launches additional types so that you no longer have to manually tune your cluster definitions. When your cluster uses spot instances, we'll select the instance types with the lowest price and least likelihood of spot termination.

For example, if a cluster has the following configuration:

  Driver: rd-fleet.xlarge
  Workers: rd-fleet.4xlarge
  Number of Workers: 3

The cluster may end up acquiring the following instance types:

  Driver: r6id.xlarge
  Workers: r6id.4xlarge, r6id.4xlarge, r5d.4xlarge

The specific instances are determined by the price and capacity conditions at the time of cluster creation. Databricks will bill you based on the normal DBU rates for each of the acquired instances.

How can I use it?

Simply create a new cluster or instance pool to use a fleet instance type. We recommend using fleet instance types for both worker and driver.

You will also need to make sure that the AWS IAM role for your databricks workspace includes the most up-to-date permissions listed in our documentation. If any permissions are missing, you'll be prompted to add them when attempting to start a fleet cluster.

NOTE: Older workspaces will be updated to support fleet clusters over the coming weeks. If you have any issues with access, please contact customer support.