Prioritize VM types with flexible VMs

Flexible VMs is a Dataproc feature that lets you specify prioritized lists of VM types for Dataproc master, primary, and secondary worker nodes when you create a Dataproc cluster.

Why use flexible VMs?

Previously, if a VM type was unavailable when you submitted a cluster creation request, the request failed, and you needed to update your request, script, or code to specify a "next-best" VM type. This re-request process could involve multiple iterations until you specified a VM type that was available.

The Dataproc flexible VM feature helps your cluster creation request succeed by selecting master, primary, and secondary worker VM types from your ranked VM lists, and then searching for zones within your specified cluster region with availability of the listed VM types.

Terminology

  • VM type: The family, memory capacity, and number of CPU cores of a VM instance. Dataproc supports the use of predefined and custom VM types.
  • Master and primary worker nodes: A cluster has a master node and at least two primary workers.
  • Secondary workers: Secondary workers are optional, and don't store data. They function only as processing nodes. You can use secondary workers to scale compute without scaling storage. The default flexible VM secondary worker type is a Spot VM, which is a preemptible type (see Dataproc secondary workers).

Usage

  • Flexible VMs are available in Dataproc on Compute Engine 2.0.74+, 2.1.76+, 2.2.42+, and later image versions.
  • Cluster creation with master or primary worker Flex VMs takes longer (approximately an additional 32 seconds).
  • The cluster name must not exceed 45 characters.
  • The same disk type is required for all VM types.
  • You can specify up to five ranked VM type lists, with up to 10 VM types in a list. For more information, see How to request flexible VMs.
  • The creation of a cluster with flexible VMs requires the use of Dataproc autozone placement, which allows Dataproc to choose the zone that has the capacity to fulfill your VM type requests.
  • If your cluster creation request includes an autoscaling policy, flexible VMs can be from different VM families, but they must have the same amount of memory and core count.
  • When provisioning flexible VMs, Dataproc consumes "any matching" available reservations, but not "specific" reservations (see Consume reserved instances). Machine types that match reservations are first selected within a rank, followed by VM types with the largest number of CPUs.
  • Dataproc applies Google Cloud quotas to flexible VM provisioning.
  • Although you can specify different CPU-to-memory ratios for primary and secondary worker V types in a cluster, this can lead to performance degradation because the smallest CPU-to-memory ratio is used as the smallest container unit.
  • If you update a cluster that was created using flexible VMs, Dataproc selects and adds workers from the flexible VM lists that you provided when you created your cluster.

How to request flexible VMs

You can specify flexible VMs when you create a Dataproc cluster using the Google Cloud console, Google Cloud CLI, or Dataproc API.

  • You can specify up to five ranked VM type lists, with up to 10 VM types in a list. Lowest ranked lists have the highest priority. By default, flexible VM lists have a rank of 0. Within a list, Dataproc prioritizes VM types with unused reservations, followed by the largest VM sizes. VM types within a list with the same CPU count are treated equally.

Console

To create a cluster with secondary worker flexible VMs, do the following:

  1. Open the Dataproc Create a cluster on Compute Engine page in the Google Cloud console.
  2. The Set up cluster panel is selected with fields filled in with default values. You can change the suggested name and the cluster region, and make other changes. Make sure that Any is selected as the cluster Zone to allow Dataproc autozone placement to choose the zone that has the best availability of the VM types specified in your flexible VM lists.
  3. Select the Configure nodes panel. In the Secondary worker nodes section, specify the number and preemptibility of secondary workers.
    • Click Add a secondary worker for each rank of secondary workers, specifying one or more machine types to include in each rank.
  4. After confirming and specifying cluster details in the cluster create panels, click Create.

gcloud

Use the gcloud dataproc clusters create command to add multiple master-machine-types, worker-machine-types and secondary-worker-machine-types flags to specify ranked flexible VM lists for master, primary, and secondary workers.

The following example requests master, primary, and secondary VM types with the following priorities:

  • Provision e2-standard-8 VMs if available (rank 0); if e2-standard-8 machines are not available, provision n2-standard-8 VMs (rank 1).

Since the secondary worker type isn't specified, preemptible Spot secondary VMs will be provisioned.

gcloud dataproc clusters create CLUSTER_NAME \
    --region=REGION \
    --zone="" \
    --master-machine-types="type=e2-standard-8,rank=0" \
    --master-machine-types="type=n2-standard-8,rank=1" \
    --num-workers=10 \
    --worker-machine-types="type=e2-standard-8,rank=0" \
    --worker-machine-types="type=n2-standard-8,rank=1" \
    --num-secondary-workers=4 \
    --secondary-worker-machine-types="type=e2-standard-8,rank=0" \
    --secondary-worker-machine-types="type=n2-standard-8,rank=1"

Notes:

  • --zone="": The flexible VM feature requires Dataproc autozone placement to allow Dataproc to choose the zone that has your VM types available for use. Passing an empty value ("") to the --zone flag overrides any zone selection specified in your default gcloud config list.

API

Use the instanceFlexibilityPolicy.instanceSelectionList as part of a Dataproc API clusters.create request to specify a ranked list of machineTypes for master, primary, and secondary workers.

Example: The following JSON snippet from a clusters.create request body specifies master (masterConfig), primary worker (workerConfig), and secondary worker (secondaryWorkerConfig) machine types with ranks 0 and rank 1.

{
  "projectId": "PROJECT_ID",
  "clusterName": "CLUSTER_NAME",
  "config": {
    "gceClusterConfig": {
      "zoneUri": ""
    },
    "masterConfig": {
      "numInstances": 1,
      "instanceFlexibilityPolicy": {
        "instanceSelectionList": [
          {
            "machineTypes": ["e2-standard-8"],
            "rank": 0
          },
          {
            "machineTypes": ["n2-standard-8"],
            "rank": 1
          }
        ]
      }
    },
    "workerConfig": {
      "numInstances": 10,
      "instanceFlexibilityPolicy": {
        "instanceSelectionList": [
          {
            "machineTypes": ["e2-standard-8"],
            "rank": 0
          },
          {
            "machineTypes": ["n2-standard-8"],
            "rank": 1
          }
        ]
      }
    },
    "secondaryWorkerConfig": {
      "numInstances": 4,
      "instanceFlexibilityPolicy": {
        "instanceSelectionList": [
          {
            "machineTypes": ["e2-standard-8"],
            "rank": 0
          },
          {
            "machineTypes": ["n2-standard-8"],
            "rank": 1
          }
        ]
      }
    }
  }
}

Override Flex VM properties

Dataproc sets properties at the cluster level. When you create a cluster that uses flexible VMs, you can override system-generated properties for primary and secondary worker Flex VM types.

gcloud

To override properties when you create a cluster, use the --properties flag with the following syntax:

--properties="$ROLE:$MACHINE_TYPE:$COMPONENT_PREFIX:$COMPONENT_PROPERTY=$VALUE"
  • ROLE can be either primary_worker or secondary_worker.
  • Separate multiple properties with a comma.

Run the following gcloud dataproc clusters create command to override the number of vCPUs that YARN allocates for NodeManager on secondary workers. This example sets the yarn.nodemanager.resource.cpu-vcores value in yarn-site.xml to 6 for all e2-standard-8 and n2-standard-8 secondary worker VMs.

gcloud dataproc clusters create CLUSTER_NAME \
    --num-workers=10 \
    --num-secondary-workers=4 \
    --worker-machine-types="type=e2-standard-8,rank=0" \
    --worker-machine-types="type=n2-standard-8,rank=1" \
    --master-machine-types="type=e2-standard-8,rank=0" \
    --master-machine-types="type=n2-standard-8,rank=1" \
    --secondary-worker-machine-types="type=e2-standard-8,rank=0" \
    --secondary-worker-machine-types="type=n2-standard-8,rank=1" \
    --region=us-central1 \
    --zone="" \
    --properties="secondary_worker:e2-standard-8:yarn:yarn.nodemanager.resource.cpu-vcores=6,secondary_worker:n2-standard-8:yarn:yarn.nodemanager.resource.cpu-vcores=6"

API

To override properties, define them in the properties field of the SoftwareConfig object in your cluster creation request.

Use the following syntax for the property key:

ROLE:MACHINE_TYPE:COMPONENT_PREFIX:COMPONENT_PROPERTY
  • ROLE can be either primary_worker or secondary_worker.

The following SoftwareConfig object overrides the number of vCPUs that YARN allocates for NodeManager on secondary workers. This example sets the yarn.nodemanager.resource.cpu-vcores value to 6 for all e2-standard-8 and n2-standard-8 secondary worker VMs.

{
  "imageVersion":"2.2.42",
  "properties": {
    "secondary_worker:e2-standard-8:yarn:yarn.nodemanager.resource.cpu-vcores" : "6",
    "secondary_worker:n2-standard-8:yarn:yarn.nodemanager.resource.cpu-vcores" : "6"
  }
}

What's next