Prioritize VM types with flexible VMs

Flexible VMs is a Managed Service for Apache Spark feature that lets you specify prioritized lists of VM types for Managed Service for Apache Spark master, primary, and secondary worker nodes when you create a Managed Service for Apache Spark cluster.

Why use flexible VMs?

The problem: If a VM type is unavailable when you submit a cluster creation request, the request fails, and you need to update your request, script, or code to specify a "next-best" VM type. This re-request process can involve multiple iterations until you specify a VM type that is available.

The solution: The Managed Service for Apache Spark flexible VM feature helps your cluster creation request succeed by selecting master, primary, and secondary worker VM types from your ranked VM lists, and then searching for zones within your specified cluster region with availability of the listed VM types.

Limitations

• Clusters that use flexible VMs cannot be stopped.
• Master nodes in High availability clusters cannot use flexible VMs, but HA cluster worker nodes can use flexible VMs.

Terminology

• VM type: The family, memory capacity, and number of CPU cores of a VM instance. Managed Service for Apache Spark supports the use of predefined and custom VM types.
  In this document the terms "VM type" and "instance-selection" refer to a supported machine type used to provision VMs in a cluster master or worker node.
• Master and primary worker nodes: By default, a Managed Service for Apache Spark cluster has one master node and two primary worker nodes.
  • A High availability (HA) cluster has three master nodes.
  • A single node cluster has one node that acts as a master and worker node.
  • A zero-scale cluster has a master node and secondary workers only (no primary workers).
• Secondary worker nodes: Secondary workers don't store data and function only as processing nodes. You can use secondary workers to scale compute without scaling storage. The default flexible VM secondary worker type is a Spot VM, which is a preemptible type.

Usage

• Flexible VMs are available in Managed Service for Apache Spark on Managed Service for Apache Spark 2.0.74+, 2.1.76+, 2.2.42+, and later image versions.
• Cluster creation with master or primary worker Flex VMs typically adds 32 seconds to cluster creation time.

• You can specify up to five ranked VM type lists, with up to 10 VM types in a list.

  Recommendation: Enable Managed Service for Apache Spark autozone placement, which allows Managed Service for Apache Spark to choose a zone with the capacity to provision requested VMs.

• By default, a cluster node must use one disk type. You can use disk overrides to specify different disk types for different machine types specified for a Flex VM cluster node.

• Although you can specify different CPU-to-memory ratios for primary and secondary worker VM types in a cluster, this can lead to performance degradation because the smallest CPU-to-memory ratio is used as the smallest container unit.

  Use a uniform CPU-to-memory ratio for primary and secondary workers, including flexible VMs.

• If your cluster creation request includes an autoscaling policy, flexible VMs can be from different VM families, but they must have the same amount of memory and core count.

• When provisioning flexible VMs, Managed Service for Apache Spark consumes "any matching" available reservations, but not "specific" reservations (see Consume reserved instances). Machine types that match reservations are first selected within a rank, followed by VM types with the largest number of CPUs.

• Managed Service for Apache Spark applies Google Cloud quotas to flexible VM provisioning.

• If you update a cluster that was created using flexible VMs, Managed Service for Apache Spark selects and adds workers from the flexible VM lists that you provided when you created your cluster.

How to request flexible VMs

You can specify up to five ranked VM type lists, with up to 10 VM types in a list. Lowest ranked lists have the highest priority. By default, flexible VM lists have a rank of 0. Within a list, Managed Service for Apache Spark prioritizes VM types with unused reservations, followed by the largest VM sizes. VM types within a list with the same CPU count are treated equally.

You can request flexible VMs when you create a Managed Service for Apache Spark cluster using the Google Cloud console, Google Cloud CLI, or Dataproc API.

Console

To create a cluster with flexible VMs, do the following:

Note: Console support for manager (master) node Flex VMs is not available.

1. Open the Create cluster page.
2. Click Additional configuration to expand that section.
3. Edit Primary workers or Secondary workers. Under Add worker types, specify additional ranked VMs.

gcloud

Use the gcloud dataproc clusters create command with master-instance-selection, worker-instance-selection and secondary-worker-instance-selection flags to specify ranked flexible VM lists for master, primary, and secondary workers.

Deprecated: The previous --master-machine-types, --worker-machine-types, and --secondary-worker-machine-types flags are deprecated.

The following example requests master, primary, and secondary VM types with the following priorities:

• Provision e2-standard-8 VMs if available (rank 0); if e2-standard-8 machines are not available, provision n2-standard-8 VMs (rank 1).

Since the secondary worker type isn't specified, preemptible Spot secondary VMs will be provisioned.

gcloud dataproc clusters create CLUSTER_NAME \
    --region=REGION \
    --zone="" \
    --master-instance-selection='{"machineTypes":["e2-standard-8"],"rank":0}' \
    --master-instance-selection='{"machineTypes":["n2-standard-8"],"rank":1}' \
    --num-workers=10 \
    --worker-instance-selection='{"machineTypes":["e2-standard-8"],"rank":0}' \
    --worker-instance-selection='{"machineTypes":["n2-standard-8"],"rank":1}' \
    --num-secondary-workers=4 \
    --secondary-worker-instance-selection='{"machineTypes":["e2-standard-8"],"rank":0}' \
    --secondary-worker-instance-selection='{"machineTypes":["n2-standard-8"],"rank":1}'

Notes:

• --zone="": Setting this flag to an empty value enables autozone placement, which allows Managed Service for Apache Spark to choose a zone that has the requested VM types available for use. This flag value overrides any zone selection specified in your default gcloud config list.

API

Use the instanceFlexibilityPolicy.instanceSelectionList as part of a Dataproc API clusters.create request to specify a ranked list of machineTypes for master, primary, and secondary workers.

Example: The following JSON snippet from a clusters.create request body specifies master (masterConfig), primary worker (workerConfig), and secondary worker (secondaryWorkerConfig) machine types with ranks 0 and rank 1.

{
  "projectId": "PROJECT_ID",
  "clusterName": "CLUSTER_NAME",
  "config": {
    "gceClusterConfig": {
      "zoneUri": ""
    },
    "masterConfig": {
      "numInstances": 1,
      "instanceFlexibilityPolicy": {
        "instanceSelectionList": [
          {
            "machineTypes": ["e2-standard-8"],
            "rank": 0
          },
          {
            "machineTypes": ["n2-standard-8"],
            "rank": 1
          }
        ]
      }
    },
    "workerConfig": {
      "numInstances": 10,
      "instanceFlexibilityPolicy": {
        "instanceSelectionList": [
          {
            "machineTypes": ["e2-standard-8"],
            "rank": 0
          },
          {
            "machineTypes": ["n2-standard-8"],
            "rank": 1
          }
        ]
      }
    },
    "secondaryWorkerConfig": {
      "numInstances": 4,
      "instanceFlexibilityPolicy": {
        "instanceSelectionList": [
          {
            "machineTypes": ["e2-standard-8"],
            "rank": 0
          },
          {
            "machineTypes": ["n2-standard-8"],
            "rank": 1
          }
        ]
      }
    }
  }
}

Disk overrides

You can specify disk overrides for each machine type (instance selection) in your flexible VM specification. This lets you customize boot disks, override local ssds and attach additional disks for specific machine types.

Disk override options and rules

Disk override configuration options:

• Base disk configuration: A disk configuration specified for a cluster node, for example, specifying a boot disk size for primary workers using the gcloud CLI --worker-boot-disk-size flag or the workerConfig.diskConfig.bootDiskSizeGb Dataproc API field.
• Instance selection disk overrides: Disk configurations for machine types specified for a cluster node.

Disk override configuration rules:

• Base disk configuration: If no instance selection for a cluster node includes a diskConfig override, you can define a base disk configuration for the node. This base disk configuration is applied to all instance selections for the node.

• Instance selection disk configurations: If an instance selection for a cluster node includes a diskConfig override, all instance selections in the node group must include a diskConfig (if you also define a base disk configuration for the node, a validation error occurs).

• Machine type compatibility: All machine types within a single InstanceSelection must be compatible with the specified diskConfig. For example, you can't group an e2-standard-4 machine type, which does not support a hyperdisk, with an n4-standard-4 machine type, which requires a hyperdisk, in the same instance selection since the diskConfig can't satisfy both machine types.

• Local SSD support: If you configure local SSDs (numLocalSsds > 0) in a disk override configuration, all machine types in the instance selection must support local SSDs.

• Mandatory disk override configuration fields:

  • If you define diskConfig for an instance selection, bootDiskType is mandatory.
  • If you define attachedDiskConfigs, both diskType and diskSizeGb are mandatory for each attached disk.

Disk override configuration examples

Note: gcloud CLI support for disk overrides will be available soon.

The following example specifies the following disk override configuration options for the following cluster nodes:

• Master nodes: Use default boot disks.
• Primary workers: Use customized disks per instance selection: for example n4-standard-4 uses hyperdisk-balanced, while n2-standard-4 uses pd-standard.
• Secondary workers: Use a custom base disk configuration: pd-ssd with 200 GB, which is applied to all instance selections.

Dataproc API

Use the diskConfig field within an instanceFlexibilityPolicy.instanceSelectionList in a Dataproc API clusters.create request.

Example JSON request body:

{
  "projectId": "PROJECT_ID",
  "clusterName": "CLUSTER_NAME",
  "config": {
    "gceClusterConfig": {
      "zoneUri": ""
    },
    "masterConfig": {
      "numInstances": 1,
      "instanceFlexibilityPolicy": {
        "instanceSelectionList": [
          {
            "machineTypes": ["e2-standard-8"],
            "rank": 0
          },
          {
            "machineTypes": ["n2-standard-8"],
            "rank": 1
          }
        ]
      }
    },
    "workerConfig": {
      "numInstances": 10,
      "instanceFlexibilityPolicy": {
        "instanceSelectionList": [
          {
            "machineTypes": ["n4-standard-4"],
            "rank": 0,
            "diskConfig": {
              "bootDiskType": "hyperdisk-balanced",
              "bootDiskSizeGb": 100,
              "bootDiskProvisionedIops": 6000,
              "bootDiskProvisionedThroughput": 400,
              "attachedDiskConfigs": [
                {
                  "diskType": "HYPERDISK_THROUGHPUT",
                  "diskSizeGb": 2048
                }
              ]
            }
          },
          {
            "machineTypes": ["n2-standard-4"],
            "rank": 0,
            "diskConfig": {
              "bootDiskType": "pd-standard",
              "bootDiskSizeGb": 400
            }
          }
        ]
      }
    },
    "secondaryWorkerConfig": {
      "numInstances": 4,
      "diskConfig": {
        "bootDiskType": "pd-ssd",
        "bootDiskSizeGb": 200
      },
      "instanceFlexibilityPolicy": {
        "instanceSelectionList": [
          {
            "machineTypes": ["e2-standard-8"],
            "rank": 0
          },
          {
            "machineTypes": ["n2-standard-8"],
            "rank": 1
          }
        ]
      }
    }
  }
}

Override Flex VM properties

Managed Service for Apache Spark sets properties at the cluster level. When you create a cluster that uses flexible VMs, you can override system-generated properties for primary and secondary worker Flex VM types.

gcloud

To override properties when you create a cluster, use the --properties flag with the following syntax:

--properties="$ROLE:$MACHINE_TYPE:$COMPONENT_PREFIX:$COMPONENT_PROPERTY=$VALUE"

• ROLE can be either primary_worker or secondary_worker.
• Separate multiple properties with a comma.

The following gcloud dataproc clusters create command overrides the number of vCPUs that YARN allocates for NodeManager on secondary workers. This example sets the yarn.nodemanager.resource.cpu-vcores value in yarn-site.xml to 6 for all e2-standard-8 and n2-standard-8 secondary worker VMs.

gcloud dataproc clusters create CLUSTER_NAME \
    --num-workers=10 \
    --num-secondary-workers=4 \
    --worker-machine-types="type=e2-standard-8,rank=0" \
    --worker-machine-types="type=n2-standard-8,rank=1" \
    --master-machine-types="type=e2-standard-8,rank=0" \
    --master-machine-types="type=n2-standard-8,rank=1" \
    --secondary-worker-machine-types="type=e2-standard-8,rank=0" \
    --secondary-worker-machine-types="type=n2-standard-8,rank=1" \
    --region=us-central1 \
    --zone="" \
    --properties="secondary_worker:e2-standard-8:yarn:yarn.nodemanager.resource.cpu-vcores=6,secondary_worker:n2-standard-8:yarn:yarn.nodemanager.resource.cpu-vcores=6"

API

To override properties, define them in the properties field of the SoftwareConfig object in your cluster creation request.

Use the following syntax for the property key:

ROLE:MACHINE_TYPE:COMPONENT_PREFIX:COMPONENT_PROPERTY

• ROLE can be either primary_worker or secondary_worker.

The following SoftwareConfig object overrides the number of vCPUs that YARN allocates for NodeManager on secondary workers. This example sets the yarn.nodemanager.resource.cpu-vcores value to 6 for all e2-standard-8 and n2-standard-8 secondary worker VMs.

{
  "imageVersion":"2.2.42",
  "properties": {
    "secondary_worker:e2-standard-8:yarn:yarn.nodemanager.resource.cpu-vcores" : "6",
    "secondary_worker:n2-standard-8:yarn:yarn.nodemanager.resource.cpu-vcores" : "6"
  }
}

What's next

• Learn more about Managed Service for Apache Spark cluster properties.
• Learn how to create a Managed Service for Apache Spark cluster.