Prioritize VM types with flexible VMs

Flexible VMs is a Managed Service for Apache Spark feature that lets you specify prioritized lists of VM types for Managed Service for Apache Spark master, primary, and secondary worker nodes when you create a Managed Service for Apache Spark cluster.

Why use flexible VMs?

Previously, if a VM type was unavailable when you submitted a cluster creation request, the request failed, and you needed to update your request, script, or code to specify a "next-best" VM type. This re-request process could involve multiple iterations until you specified a VM type that was available.

The Managed Service for Apache Spark flexible VM feature helps your cluster creation request succeed by selecting master, primary, and secondary worker VM types from your ranked VM lists, and then searching for zones within your specified cluster region with availability of the listed VM types.

Limitations

Clusters that use flexible VMs cannot be stopped.

Terminology

  • VM type: The family, memory capacity, and number of CPU cores of a VM instance. Managed Service for Apache Spark supports the use of predefined and custom VM types.
  • Master and primary worker nodes: A cluster has a master node and at least two primary workers.
  • Secondary workers: Secondary workers are optional, and don't store data. They function only as processing nodes. You can use secondary workers to scale compute without scaling storage. The default flexible VM secondary worker type is a Spot VM, which is a preemptible type (see Managed Service for Apache Spark secondary workers).

Usage

  • Flexible VMs are available in Managed Service for Apache Spark on Compute Engine 2.0.74+, 2.1.76+, 2.2.42+, and later image versions.
  • Cluster creation with master or primary worker Flex VMs takes longer (approximately an additional 32 seconds).
  • The cluster name must not exceed 45 characters.
  • The same disk type is required for all VM types.
  • You can specify up to five ranked VM type lists, with up to 10 VM types in a list. For more information, see How to request flexible VMs.
  • The creation of a cluster with flexible VMs requires the use of Managed Service for Apache Spark autozone placement, which allows Managed Service for Apache Spark to choose the zone that has the capacity to fulfill your VM type requests.
  • If your cluster creation request includes an autoscaling policy, flexible VMs can be from different VM families, but they must have the same amount of memory and core count.
  • When provisioning flexible VMs, Managed Service for Apache Spark consumes "any matching" available reservations, but not "specific" reservations (see Consume reserved instances). Machine types that match reservations are first selected within a rank, followed by VM types with the largest number of CPUs.
  • Managed Service for Apache Spark applies Google Cloud quotas to flexible VM provisioning.
  • Although you can specify different CPU-to-memory ratios for primary and secondary worker V types in a cluster, this can lead to performance degradation because the smallest CPU-to-memory ratio is used as the smallest container unit.
  • If you update a cluster that was created using flexible VMs, Managed Service for Apache Spark selects and adds workers from the flexible VM lists that you provided when you created your cluster.

How to request flexible VMs

You can specify flexible VMs when you create a Managed Service for Apache Spark cluster using the Google Cloud console, Google Cloud CLI, or Managed Service for Apache Spark API.

  • You can specify up to five ranked VM type lists, with up to 10 VM types in a list. Lowest ranked lists have the highest priority. By default, flexible VM lists have a rank of 0. Within a list, Managed Service for Apache Spark prioritizes VM types with unused reservations, followed by the largest VM sizes. VM types within a list with the same CPU count are treated equally.

Console

To create a cluster with secondary worker flexible VMs, do the following:

  1. Open the Managed Service for Apache Spark Create a cluster on Compute Engine page in the Google Cloud console.
  2. The Set up cluster panel is selected with fields filled in with default values. You can change the suggested name and the cluster region, and make other changes. Make sure that Any is selected as the cluster Zone to allow Managed Service for Apache Spark autozone placement to choose the zone that has the best availability of the VM types specified in your flexible VM lists.
  3. Select the Configure nodes panel. In the Secondary worker nodes section, specify the number and preemptibility of secondary workers.
    • Click Add a secondary worker for each rank of secondary workers, specifying one or more machine types to include in each rank.
  4. After confirming and specifying cluster details in the cluster create panels, click Create.

gcloud

Use the gcloud dataproc clusters create command to add multiple master-machine-types, worker-machine-types and secondary-worker-machine-types flags to specify ranked flexible VM lists for master, primary, and secondary workers.

The following example requests master, primary, and secondary VM types with the following priorities:

  • Provision e2-standard-8 VMs if available (rank 0); if e2-standard-8 machines are not available, provision n2-standard-8 VMs (rank 1).

Since the secondary worker type isn't specified, preemptible Spot secondary VMs will be provisioned.

gcloud dataproc clusters create CLUSTER_NAME \
    --region=REGION \
    --zone="" \
    --master-machine-types="type=e2-standard-8,rank=0" \
    --master-machine-types="type=n2-standard-8,rank=1" \
    --num-workers=10 \
    --worker-machine-types="type=e2-standard-8,rank=0" \
    --worker-machine-types="type=n2-standard-8,rank=1" \
    --num-secondary-workers=4 \
    --secondary-worker-machine-types="type=e2-standard-8,rank=0" \
    --secondary-worker-machine-types="type=n2-standard-8,rank=1"

Notes:

  • --zone="": The flexible VM feature requires Managed Service for Apache Spark autozone placement to allow Managed Service for Apache Spark to choose the zone that has your VM types available for use. Passing an empty value ("") to the --zone flag overrides any zone selection specified in your default gcloud config list.

API

Use the instanceFlexibilityPolicy.instanceSelectionList as part of a Managed Service for Apache Spark API clusters.create request to specify a ranked list of machineTypes for master, primary, and secondary workers.

Example: The following JSON snippet from a clusters.create request body specifies master (masterConfig), primary worker (workerConfig), and secondary worker (secondaryWorkerConfig) machine types with ranks 0 and rank 1.

{
  "projectId": "PROJECT_ID",
  "clusterName": "CLUSTER_NAME",
  "config": {
    "gceClusterConfig": {
      "zoneUri": ""
    },
    "masterConfig": {
      "numInstances": 1,
      "instanceFlexibilityPolicy": {
        "instanceSelectionList": [
          {
            "machineTypes": ["e2-standard-8"],
            "rank": 0
          },
          {
            "machineTypes": ["n2-standard-8"],
            "rank": 1
          }
        ]
      }
    },
    "workerConfig": {
      "numInstances": 10,
      "instanceFlexibilityPolicy": {
        "instanceSelectionList": [
          {
            "machineTypes": ["e2-standard-8"],
            "rank": 0
          },
          {
            "machineTypes": ["n2-standard-8"],
            "rank": 1
          }
        ]
      }
    },
    "secondaryWorkerConfig": {
      "numInstances": 4,
      "instanceFlexibilityPolicy": {
        "instanceSelectionList": [
          {
            "machineTypes": ["e2-standard-8"],
            "rank": 0
          },
          {
            "machineTypes": ["n2-standard-8"],
            "rank": 1
          }
        ]
      }
    }
  }
}

Override Flex VM properties

Managed Service for Apache Spark sets properties at the cluster level. When you create a cluster that uses flexible VMs, you can override system-generated properties for primary and secondary worker Flex VM types.

gcloud

To override properties when you create a cluster, use the --properties flag with the following syntax:

--properties="$ROLE:$MACHINE_TYPE:$COMPONENT_PREFIX:$COMPONENT_PROPERTY=$VALUE"
  • ROLE can be either primary_worker or secondary_worker.
  • Separate multiple properties with a comma.

Run the following gcloud dataproc clusters create command to override the number of vCPUs that YARN allocates for NodeManager on secondary workers. This example sets the yarn.nodemanager.resource.cpu-vcores value in yarn-site.xml to 6 for all e2-standard-8 and n2-standard-8 secondary worker VMs.

gcloud dataproc clusters create CLUSTER_NAME \
    --num-workers=10 \
    --num-secondary-workers=4 \
    --worker-machine-types="type=e2-standard-8,rank=0" \
    --worker-machine-types="type=n2-standard-8,rank=1" \
    --master-machine-types="type=e2-standard-8,rank=0" \
    --master-machine-types="type=n2-standard-8,rank=1" \
    --secondary-worker-machine-types="type=e2-standard-8,rank=0" \
    --secondary-worker-machine-types="type=n2-standard-8,rank=1" \
    --region=us-central1 \
    --zone="" \
    --properties="secondary_worker:e2-standard-8:yarn:yarn.nodemanager.resource.cpu-vcores=6,secondary_worker:n2-standard-8:yarn:yarn.nodemanager.resource.cpu-vcores=6"

API

To override properties, define them in the properties field of the SoftwareConfig object in your cluster creation request.

Use the following syntax for the property key:

ROLE:MACHINE_TYPE:COMPONENT_PREFIX:COMPONENT_PROPERTY
  • ROLE can be either primary_worker or secondary_worker.

The following SoftwareConfig object overrides the number of vCPUs that YARN allocates for NodeManager on secondary workers. This example sets the yarn.nodemanager.resource.cpu-vcores value to 6 for all e2-standard-8 and n2-standard-8 secondary worker VMs.

{
  "imageVersion":"2.2.42",
  "properties": {
    "secondary_worker:e2-standard-8:yarn:yarn.nodemanager.resource.cpu-vcores" : "6",
    "secondary_worker:n2-standard-8:yarn:yarn.nodemanager.resource.cpu-vcores" : "6"
  }
}

What's next