About instance flexibility in MIGs

Instance flexibility in managed instance groups (MIGs) lets you specify multiple machine types for the virtual machine (VM) instances in the group, rather than being limited to a single machine type. If a machine type is unavailable, the MIG can automatically select another compatible type from your list based on the current resource availability. This flexibility helps to improve resource obtainability, particularly for applications that can operate on different machine types and that require large-scale capacity or hardware with high demand.

This document describes how instance flexibility works in a MIG and its limitations. To know how to configure a MIG to use instance flexibility, see Add instance flexibility.

Instance flexibility

A MIG creates all its VMs based on the instance template that you select. The instance template specifies the machine type to use for each VM. With instance flexibility, you can list all the machine types that are compatible with your application.

The following image compares a MIG that uses a single machine type with a MIG that uses instance flexibility. As depicted in the image, a MIG with a single machine type creates identical VMs based on an instance template, whereas a MIG with instance flexibility can create VMs with different machine types and sizes.

Comparison of a MIG that uses a single machine type with a MIG that uses instance flexibility

How instance flexibility works

By default, a MIG creates all VMs using a single machine type from the instance template. You can set multiple machine types in a MIG by configuring an instance selection (instanceSelections) in the instance flexibility policy (instanceFlexibilityPolicy).

You can also specify machine types that run on different CPU platforms and those with different architectures (x86 and Arm architecture). To use machine types with different requirements for minimum CPU platform or disks, use the instance flexibility policy to override corresponding properties from the instance template. For more information, see How minCpuPlatform and disks overrides work.

Example for machine type overrides

If your workload can operate on a number of different machine types, you can configure a list of all machine types compatible with your application in a single instanceSelection as follows:

  "instanceFlexibilityPolicy": {
    "instanceSelections": {
      "instance-selection-1": {
        "machineTypes": ["c2-standard-8","c2d-standard-8","c3-standard-8", "c3d-standard-8"]
      }
    }
  }

The machine types within an instance selection have equal preference. If you want a MIG to choose machine types in a specific order, you can configure multiple instance selections, each with a list of machine types and a rank to define preference. A lower value of rank means higher preference. The MIG attempts to create its VMs using machine types with a higher preference. If these preferred machine types are not available, the MIG uses a machine type with a lower preference.

  "instanceFlexibilityPolicy": {
    "instanceSelections": {
      "most-preferred": {
        "rank": 1,
        "machineTypes": ["c3-standard-8","c2-standard-8"]
      },
      "least-preferred": {
        "rank": 2,
        "machineTypes": ["n2-standard-16","n1-standard-16"]
      }
    }
  }

Your application needs to be ready to run on any of the listed machine types as the availability of hardware changes over time and specific hardware is not assured unless you use reservations.

Instance selections can also override the instance properties like minCpuPlatform and disks. For more information, see the following section about How instance selection overrides work.

How minCpuPlatform and disks overrides work

In an instance selection, you can specify additional instance properties, such as minimum CPU platform (minCpuPlatform) and disks (disks), to override those defined in the instance template. When a MIG selects a machine type from an instance selection, the MIG creates a VM by using the instance properties specified in that instance selection. If you don't specify any of these properties for an instance selection, the MIG will use the corresponding properties from the template for that selection.

Overriding these properties at the instance selection level provides more control and flexibility, especially when using different hardware in a MIG.

  • Override minimum CPU platform when mixing machine types that run on different CPU platforms. You can specify a minimum CPU platform for instances created from the machine types within a selection. This override lets you mix different machine types but selectively apply minimum CPU platforms. Alternatively, you can configure the instance selection to automatically select a CPU platform by setting the minCpuPlatform field to AUTOMATIC. This automatic selection overrides the instance template's minimum CPU platform setting and allows the MIG to use any compatible CPU platform for the selected machine types.

    For example, you can mix machine types from N1 and N2 machine series in a selection, while ensuring that N1 machines run on Intel Skylake to achieve performance gains.

  • Override disks when mixing machine types that have different architectures. You can specify a custom disk configuration to override the boot disk or additional disks specified in the instance properties, or to attach a specific storage. This override lets you mix machine types that have different architectures or those that don't share disk technologies, and specify the appropriate boot images or disks required for each machine type.

    For example, you can mix C4 (x86 architecture) and C4A (Arm architecture) machine types. As x86 and Arm machines require different boot images, you can use separate instance selections to override the boot disk for each architecture.

    When you specify the disks for an instance selection, the MIG merges the disks in the instance template and in the instance selection by their deviceName keys. The following diagram shows an example.

    Disks from instance template and instance selection merges

To know about the limitations when using instance selection overrides, see the Limitations section in this document.

Example for minimum CPU platform and disk overrides

Consider an instance template with the following configuration:

  {
    ...
      "disks": [
        {
          "autoDelete": true,
          "deviceName": "boot",
          "boot": true,
          "diskEncryptionKey": {},
          "initializeParams": {
            "sourceImage": "projects/debian-cloud/global/images/family/debian-12",
            "diskSizeGb": "10",
            "diskType": "pd-balanced"
          }
        }
      ],
      "machineType": "n2-standard-2",
      ...
    }

The following example of an instance flexibility policy overrides the template configuration by specifying a minimum CPU platform and disk configuration.

  "instanceFlexibilityPolicy": {
    "instanceSelections": {
      "first-preference": {
        "rank": 1,
        "machineTypes": ["n4-standard-4 "],
        "disks": [
          {
            "autoDelete": true,
            "deviceName": "boot",
            "boot": true,
            "diskEncryptionKey": {},
            "initializeParams": {
              "sourceImage": "projects/debian-cloud/global/images/family/debian-12",
              "diskSizeGb": "10",
              "diskType": "hyperdisk-balanced"
            }
          }
        ]
      },
      "second-preference": {
        "rank": 2,
        "machineTypes": ["n2-standard-4"],
        "minCpuPlatform": "Intel Ice Lake",
        "disks": [
          {
            "autoDelete": true,
            "deviceName": "boot",
            "boot": true,
            "initializeParams": {
              "sourceImage": "projects/debian-cloud/global/images/family/debian-12",
              "diskSizeGb": "10",
              "diskType": "pd-ssd"
            }
          }
        ]
      },
      "third-preference": {
        "rank": 3,
        "machineTypes": ["n1-standard-4"],
        "minCpuPlatform": "Intel Skylake",
      }
    }
  }

How a MIG selects machine types

When you configure multiple machine types, the MIG selects a machine type based on current resource availability and prioritizes unused reservations. If you've configured ranks, the MIG first considers your preference order. Within a ranked instance selection, the MIG then checks resource availability, followed by reservations.

The zones in the MIG's region might also impact the machine type selection because resource availability varies across zones. For more information, see the following section about how a MIG selects zones.

How a MIG selects zones

You can configure instance flexibility in a regional MIG that has a BALANCED, ANY, or ANY_SINGLE_ZONE target distribution shape. Instance flexibility doesn't support EVEN target distribution shape.

With instance flexibility, when a regional MIG creates a VM, the MIG selects a zone depending on the selected target distribution shape as follows:

  • BALANCED: The MIG selects a zone with available resources that can balance the VM distribution across zones as evenly as possible. Within a selected zone, the MIG prioritizes machine types based on ranks, if configured, and available reservations.

  • ANY: The MIG selects a zone with available resources and reservations. If ranks are configured, then the MIG prioritizes the zone in which a machine type based on ranks is available.

  • ANY_SINGLE_ZONE: The MIG selects a zone which has the most machine types based on the ranks, if configured, and where the MIG can create most VMs out of unused reservations.

For more information about how a target distribution shape works, see Regional MIG target distribution shape.

Instance flexibility and reservations

You can use instance flexibility with reservations. Based on the reservation affinity property you set in the instance template, a MIG uses reservations as follows:

  • ANY_RESERVATION (recommended for instance flexibility): When using any reservation, the MIG selects the machine type with unused reservation first. If you want to change the order in which the MIG selects machine types, you can configure multiple instance selections and specify your preference using ranks. The MIG then first follows the ranks and then considers available reservations for the machine types within a rank.

  • SPECIFIC: When using a specific reservation, the MIG selects the machine type that matches the reservation properties. If your instance selections don't include the machine type that matches the specific reservation properties, then instance creation fails.

For more information about reservations, see About reservations.

Instance flexibility and Spot VMs

If your MIG with instance flexibility uses Spot VMs, then the MIG automatically selects a machine type with the lowest observed preemption rate to ensure least disruption for your workload. If you want to override this behavior, you can configure multiple instance selections and specify your preference using ranks. The MIG then first follows the preference order and then considers the Spot VMs reliability within an instance selection.

For more information about Spot VMs, see Spot VMs in a managed instance group.

Instance flexibility and VM repairs

By default, if a VM in a MIG fails, the MIG repairs the VM by recreating it using the same configuration, including the machine type. If you configure instance flexibility and enable updates on repair, then you can improve the chances to obtain resources if a stockout occurs. When a MIG attempts to recreate a VM, if the original machine type of the VM is unavailable, then you will get a stockout error. With updates on repair enabled, the MIG then uses your specified instance selections and current availability to choose another machine type for the VM creation. Enabling updates on repair also helps to ensure that the recreated VM uses the latest instance flexibility configuration.

For more information about repairs, see About repairing VMs for high availability.

Instance flexibility and autoscaling

When you configure autoscaling in a MIG that has multiple machine types, the following happens:

  • When an autoscaler scales out a MIG, the MIG creates VMs by selecting machine types in the same manner as in a non-autoscaled MIG. For more information, see the How a MIG selects machine types in this document.

  • When an autoscaler scales in a MIG, the MIG selects the VMs to delete as follows:

    1. The MIG first prioritizes deleting VMs that best preserve the target distribution shape.
    2. If you configured ranks, then among the prioritized VMs, the MIG prioritizes deleting VMs that use lower-ranked machine types.
    3. Finally, the MIG checks the autoscaler recommendation to select the VMs to delete.

Autoscaling works effectively when your MIG has machine types of similar performance—for example, machines with the same vCPU count. Otherwise, when the MIG scales out or scales in, the autoscaler might attempt to add or remove the highest-performing machine type more frequently. This behavior can happen regardless of the rank that you configured for that machine type.

For more information about autoscaling, see Autoscaling groups of instances.

Use cases

The following sections describe some use cases of instance flexibility in MIGs.

Optimize costs

A MIG can automatically provision the most cost-effective instances for your infrastructure when you configure the instance flexibility settings of your MIG as follows:

  • Machine types: Select the machine types that offer the preferred cost-performance ratio for your workload. You don't need to specify all the available sizes of each machine type; specify only the sizes that are most cost-optimal for your needs.

  • Rank: Assign ranks to instance selections according to your cost preferences, with the lowest-cost machine type as most preferred. This helps to ensure that you obtain the lowest-cost machine type based on availability.

  • Location: When creating the MIG, select all zones and the ANY target distribution shape. This increases the chances of obtaining the lowest-cost machine types in zones where they are available.

Based on your instance flexibility settings, the MIG automatically provisions the best possible combination of available machine types to meet your cost criteria and the total capacity requested. This ensures the lowest possible cost at a given time.

Improve obtainability

You can configure MIG instance flexibility to improve obtainability. To do so, select as many machine types as possible that are compatible with your application within the same rank. You can further increase the chance of obtaining the resources by choosing all zones when creating your MIG and using the ANY target distribution shape, which prioritizes zones where resources are available. Based on the selected machine types and the ANY shape, the MIG finds capacity where it is most available.

To maximize the chances of acquiring Spot VMs and to reduce the chances of preemption, Google recommends you include a wide range of machine types within the same rank. This widens the MIG's search for available capacity, leading to the best possible outcomes with the spot provisioning model.

Maximize performance

You can configure MIG instance flexibility to maximize performance. To do so, select machine families and sizes that best serve your workload. Then rank the machine types based on performance, from highest to lowest, using the most preferred size within each family. Based on the ranking and the machine types, the MIG selects the highest-performing machines from all available resources.

Additionally, specifying a minCpuPlatform for the instance selection can ensure that the MIG selects a high-performing CPU platform.

Seamless transition to latest machine types

You can configure MIG instance flexibility to prioritize the adoption of latest generation machine types while using older generation machine types as a fallback for capacity. To do so, create an instance selection containing the latest generation machine types and assign a lower rank (higher preference). If these machine types require different disk configurations, use the disk overrides within this instance selection to define the specific disk properties. Create a second instance selection containing the older generation machine types to serve as a fallback. Assign a higher rank (lower preference) to this instance selection.

When creating the MIG, select all zones in the region and the ANY target distribution shape. This configuration ensures that the MIG first searches for the latest generation machines across all zones in the region. The MIG will only provision the older generation machines if the MIG can't find enough latest generation capacity, ensuring that your requested total capacity is always met while maximizing the share of the newest hardware.

Limitations

Instance flexibility has the following limitations:

  • Instance flexibility is not supported in zonal MIGs. You can use a regional MIG with a single zone instead.

  • Instance flexibility doesn't support the following MIG features:

  • When overriding minCpuPlatform and disks, the following limitations also apply:

    • You must enable updates on repairs.

    • When configuring the minCpuPlatform or disks for an existing instance selection, you must also include the machineTypes field. Without the machineTypes field, the operation fails with an error.

    • You can't apply stateful policy or per-instance configuration to disks overridden through an instance selection.

    • You can override local SSDs only for those machine types that allow you to choose a number of local SSD disks. You can't override local SSD for -lssd variants of machine types, which automatically attach local SSD disks.

    • You must specify unique machine type across all instance selections. For example, you cannot specify n1-standard-2 in one selection with a Skylake CPU and also in another selection with a Broadwell CPU.

  • To help ensure effective load balancing or autoscaling in a MIG that has instance flexibility, use machine types with similar performance—for example, the same vCPU count.

    • In Cloud Load Balancing, similar performance machine types can prevent over-utilization of smaller hardware and under-utilization of faster machines.

    • In autoscaling, similar performance machine types can prevent addition or deletion of the highest-performing machine type.

What's next?