This page describes how to provision VMs on sole-tenant nodes, which are physical servers that run VMs only from a single project. Before provisioning VMs on sole-tenant nodes, read the [sole-tenant node overview](https://docs.cloud.google.com/compute/docs/nodes/sole-tenant-nodes). ## Before you begin - Before provisioning VMs on sole-tenant nodes, [check your quota](https://docs.cloud.google.com/compute/quotas-limits#check_your_allocation_quota). Depending on the number and size of nodes that you reserve, you might need to [request additional quota](https://docs.cloud.google.com/compute/quotas-limits#edit_your_quota). - [Create a sole-tenant node template](https://docs.cloud.google.com/compute/docs/nodes/sole-tenancy-node-templates). - [Create a sole-tenant node group](https://docs.cloud.google.com/compute/docs/nodes/sole-tenancy-node-groups). - If you haven't already, set up [authentication](https://docs.cloud.google.com/compute/docs/authentication). Authentication verifies your identity for access to Google Cloud services and APIs. To run code or samples from a local development environment, you can authenticate to Compute Engine by selecting one of the following options: Select the tab for how you plan to use the samples on this page: ### Console When you use the Google Cloud console to access Google Cloud services and APIs, you don't need to set up authentication. ### gcloud 1. [Install](https://docs.cloud.google.com/sdk/docs/install) the Google Cloud CLI. After installation, [initialize](https://docs.cloud.google.com/sdk/docs/initializing) the Google Cloud CLI by running the following command: ```bash gcloud init ``` If you're using an external identity provider (IdP), you must first [sign in to the gcloud CLI with your federated identity](https://docs.cloud.google.com/iam/docs/workforce-log-in-gcloud). > [!NOTE] > **Note:** If you installed the gcloud CLI previously, make sure you have the latest version by running `gcloud components update`. 2. [Set a default region and zone](https://docs.cloud.google.com/compute/docs/gcloud-compute#set_default_zone_and_region_in_your_local_client). ### REST To use the REST API samples on this page in a local development environment, you use the credentials you provide to the gcloud CLI. 1. [Install](https://docs.cloud.google.com/sdk/docs/install) the Google Cloud CLI. 2. If you're using an external identity provider (IdP), you must first [sign in to the gcloud CLI with your federated identity](https://docs.cloud.google.com/iam/docs/workforce-log-in-gcloud). For more information, see [Authenticate for using REST](https://docs.cloud.google.com/docs/authentication/rest) in the Google Cloud authentication documentation. ## Create a sole-tenant node template Before creating a node group and provisioning sole-tenant VMs, you must create a sole-tenant node template. A node template is a regional resource that defines the properties (such as node type, affinity labels, local SSDs, accelerators, and CPU overcommit) for each node in a node group. To configure and create a node template using the Google Cloud console, the Google Cloud CLI, or the Compute Engine REST API, see [Create a sole-tenant node template](https://docs.cloud.google.com/compute/docs/nodes/sole-tenancy-node-templates). ## Create a sole-tenant node group After creating a node template, you must create a sole-tenant node group. A node group is a homogeneous cluster of sole-tenant nodes in a specific zone that inherits properties from its template. You can configure host maintenance policies, autoscaling, and cross-project sharing settings on the node group. To configure and create a node group using the Google Cloud console, the Google Cloud CLI, or the Compute Engine REST API, see [Create a sole-tenant node group](https://docs.cloud.google.com/compute/docs/nodes/sole-tenancy-node-groups). ## Provision a sole-tenant VM After creating a node group based on a previously created node template, you can provision individual VMs on a sole-tenant node group. To provision a VM on a specific node or node group that has affinity labels that match those you previously assigned to the node template, follow the [standard procedure for creating a VM instance](https://docs.cloud.google.com/compute/docs/instances/create-start-instance), and assign [affinity labels](https://docs.cloud.google.com/compute/docs/nodes/sole-tenant-nodes#node_affinity_and_anti-affinity) to the VM. Or, you can use the following procedure to provision a VM on a sole-tenant node from the node group details page. Based on the node group that you provision VMs on, Compute Engine assigns [affinity labels](https://docs.cloud.google.com/compute/docs/nodes/sole-tenant-nodes#node_affinity_and_anti-affinity). ### Console 1. In the Google Cloud console, go to the **Sole-tenant nodes** page. [Go to Sole-tenant nodes](https://console.cloud.google.com/compute/soleTenancy) 2. Click **Node groups**. 3. Click the **Name** of the node group to provision a VM instance on, and then optionally, to provision a VM on a specific sole-tenant node, click the name of the specific sole-tenant node to provision the VM. 4. Click **Create instance** to provision a VM instance on this node group, note the values automatically applied for the **Name** , **Region** , and **Zone**, and modify those values as necessary. 5. Select a **Machine configuration** by specifying the **Machine family** , **Series** , and **Machine type** . Choose the **Series** that corresponds to the [sole-tenant node type](https://docs.cloud.google.com/compute/docs/nodes/sole-tenant-nodes#node_types). 6. Modify the **Boot disk** , **Firewall**, and other settings as necessary. 7. Click **Sole Tenancy** , note the automatically assigned **Node affinity labels** , and use **Browse** to adjust as necessary. 8. Click **Management** , and for **On host maintenance**, choose one of the following: - **Migrate VM instance (recommended)**: VM migrated to another node in the node group during maintenance events. - **Terminate**: VM stopped during maintenance events. 9. Choose one of the following for the **Automatic restart**: - **On (recommended)**: Automatically restarts VMs if they are stopped for maintenance events. - **Off**: Does not automatically restart VMs after a maintenance event. 10. Click **Create** to finish creating your sole-tenant VM. ### gcloud Use the [`gcloud compute instances create` command](https://docs.cloud.google.com/sdk/gcloud/reference/compute/instances/create) command to provision a VM on a sole-tenant node group: ``` gcloud compute instances create VM_NAME \ [--zone=ZONE \] --image-family=IMAGE_FAMILY \ --image-project=IMAGE_PROJECT \ --node-group=GROUP_NAME \ --machine-type=MACHINE_TYPE \ [--maintenance-policy=MAINTENANCE_POLICY \] [--accelerator type=GPU_TYPE,count=GPU_COUNT \] [--local-ssd interface=SSD_INTERFACE \] [--restart-on-failure] ``` The `--restart-on-failure` flag indicates whether sole-tenant VMs restart after stopping. This flag is enabled by default. Use `--no-restart-on-failure` to disable. Replace the following: - `VM_NAME`: the name of the new sole-tenant VM. - `ZONE`: the zone to provision the sole-tenant VM in. - `IMAGE_FAMILY`: the [image family](https://docs.cloud.google.com/compute/docs/images/os-details) of the image to use to create the VM. - `IMAGE_PROJECT`: the [image project](https://docs.cloud.google.com/compute/docs/images/os-details) of the image family. - `GROUP_NAME`: the name of the node group to provision the VM on. - `MACHINE_TYPE`: the machine type of the sole-tenant VM. Use the [`gcloud compute machine-types list` command](https://docs.cloud.google.com/sdk/gcloud/reference/compute/machine-types/list) to get a list of available machine types for the project. - `MAINTENANCE_POLICY`: specifies restart behavior of sole-tenant VMs during maintenance events. Set to one of the following: - `MIGRATE`: VM migrated to another node in the node group during maintenance events. - `TERMINATE`: VM stopped during maintenance events. - `GPU_TYPE`: type of GPU. Set to one of the accelerator types specified when the node template was created. - `GPU_COUNT`: number of GPUs of the total specified by the node template to attach to this VM. Default value is `1`. - `SSD_INTERFACE`: type of local SSD interface. You can only set this for instances created from a node template with local SSD support. If you specify this while creating the instance, and the node template does not support local SSD, instance creation fails. Set to `nvme` if the boot disk image drivers are optimized for NVMe, otherwise set to `scsi`. Specify this flag and a corresponding value once for each local SSD partition. ### REST Use the [`instances.insert` method](https://docs.cloud.google.com/compute/docs/reference/rest/v1/instances/insert) to provision a VM on a sole-tenant node group: ``` POST https://compute.googleapis.com/compute/v1/projects/PROJECT_ID/zones/VM_ZONE/instances { "machineType": "zones/MACHINE_TYPE_ZONE/machineTypes/MACHINE_TYPE", "name": "VM_NAME", "scheduling": { "onHostMaintenance": MAINTENANCE_POLICY, "automaticRestart": RESTART_ON_FAILURE, "nodeAffinities": [ { "key": "compute.googleapis.com/node-group-name", "operator": "IN", "values": [ "GROUP_NAME" ] } ] }, "networkInterfaces": [ { "network": "global/networks/NETWORK", "subnetwork": "regions/REGION/subnetworks/SUBNETWORK" } ], "guestAccelerators": [ { "acceleratorType": GPU_TYPE, "acceleratorCount": GPU_COUNT } ], "disks": [ { "boot": true, "initializeParams": { "sourceImage": "projects/IMAGE_PROJECT/global/images/family/IMAGE_FAMILY" } }, { "type":"SCRATCH", "initializeParams":{ "diskType":"zones/LOCAL_SSD_ZONE/diskTypes/local-ssd" }, "autoDelete":true, "interface":"SSD_INTERFACE" } ] } ``` Replace the following: - `PROJECT_ID`: the project ID. - `VM_ZONE`: the zone to provision the sole-tenant VM in. - `MACHINE_TYPE_ZONE`: the zone of the machine type. - `MACHINE_TYPE`: the machine type of the sole-tenant VM. Use the [`machineTypes.list` method](https://docs.cloud.google.com/compute/docs/reference/rest/v1/machineTypes/list) to get a list of available machine types for the project. - `VM_NAME`: the name of the new sole-tenant VM. - `MAINTENANCE_POLICY`: specifies restart behavior of sole-tenant VMs during maintenance events. Set to one of the following: - `MIGRATE`: VM migrated to another node in the node group during maintenance events. - `TERMINATE`: VM stopped during maintenance events. - `RESTART_ON_FAILURE`: indicates whether sole-tenant VMs restart after stopping. Default is `true`. - `GROUP_NAME`: the name of the node group to provision the VM on. - `NETWORK`: the URL of the network resource for this VM. - `REGION`: the region containing the subnetwork for this VM. - `SUBNETWORK`: the URL of the subnetwork resource for this VM. - `GPU_TYPE`: the type of GPU. Set to one of the accelerator types specified when the node template was created. - `GPU_COUNT`: the number of GPUs of the total specified by the node template to attach to this VM. Default value is `1`. - `IMAGE_PROJECT`: [image project](https://docs.cloud.google.com/compute/docs/images/os-details) of the image family. - `IMAGE_FAMILY`: [image family](https://docs.cloud.google.com/compute/docs/images/os-details) of the image to use to create the VM. - `LOCAL_SSD_ZONE`: the local SSD's zone. - `SSD_INTERFACE`: the type of local SSD interface. Set to `NVME` if the boot disk image drivers are optimized for NVMe, otherwise set to `SCSI`. ## Provision a group of sole-tenant VMs [Managed instance groups (MIGs)](https://docs.cloud.google.com/compute/docs/instance-groups) let you provision a group of identical sole-tenant VMs. [Affinity labels](https://docs.cloud.google.com/compute/docs/nodes/sole-tenant-nodes#node_affinity_and_anti-affinity) let you specify the sole-tenant node or node group on which to provision the group of sole-tenant VMs. For regional MIGs, you must create node groups in each of the regional MIG's zones and you must specify node affinities for those node groups in the regional MIG's instance template. > [!WARNING] > **Warning:** For regional MIGs, you must set the MIG's [target distribution shape](https://docs.cloud.google.com/compute/docs/instance-groups/regional-mig-distribution-shape) to `EVEN`. If you use a different shape, the MIG's VMs might not be created in your sole-tenant nodes. ### gcloud 1. Use the [`gcloud compute instance-templates create` command](https://docs.cloud.google.com/sdk/gcloud/reference/compute/instance-templates/create) to create a managed instance group template for a group of VMs to create on a sole-tenant node group: ``` gcloud compute instance-templates create INSTANCE_TEMPLATE \ --machine-type=MACHINE_TYPE \ --image-project=IMAGE_PROJECT \ --image-family=IMAGE_FAMILY \ --node-group=GROUP_NAME \ [--accelerator type=GPU_TYPE,count=GPU_COUNT \] [--local-ssd interface=SSD_INTERFACE] ``` Replace the following: - `INSTANCE_TEMPLATE`: the name for the new instance template. - `MACHINE_TYPE`: the machine type of the sole-tenant VM. Use the [`gcloud compute machine-types list` command](https://docs.cloud.google.com/sdk/gcloud/reference/compute/machine-types/list) to get a list of available machine types for the project. - `IMAGE_PROJECT`: the [image project](https://docs.cloud.google.com/compute/docs/images/os-details) of the image family. - `IMAGE_FAMILY`: the [image family](https://docs.cloud.google.com/compute/docs/images/os-details) of the image to use to create the VM. - `GROUP_NAME`: the name of the node group to provision the VM on. Alternatively, if you want to use this instance template to create a regional MIG that exists in more than one zone, use the [`--node-affinity-file` flag](https://docs.cloud.google.com/sdk/gcloud/reference/compute/instance-templates/create#--node-affinity-file) to specify a list of values for the regional MIG's node groups. - `GPU_TYPE`: type of GPU. Set to one of the accelerator types specified when the node template was created. - `GPU_COUNT`: number of GPUs of the total specified by the node template to attach to this VM. Default value is `1`. - `SSD_INTERFACE`: type of local SSD interface. Set to `nvme` if the boot disk image drivers are optimized for NVMe, otherwise set to `scsi`. Specify this flag and a corresponding value once for each local SSD partition. 2. Use the [`gcloud compute instance-groups managed create` command](https://docs.cloud.google.com/sdk/gcloud/reference/compute/instance-groups/managed/create) command to create a managed instance group within your sole-tenant node group: ``` gcloud compute instance-groups managed create INSTANCE_GROUP_NAME \ --size=SIZE \ --template=INSTANCE_TEMPLATE \ --zone=ZONE ``` Replace the following: - `INSTANCE_GROUP_NAME`: the name for this instance group. - `SIZE`: the number of VMs to include in this instance group. Your node group must have enough resources to accommodate the instances in this managed instance group. Use the [managed instance group autoscaler](https://docs.cloud.google.com/compute/docs/autoscaler) to automatically manage the size of managed instance groups. - `INSTANCE_TEMPLATE`: the name of the instance template to use to create this MIG. The template must have one or more [node affinity labels](https://docs.cloud.google.com/compute/docs/nodes/sole-tenant-nodes#node_affinity_and_anti-affinity) pointing to the appropriate node groups. - `ZONE`: the zone to create the managed instance group in. For a regional MIG, replace the `--zone` flag with the `--region` flag and specify a region; also add the `--zones` flag to specify all of the zones where the node groups exist. ### REST 1. Use the [`instanceTemplates.insert` method](https://docs.cloud.google.com/compute/docs/reference/rest/v1/instanceTemplates/insert) to create a managed instance group template within your sole-tenant node group: ``` POST https://compute.googleapis.com/compute/v1/projects/PROJECT_ID/zones/TEMPLATE_ZONE/instance-templates { "name": "INSTANCE_TEMPLATE", "properties": { "machineType": "zones/MACHINE_TYPE_ZONE/machineTypes/MACHINE_TYPE", "scheduling": { "nodeAffinities": [ { "key": "compute.googleapis.com/node-group-name", "operator": "IN", "values": [ "GROUP_NAME" ] } ] }, "networkInterfaces": [ { "network": "global/networks/NETWORK", "subnetwork": "regions/REGION/subnetworks/SUBNETWORK" } ], "guestAccelerators": [ { "acceleratorType": GPU_TYPE, "acceleratorCount": GPU_COUNT } ], "disks": [ { "boot": true, "initializeParams": { "sourceImage": "projects/IMAGE_PROJECT/global/images/family/IMAGE_FAMILY" } }, { "type":"SCRATCH", "initializeParams":{ "diskType":"zones/LOCAL_SSD_ZONE/diskTypes/local-ssd" }, "autoDelete":true, "interface":"SSD_INTERFACE" } ] } } ``` Replace the following: - `PROJECT_ID`: the project ID. - `TEMPLATE_ZONE`: the zone to create the instance template in. - `INSTANCE_TEMPLATE`: the name of the new instance template. - `MACHINE_TYPE_ZONE`: the zone of the machine type. - `MACHINE_TYPE`: the machine type of the sole-tenant VM. Use the [`machineTypes.list` method](https://docs.cloud.google.com/compute/docs/reference/rest/v1/machineTypes/list) to get a list of available machine types for the project. - `GROUP_NAME`: name of the node group to provision the VM on. If you want to use this instance template to create a regional MIG that exists in more than one zone, specify a list of node groups that exist in the same zones as the regional MIG's zones. - `NETWORK`: the URL of the network resource for this instance template. - `REGION`: the region containing the subnetwork for this instance template. - `SUBNETWORK`: the URL of the subnetwork resource for this instance template. - `GPU_TYPE`: the type of GPU. Set to one of the accelerator types specified when the node template was created. - `GPU_COUNT`: the number of GPUs of the total specified by the node template to attach to this VM. Default value is `1`. - `IMAGE_PROJECT`: the [image project](https://docs.cloud.google.com/compute/docs/images/os-details) of the image family. - `IMAGE_FAMILY`: the [image family](https://docs.cloud.google.com/compute/docs/images/os-details) of the image to use to create the VM. - `LOCAL_SSD_ZONE`: the local SSD's zone. - `SSD_INTERFACE`: the type of local SSD interface. Set to `NVME` if the boot disk image drivers are optimized for NVMe, otherwise set to `SCSI`. 2. Use the [`instanceGroupManagers.insert` method](https://docs.cloud.google.com/compute/docs/reference/rest/v1/instanceGroupManagers/insert) to create a MIG within your sole-tenant node group based on the previously created instance template. Or, if you want to create a regional MIG, use the [`regionInstanceGroupManagers.insert` method](https://docs.cloud.google.com/compute/docs/reference/rest/v1/regionInstanceGroupManagers/insert) and specify the region and zones of all of the node groups as specified in the instance template. For example, to create a zonal MIG, use the following request: ``` POST https://compute.googleapis.com/compute/v1/projects/PROJECT_ID/zones/ZONE/instanceGroupManagers { "baseInstanceName": "NAME_PREFIX", "name": "INSTANCE_GROUP_NAME", "targetSize": SIZE, "instanceTemplate": "global/instanceTemplates/INSTANCE_TEMPLATE" } ``` Replace the following: - `PROJECT_ID`: the project ID. - `ZONE`: the zone to create the managed instance group in. - `NAME_PREFIX`: the prefix name for each of the instances in the managed instance group. - `INSTANCE_GROUP_NAME`: the name for the instance group. - `SIZE`: the number of VMs to include in this instance group. Your node group must have enough resources to accommodate the instances in this managed instance group. Use the [managed instance group autoscaler](https://docs.cloud.google.com/compute/docs/autoscaler) to automatically manage the size of managed instance groups. - `INSTANCE_TEMPLATE`: the URL of the instance template to use to create this group. The template must have a [node affinity label](https://docs.cloud.google.com/compute/docs/nodes/sole-tenant-nodes#node_affinity_and_anti-affinity) pointing to the appropriate node group. ## Configure node affinity labels Node affinity labels let you logically group node groups and schedule VMs on a specific set of node groups. You can also use node affinity labels to schedule VMs on node groups across different zones, and still keep the node groups in a logical group. The following procedure is an example of using affinity labels to associate VMs with a specific node group that is used for production workloads. This example shows how to schedule a single VM, but you could also use [managed instance groups](https://docs.cloud.google.com/compute/docs/instance-groups) to schedule a group of VMs. ### gcloud 1. Use the [`gcloud compute sole-tenancy node-templates create` command](https://docs.cloud.google.com/sdk/gcloud/reference/compute/sole-tenancy/node-templates/create) to create a node template with a set of affinity labels for a production workload: ``` gcloud compute sole-tenancy node-templates create prod-template \ --node-type=n1-node-96-624 \ --node-affinity-labels workload=frontend,environment=prod ``` 2. Use the [`gcloud compute sole-tenancy node-templates describe` command](https://docs.cloud.google.com/sdk/gcloud/reference/compute/sole-tenancy/node-templates/describe) to view the node affinity labels assigned to the node template. 3. Use the [`gcloud compute sole-tenancy node-groups create` command](https://docs.cloud.google.com/sdk/gcloud/reference/compute/sole-tenancy/node-groups/create) to create a node group that uses the production template: ``` gcloud compute sole-tenancy node-groups create prod-group \ --node-template=prod-template \ --target-size=1 ``` 4. For your production VMs, create a `node-affinity-prod.json` file to specify the affinity of your production VMs. For example, you might create a file that specifies that VMs run only on nodes with both the `workload=frontend` and `environment=prod` affinities. Create the node affinity file by using [Cloud Shell](https://docs.cloud.google.com/shell/docs/viewing-and-editing-files) or create it in a location of your choice. ``` [ { "key" : "workload", "operator" : "IN", "values" : ["frontend"] }, { "key" : "environment", "operator" : "IN", "values" : ["prod"] } ] ``` 5. Use the `node-affinity-prod.json` file with the [`gcloud compute instances create` command](https://docs.cloud.google.com/sdk/gcloud/reference/compute/instances/create) to schedule a VM on the node group with matching affinity labels. ``` gcloud compute instances create prod-vm \ --node-affinity-file node-affinity-prod.json \ --machine-type=n1-standard-2 ``` 6. Use the [`gcloud compute instances describe` command](https://docs.cloud.google.com/sdk/gcloud/reference/compute/instances/describe) and check the `scheduling` field to view the node affinities assigned to the VM. ## Configure node anti-affinity labels Node affinity labels can be configured as anti-affinity labels to prevent VMs from running on specific nodes. For example, you can use anti-affinity labels to prevent VMs that you are using for development purposes from being scheduled on the same nodes as your production VM. The following example shows how to use affinity labels to prevent VMs from running on specific node groups. This example shows how to schedule a single VM, but you could also use [managed instance groups](https://docs.cloud.google.com/compute/docs/instance-groups) to schedule a group of VMs. ### gcloud 1. For development VMs, specify the affinity of your development VMs by creating a `node-affinity-dev.json` with [Cloud Shell](https://docs.cloud.google.com/shell/docs/viewing-and-editing-files), or by creating it in a location of your choice. For example, create a file that configures VMs to run on any node group with the `workload=frontend` affinity as long as it is not `environment=prod`: ``` [ { "key" : "workload", "operator" : "IN", "values" : ["frontend"] }, { "key" : "environment", "operator" : "NOT_IN", "values" : ["prod"] } ] ``` 2. Use the `node-affinity-dev.json` file with the [`gcloud compute instances create` command](https://docs.cloud.google.com/sdk/gcloud/reference/compute/instances/create) to create the development VM: ``` gcloud compute instances create dev-vm \ --node-affinity-file=node-affinity-dev.json \ --machine-type=n1-standard-2 ``` 3. Use the [`gcloud compute instances describe` command](https://docs.cloud.google.com/sdk/gcloud/reference/compute/instances/describe) and check the `scheduling` field to view the node anti-affinities assigned to the VM. ## What's next - [Create sole-tenant node templates](https://docs.cloud.google.com/compute/docs/nodes/sole-tenancy-node-templates). - [Create sole-tenant node groups](https://docs.cloud.google.com/compute/docs/nodes/sole-tenancy-node-groups). - For information about sole-tenant node pricing, see [Sole-tenant nodes pricing](https://cloud.google.com/compute/sole-tenant-pricing). - For information about how to enable autoscaling on sole-tenant node groups, see [Node group autoscaler](https://docs.cloud.google.com/compute/docs/nodes/autoscaling-node-groups). - For more information about bringing existing licenses to Google Cloud, see [Bring existing licenses](https://docs.cloud.google.com/compute/docs/nodes/bringing-your-own-licenses). - For more information about sole-tenant nodes, see [Sole-tenant nodes](https://docs.cloud.google.com/compute/docs/nodes/sole-tenant-nodes).