Configure VM host Group affinity

This document gives an example of how to configure a Google Distributed Cloud cluster to use VM host affinity.

VM host group affinity is one of the mechanisms that Google Distributed Cloud provides to ensure high availability. With VM host Group affinity, you create groups of physical ESXi hosts. Then you configure your cluster to associate VM groups with host groups.

For example, you could configure all VMs in one node pool to run on a particular host group. And you could configure all VMs in a second node pool to run on a different host group. You could then treat each node pool as a failure domain. To differentiate the failure domains, you could add labels to the VMs in the various node pools.

On advanced clusters, VM host Group affinity is supported in version 1.32.200 and version 1.33 and later. If your cluster uses topology domains, configure host groups in the topologyDomains[i].compute.hostGroups field of the vSphere infrastructure configuration file instead of the nodePools[i].vsphere.hostgroups field in the user cluster configuration file.

Before you begin

For this exercise, you need to have at least six ESXi hosts in your vSphere environment.

Create host groups

Create two or more host DRS groups in your vSphere environment. For this exercise, two host groups with three hosts each would be appropriate. For instructions, see Create a Host DRS Group.

Create a user cluster

This section gives an example of how to create a user cluster that uses VM host Group affinity. The cluster in this example uses Controlplane V2. The cluster has a high-availability control plane, so there are three control-plane nodes. In addition to the control-plane nodes, there are six worker nodes: three in one node pool and three in a second node pool. All nodes use static IP addresses.

Start by following the instructions in Create a user cluster (or Create a user cluster for use with topology domains if your cluster uses topology domains).

As you fill in your configuration files:

Without topology domains: In your user cluster configuration file, specify two node pools for worker nodes. For each node pool, set replicas to 3, and provide the name of an existing host group in nodePools[i].vsphere.hostgroups.
With topology domains: In your vSphere infrastructure configuration file, specify two topology domains and provide the name of an existing host group in topologyDomains[i].compute.hostGroups for each domain. In your user cluster configuration file, specify two node pools for worker nodes, set replicas to 3, and set nodePools[i].topologyDomains to associate each node pool with a topology domain.

Example configuration file

The following examples show how to configure VM host Group affinity without topology domains and with topology domains.

Without topology domains

Here is an example of an IP block file and a portion of a user cluster configuration file for a cluster that doesn't use topology domains.

user-ipblock.yaml

blocks:
  - netmask: 255.255.255.0
    gateway: 172.16.21.1
    ips:
    - ip: 172.16.21.2
    - ip: 172.16.21.3
    - ip: 172.16.21.4
    - ip: 172.16.21.5
    - ip: 172.16.21.6
    - ip: 172.16.21.7
    - ip: 172.16.21.8

user-cluster-yaml

apiVersion: v1
kind: UserCluster
...
network:
  hostConfig:
    dnsServers:
    - "203.0.113.2"
    - "198.51.100.2"
    ntpServers:
    - "216.239.35.4"
  ipMode:
    type: "static"
    ipBlockFilePath: "user-ipblock.yaml"
  controlPlaneIPBlock:
    netmask: "255.255.255.0"
    gateway: "172.16.21.1"
    ips:
    - ip: "172.16.21.9"
      hostname: "cp-vm-1"
    - ip: "172.16.21.10"
      hostname: "cp-vm-2"
    - ip: "172.16.21.11"
      hostname: "cp-vm-3"
loadBalancer:
  vips:
    controlPlaneVIP: "172.16.21.40"
    ingressVIP: "172.16.21.30"
  kind: MetalLB
  metalLB:
    addressPools:
    - name: "address-pool-1"
      addresses:
    - "172.16.21.30-172.16.21.39"
...
enableAdvancedCluster: true
enableControlplaneV2: true
masterNode:
  cpus: 4
  memoryMB: 8192
  replicas: 3
nodePools:
- name: "worker-pool-1"
  enableLoadBalancer: true
  replicas: 3
  vsphere:
    hostgroups:
    - "hostgroup-1"
  labels:
    failuredomain: "failuredomain-1"
- name: "worker-pool-2"
  replicas: 3
  vsphere:
    hostgroups:
    - "hostgroup-2"
  labels:
    failuredomain: "failuredomain-2"
...

These are the important points to understand in the preceding example:

The static IP addresses for the worker nodes are specified in an IP block file. The IP block file has seven addresses even though there are only six worker nodes. The extra IP address is needed during cluster upgrade, update, and auto repair.
The static IP addresses for the three control-plane nodes are specified in the network.controlPlaneIPBlock section of the user cluster configuration file. There is no need for an extra IP address in this block.
The masterNode.replicas field is set to 3, so there will be three control-plane nodes.
A cluster controller will create a VM DRS group that has the three nodes in the worker-pool-1 node pool. A controller will also create a VM host affinity rule that ensures nodes in worker-pool-1 will run on hosts that are in hostgroup-1. The nodes in worker-pool-1 have the label failuredomain: "failuredomain-1"
A cluster controller will create a VM DRS group that has the three nodes in the worker-pool-2 node pool. A controller will also create a VM host affinity rule that ensures nodes in worker-pool-2 will run on hosts that are in hostgroup-2. The nodes in worker-pool-2 have the label failuredomain: "failuredomain-2"

Continue creating your user cluster as described in Create a user cluster.

With topology domains

Here is an example of a portion of a vSphere infrastructure configuration file and a user cluster configuration file for an advanced cluster that uses topology domains.

vsphere-infra-config.yaml

Each topology domain in the vSphere infrastructure configuration file specifies a host group in topologyDomains[i].compute.hostGroups and a failure-domain label in topologyDomains[i].topologyLabels.

apiVersion: vmware.cluster.gke.io/v1alpha1
kind: VSphereInfraConfig
metadata:
  name: default
...
topologyDomains:
- name: "topology-domain-1"
  topologyLabels:
    "topology.kubernetes.io/zone": "zone-1"
  compute:
    vcenter: "vc01.example"
    datacenter: "dc-1"
    cluster: "cls-1"
    resourcePool: "rp-1"
    hostGroups:
    - "hostgroup-1"
  storage:
    datastore: "vsanDatastore"
  network:
    vsphereNetwork: "vm-network-1"
    gateway: "172.16.21.1"
    dnsServers:
    - "203.0.113.2"
    - "198.51.100.2"
    ntpServers:
    - "216.239.35.4"
- name: "topology-domain-2"
  topologyLabels:
    "topology.kubernetes.io/zone": "zone-2"
  compute:
    vcenter: "vc01.example"
    datacenter: "dc-1"
    cluster: "cls-1"
    resourcePool: "rp-1"
    hostGroups:
    - "hostgroup-2"
  storage:
    datastore: "vsanDatastore"
  network:
    vsphereNetwork: "vm-network-1"
    gateway: "172.16.21.1"
    dnsServers:
    - "203.0.113.2"
    - "198.51.100.2"
    ntpServers:
    - "216.239.35.4"

user-cluster.yaml

In the following example of a user cluster configuration file, the nodePools[i].vsphere section is removed, and each node pool references its topology domain in nodePools[i].topologyDomains.

A cluster controller creates a VM DRS group and a VM host affinity rule for worker-pool-1 (in topology-domain-1) so that its nodes run on hosts in hostgroup-1 and receive the label topology.kubernetes.io/zone: "zone-1". Similarly, a cluster controller creates a VM DRS group and a VM host affinity rule for worker-pool-2 (in topology-domain-2) so that its nodes run on hosts in hostgroup-2 and receive the label topology.kubernetes.io/zone: "zone-2".

apiVersion: v1
kind: UserCluster
...
enableAdvancedCluster: true
enableControlplaneV2: true
masterNode:
  cpus: 4
  memoryMB: 8192
  replicas: 3
  topologyDomains:
  - "topology-domain-1"
nodePools:
- name: "worker-pool-1"
  replicas: 3
  osImageType: "ubuntu_cgv2"
  topologyDomains:
  - "topology-domain-1"
- name: "worker-pool-2"
  replicas: 3
  osImageType: "ubuntu_cgv2"
  topologyDomains:
  - "topology-domain-2"
...

Continue to create a user cluster for use with topology domains.

Configure VM host Group affinity Stay organized with collections Save and categorize content based on your preferences.