Update or upgrade a cluster to an advanced cluster

This document shows how to update or upgrade a cluster to an advanced cluster from version 1.32.

Important: Non-advanced clusters will be upgraded to advanced cluster automatically in 1.33.

Prerequisites

• The gkectl version must be the same as the target cluster version.

  • If needed, see Download gkectl to get a supported version of gkectl.

• Always diagnose your cluster before updating or upgrading.

• Make sure your clusters use only recommended features.

• Verify your clusters don't use any features not supported by advanced clusters before starting the update or upgrade. If any unsupported features are active, disable them using update first.

  • Although highly-available (HA) user clusters are required on advanced clusters, you can still update or upgrade a non-HA user cluster to an HA advanced cluster following this guide.

• You must update or upgrade the admin cluster to an advanced cluster before you update or upgrade any user clusters.

• About cert-manager installation: cert-manager is automatically installed on advanced clusters. When you migrate from a non-advanced cluster to an advanced cluster, the integrated cert-manager automatically overrides any customer-installed versions of cert-manager. Make sure you don't have critical custom configurations or features applied to your existing cert-manager before you update or upgrade to advanced clusters. For more information, see Upgrade to cert-manager bundled with advanced clusters.

Update or upgrade the admin cluster to advanced cluster

Update

Important: Make sure your cluster version is at least 1.32.0.

1. Prepare for the advanced cluster:

   gkectl prepare \
       --bundle-path BUNDLE_PATH \
       --kubeconfig ADMIN_CLUSTER_KUBECONFIG \
       --advanced-cluster
   

   Replace the following:

   • BUNDLE_PATH: the path of the bundle file. This file is on your admin workstation in /var/lib/gke/bundles/. For example:

     /var/lib/gke/bundles/gke-onprem-vsphere-1.32.0-gke.1085-full.tgz
     

   • ADMIN_CLUSTER_KUBECONFIG: the path of the admin cluster kubeconfig file.

2. Set enableAdvancedCluster in your admin cluster configuration file to true.

3. Update the admin cluster to the advanced cluster:

   gkectl update admin \
       --kubeconfig ADMIN_CLUSTER_KUBECONFIG \
       --config ADMIN_CLUSTER_CONFIG
   

   Replace the following:

   • ADMIN_CLUSTER_KUBECONFIG: the path of the admin cluster kubeconfig file.
   • ADMIN_CLUSTER_CONFIG: the path of the admin cluster configuration file.

   The command will recreate all the nodes in the admin cluster.

Upgrade

Caution: If your cluster is using cos or cos_cgv2 as the control plane OS image type or node pool OS image type, make sure your cluster version is at least 1.32.0 before upgrading. Otherwise, make sure your cluster version is at least 1.31.100.

1. Upgrade your admin workstation if needed.

2. Prepare for the advanced cluster:

   gkectl prepare \
       --bundle-path BUNDLE_PATH \
       --kubeconfig ADMIN_CLUSTER_KUBECONFIG \
       --advanced-cluster
   

   Replace the following:

   • BUNDLE_PATH: the path of the bundle file. This file is on your admin workstation in /var/lib/gke/bundles/. For example:

     /var/lib/gke/bundles/gke-onprem-vsphere-1.32.0-gke.1085-full.tgz
     

   • ADMIN_CLUSTER_KUBECONFIG: the path of the admin cluster kubeconfig file.

3. Set enableAdvancedCluster in your admin cluster configuration file to true.

4. Make sure the bundlepath field in the admin cluster configuration file matches the path of the bundle to which you want to upgrade.

5. Upgrade the admin cluster to the advanced cluster:

   gkectl upgrade admin \
       --kubeconfig ADMIN_CLUSTER_KUBECONFIG \
       --config ADMIN_CLUSTER_CONFIG
   

   Replace the following:

   • ADMIN_CLUSTER_KUBECONFIG: the path of the admin cluster kubeconfig file.
   • ADMIN_CLUSTER_CONFIG: the path of the admin cluster configuration file.

   The command will rollout recreate all the nodes in the admin cluster. With the rollout recreate deployment strategy, the nodes are upgraded one at a time. During the node upgrade, the VM is deleted and then recreated with the new version.

After the admin cluster has been updated or upgraded to an advanced cluster:

• Your existing non-advanced user clusters continue to function normally.

• You can create only advanced user clusters that are managed by the advanced admin cluster.

We recommend that you upgrade all user clusters to advanced clusters soon afterward to maintain a consistent and fully-featured environment.

Update or upgrade the user cluster to advanced cluster

Important: Make sure the admin cluster has already been updated or upgraded to the advanced cluster.

Update

Important: Make sure your cluster version is at least 1.32.0.

1. If your user cluster control plane is not highly available (HA):

   Caution: Due to a known issue with the masterNode.replicas field, to update a version 1.32 or version 1.33 non-HA user cluster to an HA advanced cluster, your cluster must be at a version higher than 1.32.600 or higher than 1.33.100 respectively.

   1. Change masterNode.replicas from 1 to 3.

   2. Add the static IP addresses for the user cluster control-plane nodes to the network.controlPlaneIPBlock.ips section.

2. Prepare for the advanced cluster:

   gkectl prepare \
       --bundle-path BUNDLE_PATH \
       --kubeconfig ADMIN_CLUSTER_KUBECONFIG \
       --advanced-cluster
   

   Replace the following:

   • BUNDLE_PATH: the path of the bundle file. This file is on your admin workstation in /var/lib/gke/bundles/. For example:

     /var/lib/gke/bundles/gke-onprem-vsphere-1.32.0-gke.1085-full.tgz
     

   • ADMIN_CLUSTER_KUBECONFIG: the path of the admin cluster kubeconfig file.

3. Set enableAdvancedCluster in your user cluster configuration file to true.

4. Update the user cluster to the advanced cluster:

   gkectl update cluster \
       --kubeconfig ADMIN_CLUSTER_KUBECONFIG \
       --config USER_CLUSTER_CONFIG
   

   Replace the following:

   • ADMIN_CLUSTER_KUBECONFIG: the path of the admin cluster kubeconfig file.
   • USER_CLUSTER_CONFIG: the path of the user cluster configuration file.

   The command will rollout recreate all the nodes in the user cluster. With the rollout recreate deployment strategy, the nodes are updated one at a time. During the node update, the VM is deleted and then recreated with the new configuration.

Upgrade

Caution: If your cluster is using cos or cos_cgv2 as the control plane OS image type or node pool OS image type, make sure your cluster version is at least 1.32.0 before upgrading. Otherwise, make sure your cluster version is at least 1.31.100.

1. If your user cluster control plane is not highly available (HA):

   1. Change masterNode.replicas from 1 to 3.

   2. Add the static IP addresses for the user cluster control-plane nodes to the network.controlPlaneIPBlock.ips section.

2. Prepare for the advanced cluster:

   gkectl prepare \
       --bundle-path BUNDLE_PATH \
       --kubeconfig ADMIN_CLUSTER_KUBECONFIG \
       --advanced-cluster
   

   Replace the following:

   • BUNDLE_PATH: the path of the bundle file. This file is on your admin workstation in /var/lib/gke/bundles/. For example:

     /var/lib/gke/bundles/gke-onprem-vsphere-1.32.0-gke.1085-full.tgz
     

   • ADMIN_CLUSTER_KUBECONFIG: the path of the admin cluster kubeconfig file.

3. Set enableAdvancedCluster in your user cluster configuration file to true.

4. Make sure the gkeOnPremVersion field in the user cluster configuration file matches the target version of your upgrade.

5. Upgrade the user cluster to the advanced cluster:

   gkectl upgrade cluster \
       --kubeconfig ADMIN_CLUSTER_KUBECONFIG \
       --config USER_CLUSTER_CONFIG
   

   Replace the following:

   • ADMIN_CLUSTER_KUBECONFIG: the path of the admin cluster kubeconfig file.
   • USER_CLUSTER_CONFIG: the path of the user cluster configuration file.

   The command will recreate all the nodes in the user cluster.

cert-manager bundled with advanced clusters

If you use cert-manager without advanced clusters, check the following before you upgrade to advanced clusters:

• There are no custom configurations or feature flags in your cert-manager.
• Your cert-manager version is the same version or one version behind the bundled version.

You can do the following with the bundled cert-manager:

• Create and edit Issuer and ClusterIssuer resources.
• Create and edit certificate resources that reference issuers and standard issuer types.
• Create and edit standard Kubernetes resources that cert-manager uses.
• Create Kubernetes Secrets containing API keys and reference them in your issuers.
• Configure your issuers to use specific HTTP-01 or DNS-01 solvers.

You cannot customize the following in the bundled cert-manager:

• Set the deployment or helm values.
• Add command line arguments or flags to the cert-manager binary.
• Change resource limits or node selectors for cert-manager pods.
• Modify the installation logic of the cert-manager software.
• Turn on experimental features.
• Force cert-manager control plane pods to run on specific infrastructure nodes.

If you have specific constraints that prevent you from using the bundled cert-manager, contact Google Cloud support for assistance with advanced configuration options.

The following table lists the bundled cert-manager versions for each version of Google Distributed Cloud (software only) for VMware.

Google Distributed Cloud (software only) for VMware	Bundled cert-manager version
1.32	1.17
1.33	1.18
1.34	1.19

Key differences after moving to advanced clusters

Upgrading or updating your cluster to an advanced cluster introduces several key differences compared to standard clusters. Here are the major changes in advanced clusters:

• Resource location: Cluster and machine resources are no longer located in user clusters. These resources are located exclusively within the admin cluster.

• NodeExternalIP: The NodeExternalIP isn't set on the nodes.

• cert-manager installation: cert-manager is automatically installed on advanced clusters. If you installed cert-manager in your cluster, updating or upgrading to advanced cluster overwrites it with the advanced cluster version.

For a comprehensive overview of feature differences, refer to the feature comparison table.