You can convert a version 1.32 cluster to an advanced cluster by updating it in place at version 1.32 or by upgrading it to version 1.33. Converting to an advanced cluster prepares your environment for version 1.33 and later, where advanced clusters are the default or required architecture.
Key differences after moving to advanced clusters
Upgrading or updating your cluster to an advanced cluster introduces several key differences compared to standard clusters. Here are the major changes in advanced clusters:
Resource location: Cluster and machine resources are no longer located in user clusters. These resources are located exclusively within the admin cluster.
NodeExternalIP: TheNodeExternalIPisn't set on the nodes.cert-managerinstallation:cert-manageris automatically installed on advanced clusters. If you installedcert-managerin your cluster, updating or upgrading to advanced cluster overwrites it with the advanced cluster version.
Feature differences between advanced and non-advanced clusters
When you enable advanced clusters, some features or capabilities won't work the same way as they do in non-advanced clusters. Some of these changes apply only to the Preview version of advanced clusters in version 1.31. The differences are noted throughout the documentation.
The following table lists the features supported for Google Distributed Cloud software only for VMware, their availability status when advanced clusters is enabled, and any differences in behavior or use:
| Feature/capability | 1.31 Adv. clusters (Preview) | 1.32 Not enabled | 1.32 Advanced clusters (GA) | 1.33 Advanced clusters (GA) | Differences |
|---|---|---|---|---|---|
| Advanced clusters | Preview | GA | GA | GA | Not supported |
| Topology domains | Preview | Preview | Preview | Changes apply. For more information see Set up topology domains and related cluster configuration references. | |
| Regional fleet membership | GA | GA | GA | GA | No differences. |
| Version skew n+2 admin cluster - user cluster | GA | GA | GA | Not recommended: Because managing a lower-version preview user cluster isn't supported, don't use this feature. | |
| Version skew n+2 node pool - user cluster | GA | GA | GA | Not recommended: Because managing a lower-version preview user cluster isn't supported, don't use this feature. | |
| Max surge setting for node pool updates | Preview | Preview | Preview | Preview | No differences. |
| cgroup v2 for nodes | GA | GA | GA | GA | No differences. |
| DSR mode for Dataplane V2 | GA | GA | GA | GA | No differences. |
| Binary Authorization for Controlplane V2 user clusters | GA | GA | GA | GA | No differences. |
| User-managed admin workstation | GA | GA | GA | GA | No differences. |
| StatefulSet CSI migration tool | GA | GA | GA | GA | No differences. |
| Seesaw to MetalLB migration | GA | GA | GA | GA | No differences. |
| Disable bundled ingress | GA | GA | GA | GA | No differences. |
| Admin cluster prepared credentials | GA | GA | GA | No differences. | |
| Storage policy for a user cluster | GA | GA | GA | GA | No differences. |
| Storage policy for an admin cluster | GA | GA | GA | GA | No differences. |
| Node auto repair | GA | GA | GA | GA | Changes apply: If advanced cluster is enabled, the periodic health checks aren't run as part of auto repair. For information and other differences, see Automatic node repair and health checking. |
| High availability admin cluster | GA | GA | GA | GA | No differences. |
| VM host affinity | GA | GA | GA | Supported on advanced clusters in version 1.32.200 and 1.33 and later. When you use
topology domains,
configure host groups in
topologyDomains[i].compute.hostGroups
in the vSphere infrastructure configuration file instead of
nodePools[i].vsphere.hostgroups in the user cluster
configuration file. |
|
| Generate configuration files from an existing cluster | GA | GA | GA | GA | Changes apply. Clusters that use topology domains aren't supported. |
| Google Cloud Managed Service for Prometheus system metrics collection | GA | GA | GA | GA | No differences. |
| Upgrade and rollback of node pools | GA | GA | GA | No differences. | |
| Update private registry credentials | GA | GA | GA | GA | No differences. |
| Admin cluster backup and restore with gkectl | Preview | Preview | GA | Changes apply. Updated the gkectl command for improved
restoring of admin clusters and to support backing up and restoring user
clusters. For more information, see
Back up and restore advanced clusters with gkectl. |
|
| User cluster node pool autoscaling | GA | GA | GA | GA | No differences. |
| Cluster automatic node resizing | GA | GA | No differences. | ||
| Multiple vSphere cluster support | GA | GA | GA | GA | No differences. |
| Multiple vSphere data centers support | GA | GA | GA | GA | No differences. |
| OpenID Connect (OIDC) support for authentication to clusters | GA | GA | GA | GA | No differences. |
| CA certificate rotation | GA | GA | GA | No differences. | |
| Workload Identity Federation support | GA | GA | GA | GA | No differences. |
| GKE Identity Service with LDAP authentication support | GA | GA | GA | GA | No differences. |
| Always-on secrets encryption without hardware security module (HSM) | GA | GA | GA | No differences. | |
| Update vCenter CA certificates with gkectl | GA | GA | GA | GA | No differences. |
| Egress NAT gateway | GA | GA | GA | GA | No differences. |
| Admin cluster fleet registration | GA | GA | GA | GA | No differences. |
| Windows node pool support | GA | Not supported. | |||
containerd runtime for Windows node pool |
GA | Not supported. | |||
| Container-Optimized OS node pool support | GA | GA | GA | No differences. | |
| CoreDNS as the cluster DNS provider | GA | GA | GA | GA | No differences. |
| User cluster lifecycle in the Google Cloud console | GA | Partial support | GA | Changes apply. For more information, see Advanced cluster limitations. | |
| Admin cluster node creation with Container-Optimized OS | GA | GA | GA | No differences. | |
| Multi-NIC capability for Pods | GA | GA | GA | GA | No differences. |
| MetalLB load balancer option | GA | GA | GA | GA | Changes apply. For details, see Bundled load balancing with MetalLB. |
gkectl update admin support for enabling and disabling Cloud Logging and Cloud Monitoring |
GA | GA | GA | GA | No differences. |
| Windows Dataplane V2 support | GA | Not supported. | |||
| Summary API metrics | GA | GA | GA | GA | No differences. |
gkectl update credentials support for updating component access service account key |
Partial support | GA | Partial support | Partial support | Changes apply. For details about what is or isn't supported for updating credentials for advanced clusters, see Advanced cluster differences for rotating service account keys. |
| Prepared credentials for user cluster | GA | GA | GA | No differences. | |
| Dry run upgrade of user cluster | GA | GA | GA | GA | No differences. |
| Asynchronous upgrade of user cluster | GA | GA | GA | GA | No differences. |
| Dry run upgrade of admin cluster | GA | GA | No differences. | ||
| Asynchronous upgrade of admin cluster | GA | GA | GA | GA | No differences. |
| Sequential update of node pools | GA | GA | GA | No differences. | |
| Create volume snapshot with vSphere CSI driver | Preview | Preview | Preview | Preview | No differences. |
| Create user cluster with Controlplane V2 enabled | GA | GA | GA | GA | No differences. |
| Storage migration with SPBM | GA | GA | GA | GA | No differences. |
| Migrate a datastore to an SPBM | GA | GA | GA | GA | No differences. |
| Migrate a user cluster to Controlplane V2 | GA | GA | GA | GA | No differences. |
| Migrate to an HA admin cluster | GA | GA | GA | GA | No differences. |
| Migrate configuration settings for F5 BIG-IP | GA | GA | GA | GA | No differences. |
| Load balancer | Partial support | GA | GA | GA | Changes apply. For more information, refer to the relevant load balancer and cluster configuration reference documentation. |
| Preflight checks | Partial support | GA | GA | GA | No differences. |
| Hardware requirements | GA | GA | GA | GA | Changes apply. For details, see CPU, RAM, and storage requirements. |
| Snapshot | GA | GA | GA | GA | No differences. |
| Private registry | GA | GA | GA | GA | Changes apply. For details about using a private registry on an advanced cluster, see Limitations with advanced clusters and the full bundle. |
| IPAM, DNS, NTP and multi-NIC | GA | GA | GA | GA | No differences. |
| Cluster creation/upgrade/deletion | GA | GA | GA | GA | Changes apply. For more information, refer to the relevant cluster operation documentation and Advanced cluster limitations. |
| Proxy | GA | GA | GA | GA | No differences. |
| NodePool update policy | GA | GA | GA | No differences. | |
| Diagnose | GA | GA | GA | Changes apply. vCenter-related checks aren't available for advanced
clusters. Since there's no longer an onprem-user-cluster-controller
controller, the Diagnose
cluster issues guide is obsolete for advanced clusters. For version
1.32 advanced clusters, the gkectl diagnose cluster command
runs health checks
for the specified cluster. |
|
| vSphere tags and VM tracking | GA | GA | No differences. | ||
| vSphere host group | GA | GA | No differences. | ||
| Logging and monitoring | GA | GA | GA | GA | No differences. |
| Anti-affinity groups | GA | GA | GA | No differences. | |
| Non-HA user cluster control plane | GA | Not supported. | |||
gkectl check-config |
GA | GA | GA | Changes apply. Support for the command on advanced user clusters was added in the following versions: 1.32.700+, 1.33.300+, and 1.34.0 and higher. |
Prerequisites
The
gkectlversion must be the same as the target cluster version.- If needed, see
Download
gkectlto get a supported version ofgkectl.
- If needed, see
Download
Always diagnose your cluster before updating or upgrading to an advanced cluster:
- Run
gkectl diagnose clusteron your admin and user clusters to ensure that they are healthy. - If the command reports any errors or warnings, resolve them before you start the update or upgrade to an advanced cluster.
- Run
Make sure your clusters use only recommended features.
Verify your clusters don't use any features not supported by advanced clusters before starting the update or upgrade. If any unsupported features are active, disable them using update first.
- Although highly-available (HA) user clusters are required on advanced clusters, you can still update or upgrade a non-HA user cluster to an HA advanced cluster following this guide.
You must update or upgrade the admin cluster to an advanced cluster before you update or upgrade any user clusters.
About
cert-managerinstallation:cert-manageris automatically installed on advanced clusters. When you migrate from a non-advanced cluster to an advanced cluster, the integratedcert-managerautomatically overrides any customer-installed versions ofcert-manager. Make sure you don't have critical custom configurations or features applied to your existingcert-managerbefore you update or upgrade to advanced clusters.
Update or upgrade the admin cluster to advanced cluster
Update
Diagnose your cluster to make sure it is healthy:
gkectl diagnose cluster \ --kubeconfig ADMIN_CLUSTER_KUBECONFIG- If the command reports any errors or warnings, refer to Diagnose cluster issues and resolve them before proceeding.
Prepare for the advanced cluster:
gkectl prepare \ --bundle-path BUNDLE_PATH \ --kubeconfig ADMIN_CLUSTER_KUBECONFIG \ --advanced-clusterReplace the following:
BUNDLE_PATH: the path of the bundle file. This file is on your admin workstation in/var/lib/gke/bundles/. For example:/var/lib/gke/bundles/gke-onprem-vsphere-1.32.0-gke.1085-full.tgzADMIN_CLUSTER_KUBECONFIG: the path of the admin cluster kubeconfig file.
Set
enableAdvancedClusterin your admin cluster configuration file totrue.Update the admin cluster to the advanced cluster:
gkectl update admin \ --kubeconfig ADMIN_CLUSTER_KUBECONFIG \ --config ADMIN_CLUSTER_CONFIGReplace the following:
ADMIN_CLUSTER_KUBECONFIG: the path of the admin cluster kubeconfig file.ADMIN_CLUSTER_CONFIG: the path of the admin cluster configuration file.
The command will recreate all the nodes in the admin cluster.
Upgrade
Upgrade your admin workstation if needed.
Diagnose your cluster to make sure it is healthy:
gkectl diagnose cluster \ --kubeconfig ADMIN_CLUSTER_KUBECONFIG- If the command reports any errors or warnings, refer to Diagnose cluster issues and resolve them before proceeding.
Prepare for the advanced cluster:
gkectl prepare \ --bundle-path BUNDLE_PATH \ --kubeconfig ADMIN_CLUSTER_KUBECONFIG \ --advanced-clusterReplace the following:
BUNDLE_PATH: the path of the bundle file. This file is on your admin workstation in/var/lib/gke/bundles/. For example:/var/lib/gke/bundles/gke-onprem-vsphere-1.32.0-gke.1085-full.tgzADMIN_CLUSTER_KUBECONFIG: the path of the admin cluster kubeconfig file.
Set
enableAdvancedClusterin your admin cluster configuration file totrue.Make sure the
bundlepathfield in the admin cluster configuration file matches the path of the bundle to which you want to upgrade.Upgrade the admin cluster to the advanced cluster:
gkectl upgrade admin \ --kubeconfig ADMIN_CLUSTER_KUBECONFIG \ --config ADMIN_CLUSTER_CONFIGReplace the following:
ADMIN_CLUSTER_KUBECONFIG: the path of the admin cluster kubeconfig file.ADMIN_CLUSTER_CONFIG: the path of the admin cluster configuration file.
The command will rollout recreate all the nodes in the admin cluster. With the rollout recreate deployment strategy, the nodes are upgraded one at a time. During the node upgrade, the VM is deleted and then recreated with the new version.
After the admin cluster has been updated or upgraded to an advanced cluster:
Your existing non-advanced user clusters continue to function normally.
You can create only advanced user clusters that are managed by the advanced admin cluster.
Upgrade all user clusters to advanced clusters soon afterward to maintain a consistent and fully-featured environment.
Update or upgrade the user cluster to advanced cluster
Update
Diagnose your cluster to make sure it is healthy:
gkectl diagnose cluster \ --kubeconfig ADMIN_CLUSTER_KUBECONFIG \ --cluster-name USER_CLUSTER_NAME- If the command reports any errors or warnings, refer to Diagnose cluster issues and resolve them before proceeding.
If your user cluster control plane is not highly available (HA):
Change
masterNode.replicasfrom 1 to 3.Add the static IP addresses for the user cluster control-plane nodes to the
network.controlPlaneIPBlock.ipssection.
Prepare for the advanced cluster:
gkectl prepare \ --bundle-path BUNDLE_PATH \ --kubeconfig ADMIN_CLUSTER_KUBECONFIG \ --advanced-clusterReplace the following:
BUNDLE_PATH: the path of the bundle file. This file is on your admin workstation in/var/lib/gke/bundles/. For example:/var/lib/gke/bundles/gke-onprem-vsphere-1.32.0-gke.1085-full.tgzADMIN_CLUSTER_KUBECONFIG: the path of the admin cluster kubeconfig file.
Set
enableAdvancedClusterin your user cluster configuration file totrue.Update the user cluster to the advanced cluster:
gkectl update cluster \ --kubeconfig ADMIN_CLUSTER_KUBECONFIG \ --config USER_CLUSTER_CONFIGReplace the following:
ADMIN_CLUSTER_KUBECONFIG: the path of the admin cluster kubeconfig file.USER_CLUSTER_CONFIG: the path of the user cluster configuration file.
The command will rollout recreate all the nodes in the user cluster. With the rollout recreate deployment strategy, the nodes are updated one at a time. During the node update, the VM is deleted and then recreated with the new configuration.
Upgrade
Diagnose your cluster to make sure it is healthy:
gkectl diagnose cluster \ --kubeconfig ADMIN_CLUSTER_KUBECONFIG \ --cluster-name USER_CLUSTER_NAME- If the command reports any errors or warnings, refer to Diagnose cluster issues and resolve them before proceeding.
If your user cluster control plane is not highly available (HA):
Change
masterNode.replicasfrom 1 to 3.Add the static IP addresses for the user cluster control-plane nodes to the
network.controlPlaneIPBlock.ipssection.
Prepare for the advanced cluster:
gkectl prepare \ --bundle-path BUNDLE_PATH \ --kubeconfig ADMIN_CLUSTER_KUBECONFIG \ --advanced-clusterReplace the following:
BUNDLE_PATH: the path of the bundle file. This file is on your admin workstation in/var/lib/gke/bundles/. For example:/var/lib/gke/bundles/gke-onprem-vsphere-1.32.0-gke.1085-full.tgzADMIN_CLUSTER_KUBECONFIG: the path of the admin cluster kubeconfig file.
Set
enableAdvancedClusterin your user cluster configuration file totrue.Make sure the
gkeOnPremVersionfield in the user cluster configuration file matches the target version of your upgrade.Upgrade the user cluster to the advanced cluster:
gkectl upgrade cluster \ --kubeconfig ADMIN_CLUSTER_KUBECONFIG \ --config USER_CLUSTER_CONFIGReplace the following:
ADMIN_CLUSTER_KUBECONFIG: the path of the admin cluster kubeconfig file.USER_CLUSTER_CONFIG: the path of the user cluster configuration file.
The command will recreate all the nodes in the user cluster.
cert-manager bundled with advanced clusters
cert-manager is a mandatory, system-managed component in advanced clusters.
If you use cert-manager without advanced clusters, check the following before
you upgrade to advanced clusters:
- There are no custom configurations or feature flags in your
cert-manager. - Your
cert-managerversion is the same version or one version behind the bundled version.
Apigee Hybrid compatibility
The bundled cert-manager is fully compatible with Apigee Hybrid workloads. When you upgrade to an advanced cluster, the installer might warn you that the existing cert-manager installation will be replaced. This is expected behavior that the system handles automatically.
To ensure a smooth transition for your Apigee workloads:
- Do not uninstall your existing
cert-managermanually before the upgrade. - Revert customizations: If you manage
cert-managerusing GitOps or have applied custom configurations to thecert-managerdeployment, you must revert these customizations or suspend sync before the upgrade to avoid configuration drift and potential upgrade pauses.
Capabilities and limitations
You can do the following with the bundled cert-manager:
- Create and edit
IssuerandClusterIssuerresources. - Create and edit certificate resources that reference issuers and standard issuer types.
- Create and edit standard Kubernetes resources that
cert-manageruses. - Create Kubernetes Secrets containing API keys and reference them in your issuers.
- Configure your issuers to use specific HTTP-01 or DNS-01 solvers.
You cannot customize the following in the bundled cert-manager:
- Set the
deploymentorhelmvalues. - Add command line arguments or flags to the
cert-managerbinary. - Change resource limits or node selectors for
cert-managerpods. - Modify the installation logic of the
cert-managersoftware. - Turn on experimental features.
- Force
cert-managercontrol plane pods to run on specific infrastructure nodes.
If you have specific constraints that prevent you from using the bundled
cert-manager, contact Google Cloud support
for assistance with advanced configuration options.
Bundled versions
The following table lists the bundled cert-manager versions for each version
of Google Distributed Cloud (software only) for VMware.
| Google Distributed Cloud (software only) for VMware | Bundled cert-manager version |
|---|---|
| 1.32 | 1.17 |
| 1.33 | 1.18 |
| 1.34 | 1.19 |