Distributed Cloud release notes

This page documents production updates to Google Distributed Cloud, which is a component of Google Distributed Cloud. You can periodically check this page for announcements about new or updated features, bug fixes, known issues, and deprecated features.

See also:

You can see the latest product updates for all of Google Cloud on the Google Cloud page, browse and filter all release notes in the Google Cloud console, or programmatically access release notes in BigQuery.

To get the latest product updates delivered to you, add the URL of this page to your feed reader, or add the feed URL directly.

July 22, 2025

Google Distributed Cloud (software only) for VMware 1.31.700-gke.72 is now available for download. To upgrade, see Upgrade a cluster. Distributed Cloud 1.31.700-gke.72 runs on Kubernetes v1.31.10-gke.200.

If you are using a third-party storage vendor, check the GDC Ready storage partners document to make sure the storage vendor has already passed the qualification for this release.

After a release, it takes approximately 7 to 14 days for the version to become available for use with GKE On-Prem API clients: the Google Cloud console, the gcloud CLI, and Terraform.

May 22, 2025

The following issues were fixed in 1.31.500-gke.68:

February 13, 2025

The following issues are fixed in 1.31.200-gke.58:

  • Fixed an issue that caused Runtime: out of memory errors after running gkeadm to create or upgrade clusters.

  • Fixed the issue that caused Node CIDR based Network Policies to not get enforced.

February 05, 2025

The 1.31.100-gke.136 release includes many vulnerability fixes. For a list of all vulnerabilities fixed in this release, see Vulnerability fixes.

October 02, 2024

Google Distributed Cloud (software only) for VMware 1.30.100-gke.96 is now available for download. To upgrade, see Upgrade a cluster or a node pool. Google Distributed Cloud 1.30.100-gke.96 runs on Kubernetes v1.30.4-gke.1800.

If you are using a third-party storage vendor, check the GDCV Ready storage partners document to make sure the storage vendor has already passed the qualification for this release.

After a release, it takes approximately 7 to 14 days for the version to become available for use with GKE On-Prem API clients: the Google Cloud console, the gcloud CLI, and Terraform.

August 29, 2024

Version changes in 1.30.0-gke.1930:

  • Existing Seesaw load balancers now require TLS 1.2.
  • COS was upgraded to m109
  • Updated Dataplane V2 to use Cilium 1.13

Other changes in1.30.0-gke.1930:

  • Enhanced the upgrade process to include an automatic pre-upgrade check. Before you upgrade your admin or user cluster, the system runs this check to detect known issues. The check also provides guidance to ensure a smooth upgrade experience.
  • Ingress node ports are optional for ControlplaneV2 clusters.
  • Admin clusters created in 1.30 will use Dataplane V2, Google's Container Network Interface (CNI) implementation, which is based on Cilium.
  • Admin clusters upgraded to 1.30 from 1.29 will use Dataplane V2.
  • Removed mTLS on system metrics scrape endpoints, which makes it easier to integrate with 3rd party monitoring systems.

  • Stopped bundling cert-manager and removed the monitoring-operator because system components no longer depend on them. Cert-manager from existing 1.29 clusters will continue running, but stop being managed by Google after upgrading to 1.30. If you don't use cert-manager, you can delete cert-manager after upgrade. New clusters in 1.30 and higher won't come with cert-manager. If you rely on the bundled cert-manager for their own use case, you should install their own in new clusters.

  • We recommend that you don't use the preview feature usage metering when you create new clusters. However, existing clusters using this feature will continue to function. As the alternative, we recommend that you use the predefined dashboard, Anthos Cluster Utilization Metering, to understand resource usage at different levels.

August 07, 2024

Fixed
The following vulnerabilities are fixed in 1.28.800-gke.109:

High-severity container vulnerabilities:

Ubuntu vulnerabilities:

August 01, 2024

Existing Seesaw load balancers now require TLS 1.2.

July 09, 2024

Google Distributed Cloud for VMware 1.29.200-gke.245 is now available for download. To upgrade, see Upgrade a cluster or a node pool. Google Distributed Cloud 1.29.200-gke.245 runs on Kubernetes v1.29.5-gke.800.

If you are using a third-party storage vendor, check the GDCV Ready storage partners document to make sure the storage vendor has already passed the qualification for this release.

June 10, 2024

A vulnerability (CVE-2022-23222) was discovered in the Linux kernel that can lead to a privilege escalation on Container-Optimized OS nodes:

For more information, see the GCP-2024-033 security bulletin.

April 09, 2024

GKE on VMware 1.16.7-gke.46 is now available. To upgrade, see Upgrading GKE on VMware. GKE on VMware 1.16.7-gke.46 runs on Kubernetes v1.27.10-gke.500.

If you are using a third-party storage vendor, check the GDCV Ready storage partners document to make sure the storage vendor has already passed the qualification for this release of GKE on VMware.

February 16, 2024

The following vulnerability was discovered in the Linux kernel that can lead to a privilege escalation on Container-Optimized OS and Ubuntu nodes:

  • CVE-2023-6932

For more information, see the GCP-2024-011 security bulletin.

February 01, 2024

The following issues are fixed in 1.15.8-gke.41:

  • Fixed Seesaw crashing on duplicated service IP.
  • Fixed a warning in the storage preflight check.

The following vulnerabilities are fixed in 1.15.8-gke.41:

January 25, 2024

GKE for VMware 1.16.5-gke.28 is now available. To upgrade, see Upgrading GKE on VMware. GDCV for VMware 1.16.5-gke.28 runs on Kubernetes 1.27.6-gke.2500.

If you use a third-party storage vendor, check the GDCV Ready storage partners document to make sure the storage vendor has already passed the qualification for this release of GKE on VMware.

The following issues are fixed in 1.16.5-gke.28:

  • Fixed an issue where duplicate Service IP addresses caused the Seesaw load balancer to fail.

The following vulnerabilities are fixed in 1.16.5-gke.28:

December 12, 2023

The following issues are fixed in 1.15.7-gke.40:

  • Fixed the etcd hostname mismatch issue when using a FQDN.
  • Fixed an issue where the cluster-health-controller might leak vSphere sessions.
    Fixed the known issue where the CSI workload preflight check fails due to Pod startup failure.

The following vulnerabilities are fixed in 1.15.7-gke.40:

November 13, 2023

The following vulnerabilities were discovered in the Linux kernel that can lead to a privilege escalation on Container-Optimized OS and Ubuntu nodes.

  • CVE-2023-4147

For more information, see the GCP-2023-042 security bulletin.

September 01, 2023

The following issues are fixed in 1.15.4-gke.37:

  • Fixed a known issue where incorrect log rotation configuration for fluent-bit caused low disk space on the Seesaw VM.

  • Fixed a known issue that GARP reply sent by Seesaw doesn't set target IP.

  • Fixed an issue where /etc/vsphere/certificate/ca.crt wasn't updated after vsphere CA rotation on the Controlplane v2 user cluster control plane machines.

  • Fixed a known issue where the admin SSH public key has error after admin cluster upgrade or update.

August 23, 2023

The following issues are fixed in 1.16.0-gke.669:

  • Fixed the known issue that caused intermittent ssh errors on non-HA admin master after update or upgrade.
  • Fixed the known issue where upgrading enrolled admin cluster could fail due to membership update failure.

  • Fixed the issue where the CPv1 stackdriver operator had --is-kubeception-less=true specified by mistake.

  • Fixed the issue where clusters used the non-high-availability (HA) Connect Agent after an upgrade to 1.15.

  • Fixed the known issue of Cloud Audit Logging failure due to permission denied.

  • Fixed a known issue where the update operation cannot be fulfilled due to KSA signing key version unmatched.

  • Fixed a known issue where $ in the private registry username caused admin control plane machine startup failure.

  • Fixed a known issue where gkectl diagnose snapshot failed to limit the time window for journalctlcommands running on the cluster nodes when you take a cluster snapshot with the --log-since flag.

  • Fixed a known issue where node ID verification failed to handle hostnames with dots.

  • Fixed continuous increase of logging agent memory.

  • Fixed the issue that caused gcloud to fail to update the platform when the required-platform-version is already the current platform version.

  • Fixed an issue where cluster-api-controllers in a high-availability admin cluster had no Pod anti-affinity. This could allow the three clusterapi-controllers Pods not to be scheduled on different control-plane nodes.

  • Fixed the wrong admin cluster resource link annotation key that can cause the cluster to be enrolled again by mistake.

  • Fixed a known issue where node pool creation failed because of duplicated VM-Host affinity rules.

  • The preflight check for StorageClass parameter validations now throws a warning instead of a failure on ignored parameters after CSI Migration. StorageClass parameter diskformat=thin is now allowed and does not generate a warning.

  • Fixed a false error message for gkectl prepare when using a high-availability admin cluster.

  • Fixed an issue during the migration from the Seesaw load balancer to MetalLB that caused 'DeprecatedKubeception' always shows up in the diff.

  • Fixed a known issue where some cluster nodes couldn't access the HA control plane when the underlying network performs ARP suppression.

  • Removed unused Pod disruption budgets (such as kube-apiserver-pdb, kube-controller-manager-pdb, and kube-etcd-pdb) for Controlplane V2 user clusters

August 17, 2023

The following issues are fixed in 1.14.7-gke.42:

  • Fixed a known issue that admin SSH public key has error after admin cluster upgrade or update.
  • Fixed a known issue that GARP reply sent by Seesaw doesn't set target IP.
  • Fixed an issue that /etc/vsphere/certificate/ca.crt was not updated after vsphere CA rotation on the Controlplane v2 user cluster control plane machines.
  • Fixed an issue that the CPv1 stackdriver operator had --is-kubeception-less=true specified by mistake.

August 10, 2023

The following issues are fixed in 1.15.3-gke.47:

  • Fixed a known issue. that caused upgrading an admin cluster enrolled in the Anthos On-Prem API to fail.
  • Fixed an issue where audit logs are duplicated into an offline buffer even when they are successfully sent to Cloud Audit Logging.

July 20, 2023

  • Upgraded VMware vSphere Container Storage Plug-in from 2.6.2 to 2.7.2.
  • Added short names for Volume Snapshot CRDs.

July 10, 2023

Anthos clusters on VMware 1.15.2-gke.44 is now available. To upgrade, see Upgrading Anthos clusters on VMware. Anthos clusters on VMware. 1.15.2-gke.44 runs on Kubernetes 1.26.2-gke.1001.

If you use a third-party storage vendor, check the GDCV Ready storage partners document to make sure the storage vendor has already passed the qualification for this release of GKE on VMware.

June 05, 2023

Known issue

If you create a version 1.13.8 or version 1.14.4 admin cluster, or upgrade an admin cluster to version 1.13.8 or 1.14.4, the kind cluster pulls the following container images from docker.io:

  • docker.io/kindest/kindnetd
  • docker.io/kindest/local-path-provisioner
  • docker.io/kindest/local-path-helper

If docker.io isn't accessible from your admin workstation, the admin cluster creation or upgrade fails to bring up the kind cluster.

This issue affects the following versions of Anthos clusters on VMware:

  • 1.14.4
  • 1.13.8

For more information, including a workaround, see kind cluster pulls container images from docker.io on the Known issues page.

June 01, 2023

  • Fixed a known issue where node ID verification failed to handle hostnames with dots.

  • Fixed continuous increase of logging agent memory.

  • Fixed an issue where cluster-api-controllers in a high-availability admin cluster had no Pod anti-affinity. This could allow the three clusterapi-controllers Pods not to be scheduled on different control-plane nodes.

  • Fixed the wrong admin cluster resource link annotation key that can cause the cluster to be enrolled again by mistake.

  • Fixed a known issue where node pool creation failed because of duplicated VM-Host affinity rules.

  • The preflight check for StorageClass parameter validations now throws a warning instead of a failure on ignored parameters after CSI Migration. StorageClass parameter diskformat=thin is now allowed and does not generate a warning.

  • Fixed an issue where gkectl repair admin-master might fail with Failed to repair: failed to delete the admin master node object and reboot the admin master VM.

  • Fixed a race condition where some cluster nodes couldn't access the high-availability control plane when the underlying network performed ARP suppression.

  • Fixed a false error message for gkectl prepare when using a high-availability admin cluster.

  • Fixed an issue where during user cluster update, DeprecatedKubeception always shows up in the diff.

  • Fixed an issue where there were leftover Pods with failed status due to Predicate NodeAffinity failed during node re-creation.

May 02, 2023

Deprecations

  • Support for gkeadm on MAC and Windows is deprecated.

  • The enableWindowsDataplaneV2 field in the user cluster configuration file is deprecated.

  • The gkectl enroll cluster command is deprecated. Use gcloud to enroll a user cluster instead.

  • The following dashboards in the Cloud Monitoring Sample Library will be deprecated in a future release:

    • Anthos cluster control plane uptime
    • Anthos cluster node status
    • Anthos cluster pod status
    • Anthos utilization metering
    • GKE on-prem node status
    • GKE on-prem control plane uptime
    • GKE on-prem pod status
    • GKE on-prem vSphere vm health status

  • In a future release, the following customized dashboards will not be created when you create a new cluster:

    • GKE on-prem node status
    • GKE on-prem control plane uptime
    • GKE on-prem pod status
    • GKE on-prem vSphere vm health status
    • GKE on-prem Windows pod status
    • GKE on-prem Windows node status

April 13, 2023

Anthos clusters on VMware 1.12.7-gke.20 is now available. To upgrade, see Upgrading Anthos clusters on VMware. Anthos clusters on VMware 1.12.7-gke.20 runs on Kubernetes 1.23.17-gke.900.

The supported versions offering the latest patches and updates for security vulnerabilities, exposures, and issues impacting Anthos clusters on VMware are 1.14, 1.13, and 1.12.

April 11, 2023

Security bulletin

Two new vulnerabilities, CVE-2023-0240 and CVE-2023-23586, have been discovered in the Linux kernel that could allow an unprivileged user to escalate privileges. For more information, see the GCP-2023-003 security bulletin.

February 13, 2023

During preflight checks and cluster diagnosis, we now skip PVs and PVCs that use non-vSphere drivers.

January 12, 2023

  • If you specify a CIDR range (subnet) in the IP block file for your cluster nodes, the broadcast IP of the subnet, the network CIDR IP, and the network gateway IP will be excluded from the pool of addresses that get assigned to nodes.
  • Fixed a bug where CIDR ranges cannot be used in an IP block file.

December 21, 2022

  • Support for user cluster creation with Controlplane V2 enabled is now generally available. For more details on how to create a user cluster with this model, see Create a user cluster with Controlplane V2.
  • Preview: You can now roll back node pools to a previous working version if you detect an issue in the new version after a cluster upgrade. For more information, see Rolling back a node pool after an upgrade.
  • Preview: The following private registry updates are now available:
    • Support for private registry credentials using prepared Secrets is now available as a preview feature. A new privateRegistry field has been added in the Secrets configuration file.
    • Added a new privateRegistry section in the user cluster configuration file. You can use different private registry credentials for the user cluster and admin cluster. You can also use a different private registry address for user clusters with Controlplane V2 enabled.
    • You can also update private registry credentials for an admin cluster or user cluster with the gkectl update credentials command. For more information, see Update private registry credentials.
  • Cluster names are now included in kubeconfig files when creating a new admin cluster or user cluster. If you are upgrading your existing cluster to 1.14.0 or higher, the existing kubeconfig file is updated with the cluster name.
  • cluster-health-controller is now integrated with health-check-exporter to emit metrics based on the periodic health check results, making it easy to monitor and detect cluster health problems.
  • GA: The node pool update policy is generally available. With this feature, you can configure the value of maximumConcurrentNodePoolUpdate in the user cluster configuration file to 1. This will configure the maximum number of additional nodes spawned during cluster upgrade or update, which can potentially avoid two issues — resource quota limit issue and PDB deadlock issue. For more information, see Configure node pool update policy.
  • Support for vSphere cluster/host/network/datastore folders is generally available. You can use folders to group objects of the same type for easier management. For more information, see Specify vSphere folders in cluster configuration and the relevant sections in the admin cluster and user cluster configuration files.
  • Added a feature enabling cluster administrators to configure RBAC policies based on Azure Active Directory (AD) groups. Group information for users belonging to more than 200 groups can now be retrieved.

A known issue has been discovered. See the January 25, 2023 release note.

June 16, 2022

Fixed for version 1.10.5

  • Fixed the issue where admin cluster backup did not back up always-on secrets encryption keys. This caused repairing an admin cluster using gkectl repair master --restore-from-backup to fail when always-on secrets encryption was enabled.

  • Fixed the issue of high resource usage when AIDE runs as a cron job by disabling AIDE by default. This fix will affect compliance with CIS L1 Server benchmark 1.4.2: Ensure filesystem integrity is regularly checked.

    To re-enable the AIDE cron job, see Configure AIDE cron job.

Fixed the following vulnerabilities

May 19, 2022

Secret encryption key rotation does not fail when the cluster has more than 1000 secrets.

April 27, 2022

We have removed the over-privileged RBAC permissions for the following components.

RBAC policies applied to service account on the admin cluster

When you register a 1.11.0+ admin cluster to a fleet, a service account is created with the needed role-based access control (RBAC) policies that lets the Connect agent send requests to the admin cluster's Kubernetes API server on behalf of the service account. The service account and RBAC policies are needed so that you can manage the lifecycle of your user clusters in the Google Cloud console. For more information, see Admin cluster RBAC policies.

March 03, 2022

Changes

  • gkectl diagnose now reports a broken cluster caused by an admin cluster registration error during creation.

January 24, 2022

Anthos clusters on VMware 1.8.6-gke.4 is now available. To upgrade, see Upgrading Anthos clusters on VMware. Anthos clusters on VMware 1.8.6-gke.4 runs on Kubernetes 1.20.12-gke.1500.

November 30, 2021

Anthos clusters on VMware 1.7.6-gke.6 is now available. To upgrade, see Upgrading Anthos clusters on VMware. Anthos clusters on VMware 1.7.6-gke.6 runs on Kubernetes v1.19.15-gke.1900.

The supported versions offering the latest patches and updates for security vulnerabilities, exposures, and issues impacting Anthos clusters on VMware are 1.9, 1.8, and 1.7.

November 18, 2021

With version 1.9.2, cert-manager is installed in the cert-manager namespace. Previously, for versions 1.8.2 to 1.9.1, cert-manager was installed in the kube-system namespace.

The cert-manager version is upgraded from 1.0.3 to 1.5.4.

If you already use any ClusterIssuer with a different cluster resource namespace from the default cert-manager namespace, follow these steps if you upgrade to version 1.9.2.

   * Manually copy the related certificates, secrets, or issuers to the cert-manager namespace to use the installed cert-manager after upgrading to 1.9.2.    

   * If you need to use a different version of cert-manager, or if you need to install it in a different namespace, follow these instructions each time that you upgrade your cluster.

October 27, 2021

Anthos clusters on VMware 1.8.4-gke.1 is now available. To upgrade, see Upgrading Anthos clusters on VMware. Anthos clusters on VMware 1.8.4-gke.1 runs on Kubernetes v1.20.9-gke.701.

The supported versions offering the latest patches and updates for security vulnerabilities, exposures, and issues impacting Anthos clusters on VMware are 1.9, 1.8, and 1.7.

October 21, 2021

A security issue was discovered in the Kubernetes ingress-nginx controller, CVE-2021-25742. Ingress-nginx custom snippets allow retrieval of ingress-nginx service account tokens and secrets across all namespaces. For more information, see the GCP-2021-024 security bulletin.

September 29, 2021

Features:

Cluster lifecycle Improvements:

  • GA: You can register an admin cluster during its creation by filling in the gkeConnect section in the admin cluster configuration file, similar to user cluster registration.

Platform enhancements:

  • Preview: User clusters can now be in a different vSphere datacenter from the admin cluster, resulting in datacenter isolation between the admin cluster and user clusters. This provides greater resiliency in the case of vSphere environment failures.

  • GA: Support for Windows node pools is generally available.This release adds:

    • Preview: Windows DataplaneV2 support, which allows for using Windows Network Policy
    • Node Problem Detector (NPD) support on Windows
    • Streamlined process for preparing Windows images in a private registry
    • Enhanced Flannel CNI support on Windows

    The upstream fixes for the "Windows Pod stuck at terminating status" error are also applied to this release, which improves the stability of running Windows workloads.

  • GA: Support for Container-Optimized OS (COS) node pools is generally available.

  • GA: CoreDNS is now the cluster DNS provider.

    • Clusters that are upgraded to 1.9 will have their KubeDNS provider replaced with CoreDNS. During the upgrade, CoreDNS is first deployed and then KubeDNS is removed, so applications should not observe DNS unavailability. However before upgrading, ensure that your cluster has enough additional resources to deploy CoreDNS. CoreDNS requires 100 millicpu and 170 MiB of memory per instance, all clusters require a minimum of 2 instances, and there is an additional instance deployed for every 16 nodes in the cluster.
    • You can configure cluster DNS options such as upstream name servers by using the new ClusterDNS custom resource.

Security enhancements:

  • GA: Always-on secrets encryption: You can enable secrets encryption with internally generated keys instead of a hardware security module (HSM). Use the gkectl update command to rotate these keys or to enable or disable secrets encryption after cluster creation.
  • Preview: Windows network policy support. This release introduces a new network plugin, Antrea, for Windows nodes. In addition to network connectivity and services support, it provides network policy support. When creating a user cluster, you can set enableWindowsDataplaneV2 to true to enable this feature. Enabling this feature replaces Flannel with Antrea on Windows nodes.
  • Preview: Azure AD group support for Authentication: This feature allows cluster admins to configure RBAC policies based on Azure AD groups for authorization in clusters. This supports retrieval of groups information for users belonging to more than 200 groups, thus overcoming a limitation of regular OIDC configured with Azure AD as the identity provider.

Simplify day-2 operations:

  • Preview: When creating a user cluster, you can set enableVMTracking in the configuration file to true to enable vSphere tag creation and attachment to the VMs in the user cluster. This allows easy mapping of VMs to clusters and node pools. See Enable VM tracking.
  • GA: New metrics agents based on open telemetry are introduced to improve reliability, scalability and resource usage.
  • Preview: You can enable or disable Stackdriver with gkectl update on existing user clusters. You can enable or disable cloud audit logging and monitoring with gkectl update on both admin and user clusters.

Breaking changes:

  • User cluster registration is now required and enforced. You must fill in the gkeConnect section of the user cluster configuration file before creating a new user cluster. You cannot upgrade a user cluster unless that cluster is registered. To unblock the cluster upgrade, add the gkeConnect section to the configuration file and run gkectl update cluster to register an existing 1.8 user cluster.

  • User clusters must be upgraded before the admin cluster. The flag --force-upgrade-admin to allow the old upgrade flow (admin cluster upgrade first) is no longer supported.

  • The following requirements are now enforced when you create a cluster that has logging and monitoring enabled.

    • The Config Monitoring for Ops API is enabled in your logging-monitoring project.
    • The Ops Config Monitoring Resource Metadata Writer role is granted to your logging-monitoring service account.
    • The URL opsconfigmonitoring.googleapis.com is added to your proxy allowlist (if applicable).

September 03, 2021

Anthos clusters on VMware 1.7.3-gke.6 is now available. To upgrade, see Upgrading Anthos clusters on VMware. Anthos clusters on VMware 1.7.3-gke.X runs on Kubernetes v1.19.12-gke.1100

The supported versions offering the latest patches and updates for security vulnerabilities, exposures, and issues impacting Anthos clusters on VMware are 1.8, 1.7, and 1.6.

July 08, 2021

Fixes:

Fixed CVE-2021-34824 that could expose private keys and certificates from Kubernetes secrets through the credentialName field when using Gateway or DestinationRule. This vulnerability affects all clusters created or upgraded with Anthos clusters on VMware version 1.8.0.21. For more information, see the GCP-2021-012 security bulletin.

June 28, 2021

Platform enhancements:

  • Preview: Cluster autoscaling is now available in preview. With cluster autoscaling, you can horizontally scale node pools in proportion to workload demand. When demand is high, the cluster autoscaler adds nodes to the node pool. When demand is low, the cluster autoscaler removes nodes from the node pool, scaling back down to a minimum size that you designate. Cluster autoscaling can increase the availability of your workloads while controlling costs.

  • Preview: User cluster control-plane node and admin cluster add-on node auto sizing are now available in preview. The features can be enabled separately in user cluster or admin cluster configurations. When you enable user cluster control-plane node auto sizing, user cluster control-plane nodes are automatically resized in proportion to the number of node pool nodes in the given user cluster. When you enable admin cluster add-on node auto sizing, admin cluster add-on nodes are automatically resized in proportion to the number nodes in the admin cluster.

  • Preview: Windows Server container support for Anthos clusters on VMware is now available in preview. This allows you to modernize and run your Windows-based apps more efficiently in your data centers without having to go through risky application rewrites. You can use Windows containers alongside Linux containers for your container workloads. The same experience and benefits that you have come to enjoy with Anthos clusters on VMware using Linux--application portability, consolidation, cost savings, and agility--can now be applied to Windows Server applications also.

  • Preview: Admin cluster backup is now available in preview. With this feature enabled, admin cluster backups are automatically performed before and after user and admin cluster creation, update, and upgrade. A new gkectl backup admin command performs manual backup. Upon admin cluster storage failure, you can restore the admin cluster from a backup with the gkectl repair admin-cluster --restore-from-backup command.

May 11, 2021

A recently discovered vulnerability, CVE-2021-31920, affects Istio in respect to its authorization policies. Istio contains a remotely exploitable vulnerability where an HTTP request with multiple slashes or escaped slash characters can bypass Istio authorization policy when path-based authorization rules are used. While Anthos clusters on VMware uses an Istio Gateway object for network ingress traffic into clusters, authorization policies are not a supported or intended use case for Istio as part of the Anthos clusters on VMware prerequisites. For more details, refer to the Istio security bulletin.

May 05, 2021

If you upgrade the admin cluster before you upgrade the associated user clusters within the same minor version, such as from 1.7.0 to 1.7.1, the user cluster control-planes will be upgraded together with the admin cluster. This applies even if you use the flag --force-upgrade-admin. This behavior, in versions 1.7.0 and later, is different from versions 1.6 and earlier, and is expected behavior.

February 26, 2021

Anthos clusters on VMware (GKE on-prem) 1.6.2-gke.0 is now available. To upgrade, see Upgrading Anthos clusters on VMware. Anthos clusters on VMware 1.6.2-gke.0 clusters run on Kubernetes 1.18.13-gke.400.

December 10, 2020

Node Problem Detector and Node Auto Repair automatically detect and repair additional failures, such as Kubelet-API server connection loss (an OSS issue) and long-lasting DiskPressure conditions.

September 24, 2020

Improved support for Day-2 operations:

  • The gkectl update cluster command is now generally available. Users can use it to change supported features in the user cluster configurations after cluster creation.
  • The gkectl update credentials command for vSphere and F5 credentials is now generally available.
  • Improves scalability with 20 user clusters per admin cluster, and 250 nodes, 7500 pods, 500 load balancing services (using Seesaw), and 250 load balancing services (using F5) per user cluster.
  • Introduces vSphere CSI driver in preview.

Fixes:

  • Fixed an issue that caused approximately 50 seconds of downtime for the user cluster API service during cluster upgrade or update.
  • Corrected the default log verbosity setting in gkectl and gkeadm help messages.

June 25, 2020

Functionality changes:

  • Enabled Horizontal Pod Autoscaler (HPA) for the Istio ingress gateway.
  • Removed ingress controller from admin cluster.
  • Consolidated sysctl configs with Google Kubernetes Engine.
  • Added etcd defrag pod in admin cluster and user cluster, which will be responsible for monitoring etcd's database size and defragmenting it as needed. This helps reclaim etcd database size and recover etcd when its disk space is exceeded.

May 21, 2020

Preflight check for internet validation is updated to not follow redirect. If your organization requires outbound traffic to pass through a proxy server, you no longer need to allowlist the following addresses in your proxy server:

  • console.cloud.google.com
  • cloud.google.com

April 23, 2020

OVA download displays download progress.

Fix for Docker failed DNS resolution on admin workstation at startup.

March 23, 2020

Added support for up to 100 nodes per user cluster.

gkectl automatically uses the proxy URL from config.yaml to configure the proxy on the admin workstation.

Preview Feature: Introducing User cluster Nodepools. A node pool is a group of nodes within a cluster that all have the same configuration. In GKE on-prem 1.3.0, node pools are a preview feature in the user clusters. This feature lets users create multiple node pools in a cluster, and update them as needed.

February 21, 2020

Resolved a known issue of cluster upgrade when using a vSAN datastore associated with a GKE on-prem version before 1.2

December 20, 2019

GKE on-prem now supports vSphere 6.7 Update 3. Read its release notes.

October 25, 2019

Known Issues

September 26, 2019

GKE On-Prem version 1.1.0-gke.6 is now available. To download version 1.1.0-gke.6's gkectl and upgrade bundle, see Downloads. Then, see Upgrading clusters.

This minor version includes the following changes:

You can now diagnose node issues using the debug-toolbox container image.

August 22, 2019

July 30, 2019

Upgrades F5 BIG-IP controller to version 1.9.0.

June 17, 2019

Upgrading from beta-1.4 to 1.0.10

Before upgrading your beta clusters to the first general availability version, perform the steps described in Installing GKE On-Prem, and review the following points:

  • If you are running a beta version before beta-1.4, be sure to upgrade to beta-1.4 first.

  • If your beta clusters are running their own L4 load balancers (not the default, F5 BIG-IP), you need to delete and recreate your clusters to run the latest GKE On-Prem version.

  • If your clusters were upgraded to beta-1.4 from beta-1.3, run the following command for each user cluster before upgrading:

    kubectl delete crd networkpolicies.crd.projectcalico.org

  • vCenter certificate verification is now required. (vsphereinsecure is no longer supported.) If you're upgrading your beta 1.4 clusters to 1.0.10, you need to provide a vCenter trusted root CA public certificate in the upgrade configuration file.

  • You need to upgrade all of your running clusters. For this upgrade to succeed, your clusters can't run in a mixed version state.

  • You need to upgrade your admin clusters to the latest version first, then upgrade your user clusters.

May 13, 2019

Clusters upgraded from version beta-1.2 to beta-1.3 might be affected by a known issue that damages the cluster's configuration file and prevents future cluster upgrades. This issue affects all future cluster upgrades.

You can resolve this issue by deleting and recreating clusters upgraded from beta-1.2 to beta-1.3.

To resolve the issue without deleting and recreating the cluster, you need to re-encode and apply each cluster's Secrets. Perform the following steps:

  1. Get the contents of the create-config Secrets stored in the admin cluster. This must be done for the create-config Secret in the kube-system namespace, and for the create-config Secrets in each user cluster's namespace:

    kubectl get secret create-config -n [USER_CLUSTER_NAME] -o jsonpath={.data.cfg} | base64 -d > [USER_CLUSTER_NAME]_create_secret.yaml

    For example:

    kubectl get secret create-config -n kube-system -o jsonpath={.data.cfg} | base64 -d > kube-system_create_secret.yaml

  2. For each user cluster, open the [USER_CLUSTER_NAME]_create_secret.yaml file in an editor.

    If the values for registerserviceaccountkey and connectserviceaccountkey are not REDACTED, no further action is required: the Secrets do not need to be re-encoded and written to the cluster.

  3. Open the original create_config.yaml file in another editor.

  4. In [USER_CLUSTER_NAME]_create_secret.yaml, replace the registerserviceaccountkey and connectserviceaccountkey values with the values from the original create_config.yaml file. Save the changed file.

  5. Repeat steps 2-4 for each [USER_CLUSTER_NAME]_create_secret.yaml, and for the kube-system_create_secret.yaml file.

  6. Base64-encode each [USER_CLUSTER_NAME]_create_secret.yaml file and the kube-system_create_secret.yaml file:

    cat [USER_CLUSTER_NAME]_create_secret.yaml | base64 > [USER_CLUSTER_NAME]_create_secret_create_secret.b64

    cat kube-system-cluster_create_secret.yaml | base64 > kube-system-cluster_create_secret.b64

  7. Replace the data[cfg] field in each Secret in the cluster with the contents of the corresponding file:

    kubectl edit secret create-config -n [USER_CLUSTER_NAME]
  # kubectl edit opens the file in the shell's default text editor
  # Open `first-user-cluster_create_secret.b64` in another editor, and replace
  # the `cfg` value with the copied value
  # Make sure the copied string has no newlines in it

  8. Repeat step 7 for each [USER_CLUSTER_NAME]_create_secret.yaml Secret, and for the kube-system_create_secret.yaml Secret.

  9. To ensure that the update was successful, repeat step 1.

April 25, 2019

GKE On-Prem's ingress controller uses Istio 1.1 with automatic Secret discovery. However, the node agent for Secret discovery may fail to get Secret updates after Secret deletion. So avoid deleting Secrets. If you must delete a Secret and Ingress TLS fails afterwards, manually restart the Ingress Pod in the gke-system namespace.

April 11, 2019

GKE On-Prem clusters now automatically connect back to Google using Connect.

A regression causes gkectl diagnose snapshot commands to use the wrong SSH key, which prevents the command from collecting information from user clusters. As a workaround for support cases, you might need to SSH into individual user cluster nodes and manually gather data.

April 02, 2019

Added documentation for backing up and restoring clusters.

You can now configure authentication for clusters using OIDC and ADFS. To learn more, refer to Authenticating with OIDC and AD FS and Authentication.

March 04, 2019

Kubernetes Network Policies are now supported.

February 07, 2019

You now need to provision a 100GB disk in vSphere Datastore. GKE On-Prem uses the disk to store some of its vital data, such as etcd. See Data center requirements.

External communication by Grafana is disabled.

January 23, 2019

Changes

January 14, 2019

GKE On-Prem now runs Kubernetes version 1.11.2-gke.19.

vSphere credentials are now pulled from credential files.

gkectl diagnose snapshot can now take snapshots of remote files on the node, results of remote commands on the nodes, and Prometheus queries.

Resizing IPAM address blocks if using static IP allocation for nodes, is not supported in alpha. To work around this, consider allocating more IP addresses than you currently need.

November 30, 2018

GKE On-Prem alpha 1.0 is now available. The following changes are included in this release:

Support for high-availability Prometheus setup.

October 17, 2018

Cluster upgrades are not supported in EAP 2.0.

As part of the cluster bootstrapping process, a short-lived minikube instance is run. The minikube version used has security vulnerability CVE-2018-1002103.