Google Distributed Cloud (software only) for bare metal combined release notes (all minor versions)

This document lists production updates to Google Distributed Cloud (software only) for bare metal (formerly known as Google Distributed Cloud Virtual, previously known as Anthos clusters on bare metal). Check this page periodically for any new announcements.

You can see the latest product updates for all of Google Cloud on the Google Cloud page, browse and filter all release notes in the Google Cloud console, or programmatically access release notes in BigQuery.

To get the latest product updates delivered to you, add the URL of this page to your feed reader, or add the feed URL directly.

August 28, 2025

1.32
Fixed

The following issues were fixed in 1.32.400-gke.68:

  • Fixed an issue that caused the Ansible playbook for handling Customer-Acquired Licenses (CAL) to fail and not complete.

  • Fixed vulnerabilities listed in Vulnerability fixes.

August 08, 2025

1.32
Issue

For information about the latest known issues, see Google Distributed Cloud for bare metal known issues in the Troubleshooting section.

July 21, 2025

1.30 
Announcement

Google Distributed Cloud for bare metal 1.30.1100-gke.67 is now available for download. To upgrade, see Upgrade clusters. Google Distributed Cloud for bare metal 1.30.1100-gke.67 runs on Kubernetes v1.30.12-gke.800.

After a release, it takes approximately 7 to 14 days for the version to become available for installations or upgrades with the GKE On-Prem API clients: the Google Cloud console, the gcloud CLI, and Terraform.

If you use a third-party storage vendor, check the Ready storage partners document to make sure the storage vendor has already passed the qualification for this release of Google Distributed Cloud for bare metal.

July 17, 2025

1.32
Issue

For information about the latest known issues, see Google Distributed Cloud for bare metal known issues in the Troubleshooting section.

June 23, 2025

1.31
Fixed

The following issues were fixed in 1.31.600-gke.85:

April 25, 2025

1.31
Change

The following functional change was made in 1.31.400-gke.110:

  • Updated the cluster upgrade operation to keep only the three latest kubeadm backups of etcd and configuration information for a node. Previously, kubeadm kept node backups for every attempted upgrade.

  • Upgraded etcd to v3.4.33-0-gke.3.

March 28, 2025

1.29
Fixed

Fixed an issue that caused cluster creation to fail because kubelet restarted before required static pods are running.

March 25, 2025

1.31
Breaking

Since release 1.30.0-gke.1930, the featureGates.enableGMPForSystemMetrics field in the stackdriver custom resource is always on and can't be disabled. It has been enabled by default since 1.16. If you've manually turned this feature off, upgrading clusters to version 1.30 means a breaking change in the format of some system metrics. For information on this feature, see Use Google Cloud Managed Service for Prometheus for selected system components.

February 27, 2025

1.30 
Announcement

Release 1.30.500-gke.127

Google Distributed Cloud for bare metal 1.30.500-gke.127 is now available for download. To upgrade, see Upgrade clusters. Google Distributed Cloud for bare metal 1.30.500-gke.127 runs on Kubernetes 1.30.

After a release, it takes approximately 7 to 14 days for the version to become available for installations or upgrades with the GKE On-Prem API clients: the Google Cloud console, the gcloud CLI, and Terraform.

If you use a third-party storage vendor, check the Ready storage partners document to make sure the storage vendor has already passed the qualification for this release of Google Distributed Cloud for bare metal.

February 13, 2025

1.31
Announcement

Release 1.31.200-gke.58

Google Distributed Cloud for bare metal 1.31.200-gke.58 is now available for download. To upgrade, see Upgrade clusters. Google Distributed Cloud for bare metal 1.31.200-gke.58 runs on Kubernetes v1.31.5-gke.300.

After a release, it takes approximately 7 to 14 days for the version to become available for installations or upgrades with the GKE On-Prem API clients: the Google Cloud console, the gcloud CLI, and Terraform.

If you use a third-party storage vendor, check the Ready storage partners document to make sure the storage vendor has already passed the qualification for this release of Google Distributed Cloud for bare metal.

1.31
Fixed

The 1.31.200-gke.58 release includes many vulnerability fixes. For a list of all vulnerabilities fixed in this release, see Vulnerability fixes.

1.31
Issue

For information about the latest known issues, see Google Distributed Cloud for bare metal known issues in the Troubleshooting section.

February 05, 2025

1.29
Fixed

The 1.29.1000-gke.93 release includes many vulnerability fixes. For a list of all vulnerabilities fixed in this release, see Vulnerability fixes.

January 07, 2025

1.29
Fixed

The following container image security vulnerabilities have been fixed in 1.29.900-gke.180:

December 10, 2024

1.30 
Fixed

Fixes:

  • Fixed the issue where non-root users can't run bmctl restore to restore quorum.

  • Fixed an issue where CronJob specs for periodic health checks weren't updated to reflect cluster annotation changes.

  • Fixed an issue that blocked user cluster create and upgrade operations to patch versions 1.30.100, 1.30.200, or 1.30.300. This issue applies only when kubectl or a GKE On-Prem API client (console, gcloud CLI, or Terraform) is used for user cluster creation and upgrades.

October 10, 2024

1.30 
Fixed

The following container image security vulnerabilities have been fixed in 1.30.200-gke.101:

September 25, 2024

1.30 
Announcement

Release 1.30.100-gke.96

Google Distributed Cloud for bare metal 1.30.100-gke.96 is now available for download. To upgrade, see Upgrade clusters. Google Distributed Cloud for bare metal 1.30.100-gke.96 runs on Kubernetes 1.30.

After a release, it takes approximately 7 to 14 days for the version to become available for installations or upgrades with the GKE On-Prem API clients: the Google Cloud console, the gcloud CLI, and Terraform.

If you use a third-party storage vendor, check the Ready storage partners document to make sure the storage vendor has already passed the qualification for this release of Google Distributed Cloud for bare metal.

September 12, 2024

1.29
Announcement

Release 1.29.500-gke.163

Google Distributed Cloud for bare metal 1.29.500-gke.163 is now available for download. To upgrade, see Upgrade clusters. Google Distributed Cloud for bare metal 1.29.500-gke.163 runs on Kubernetes v1.29.7-gke.1200.

After a release, it takes approximately 7 to 14 days for the version to become available for installations or upgrades with the GKE On-Prem API clients: the Google Cloud console, the gcloud CLI, and Terraform.

If you use a third-party storage vendor, check the Ready storage partners document to make sure the storage vendor has already passed the qualification for this release of Google Distributed Cloud for bare metal.

August 29, 2024

1.30 
Fixed

Fixes:

  • Fixed an issue where old, inoperable WebHook resources caused problems with cluster upgrades.

  • Fixed an issue where upgraded clusters didn't get label updates that match the labels applied for newly created clusters, for a given version.

  • Fixed an issue where service accounts created by using the --create-service-accounts flag with the bmctl create config command don't have enough permissions.

  • Fixed an issue where the kubelet doesn't honor shortened, 1-second grace period for pod deletion during eviction-based draining.

July 25, 2024

1.29
Change

Functionality changes:

  • Updated registry mirror support to allow you to specify a port for host addresses.

  • Updated Kubernetes audit logging to include request and response payloads from the Kubernetes API server for bare metal custom resources, such as Cluster, NodePool, BareMetalMachine, and BareMetalCluster.

July 01, 2024

1.16
Fixed

Fixes:

  • Fixed an issue where upgraded clusters didn't get label updates that match the labels applied for newly created clusters, for a given version.

May 02, 2024

1.28
Issue

Known issues:

For information about the latest known issues, see GKE on Bare Metal known issues in the Troubleshooting section.

April 03, 2024

1.16
Security

A Denial-of-Service (DoS) vulnerability (CVE-2023-45288) was recently discovered in multiple implementations of the HTTP/2 protocol, including the golang HTTP server used by Kubernetes. The vulnerability could lead to a DoS of the Google Kubernetes Engine (GKE) control plane. For more information, see the GCP-2024-022 security bulletin.

March 04, 2024

1.15
Feature

GKE on Bare Metal version 1.15.10 and later has been qualified on and supports Red Hat Enterprise Linux (RHEL) version 8.9.

1.28
Issue

Known issues:

For information about the latest known issues, see GKE on Bare Metal known issues in the Troubleshooting section.

February 20, 2024

1.16
Feature

Release 1.16.6

GKE on Bare Metal 1.16.6 is now available for download. To upgrade, see Upgrade clusters. GKE on Bare Metal 1.16.6 runs on Kubernetes 1.27.

If you use a third-party storage vendor, check the GDCV Ready storage partners document to make sure the storage vendor has already passed the qualification for this release of GKE on Bare Metal.

January 31, 2024

1.6
Security

Security bulletin (all minor versions)

A security vulnerability, CVE-2024-21626, has been discovered in runc where a user with permission to create Pods might be able to gain full access to the node filesystem.

For instructions and more details, see the GCP-2024-005 security bulletin.

1.7
Security

Security bulletin (all minor versions)

A security vulnerability, CVE-2024-21626, has been discovered in runc where a user with permission to create Pods might be able to gain full access to the node filesystem.

For instructions and more details, see the GCP-2024-005 security bulletin.

1.8
Security

Security bulletin (all minor versions)

A security vulnerability, CVE-2024-21626, has been discovered in runc where a user with permission to create Pods might be able to gain full access to the node filesystem.

For instructions and more details, see the GCP-2024-005 security bulletin.

1.9
Security

Security bulletin (all minor versions)

A security vulnerability, CVE-2024-21626, has been discovered in runc where a user with permission to create Pods might be able to gain full access to the node filesystem.

For instructions and more details, see the GCP-2024-005 security bulletin.

December 15, 2023

1.16
Feature

Release 1.16.4

GKE on Bare Metal 1.16.4 is now available for download. To upgrade, see Upgrade clusters. GKE on Bare Metal 1.16.4 runs on Kubernetes 1.27.

If you use a third-party storage vendor, check the GDCV Ready storage partners document to make sure the storage vendor has already passed the qualification for this release of GKE on Bare Metal.

1.16
Change

Supported node pool versions:

If you use selective worker node pool upgrades to upgrade a cluster to version 1.16.4, see Node pool versioning rules for a list of the versions that are supported for the worker node pools.

1.28
Announcement

Version alignment

For easier identification of the Kubernetes version for a given release, we are aligning Anthos clusters on bare metal version numbering with GKE version numbering. This change starts with this minor release, which is version 1.28. The version alignment is for major and minor versions only, patch versions are product specific. In addition to this version alignment, the Anthos clusters on bare metal release versions will follow the GKE semantic versioning scheme (x.y.z-gke.N), including the addition of a GKE patch version (-gke.N). Unlike GKE, however, the patch version (z) increments by 100.

Example version numbers for Anthos clusters on bare metal:

  • Minor release: 1.28.0-gke.435
  • Initial patch release: 1.28.100-gke.27
  • Second patch release: 1.28.200-gke.19

This change affects numbering only. Upgrades from 1.16 to 1.28 follow the same process as upgrades between prior minor releases. However, downloads, upgrades, and cluster creation for 1.28 and higher versions require the fully qualified version number, including the GKE patch version.

1.28
Issue

Known issues:

For information about the latest known issues, see GKE on Bare Metal known issues in the Troubleshooting section.

November 21, 2023

1.14
Issue

Known issues:

For information about the latest known issues, see Anthos clusters on bare metal known issues in the Troubleshooting section.

November 06, 2023

1.14
Change

Functionality changes:

  • Added NODEPOOL-NAME, NODEPOOL-NAMESPACE, and STATUS columns for the InventoryMachine resource to improve troubleshooting.

  • Removed hardcoded timeout value for the bmctl backup operation.

1.14
Fixed

Fixes:

The following container image security vulnerabilities have been fixed in version 1.14.10:

September 29, 2023

1.14
Issue

Known issues:

For information about the latest known issues, see Anthos clusters on bare metal known issues in the Troubleshooting section.

August 16, 2023

1.15
Feature

Release 1.15.4

Anthos clusters on bare metal 1.15.4 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.15.4 runs on Kubernetes 1.26.

If you use a third-party storage vendor, check the GDCV Ready storage partners document to make sure the storage vendor has already passed the qualification for this release of GKE on Bare Metal.

1.15
Fixed

Fixes:

  • Fixed an issue for clusters configured with manual load balancing where CA rotation reported that there were no (0) control plane nodes.

August 01, 2023

1.13
Fixed

June 27, 2023

1.6
Security

Security bulletin (all minor versions)

A number of vulnerabilities have been discovered in Envoy, which is used in Anthos Service Mesh (ASM). These were reported separately as GCP-2023-002.

For more information, see the GCP-2023-016 security bulletin.

1.7
Security

Security bulletin (all minor versions)

A number of vulnerabilities have been discovered in Envoy, which is used in Anthos Service Mesh (ASM). These were reported separately as GCP-2023-002.

For more information, see the GCP-2023-016 security bulletin.

1.8
Security

Security bulletin (all minor versions)

A number of vulnerabilities have been discovered in Envoy, which is used in Anthos Service Mesh (ASM). These were reported separately as GCP-2023-002.

For more information, see the GCP-2023-016 security bulletin.

1.9
Security

Security bulletin (all minor versions)

A number of vulnerabilities have been discovered in Envoy, which is used in Anthos Service Mesh (ASM). These were reported separately as GCP-2023-002.

For more information, see the GCP-2023-016 security bulletin.

June 22, 2023

1.15
Fixed

Fixes:

  • Fixed an issue where containerd didn't restart when there was a version mismatch. This issue caused an inconsistent containerd version within the cluster.

  • Fixed an issue where the spec.proxy.noProxy value wasn't used in the Google Cloud connectivity preflight check (bmctl check gcp).

  • Fixed an issue that caused the logging agent to use continuously increasing amounts of memory. The following container image security vulnerabilities have been fixed:

June 16, 2023

1.6
Security

Security bulletin (all minor versions)

Two new security issues were discovered in Kubernetes where users may be able to launch containers that bypass policy restrictions when using ephemeral containers and either ImagePolicyWebhook (CVE-2023-2727) or the ServiceAccount admission plugin (CVE-2023-2728).

For more information, see the GCP-2023-014 security bulletin.

1.7
Security

Security bulletin (all minor versions)

Two new security issues were discovered in Kubernetes where users may be able to launch containers that bypass policy restrictions when using ephemeral containers and either ImagePolicyWebhook (CVE-2023-2727) or the ServiceAccount admission plugin (CVE-2023-2728).

For more information, see the GCP-2023-014 security bulletin.

1.8
Security

Security bulletin (all minor versions)

Two new security issues were discovered in Kubernetes where users may be able to launch containers that bypass policy restrictions when using ephemeral containers and either ImagePolicyWebhook (CVE-2023-2727) or the ServiceAccount admission plugin (CVE-2023-2728).

For more information, see the GCP-2023-014 security bulletin.

1.9
Security

Security bulletin (all minor versions)

Two new security issues were discovered in Kubernetes where users may be able to launch containers that bypass policy restrictions when using ephemeral containers and either ImagePolicyWebhook (CVE-2023-2727) or the ServiceAccount admission plugin (CVE-2023-2728).

For more information, see the GCP-2023-014 security bulletin.

June 01, 2023

1.13
Feature

Release 1.13.8

Anthos clusters on bare metal 1.13.8 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.13.8 runs on Kubernetes 1.24.

1.13
Issue

Known issues:

For information about the latest known issues, see Anthos clusters on bare metal known issues in the Troubleshooting section.

May 10, 2023

1.15
Deprecated

CentOS Linux 8 Support Deprecated

CentOS Linux 8 reached its end of life (EOL) on December 31st, 2021. We strongly recommend that you migrate to one of the other supported operating systems from Anthos clusters on bare metal. All support for CentOS is removed from Anthos clusters for bare metal release 1.17 (December 2023) and subsequent releases.

1.6
Deprecated

CentOS Linux 8 Support Deprecated

CentOS Linux 8 reached its end of life (EOL) on December 31st, 2021. We strongly recommend that you migrate to one of the other supported operating systems from Anthos clusters on bare metal. All support for CentOS is removed from Anthos clusters for bare metal release 1.17 (December 2023) and subsequent releases.

1.7
Deprecated

CentOS Linux 8 Support Deprecated

CentOS Linux 8 reached its end of life (EOL) on December 31st, 2021. We strongly recommend that you migrate to one of the other supported operating systems from Anthos clusters on bare metal. All support for CentOS is removed from Anthos clusters for bare metal release 1.17 (December 2023) and subsequent releases.

1.8
Deprecated

CentOS Linux 8 Support Deprecated

CentOS Linux 8 reached its end of life (EOL) on December 31st, 2021. We strongly recommend that you migrate to one of the other supported operating systems from Anthos clusters on bare metal. All support for CentOS is removed from Anthos clusters for bare metal release 1.17 (December 2023) and subsequent releases.

1.9
Deprecated

CentOS Linux 8 Support Deprecated

CentOS Linux 8 reached its end of life (EOL) on December 31st, 2021. We strongly recommend that you migrate to one of the other supported operating systems from Anthos clusters on bare metal. All support for CentOS is removed from Anthos clusters for bare metal release 1.17 (December 2023) and subsequent releases.

April 12, 2023

1.6
Announcement

Kubernetes image registry redirect

As of March 21, 2023, traffic to k8s.gcr.io is redirected to registry.k8s.io, following the community announcement. This change is happening gradually to reduce disruption, and should be transparent for most Anthos clusters.

To check for edge cases and mitigate potential impact to your clusters, follow the step-by-step guidance in k8s.gcr.io Redirect to registry.k8s.io - What You Need to Know.

1.7
Announcement

Kubernetes image registry redirect

As of March 21, 2023, traffic to k8s.gcr.io is redirected to registry.k8s.io, following the community announcement. This change is happening gradually to reduce disruption, and should be transparent for most Anthos clusters.

To check for edge cases and mitigate potential impact to your clusters, follow the step-by-step guidance in k8s.gcr.io Redirect to registry.k8s.io - What You Need to Know.

1.8
Announcement

Kubernetes image registry redirect

As of March 21, 2023, traffic to k8s.gcr.io is redirected to registry.k8s.io, following the community announcement. This change is happening gradually to reduce disruption, and should be transparent for most Anthos clusters.

To check for edge cases and mitigate potential impact to your clusters, follow the step-by-step guidance in k8s.gcr.io Redirect to registry.k8s.io - What You Need to Know.

1.9
Announcement

Kubernetes image registry redirect

As of March 21, 2023, traffic to k8s.gcr.io is redirected to registry.k8s.io, following the community announcement. This change is happening gradually to reduce disruption, and should be transparent for most Anthos clusters.

To check for edge cases and mitigate potential impact to your clusters, follow the step-by-step guidance in k8s.gcr.io Redirect to registry.k8s.io - What You Need to Know.

March 02, 2023

1.12
Issue

Known issues:

For information about the latest known issues, see Anthos clusters on bare metal known issues in the Troubleshooting section.

February 07, 2023

1.12
Fixed

December 19, 2022

1.13
Issue

Known issues:

For information about the latest known issues, see Anthos on bare metal known issues in the Troubleshooting section.

December 14, 2022

1.12
Feature

Release 1.12.6

Anthos clusters on bare metal 1.12.6 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.12.6 runs on Kubernetes 1.23.

November 22, 2022

1.13
Issue

Known issues:

For information about the latest known issues, see Anthos on bare metal known issues in the Troubleshooting section.

November 18, 2022

1.11
Fixed

November 07, 2022

1.12
Security

Security bulletin (1.11, 1.12, and 1.13)

A security vulnerability, CVE-2022-39278, has been discovered in Istio, which is used in Anthos Service Mesh, that allows a malicious attacker to crash the control plane.

For instructions and more details, see the Anthos clusters on bare metal security bulletin.

1.13
Security

Security bulletin (1.11, 1.12, and 1.13)

A security vulnerability, CVE-2022-39278, has been discovered in Istio, which is used in Anthos Service Mesh, that allows a malicious attacker to crash the control plane.

For instructions and more details, see the Anthos clusters on bare metal security bulletin.

August 01, 2022

1.10 
Issue

Known issues:

For information about the latest known issues, see Anthos on bare metal known issues in the Troubleshooting section.

June 09, 2022

1.9
Feature

Release 1.9.8

Anthos clusters on bare metal 1.9.8 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.9.8 runs on Kubernetes 1.21.

1.9
Issue

Known issues:

For information about the latest known issues, see Anthos on bare metal known issues in the Troubleshooting section.

April 27, 2022

1.9
Feature

Release 1.9.7

Anthos clusters on bare metal 1.9.7 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.9.7 runs on Kubernetes 1.21.

April 26, 2022

1.6
Security

Security bulletin (all minor versions)

Two security vulnerabilities, CVE-2022-1055 and CVE-2022-27666 have been discovered in the Linux kernel. Each can lead to a local attacker being able to perform a container breakout, privilege escalation on the host, or both. These vulnerabilities affect Linux operating systems supported by Anthos clusters on bare metal. For instructions and more details, see the GCP-2022-014 security bulletin.

1.7
Security

Security bulletin (all minor versions)

Two security vulnerabilities, CVE-2022-1055 and CVE-2022-27666 have been discovered in the Linux kernel. Each can lead to a local attacker being able to perform a container breakout, privilege escalation on the host, or both. These vulnerabilities affect Linux operating systems supported by Anthos clusters on bare metal. For instructions and more details, see the GCP-2022-014 security bulletin.

1.8
Security

Security bulletin (all minor versions)

Two security vulnerabilities, CVE-2022-1055 and CVE-2022-27666 have been discovered in the Linux kernel. Each can lead to a local attacker being able to perform a container breakout, privilege escalation on the host, or both. These vulnerabilities affect Linux operating systems supported by Anthos clusters on bare metal. For instructions and more details, see the GCP-2022-014 security bulletin.

1.9
Security

Security bulletin (all minor versions)

Two security vulnerabilities, CVE-2022-1055 and CVE-2022-27666 have been discovered in the Linux kernel. Each can lead to a local attacker being able to perform a container breakout, privilege escalation on the host, or both. These vulnerabilities affect Linux operating systems supported by Anthos clusters on bare metal. For instructions and more details, see the GCP-2022-014 security bulletin.

April 12, 2022

1.8
Security

Security bulletin (1.8, 1.9, and 1.10)

A security vulnerability, CVE-2022-23648, has been discovered in containerd's handling of path traversal in the OCI image volume specification. Containers launched through containerd's CRI implementation with a specially-crafted image configuration could gain full read access to arbitrary files and directories on the host.

For more information, see the GCP-2022-013 security bulletin.

1.9
Security

Security bulletin (1.8, 1.9, and 1.10)

A security vulnerability, CVE-2022-23648, has been discovered in containerd's handling of path traversal in the OCI image volume specification. Containers launched through containerd's CRI implementation with a specially-crafted image configuration could gain full read access to arbitrary files and directories on the host.

For more information, see the GCP-2022-013 security bulletin.

March 23, 2022

1.9
Feature

Release 1.9.6

Anthos clusters on bare metal 1.9.6 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.9.6 runs on Kubernetes 1.21.

1.9
Fixed
1.9
Issue

Known issues:

When you upgrade Anthos clusters on bare metal from a version with a security patch to the next minor release, we recommend that you upgrade to the highest patch version to ensure that you have the latest security fixes. Always review the release notes before upgrading so that you're aware of what has changed, including security fixes and known issues. Upgrading to a lower release version isn't supported.

For information about the latest known issues, see Anthos on bare metal known issues in the Troubleshooting section.

March 14, 2022

1.8
Feature

Release 1.8.9

Anthos clusters on bare metal 1.8.9 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.8.9 runs on Kubernetes 1.20.

1.8
Fixed

Fixes:

1.8
Issue

Known issues:

When upgrading Anthos clusters on bare metal from a version with a security patch to the next minor release, we recommend that you upgrade to the highest patch version to ensure that you have the latest security fixes. Always review the release notes before upgrading so that you're aware of what has changed, including security fixes and known issues. Upgrading to a lower release version isn't supported.

For information about the latest known issues, see Anthos on bare metal known issues in the Troubleshooting section.

February 28, 2022

1.9
Feature

Release 1.9.5

Anthos clusters on bare metal 1.9.5 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.9.5 runs on Kubernetes 1.21.

1.9
Fixed

Fixes:

1.9
Issue

Known issues:

When you upgrade Anthos clusters on bare metal from a version with a security patch to the next minor release, we recommend that you upgrade to the highest patch version to ensure that you have the latest security fixes. Always review the release notes before upgrading so that you're aware of what has changed, including security fixes and known issues. Upgrading to a lower release version isn't supported.

For information about the latest known issues, see Anthos on bare metal known issues in the Troubleshooting section.

February 25, 2022

1.10 
Feature

Release 1.10.2

Anthos clusters on bare metal 1.10.2 is now available for download. To upgrade, see Upgrade Anthos on bare metal. Anthos clusters on bare metal 1.10.2 runs on Kubernetes 1.21.

1.8
Security

Security bulletin (1.8, 1.9, and 1.10)

Envoy recently released multiple security vulnerability fixes. The vulnerabilities affect Anthos clusters on bare metal, because Envoy is used for Metrics Server.

For instructions and more details, see the GCP-2022-008 security bulletin.

1.9
Security

Security bulletin (1.8, 1.9, and 1.10)

Envoy recently released multiple security vulnerability fixes. The vulnerabilities affect Anthos clusters on bare metal, because Envoy is used for Metrics Server.

For instructions and more details, see the GCP-2022-008 security bulletin.

1.10 
Fixed

Fixes:

  • Fixed issue in which the edge profile's request to reserve resources is lost during the upgrade process.

  • Fixed bmctl upgrade command so that the log file upgrade-cluster.log is generated in the bmctl-workspace/cluster/logs directory.

  • Fixed issue in which the non-root login didn't have the proper permissions to perform bmctl backup or bmctl restore.

  • Fixed a Node Problem Detector service that sometimes failed to run on nodes after a cluster installation or upgrade.

  • The following container image security vulnerabilities have been fixed:

February 04, 2022

1.6
Security

Security bulletin (all minor versions)

A security vulnerability, CVE-2021-4034, has been discovered in pkexec, a part of the Linux policy kit package (polkit), that allows an authenticated user to perform a privilege escalation attack. PolicyKit is generally used only on Linux desktop systems to allow non-root users to perform actions, such as rebooting the system, installing packages, restarting services, as governed by a policy.

For instructions and more details, see the GCP-2022-004 security bulletin.

1.7
Security

Security bulletin (all minor versions)

A security vulnerability, CVE-2021-4034, has been discovered in pkexec, a part of the Linux policy kit package (polkit), that allows an authenticated user to perform a privilege escalation attack. PolicyKit is generally used only on Linux desktop systems to allow non-root users to perform actions, such as rebooting the system, installing packages, restarting services, as governed by a policy.

For instructions and more details, see the GCP-2022-004 security bulletin.

1.8
Security

Security bulletin (all minor versions)

A security vulnerability, CVE-2021-4034, has been discovered in pkexec, a part of the Linux policy kit package (polkit), that allows an authenticated user to perform a privilege escalation attack. PolicyKit is generally used only on Linux desktop systems to allow non-root users to perform actions, such as rebooting the system, installing packages, restarting services, as governed by a policy.

For instructions and more details, see the GCP-2022-004 security bulletin.

1.9
Security

Security bulletin (all minor versions)

A security vulnerability, CVE-2021-4034, has been discovered in pkexec, a part of the Linux policy kit package (polkit), that allows an authenticated user to perform a privilege escalation attack. PolicyKit is generally used only on Linux desktop systems to allow non-root users to perform actions, such as rebooting the system, installing packages, restarting services, as governed by a policy.

For instructions and more details, see the GCP-2022-004 security bulletin.

February 01, 2022

1.8
Feature

Release 1.8.8

Anthos clusters on bare metal 1.8.8 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.8.8 runs on Kubernetes 1.20.

1.8
Fixed

Fixes:

1.8
Issue

Known issues:

For information about the latest known issues, see Anthos on bare metal known issues in the Troubleshooting section.

January 27, 2022

1.10 
Feature

Release 1.10.1

Anthos clusters on bare metal 1.10.1 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.10.1 runs on Kubernetes 1.21.

1.9
Feature

Release 1.9.4

Anthos clusters on bare metal 1.9.4 is now available for download. To upgrade, see Upgrade Anthos on bare metal. Anthos clusters on bare metal 1.9.4 runs on Kubernetes 1.21.

1.9
Issue

Known issues:

When you upgrade Anthos clusters on bare metal from a version with a security patch to the next minor release, we recommend you upgrade to the highest patch version to ensure you have the latest security fixes. Always review the release notes before upgrading so you're aware of what has changed, including security fixes and known issues. Upgrading to a lower release version isn't supported.

For information about the latest known issues, see Anthos on bare metal known issues in the Troubleshooting section.

1.9
Fixed

Fixes:

December 22, 2021

1.9
Feature

Release 1.9.3

Anthos clusters on bare metal 1.9.3 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.9.3 runs on Kubernetes 1.21.

1.9
Fixed

Fixes:

  • Fixed an issue in which cluster creation fails if a cluster has more than one control plane node, and the HTTPS_PROXY environment variable has been defined on one or more of the control plane nodes.

  • Upgraded Kubernetes version from 1.21.4 to 1.21.5 to address an error in which pods become stuck in the ContainerCreating state because libcontainer mistakenly throws a "unit already exists" error.

  • The following container image security vulnerability has been fixed:

1.9
Issue

Known issues:

For information about the latest known issues, see Anthos on bare metal known issues in the Troubleshooting section.

December 14, 2021

1.8
Feature

Release 1.8.7

Anthos clusters on bare metal 1.8.7 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.8.7 runs on Kubernetes 1.20.

1.8
Fixed

Fixes:

The following container image security vulnerability has been fixed:

1.8
Issue

Known issues:

For information about the latest known issues, see Anthos on bare metal known issues in the Troubleshooting section.

December 09, 2021

1.7
Feature

Release 1.7.7

Anthos clusters on bare metal 1.7.7 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.7.7 runs on Kubernetes 1.19.

1.7
Fixed

Fixes:

  • The 1.7.6 release has a known issue that blocks upgrades of 1.7.5 clusters. The 1.7.7 release allows you to upgrade from all earlier versions to get the latest security fixes.

  • The following container image security vulnerabilities have been fixed:

1.7
Issue

Known issues:

For information about the latest known issues, see Anthos on bare metal known issues in the Troubleshooting section.

November 29, 2021

1.8
Feature

Release 1.8.6

Anthos clusters on bare metal 1.8.6 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.8.6 runs on Kubernetes 1.20.

1.8
Fixed

Fixes:

The following container image security vulnerabilities have been fixed:

1.8
Issue

Known issues:

For information about the latest known issues, see Anthos on bare metal known issues in the Troubleshooting section.

November 15, 2021

1.7
Feature

Release 1.7.6

Anthos clusters on bare metal 1.7.6 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.7.6 runs on Kubernetes 1.19.

1.9
Feature

Release 1.9.2

Anthos clusters on bare metal 1.9.2 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.9.2 runs on Kubernetes 1.21.

1.9
Fixed

Fixes:

  • Updated preflight checks to use strict mode for decoding YAML to catch issues with indentation and misplaced fields in the cluster configuration file.

  • Fixed an issue that caused containerRuntime to default to docker, instead of containerd in certain uncommon situations.

  • Fixed an issue where node_filesystem metrics report incorrect size in Cloud Monitoring for mount-points other than root.

  • Fixed an issue that caused communication failures between Cloud Logging metadata agent and the Cloud Monitoring API when the root certificate authority (CA) on the host node isn't set up properly.

  • The following container image security vulnerabilities have been fixed:

1.7
Fixed

Fixes:

The following container image security vulnerabilities have been fixed:

1.7
Issue

Known issues:

When you upgrade Anthos clusters on bare metal from a version with a security patch to the next minor release, we recommend you upgrade to the highest patch version to ensure you have the latest security fixes. Always review the release notes before upgrading so you're aware of what has changed, including security fixes and known issues. Upgrading to a lower release version isn't supported.

For information about the latest known issues, see Anthos on bare metal known issues in the Troubleshooting section.

1.9
Issue

Known issues:

For information about the latest known issues, see Anthos on bare metal known issues in the Troubleshooting section.

October 29, 2021

1.6
Security

Security bulletin (all minor versions)

The security community recently disclosed a new security vulnerability CVE-2021-30465 found in runc that has the potential to allow full access to a node filesystem.

For more information, see the GCP-2021-011 security bulletin.

1.7
Security

Security bulletin (all minor versions)

The security community recently disclosed a new security vulnerability CVE-2021-30465 found in runc that has the potential to allow full access to a node filesystem.

For more information, see the GCP-2021-011 security bulletin.

1.8
Security

Security bulletin (all minor versions)

The security community recently disclosed a new security vulnerability CVE-2021-30465 found in runc that has the potential to allow full access to a node filesystem.

For more information, see the GCP-2021-011 security bulletin.

1.9
Security

Security bulletin (all minor versions)

The security community recently disclosed a new security vulnerability CVE-2021-30465 found in runc that has the potential to allow full access to a node filesystem.

For more information, see the GCP-2021-011 security bulletin.

October 26, 2021

1.9
Feature

Release 1.9.1

Anthos clusters on bare metal 1.9.1 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.9.1 runs on Kubernetes 1.21.

1.9
Fixed

Fixes:

1.9
Change

Functionality changes:

  • Updated the bmctl reset cluster command to prevent you from resetting an admin cluster if the admin cluster is managing user clusters.
  • Updated the bmctl create cluster command to block you from enabling the Anthos VM Runtime for admin clusters.
1.9
Issue

Known issues:

For information about the latest known issues, see Anthos on bare metal known issues in the Troubleshooting section.

October 21, 2021

1.6
Security

Security bulletin (all minor versions)

A security issue was discovered in the Kubernetes ingress-nginx controller, CVE-2021-25742. Ingress-nginx custom snippets allows retrieval of ingress-nginx service account tokens and secrets across all namespaces. For more information, see the GCP-2021-024 security bulletin.

1.7
Security

Security bulletin (all minor versions)

A security issue was discovered in the Kubernetes ingress-nginx controller, CVE-2021-25742. Ingress-nginx custom snippets allows retrieval of ingress-nginx service account tokens and secrets across all namespaces. For more information, see the GCP-2021-024 security bulletin.

1.8
Security

Security bulletin (all minor versions)

A security issue was discovered in the Kubernetes ingress-nginx controller, CVE-2021-25742. Ingress-nginx custom snippets allows retrieval of ingress-nginx service account tokens and secrets across all namespaces. For more information, see the GCP-2021-024 security bulletin.

1.8
Feature

Release 1.8.5

Anthos clusters on bare metal 1.8.5 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.8.5 runs on Kubernetes 1.20.

1.9
Security

Security bulletin (all minor versions)

A security issue was discovered in the Kubernetes ingress-nginx controller, CVE-2021-25742. Ingress-nginx custom snippets allows retrieval of ingress-nginx service account tokens and secrets across all namespaces. For more information, see the GCP-2021-024 security bulletin.

1.8
Fixed

Fixes:

The following container image security vulnerabilities have been fixed:

1.8
Issue

Known issues:

For information about the latest known issues, see Anthos on bare metal known issues in the Troubleshooting section.

October 19, 2021

1.7
Issue

Known issues:

When you upgrade Anthos clusters on bare metal from a version with a security patch to the next minor release, we recommend you upgrade to the highest patch version to ensure you have the latest security fixes. Always review the release notes before upgrading so you're aware of what has changed, including security fixes and known issues. Upgrading to a lower release version isn't supported.

For information about the latest known issues, see Anthos on bare metal known issues in the Troubleshooting section.

1.7
Feature

Release 1.7.5

Anthos clusters on bare metal 1.7.5 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.7.5 runs on Kubernetes 1.19.

1.7
Fixed

Fixes:

The following container image security vulnerabilities have been fixed:

October 04, 2021

1.6
Security

Security bulletin (all minor versions)

A security vulnerability, CVE-2020-8561, has been discovered in Kubernetes where certain webhooks can be made to redirect kube-apiserver requests to private networks of that API server.

For more information, see the GCP-2021-021 security bulletin.

1.7
Security

Security bulletin (all minor versions)

A security vulnerability, CVE-2020-8561, has been discovered in Kubernetes where certain webhooks can be made to redirect kube-apiserver requests to private networks of that API server.

For more information, see the GCP-2021-021 security bulletin.

1.8
Security

Security bulletin (all minor versions)

A security vulnerability, CVE-2020-8561, has been discovered in Kubernetes where certain webhooks can be made to redirect kube-apiserver requests to private networks of that API server.

For more information, see the GCP-2021-021 security bulletin.

1.9
Security

Security bulletin (all minor versions)

A security vulnerability, CVE-2020-8561, has been discovered in Kubernetes where certain webhooks can be made to redirect kube-apiserver requests to private networks of that API server.

For more information, see the GCP-2021-021 security bulletin.

September 28, 2021

1.9
Feature

Release 1.9.0

Anthos clusters on bare metal 1.9.0 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.9.0 runs on Kubernetes 1.21.

1.9
Feature

Improved cluster lifecycle functionalities:

  • Preview: Added ability to reset individual nodes with the bmctl reset node command. To give access to the needed cluster configuration file, use the command with the -c flag.

  • Preview: Added ability to recover from HA control plane quorum loss withbmctl restore --control-plane-node command.

  • Added bmctl create ksa command to create a Kubernetes Service Account (KSA) and generate a bearer token. To log in to the registered cluster, you can use the token in Cloud Console Kubernetes Engine > Clusters.

  • Preview: Added bmctl backup cluster and bmctl restore cluster commands to facilitate disaster recovery for clusters.

Introduced new troubleshooting capabilities:

  • Updated the bmctl check cluster --snapshot command to support uploading cluster diagnostic snapshots to a Cloud Storage bucket for review by Cloud Customer Care.

  • Provided access to bootstrap cluster logs to help troubleshoot cluster creation or upgrade problems.

  • Preview: Added support for Node Problem Detector service on nodes for quick detection of common node problems.

Enhanced monitoring and logging:

  • GA: Cloud Audit Logs capability is now generally available and enabled by default. Audit logs are useful for investigating suspicious API requests and for collecting statistics. For more information, see Use Audit Logging.

  • Switched to new open telemetry-based metrics agents to improve reliability, ability to scale, and resource usage.

Improved networking capabilities:

  • GA: The multi-NIC capability to provide additional interfaces to your pods is now generally available.

  • Preview: Added the single root I/O virtualization (SR-IOV) container network interface (CNI) plugin for multi-NIC.

  • Added support to configure cluster Domain Name System (DNS) provider options, such as upstream nameservers, with the new ClusterDNS custom resource definition.

Enhanced security:

  • SELinux is now always enabled in the container runtime for CentOS and RHEL.

  • Preview: Enhanced the capability to rotate cluster certificate authorities (CAs). Updates include support for all cluster types, rotation of front-proxy and etcd CAs, and changes to the bmctl command syntax.

  • Preview: Added Okta group support for authentication in Anthos Identity Service.

1.9
Change

Functionality changes:

  • Changed default container runtime to containerd, containerRuntime: containerd for new clusters. Customers can still choose Docker as the container runtime.

  • Preview: Updated bmctl command, bmctl reset nodes --force, to support force removal of control plane nodes with etcd membership cleanup.

  • Added checks for cluster updates to verify access to cluster machines if changes to loginUser or sshKeyPrivatePath are detected. If the checks pass, Anthos clusters on bare metal saves the secret in the cluster.

  • Added new Anthos cluster control plane uptime dashboard in Cloud Monitoring with new metric kubernetes.io/anthos/container/uptime for component availability.

  • Added new alerts for control plane components availability with new metric kubernetes.io/anthos/container/uptime to replace deprecated alerts with metric kubernetes.io/anthos/up.

1.9
Fixed

Fixes:

  • Added missing registry mirror package required for Cloud Audit Logs to the Registry Mirror.

  • Fixed issue with containerd not finding crictl due to /usr/local/bin not being in the SSH user's PATH.

  • Fixed flapping node readiness issues caused by an unhealthy Pod Lifecycle Event Generator (PLEG).

  • Fixed kernel support issue for Ubuntu 18.04 and 18.04.1 that prevented the anetd networking controller from working properly. Anthos clusters on bare metal release 1.9.0 works with all kernels supplied with supported distributions.

1.9
Issue

Known issues:

  • Control group v2 (cgroup v2) is not officially supported in Anthos clusters on bare metal release 1.9.0 and later. The presence of /sys/fs/cgroup/cgroup.controllers indicates that your system uses cgroup v2.

  • Anthos Service Mesh v1.10 is incompatible with Anthos clusters on bare metal release 1.9.0 running on Red Hat Enterprise Linux (RHEL) when SELinux is enabled. If you want to use Anthos Service Mesh, you must disable SELinux or set it to permissive mode on the host.

For information about the latest known issues, see Anthos on bare metal known issues in the Troubleshooting section.

September 21, 2021

1.8
Feature

Release 1.8.4

Anthos clusters on bare metal 1.8.4 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.8.4 runs on Kubernetes 1.20.

1.8
Fixed

Fixes:

The following container image security vulnerabilities have been fixed: - CVE-2021-3711 - CVE-2021-3712 - CVE-2021-20305 - CVE-2021-33560

1.8
Issue

Known issues:

When you upgrade Anthos clusters on bare metal from a version with a security patch to the next minor release, we recommend you upgrade to the highest patch version to ensure you have the latest security fixes. Always review the release notes before upgrading so you're aware of what has changed, including security fixes and known issues. Upgrading to a lower release version isn't supported.

For information about the latest known issues, see Anthos clusters on bare metal known issues in the Troubleshooting section.

September 20, 2021

1.7
Feature

Release 1.7.4

Anthos clusters on bare metal release 1.7.4 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.7.4 runs on Kubernetes 1.19.

1.7
Fixed

Fixes:

  • Fixed vulnerability CVE-2021-25741 that might allow users to create a container with subpath volume mounts to access files and directories outside of the volume, including on the host filesystem. This vulnerability affects all clusters created or upgraded with Anthos clusters on bare metal release 1.7.0. For more information, see the GCP-2021-018 security bulletin.

  • Updated the Kubernetes patch version to address the following container image security vulnerabilities:

1.7
Security

Security bulletin (1.7 and 1.8)

A security issue was discovered in Kubernetes, CVE-2021-25741, where a user may be able to create a container with subpath volume mounts to access files and directories outside of the volume, including on the host filesystem. This vulnerability affects all clusters created or upgraded with Anthos clusters on bare metal 1.7.x and 1.8.x releases, specifically 1.7.3 and earlier and 1.8.2 and earlier.

To fix this vulnerability, upgrade your Anthos clusters to version 1.7.4 or 1.8.3. For more information, see the GCP-2021-018 security bulletin.

1.8
Security

Security bulletin (1.7 and 1.8)

A security issue was discovered in Kubernetes, CVE-2021-25741, where a user may be able to create a container with subpath volume mounts to access files and directories outside of the volume, including on the host filesystem. This vulnerability affects all clusters created or upgraded with Anthos clusters on bare metal 1.7.x and 1.8.x releases, specifically 1.7.3 and earlier and 1.8.2 and earlier.

To fix this vulnerability, upgrade your Anthos clusters to version 1.7.4 or 1.8.3. For more information, see the GCP-2021-018 security bulletin.

1.7
Issue

Known issues:

When you upgrade Anthos clusters on bare metal from a version with a security patch to the next minor release, we recommend you upgrade to the highest patch version to ensure you have the latest security fixes. Always review the release notes before upgrading so you're aware of what has changed, including security fixes and known issues. Upgrading to a lower release version isn't supported.

For information about the latest known issues, see Anthos on bare metal known issues in the Troubleshooting section.

August 27, 2021

1.8
Feature

Release 1.8.3

Anthos clusters on bare metal 1.8.3 is now available. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.8.3 runs on Kubernetes 1.20.

1.8
Breaking

Breaking changes:

In Anthos clusters on bare metal release 1.8.0, we added a kernel version requirement for Ubuntu 18.04. We required a Linux kernel version of 4.17.0 or later. Anthos clusters on bare metal release 1.8.3 again supports all Linux kernel versions that ship with Ubuntu 18.04 and 20.04 distributions. As a result of this change, however, the egress NAT gateway feature that was provided for Preview in release 1.8.0 does not work with Anthos clusters on bare metal release 1.8.3.

1.8
Feature

Features:

  • Preview: Anthos Identity Service now works with Anthos clusters on bare metal to support LDAP authentication methods in addition to OIDC. You can use AIS with Microsoft Active Directory without the need for provisioning Active Directory Federation Services. For more information, see Setting up Anthos Identity Service with LDAP.

  • Preview: Anthos Metadata Agent replaces Stackdriver Metadata Collector and collects more accurate and usable metadata for Kubernetes resources. When you configure logging and monitoring, you need to enable the Config Monitoring for Ops API and grant the opsconfigmonitoring.resourceMetadata.writer IAM role to your logging-monitoring service account. If Anthos clusters on bare metal is installed behind a proxy, your proxy server must also allow connections to opsconfigmonitoring.googleapis.com.

  • Added preflight checks to verify that specific APIs are enabled for your Google Cloud project. Preflight checks return an error if any of the following APIs aren't enabled for your project:

    • anthos.googleapis.com
    • anthosaudit.googleapis.com
    • anthosgke.googleapis.com
    • cloudresourcemanager.googleapis.com
    • gkeconnect.googleapis.com
    • gkehub.googleapis.com
    • iam.googleapis.com
    • opsconfigmonitoring.googleapis.com
    • logging.googleapis.com
    • monitoring.googleapis.com
    • stackdriver.googleapis.com

    To enable these APIs when you create a cluster configuration file, use the --enable-apis flag with the bmctl create config command. For an example that uses the --enable-apis flag, see Create an admin cluster config with bmctl.

  • Added preflight checks for the following machine requirements:

    • Minimum supported Linux kernel version
    • Minimum required CPU
    • Minimum required RAM
1.8
Fixed

Fixes:

  • Fixed the following container image security vulnerabilities:
  • Fixed cluster creation and cluster update failures for nodes running CentOS or Red Hat Enterprise Linux (RHEL) with both SELinux and Cloud Audit Logs enabled.
  • Fixed Transmission Control Protocol (TCP) connection leakage issue.
  • Fixed an issue that prevented cert-manager from issuing ACME certificates over HTTP due to ImagePullBackOff errors.
1.8
Change

Changes:

  • The Kubevirt version used for working with VM-based workloads is now v0.43.0-gke.3.
  • The bootstrap cluster is deleted when a cluster upgrade completes without errors.

For information about the latest known issues, see Anthos on bare metal known issues in the Troubleshooting section.

August 16, 2021

1.7
Feature

Release 1.7.3

Anthos clusters on bare metal 1.7.3 is now available. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.7.3 runs on Kubernetes 1.19.

1.7
Fixed

Fixes:

The following container image security vulnerabilities have been fixed:

1.7
Issue

Known issues:

When you upgrade Anthos clusters on bare metal from a version with a security patch to the next minor release, we recommend you upgrade to the highest patch version to ensure you have the latest security fixes. Always review the release notes before upgrading so you're aware of what has changed, including security fixes and known issues. Upgrading to a lower release version isn't supported.

For information about the latest known issues, see Anthos on bare metal known issues in the Troubleshooting section.

August 13, 2021

1.6
Feature

Release 1.6.4

Anthos clusters on bare metal 1.6.4 is now available. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.6.4 runs on Kubernetes 1.18.

1.6
Fixed

Fixes:

The following container image security vulnerabilities have been fixed:

1.6
Issue

Known issues:

When you upgrade Anthos clusters on bare metal from a version with a security patch to the next minor release, we recommend you upgrade to the highest patch version to ensure you have the latest security fixes. Always review the release notes before upgrading so you're aware of what has changed, including security fixes and known issues. Upgrading to a lower release version isn't supported.

For information about the latest known issues, see Anthos on bare metal known issues in the Troubleshooting section.

July 29, 2021

1.8
Feature

Features:

  • Preview: Added capability to rotate cluster certificate authorities (CAs) for user clusters. For instructions on using the bmctl cluster credentials command to rotate cluster CAs, see Rotate user cluster certificate authority.

  • Preview: Added support for AppArmor with Anthos clusters on bare metal. You don't need to disable AppArmor on Ubuntu as a prerequisite for installation. When you create new 1.8.2 clusters or upgrade clusters to version 1.8.2, you can enable AppArmor either before or after you upgrade.

1.8
Feature

Release 1.8.2

Anthos clusters on bare metal 1.8.2 is now available. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.8.2 runs on Kubernetes 1.20.

1.8
Fixed

Fixes:

  • Fixed CVE-2021-3520 vulnerability related to a flaw in lz4, which provides support for LZ4 a lossless compression algorithm. The flaw impacts availability, but has potential to impact confidentiality and integrity as well.

  • Fixed bmctl operation failures that occur for some Ubuntu 20.04 LTS distributions with a more recent Linux kernel, including Ubuntu 20.04 LTS images on the 5.8 kernel. For more information about this issue and a workaround, see Ubuntu 20.04 LTS and bmctl.

  • Fixed OpenStack support for user clusters. In prior releases, cluster creation fails for user type clusters when the baremetal.cluster.gke.io/external-cloud-provider: "true" annotation is added to the cluster configuration file.

  • Fixed PATH environment issues for executing commands as a non-root user. For more information, see Known Issues.

  • Fixed an issue that caused user cluster resets (bmctl reset cluster) to get stuck while deleting namespaces.

  • Fixed out-of-memory (OOM) conditions related to Connect Agent memory usage that resulted in pod failures.

  • Fixed issue that blocked snapshots for clusters configured for passwordless SUDO capability for machine login (nodeAccess.loginUser: <login user name>).

  • Fixed issue that blocked some 1.7.x version admin, hybrid, or standalone clusters from upgrading to the 1.8 minor release. This issue affected some clusters that were updated by applying changes from an updated cluster configuration file.

  • Fixed Address Resolution Protocol (ARP) table issue for high-availability (HA) deployments that blocked upgrades from completing.

1.8
Change

Functionality changes:

  • Expanded snapshots to include resource usage metrics to improve troubleshooting and support. Added metrics include the output of ip neigh, kubectl top nodes, and kubectl top pods commands.

For information about the latest known issues, see Anthos on bare metal known issues in the Troubleshooting section.

July 02, 2021

1.8
Feature

Release 1.8.1

Anthos clusters on bare metal release 1.8.1 is now available. To upgrade, see Upgrading Anthos on bare metal. Anthos on bare metal 1.8.1 runs on Kubernetes 1.20.

1.8
Fixed

Fixes:

Fixed CVE-2021-34824 that could expose private keys and certificates from Kubernetes secrets through the credentialName field when using Gateway or DestinationRule. This vulnerability affects all clusters created or upgraded with Anthos clusters on bare metal release 1.8.0. For more information, see the GCP-2021-012 security bulletin.

For information about the latest known issues, see Anthos on bare metal known issues in the Troubleshooting section.

June 30, 2021

1.8
Security

Security bulletin (1.8)

The Istio project recently announced a security vulnerability (CVE-2021-34824) where credentials specified in the credentialName field for Gateway or DestinationRule can be accessed from different namespaces. This vulnerability affects all clusters created or upgraded with Anthos clusters on bare metal release 1.8.0. For more information, see the GCP-2021-012 security bulletin.

June 21, 2021

1.8
Feature

Release 1.8.0

Anthos clusters on bare metal release 1.8.0 is now available. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.8.0 runs on Kubernetes 1.20.

1.8
Feature

Extended installation support:

  • Provided support to use containerd as the container runtime as GA for Anthos clusters on bare metal release 1.8.0. Cluster upgrades to 1.8.0 are blocked for 1.7.x clusters that are configured to use the preview containerd capability. For more information, see Upgrading 1.7.x clusters that use containerd in Known Issues.
  • Preview: Improved virtual machine (VM) management capability. Anthos VM Runtime uses KubeVirt to orchestrate VMs on clusters, allowing you to work with your VM-based apps and workloads in a uniform development environment. Anthos VM Runtime has worked with Anthos clusters on bare metal as a preview feature since November 2020 and we have continued to enhance its capability. For more information, see Working with VM-based workloads.
  • Added edge profile support for standalone clusters. The edge profile is recommended for edge devices with limited resources. Add profile: edge to the cluster config file when you create a standalone cluster to produce a cluster that has significantly reduced system resource requirements. The edge profile is only available for standalone clusters, it is ignored for other cluster types. For more information, see Creating standalone clusters.
  • Added support to specify provider ID for Nodes (controlPlane.nodePoolSpec.nodes.providerID) to support deploying on OpenStack using Load Balancing as a Service (LBaaS) resources. For more information, see Configure your clusters to use OpenStack.
  • Preview: Added support for installing Anthos clusters on bare metal, using your own registry service, instead of gcr.io. For instructions and additional information, see Installing Anthos Bare Metal using registry mirror.

Improved upgrade:

  • Enabled support for upgrading non-SELinux clusters to SELinux. For more information, see Enable SELinux in Upgrading Anthos clusters on bare metal.
  • Cluster upgrades are not blocked by excessive Node draining durations. During a cluster upgrade, if the draining process takes longer than 20 minutes for any specific Node, the upgrade process will carry on without waiting for draining to complete.

Updated user cluster lifecycle management:

  • Added bmctl improvements for resetting user cluster and adding additional preflight checks to confirm machine and network readiness for cluster creation:

Enhanced monitoring and logging:

  • Preview: Added Cloud Audit Logging capability, which enables audit logs to be written to Cloud Audit Logs in your Google project. Audit logs are useful for investigating suspicious API requests and for collecting statistics. For more information, see Enable Audit Logging.

Introduced new networking capabilities in preview:

  • Preview: Added multi-NIC capability to provide additional interfaces to your Pods.
  • Preview: Added egress NAT gateway capability to provide persistent, deterministic routing for the egress traffic from your clusters. For more information, see Configure an egress NAT gateway for external communication.
  • Preview: Added option for BGP bundled load balancer for Layer-3 (L3) topologies. This feature can be used with user clusters and admin clusters.

Enhanced security:

  • Workload Identity is GA. The Connect Agent Service Account Key is no longer required during installation. Connect Agent uses Workload Identity to authenticate to Google Cloud instead of an exported Google Cloud Service Account Key.

Expanded support for newer versions of operating systems:

  • Added support for installing Anthos clusters on bare metal on Red Hat Enterprise Linux (RHEL) 8.4, and CentOS 8.4
1.8
Change

Functionality changes:

  • Added --workspace-dir flag to bmctl to allow changing the path and name of the workspace directory from the default bmctl-workspace. The workspace directory contains the configuration and log files generated by bmctl. When using the bmctl command, pass in a --workspace-dir flag to specify a non-default workspace directory location. If the directory does not exist, bmctl will create it for you.
  • Moved away from iptables-based NodePort and masquerade handling to eBPF-based management. NodePort and masquerade handling are now applied to the Node IP and default gateway interfaces only.
1.8
Fixed

Fixes:

  • Resolved, as part of the GA support for using containerd as the container runtime, incorrect cgroup driver use. Newly created 1.8.0 clusters that are configured to use containerd will use the correct systemd cgroup driver.
  • Fixed issue that prevented usage metrics for the containerd process from being collected by Cloud Logging. This fix applies to newly created 1.8.0 clusters only.
1.8
Issue

Known issues:

  • If a Node is out of reach, Anthos clusters on bare metal can't start the draining process, which may impact the cluster upgrade process. For more information, see Node draining can't start when Node is out of reach.
  • Upgrading from 1.7.x clusters that use containerd as the container runtime to 1.8.0 is blocked. For more information, see Upgrading 1.7.x clusters that use containerd.
  • When running Anthos clusters on bare metal with firewalld enabled on either CentOS or Red Hat Enterprise Linux (RHEL), changes to firewalld can remove the Cilium iptables chains on the host network. The loss of the Cilium iptables chains causes the Pod on the Node to lose network connectivity outside of the Node. for more information, see Modifying firewalld will erase Cilium iptable chains.

For information about the latest known issues, see Anthos on bare metal known issues in the Troubleshooting section.

June 02, 2021

1.7
Feature

Release 1.7.2

Anthos clusters on bare metal release 1.7.2 is now available. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.7.2 runs on Kubernetes 1.19.

1.7
Fixed

Fixes:

  • Fixed CVE-2021-25735 that could allow node updates to bypass a Validating Admission Webhook. For more details, open the Anthos clusters on bare metal tab of the GCP-2021-003 security bulletin.
  • Resolved the bmctl snapshot command failure when the user creates a custom cluster namespace omitting "cluster-" prefix from the cluster config file. The prefix is no longer required for a custom cluster namespace.
  • Added webhook blocks to prevent users from modifying control plane node pool and load balancer node pool resources directly. Control plane and load balancer node pools for Anthos clusters on bare metal are specified in the cluster resource, using the spec.controlPlane.nodePoolSpec and spec.LoadBalancer.nodePoolSpec sections of the cluster config file respectively.
  • Fixed the cluster upgrade command, bmctl upgrade cluster, to prevent it from interfering with user-installed Anthos Service Mesh (ASM).
1.7
Change

Functionality changes:

  • Updated the bmctl check snapshot command so that it includes certificate signing requests in the snapshot.
  • Changed the upgrade process to prevent node drain issues from blocking upgrades. The upgrade process triggers a node drain. Now, if the node drain takes longer than 20 minutes, the upgrade process carries on to completion even when the draining hasn't completed. In this case, the upgrade output reports the incomplete node drain. Excessive drain times signal a problematic with pods. You may need to restart problem pods.
  • Updated cluster creation process, bmctl create cluster, to display logged errors directly on the command line. Prior to this release, detailed error messages were only available in the log files.
1.7
Issue

Known issues:

  • Node logs from nodes with a dot (".") in their name are not exported to Cloud Logging. For workaround instructions, see Node logs aren't exported to Cloud Logging in Anthos clusters on bare metal known issues.

For information about the latest known issues, see Anthos clusters on bare metal known issues in the Troubleshooting section.

May 17, 2021

1.6
Fixed

Fixed:

1.6
Issue

Known issues:

When you upgrade Anthos clusters on bare metal from a version with a security patch to the next minor release, we recommend you upgrade to the highest patch version to ensure you have the latest security fixes. Always review the release notes before upgrading so you're aware of what has changed, including security fixes and known issues. Upgrading to a lower release version isn't supported.

For information about the latest known issues, see Anthos clusters on bare metal known issues in the Troubleshooting section.

May 06, 2021

1.6
Security

Security bulletin (1.6 and 1.7)

The Envoy and Istio projects recently announced several new security vulnerabilities (CVE-2021-28683, CVE-2021-28682, and CVE-2021-29258) that could allow an attacker to crash Envoy.

For more information, see the GCP-2021-004 security bulletin.

1.7
Security

Security bulletin (1.6 and 1.7)

The Envoy and Istio projects recently announced several new security vulnerabilities (CVE-2021-28683, CVE-2021-28682, and CVE-2021-29258) that could allow an attacker to crash Envoy.

For more information, see the GCP-2021-004 security bulletin.

April 30, 2021

1.7
Feature

Release 1.7.1

Anthos clusters on bare metal release 1.7.1 is now available. To upgrade, see Upgrading Anthos clusters on bare metal. Anthos clusters on bare metal 1.7.1 runs on Kubernetes 1.19.

1.7
Change

Functionality changes:

  • Customers can now take cluster snapshots regardless of whether the admin cluster control plane is running. This is helpful for diagnosing installation issues.
  • Deploying Anthos clusters on bare metal with SELinux is now fully supported on supported versions of Redhat Enterprise Linux. This applies for new installations of Anthos clusters on bare metal cases only.
  • User cluster creation with bmctl supports credential inheritance from the admin cluster by default. Credential overrides for the user cluster can be specified in the config file during cluster creation.
1.7
Fixed

Fixes:

  • (Updated May 12, 2021) Fixed CVE-2021-28683, CVE-2021-28682, CVE-2021-29258. For more details, see the GCP-2021-004 security bulletin.
  • Fixed potential stuck upgrade from 1.6.x to 1.7.0. The bug was caused by a rare race condition when the coredns configmap failed to be backed up and restored during the upgrade.
  • Fixed potential missing GKE connect agent during installation due to a rare race condition.
  • Fixed issue that prevented automatic updates to the control plane load balancer config when adding/removing node(s) from the control plane node pool.
  • Addressed problem with syncing NodePool taints and labels that resulted in deletion of pre-existing items. Syncs will now append, update, or delete items that are added by taints and labels themselves only.
1.7
Issue

Known issues:

  • Upgrading the container runtime from containerd to Docker will fail in Anthos clusters on bare metal release 1.7.1. This operation is not supported while the containerd runtime option is in preview.
  • bmctl snapshot command fails when the user creates a custom cluster namespace omitting cluster- prefix from the cluster config file. To avoid this issue, the cluster namespace should follow the cluster-$CLUSTER_NAME naming convention.

For information about the latest known issues, see Anthos on bare metal known issues in the Troubleshooting section.

April 20, 2021

1.7
Security

Security bulletin (1.6 and 1.7)

The Kubernetes project recently announced a new security vulnerability, CVE-2021-25735, that could allow node updates to bypass a Validating Admission Webhook. For more details, see the GCP-2021-003 security bulletin.

March 25, 2021

1.7
Feature

Release 1.7.0

Anthos clusters on bare metal release 1.7.0 is now available. To upgrade, see Upgrading Anthos on bare metal. Anthos on bare metal 1.7.0 runs on Kubernetes 1.19.

Extended installation support:

  • Added requirement for Anthos clusters on bare metal connectivity with Google Cloud for install and upgrade operations. As of 1.7.0 preflight checks will check for connectivity to Google Cloud, enabled APIs, and permissions for service accounts. Existing clusters need to be registered in Google Cloud before upgrading. The connectivity checks are not overridable by the --force flag. For details, see the cluster creation and cluster upgrade documentation.

  • Added support for installing Anthos clusters on bare metal on OpenStack. For configuration instructions, see Configure your clusters to use OpenStack.

  • Added support for installing Anthos clusters on bare metal, using a private package repository instead of the default Docker APT repository. For instructions and additional information, see Use a private package repository server.

  • Removed installation prerequisite for setting Security-Enhanced Linux (SELinux) operational mode to be permissive. The related preflight check has been removed, as well.

  • Removed installation prerequisite for disabling firewalld . The related preflight check has also been removed. For information on configuring ports to use firewalld with Anthos clusters on bare metal, see Configuring firewalld ports on the Network requirements page.

  • Updated requirements for installing behind a proxy server and removed restriction on system-wide proxy configurations. For a detailed list of prerequisites, see Installing behind a proxy.

Improved upgrade:

  • Updated cluster upgrade routines to ensure worker node failures do not block cluster upgrades, providing a more consistent user experience. Control plane node failures will still block cluster upgrades.

  • Added bmctl support for running upgrade preflight checks. bmctl check preflight will run upgrade preflight checks if users specify the --kubeconfig flag. For example:
    bmctl check preflight --kubeconfig bmctl-workspace/cluster1/cluster1-kubeconfig

Updated user cluster lifecycle management:

  • Added support in bmctl for user cluster creation and upgrade functions.

  • Improved resource handling. Anthos clusters on bare metal now reconciles node pool taints and labels to nodes unless the node has a baremetal.cluster.gke.io/label-taint-no-sync annotation.

Enhanced monitoring and logging:

  • Preview: Added out-of-the-box alerts for critical cluster metrics and events. For information on working with alerting policies and getting notified, see Creating alerting policies.

  • Added support for collecting ansible job logs in admin and hybrid clusters by default.

Expanded support for newer versions of operating systems:

  • Added support for installing Anthos clusters on bare metal on Red Hat Enterprise Linux (RHEL) 8.3 and CentOS 8.3.
1.7
Change

Functionality changes:

  • Added support for configuring the number of pods per node. New clusters can be configured to run up to 250 pods per node. For more information about configuring nodes, see Pod networking. You can find additional information for configuring pods in the cluster creation documentation.
  • Preview: Added support to use containerd as the container runtime. Anthos clusters on bare metal 1.6.x supports only Docker for container runtime (dockershim). In 1.7.0, Kubelet can be configured to use either Docker or containerd, using the new containerRuntime cluster config field. You must upgrade existing clusters to 1.7.0 to add or update the containerRuntime field.
  • Added support for more load balancer addressPool entries under cluster.spec.loadBalancer.addressPools. For existing addressPools, users can use cluster.spec.loadBalancer.AddressPools[].manualAssign specify additional addressPool entries.
1.7
Issue

Known issues:

  • Under rare circumstances, bmctl upgrade may become stuck at the Moving resources to upgraded cluster stage after finishing upgrading all nodes in the cluster. The issue does not affect cluster operation, but the final step needs to be finished.

    If bmctl does not move forward after 30 minutes in this state, re-run the bmctl upgrade command to complete the upgrade.

    The issue is captured in the upgrade-cluster.log file located in .../bmctl-workspace/<cluster name>/log/upgrade-cluster-<timestamp>. The following log entry shows how the failure is reported:

    Operation failed, retrying with backoff. Cause: error creating "baremetal.cluster.gke.io/v1, Kind=Cluster" <cluster name>: Internal error occurred: failed calling webhook "vcluster.kb.io": Post "https://webhook-service.kube-system.svc:443/validate-baremetal-cluster-gke-io-v1-cluster? timeout=30s": net/http: TLS handshake timeout

For information about the latest known issues, see Anthos on bare metal known issues in the Troubleshooting section.

February 26, 2021

1.6
Fixed

Fixes:

  • Updated custom resource API to reject changes to Cluster and NodePool configuration fields that are not currently supported. For a list of supported mutable fields, see Configuration in Known Issues.
  • Updated bmctl to allow creating or upgrading Anthos clusters on bare metal to the current bmctl version (1.6.2) only. For more information about version restrictions, see Installation in Known Issues.
  • Fixed an issue that caused the automatic reset of bare metal machines to fail after deleting the user cluster.
  • Added preflight check to verify that control group v2, or cgroup v2 for short, is not in use on the cluster machine. Anthos on bare betal 1.6.x is incompatible with cgroup v2. For more information, see Control group v2 incompatibility in Known Issues.
  • Updated csi-snapshot-validation-webhook to support certification rotation. For more information about certificate rotation, see Security in Known Issues.
  • Fixed an issue to prevent constant patching for snapshot.storage.k8s.io CRDs.
  • Fixed a Certificate Signing Request (CSR) issue with kubelet to ensure fully qualified domain name(FQDN) hostnames are supported.

For information about the latest known issues, see Anthos on bare metal known issues in the Troubleshooting section.