## Hotfix 25

*** ** * ** ***

Announcement The 1.14.3-gdch.9425-25 hotfix is available.

*** ** * ** ***

Fixed This hotfix fixes the following issues:

**DNS**:

- There are failures in deploying `dns-mesh` probes when zones have duplicate names.
- DNS resolution takes a few minutes.
- There is a problem where deleting a `ManagedDNSZone` before removing its associated `ResourceRecordSets` results in orphaned records that persist in the system.

**Logging**:

- The `SyslogCollectorDroppedLogs` alert includes a fingerprint, which results in a new incident created each time that the alert fires.
- The pod logs in the `unet-system` namespace are missing.

**Managed Kubernetes Service**:

- Database instances are stuck in a deletion state caused by a race condition where virtual machines were deleted before their corresponding `InventoryMachine` resources.
- User cluster deletion is stuck due to a race condition during resource deletion.
- There is an issue with cluster scaling where downscaling a node pool followed by immediate upscaling can lead to IP address allocation failures, such as `unable to assign IP`.

**Virtual machines**:

- VM startups are delayed until a certificate is created.
- Random errors and delays occur when creating VMs in parallel. These issues manifest as CPI errors, agent timeouts, and network connectivity problems.
- Windows BYO image imports fail due to kubevirt version bump.

## Hotfix 24

*** ** * ** ***

Announcement The 1.14.3-gdch.9425-24 hotfix is available.

*** ** * ** ***

Fixed This hotfix fixes the following issues:

**Resource manager**:

- The project and project service account performance degrades over time.

## Hotfix 23

*** ** * ** ***

Announcement The 1.14.3-gdch.9425-23 hotfix is available.

*** ** * ** ***

Fixed This hotfix fixes the following issues:

**Identity and access management**:

- Role bindings are enqueued frequently in the global IAM controllers, causing a propagation delay when assigning permissions.

**Networking**:

- StatefulSet pod communication fails.
- There is a missing `node_cpu_seconds_total` metric.
- Networking controllers are stuck, resulting in load balancer and project network policy APIs not reconciling.

**OCLCM**:

- The oclcm configrunner stops working due to a certificate renewal failure.

## Hotfix 22

*** ** * ** ***

Announcement The 1.14.3-gdch.9425-22 hotfix is available.

*** ** * ** ***

Fixed This hotfix fixes the following issues:

**Identity and access management**:

- Login config request times can take minutes to complete.
- Application operators can't grant themselves access to roles in the infra cluster.
- Existing service accounts tokens become invalid.
- The `namespace-admin` project role is missing in a project.

**Managed Kubernetes Service**:

- There is a timeout waiting for a shared service cluster to become ready.

**Monitoring**:

- Project deletion is stuck due to dashboard and data source pending finalizers.
- Recurring usage fails to emit metrics.
- Metrics from KSM are not visible to PAs.
- Prometheus pods are scheduled on control plane nodes, leading to resource exhaustion and instability.
- KUB dashboards show incorrect data values.

## Hotfix 21

*** ** * ** ***

Announcement The 1.14.3-gdch.9425-21 hotfix is available.

*** ** * ** ***

Fixed This hotfix fixes the following issues:

**CLI**:

- The `gdcloud compute images import` command fails with QCOW images due to a file command version mismatch.
- The `gdcloud resource-support get-report` command produces a `failed to get support information from cluster` error.
- The `gdcloud auth activate-service-account` command resets the gdcloud config.
- Added `gdcloud get-credentials` support for vanilla clusters.
- Role bindings created with `gdcloud` bind to the platform namespace by default.
- The `gdcloud resource-support get-report` command fails with an error indicating a missing schema.
- Added SKUS to support more country-specific pricing.
- The gdcloud CLI doesn't use the correct zone when listing clusters, so the result of `gdcloud clusters list` shows one zone.

## Hotfix 20

*** ** * ** ***

Announcement The 1.14.3-gdch.9425-20 hotfix is available.

*** ** * ** ***

Fixed This hotfix fixes the following issues:

**Console**:

- Role bindings created for service accounts are not configured correctly.
- Project IAM Admins can't assign roles from the UI.
- When creating a custom role, the source role of a permission changes when you view it.
- The Roles Overview and Role Details pages show an error about not having permissions.
- The Access page is blocked even though users have some of the required permissions.

**Virtual machines**:

- The `vmm-vm-controller` subcomponent fails reconciliation due to a large config file exceeding the size limit (1MB).

## Hotfix 19

*** ** * ** ***

Announcement The 1.14.3-gdch.9425-19 hotfix is available.

*** ** * ** ***

Fixed This hotfix fixes the following issues:

**Backup and restore**:

- When deleting a snapshot in a user cluster, the corresponding snapshot in the infra cluster is not deleted.
- The backup subcomponent deployment uses a variable that prepends characters to the cluster name when creating a Kubernetes label which can sometimes violate the Kubernetes 63 character limit.

**Billing**:

- Added durable pricing SKUs.
- The prebuy calculator is not working.
- Recurring usage metrics are not emitted.
- Partner billing is not enabled.
- The prebuy calculator cannot access a community network.

**Identity and access management**:

- Service account creation fails due to a project in a deleting state causing role template reconciliation failures.
- Added P4SA support for vanilla clusters.
- Creating a service identity from the GDC console fails.

## Hotfix 18

*** ** * ** ***

Announcement The 1.14.3-gdch.9425-18 hotfix is available.

*** ** * ** ***

Fixed This hotfix fixes the following issues:

**Object storage**:

- Added support for S3 `GetBucketVersioning`.
- Cannot upload to sync dual-zone buckets using signed URLs.
- `DeleteObject` returns 500 for non-current versioned deletes.
- With dual zone buckets, S3 secrets are not generated after binding the `project-bucket-object-admin` role to a service account.

## Hotfix 17

*** ** * ** ***

Announcement The 1.14.3-gdch.9425-17 hotfix is available.

*** ** * ** ***

Fixed This hotfix fixes the following issues:

**File storage**:

- The Trident CSI driver deletes NetApp ONTAP volumes when they are offline, potentially leading to data loss.
- Multi-attach errors occur for volumes after cold reboot or node de-provision scenarios.
- The `project-fileshare-admin` role is missing patch and update access.
- Snapshots are not deleted in infra clusters when deleted in a user cluster.

**Managed Kubernetes Service**:

- Revert moving vanilla cluster VMs to user projects.

**Networking**:

- Invalid error code affects project network policies.
- A large CT ebpf map leads to create endpoint and delete endpoint failures.
- Leaked services might cause service IP duplication.

## Hotfix 16

*** ** * ** ***

Announcement The 1.14.3-gdch.9425-16 hotfix is available.

*** ** * ** ***

Fixed This hotfix fixes the following issues:

**Identity and access management**:

- There are forbidden errors when accessing vanilla Kubernetes clusters using kubeconfig from `gdcloud`.

**Endpoint detection response**:

- The endpoint detection response subcomponent gets stuck in a reconciliation error state.

**Managed Kubernetes Service**:

- Cluster validation should use the cluster's pod density when validating nodes.

**Networking**:

- The subnet predefined roles are missing verbs.

**Platform authentication**:

- CSRs for intermediary CAs are missing the basic constraint for CA.
- Added support for reusing a system domain in managed public DNS.

**Ticketing system**:

- An alert is not fired when the ticketing system is unavailable.

## Hotfix 15

*** ** * ** ***

Announcement The 1.14.3-gdch.9425-15 hotfix is available.

*** ** * ** ***

Fixed This hotfix fixes the following issues:

**Console**:

- There is an error when creating a role binding with a non-existent role.
- You can't add multiple role bindings to a service account in the Console.

**Identity and access management**:

- Added `gdcloud get-credentials` support for vanilla clusters.
- Custom roles should generate templates with the same name for global and zonal APIs.
- Exposed `CertificateAuthority` data on the well-known server.
- The identity and access management page is broken.
- There is an error when creating a role binding to a custom role.
- You can't attach user roles in the Console.

**Managed Kubernetes Service**:

- Move the vanilla cluster VMs to a user project.
- There are missing machine types for n3 type.

## Hotfix 14

*** ** * ** ***

Announcement The 1.14.3-gdch.9425-14 hotfix is available.

*** ** * ** ***

Fixed This hotfix fixes the following issues:

**Console**:

- The identity and access management page is broken.

**Multizone**:

- A pod in a vanilla cluster in zone1 cannot access the Management API server in zone2.

## Hotfix 13

*** ** * ** ***

Announcement The 1.14.3-gdch.9425-13 and 1.14.3-gdch.8490-13 hotfixes are available.

*** ** * ** ***

Fixed This hotfix fixes the following issues:

**Console**:

- Custom role creation does not work.

- Custom role creation from project scope shouldn't show the
  **Limit to selected projects** checkbox.

**DNS**:

- A pod in a vanilla cluster in zone1 cannot access the Management API server in zone2.

**Monitoring**:

- A pod in a vanilla cluster in zone1 cannot access the Management API server in zone2.

**Networking**:

- Controllers are stuck for hours in the `unet-cm-backend-controller` pod.
- Multiple clustermesh API servers reached their defined CPU limits.

- Data exfiltration protection (DEP) cannot be enabled on a global project
  that has DEP disabled.

**Object storage**:

- `GetBucketVersioning` for S3 is not supported.

- There is an error while initiating `cp` between different folders in a bucket.

**Platform authentication**:

- Cert Manager fails to issue certificates.

**SIEM**:

- You can't connect to a Splunk host from a user cluster.

## Hotfix 12

*** ** * ** ***

Announcement The 1.14.3-gdch.9425-12 and 1.14.3-gdch.8490-12 hotfixes are available.

*** ** * ** ***

Fixed This hotfix fixes the following issues:

**Console**:

- The global DNS is not resolving from a GDC VM.

**Networking**:

- Updated `allow-all-ingress` and `allow-all-egress` PNP Translation.
- Allow egress traffic from user workloads to system workloads automatically.
- The global DNS server is not reachable.

**Object storage**:

- Downloading from an S3 bucket fails.

## Hotfix 11

*** ** * ** ***

Announcement The 1.14.3-gdch.9425-11 hotfix is available.

*** ** * ** ***

Fixed This hotfix fixes the following issues:

**Endpoint detection response**:

- Nessus manager has duplicate agents and managers.

- There are gaps in EDR coverage on the perimeter, user, and service clusters.

**Identity and access management**:

- The service identity server fails to authenticate using zonal service account keys.

**Service mesh**:

- The `dataplane-ingress-gateway` pods are missing the `networking.private.gdc.goog/infra-access: enabled` label.

**Virtual machines**:

- There is a backwards compatibility issue for subnets.