Google Distributed Cloud air-gapped 1.16.2 hotfixes

Important: Starting with GDC air-gapped version 1.16.2, hotfixes are cumulative. A hotfix with a higher version number includes all fixes and changes from all preceding hotfixes for that release.

Hotfix 5

The 1.16.2-gdch.1353-5 hotfix is available.

Feature

This hotfix introduces the following features:

Storage

• Introduced a private preview of CSI drivers for customer-managed virtual machine (VM)-based Kubernetes clusters in the System Artifact Registry for integration by platform administrators and application operators.

Fixed

This hotfix fixes the following issues:

Cluster management

• Fixed an issue where containerd on Kubernetes clusters failed to verify the internal TLS certificate chain when pulling container images from harbor.org.internal by configuring harbor.org.internal with the trust-store-root-ext CA certificate secret in PrivateRegistries and deduplicating system private registry entries for platform administrators and application operators.

Hotfix 4

The 1.16.2-gdch.1353-4 hotfix is available.

Feature

This hotfix provides the following new features and capabilities:

CLI

• Added SHA-256 checksums to the GDC console CLI bundle download dialog and populated CLIBundleMetadata.Checksum from Harbor layer digests with a fixed download URL (?uncompressed=false), enabling platform administrators and application operators to verify binary authenticity and integrity before installation.

Identity and access management

• Added unique request IDs to webhook and authorization error payloads and login web templates to simplify troubleshooting authentication failures for platform administrators.

• Added API server audit ID logging to webhook authentication TokenReview debug logs to improve request correlation and diagnostics for infrastructure operators.

• Added configuration support for maximum burst rate limits on the authentication webhook server to maintain service responsiveness during high-volume authentication spikes for platform administrators.

Security

• Added support for dual Splunk 9.x and 10.x deployments to ensure continued security event forwarding and vendor support before Splunk 9.4 reaches end of life for infrastructure operators.

• Added harbor.org.internal to the Harbor gateway and decoupled its TLS certificate onto a dedicated Server Name Indication (SNI) server block signed by internal-tls-ca-issuer, preserving organization default certificate issuers across Fully Managed, Automated Certificate Management Environment (ACME), Bring Your Own Certificate (BYO Cert), and Bring Your Own Subordinate CA (BYO Sub-CA) public key infrastructure (PKI) modes for infrastructure operators.

Storage

• Added alpha support for bucket-level storage quotas, including API schemas, validation, and runbook procedures, enabling infrastructure operators to enforce capacity limits per bucket and prevent storage exhaustion.

• Added support for expiring object delete markers (ExpiredObjectDeleteMarker) in bucket lifecycle policies, enabling automatic removal of expired delete markers to optimize object storage utilization for application operators.

• Published KubeVirt CSI container images in a dedicated kvcsi Harbor project in addition to library, enabling workloads running on virtual machines and Kubernetes clusters to pull required storage drivers for platform administrators and application operators.

Fixed

This hotfix fixes the following issues:

Database service

• Upgraded Database Service to version 0.16.8 to apply the following fixes:

  • Fixed an issue where the local database operator failed to communicate with older dbdaemon instances during upgrades due to gRPC connection timeouts.
  • Fixed an issue where orphaned certificate keys remained in the dbs-certificates secret after DBCluster deletion, preventing secret size bloat.
  • Fixed an issue where stuck backup bucket deletions prevented DBCluster custom resource deletion.
  • Fixed false-positive timeouts and out-of-memory (OOM) failures in the Database Service service level objective (SLO) prober by increasing default resource allocations and enabling resource configurability.

Identity and access management

• Fixed an issue where users experienced transient 500 Internal Server Errors during tenant organization upgrades by adding readiness and startup probes with graceful connection draining to the authentication platform service.

• Fixed an issue where user deletion in the authentication platform failed if the user was associated with a non-existent realm.

• Fixed an issue where identity provider (IDP) creation failed in the GDC console when specifying a trailing hyphen (-) or colon (:) separator in the OIDC or SAML prefix by updating IDP prefix validation to accept trailing separators for platform administrators.

• Fixed an issue where creating organization-level IAM role bindings in the GDC console failed to propagate when roles resided outside the platform namespace by using the role's actual namespace, and preserved partial results when listing global IAM roles to prevent console UI errors for platform administrators.

• Fixed the user-cluster-applications-admin role name in the GDC console to ensure proper role assignment and permission enforcement for platform administrators.

Inventory

• Fixed an issue where generating hardware assets using gdcloud system assets commands for B300 (d4-highgpu2-256-gdc-metal) servers selected incorrect network ports for the primary data plane network bond, causing server provisioning failures for infrastructure operators.

• Fixed an issue where the data disk count wasn't configured for B300 (d4-highgpu2-256-gdc-metal) servers, setting the default data disk count to 8 so that spec.serverHardware.DataDiskInfo is populated on Server custom resources and local data storage volumes are properly provisioned for infrastructure operators.

• Fixed an issue in YAML cable validation that treated equivalent cable type classes (such as MMF 4x and MMF) as mismatched, preventing false-positive cable path validation warnings during inventory generation for infrastructure operators.

• Fixed an issue in media converter port validation regexes to support both lowercase (p) and uppercase (P) port names, preventing cell configuration and preflight validation failures during dynamic expansion for infrastructure operators.

• Fixed an issue where gdcloud system assets generate reported false-positive pairing warnings for inter-rack top-of-rack (ToR) switch connections passing through patch panels when a peer switch port ended at an intentionally unconnected patch panel.

• Fixed an issue where gdcloud system assets add failed to populate switch usernames in generated switch secret templates during zonal expansions for infrastructure operators.

Security

• Fixed an issue where the siem-aas-global subcomponent failed reconciliation on management API servers across organizations due to missing feature gate API resources.

• Fixed an issue where Harbor audit log purge schedule configuration wasn't idempotent, preventing configuration reconciliation errors on the container registry for infrastructure operators.

Storage

• Fixed an issue where the object storage proxy crash-looped on startup if KMS data encryption key (DEK) cache initialization failed, allowing the proxy to gracefully degrade to uncached KMS decryption and maintain service availability.

• Fixed an issue where non-empty bucket deletion errors starved the bucket controller workqueue by introducing exponential retry backoff in the bucket reconciler.

• Fixed an issue where tenant organization bootstrapping hung indefinitely due to a circular pre-install dependency in nearline-stamping Helm job hooks.

• Fixed an issue in VerifyReplicationFilter where the S3 response body was consumed during DeleteObjects requests, restoring the response body for subsequent filter processing and client reads.

• Fixed X-Amz-SignedHeaders parameter parsing by splitting values on semicolons, preventing S3 authorization failures when requests include multiple signed headers.

Upgrade

• Fixed feature gate evaluation and code generation logic to ensure runtime feature gates correctly propagate feature types and evaluate dynamic runtime states rather than static build-time stages during upgrades.

Vulnerability management

• Fixed an issue in vulnerability management enclave initialization that attempted to create the tenable-nessus-system namespace when OPA Gatekeeper constraints blocked creation in partially upgraded multi-zone environments.

Change

This hotfix provides the following changes:

Documentation

• Updated service manual runbooks to standardize customer support escalation procedures to direct users to open support cases with Google Cloud Care.

Monitoring

• Updated service level objective (SLO) export tools to anonymize organization and zone identifiers and prompt for user confirmation before data export for infrastructure operators.

Security

• Updated SIEM data retention configuration, restoring the default retention time to 89 days and configuring maximum document thresholds to prevent disk space exhaustion in Elasticsearch for infrastructure operators.

Hotfix 3

The 1.16.2-gdch.1353-3 hotfix is available.

Feature

This hotfix provides the following new features and capabilities:

Storage:

• Enhanced internal storage monitoring and diagnostics by introducing the following:

  • End-to-end file storage user journey probers.
  • Orphaned ONTAP volume detection.
  • New CSI and user journey health dashboards.
  • Secure CSI and Trident metrics scraping.

Fixed

This hotfix fixes the following issues:

Storage

• Provided the following updates to fix bugs and improve control plane stability:

  • Upgraded NetApp Trident and CSI sidecar components.
  • Increased CSI worker concurrency.
  • Added configurable ONTAP API timeouts.
  • Restricted Trident controller scheduling to control plane nodes to improve storage control plane throughput and prevent connectivity issues on worker and GPU nodes.

Hotfix 2

Fixed

The 1.16.2-gdch.1353-2 hotfix is available. This hotfix fixes the following issues:

Backup

• Added a deactivate option for virtual machine backup plans.

• Conditionally enabled liveness probes for backup cluster and virtual machine controllers on non-root-admin clusters.

CLI

• Set --auto-diff as the default run mode for the gdcloud system assets add command.

Cluster management

• Fixed an issue to populate subnet status with the correct cluster namespace.

Documentation

• Updated the GDC offline documentation hosted in the GDC console to fix an issue causing tabbed content to not render.

Identity and access management

• Added a missing rotatableSecret resource for Identity and Access Management.

• Optimized reconciliation by skipping zonalRoleRef updates when the role name is unchanged.

• Updated the ZonalRoleBindingReplica reconciler to properly handle user and administrator role combinations.

Inventory

• Updated inventory ingestion validation for B300 servers.

• Added inventory support for trunk cables, unified management ports, and an arbitrary number of data ports.

• Added support for placing power distribution units (PDUs) in any rack unit (U) slot instead of restricting them to U02.

• Removed dangling patch panel checks and added top-of-rack (ToR) switch pairing exceptions for dangling patch panels.

• Removed username and password fields from inventory switch configuration objects.

Logging

• Fixed log routing to direct standard Kubernetes cluster system logs to the application operator (AO) tenant.

Monitoring

• Added the node_name label to node exporter metrics.

Network Time Protocol (NTP)

• Increased memory limits and added caching for NTP services to improve stability.

Networking

• Added support for host network (HNETUpgrade) data plane RPM hotpatch upgrades, including OCI artifact packaging for onetd, libgdc-ipsec-offload, and libstrongswan-doca, Kubernetes Job-based Ansible execution, and upgrade failure alerts.

• Skipped checking the use verb on subnets that are propagated from the global API server.

• Enabled subnet consumer tracking in production and added cluster watch reconciliation for subnets in use.

• Fixed an issue in SwapEndFrontBacks where string replacement corrupted device names during port validation.

• Added support for three-letter rack names in management switch configurations.

• Added the commonName field to the apiserver-proxy-lancer-mgmt certificate manifest.

Platform authentication

• Enabled the Certificate Authority Service data plane feature flag in production and added CAS-Proxy service level objectives (SLOs).

Servers

• Updated server provisioning to install host network deliverables and complete ConnectX-7 (CX7) firmware provisioning jobs before marking servers as ready.

• Enabled the SERVAllowDellIDRACDowngrade feature flag in production to allow Dell iDRAC and BIOS firmware downgrades for the d4-highgpu2 machine class.

• Added a runbook to document procedures for downgrading Dell server firmware.

Storage

• Added on-demand Range-Read Helper (RRH) reconstruction, automatic recovery, orphan cleanup, and metrics to improve range read and multipart upload reliability.

• Optimized bucket reconciliation by skipping redundant StorageGRID updates for CORS, lifecycle, and locking policies when configurations are unchanged.

• Improved presigned URL lifecycle security and redacted S3 signatures and credentials from error logs.

• Fixed a request pointer race condition in the object storage proxy handler, improved response filter error logging, and added payload transfer instrumentation.

• Updated the StorageGRID hotfix version to 11.9.0.14 and skipped ObjectStorageUpgradeRequest custom resource creation on non-anchor zones.

Virtual machines

• Updated the virtual machine package repository gateway to use STRIP_LEADING_DIRECTORY_PATH for operating system (OS) package routing.

Hotfix 1

Fixed

The 1.16.2-gdch.1353-1 hotfix is available. This hotfix fixes the following issues:

CLI

• Added support for diff-based OCI package loading in the gdcloud system container-registry load-oci command to reduce download and upload sizes for hotfix releases. Diffs are always compared against the base release, not against preceding hotfix versions. When loading a diff package, the command skips duplicate image layers and blobs that already exist in the base release. Because hotfixes are cumulative, each hotfix release package contains all fixes introduced since the base release, excluding duplicate artifacts from the base release.

Database service

• Updated Database Service to version 0.16.6 and Database Service console UI to version 2.16.0-beta.31 to apply the latest bug fixes and stability improvements.

Documentation

• Updated the GDC offline documentation hosted in the GDC console Help menu with the latest content. For more information, see GDC console access.

Firewall

• Added the ZoneProtectionProfile custom resource data model, custom profile generation and claims, validation webhooks, and reconnaissance protection support for firewall nodes. Using the ZoneProtectionProfile custom resource (zoneprotectionprofiles.firewall.private.gdc.goog/v1alpha2), operators can configure protection at the organization level, including the root organization, to block malicious traffic and port scans. Operators can also customize scan thresholds, define IP address allowlists (exclusionIPs), change protection actions (alert, allow, block, block-ip), or selectively disable scan protections (enabled: false).

Gemini

• Updated the Gemini Inference Stack operator and model server templates to remove unsupported server configs and apply stability fixes.

Identity and access management

• Fixed a timeout issue with rotatableSecret in Anthos Identity Service (AIS) that caused identity service reconciliation delays.

• Enabled cookie splitting support in AIS to resolve large authentication token issues, and updated identity service container images.

Inventory

• Added support for B300 rack expansion.

• Fixed an issue where auto-generated passwords containing unsupported special characters caused switch provisioning and object storage bootstrap failures.

• Updated rack and instance IDs to support four-character identifiers for GPU pods, and removed unnecessary cable, adapter, and transceiver validations during preflight checks.

• Added support for management IP address and CIDR expansion after decommissioning a middle rack (a rack deployed after racks A and B, but before racks X and Y). Previously, if this middle rack was decommissioned, deploying or expanding to a new rack later would trigger IP address allocation issues. This fix resolves those conflicts, ensuring seamless rack expansion.

• Added support for iDRAC and OCP port names in inventory webhook validation.

Logging

• Optimized LoggingRule filter expressions to prevent high query rates (QPS) on object storage.

Monitoring

• Fixed an issue where incorrect metrics port configurations on the EZ infrastructure controller caused high-frequency EZ-E1001, EZ-E1003, and EZ-E1004 alerts to fire erroneously.

• Improved MonitoringTarget reconciler resilience by automatically requeueing targets after certificate retrieval failures and preventing unnecessary patches on persistent volume claims (PVCs).

• Fixed an issue where restartedAt annotations caused unnecessary restarts during Prometheus StatefulSet comparison.

• Removed legacy Google-Built OpenTelemetry Collector (GBOC) network policies in the monitoring namespace.

Networking

• Fixed an issue where subnetClaimRef wasn't populated when a server was located in a different rack from its management switch.

• Fixed an issue where device category CIDRs were missing for existing management switches during vertical expansion.

• Added support and generic regular expression validation for B300 server management and data port naming formats (such as fe-s6p1 and ocp-s1p1).

• Fixed JSON parsing errors in VLAN interface tables when the interface list wraps across multiple lines.

• Added a host network stage to rollout policies to ensure reliable network staging during component updates.

• Disabled ABM Gateway resource installation to avoid reconciliation errors in unsupported environments.

• Fixed connectivity tests to treat media converters as passive devices during validation.

• Improved error handling for invalid port formats during switch port connection validation.

Operating system

• Improved file access policy daemon (fapolicyd) performance and stability on nodes by increasing cache size and optimizing trust database rules.

Security

• Added support for multiple root certificates during package validation preflight checks.

Servers

• Added support for Dell iDRAC and BIOS firmware downgrades for the d4-highgpu2 machine class.

Upgrade

• Added safeguards to prevent re-evaluation of completed upgrades and improved client validation error handling during upgrade orchestration.

• Added support for a new diff-download mode for hotfixes, to download only the changed artifacts compared to the base release, drastically reducing the download size.

Vertex AI

• Updated Vertex AI Search container images and prediction_proxy to apply the latest security and stability updates.

Virtual machines

• Added support for running workloads that require GPUs on standard Kubernetes clusters.

• Marked shared VM and virtual machine image resources as audit-relevant in AuditlogInventory to ensure comprehensive audit logging.

• Fixed an issue in image-controller where missing cache checks caused excessive HTTP 409 conflict errors during concurrent image creation.

• Fixed an issue affecting virtual machine image import stability.

Vulnerability management

• Fixed vulnerability report import workflows to wait for completion before proceeding.

• Skipped unnecessary Security Compliance Framework for Google Cloud (SCFGM) generation for Windows OIC nodes.

• Made database CPU and memory resource allocations configurable and increased default database resource limits for the vulnerability scanning database.