Troubleshoot Windows Server node pools

When running Windows Server node pools in Google Kubernetes Engine (GKE), you might encounter issues such as Pods failing to start, errors pulling Windows container images, network connectivity problems, or nodes that fail to start.

Use this document to diagnose and resolve these common issues and keep your Windows-based applications running reliably.

This information is important for Platform admins and operators who manage GKE clusters with Windows node pools, and for Application developers who deploy and run Windows-based applications on GKE. For more information about the common roles and example tasks that we reference in Google Cloud content, see Common GKE user roles and tasks.

For more general guidance, see the Kubernetes documentation on debugging Pods and Services.

Containerd node issues

For information about resolving issues if you use a containerd node image, see Issues on Windows Server node pools.

Windows Pods fail to start

Incompatibilities between the base image and Windows Server host OS versions can prevent Pods from starting.

Symptoms

  • Windows Pods fail to start.
  • The node reports the NotReady status.

Cause

The container image was built on an older base Windows image that is incompatible with the host node's Windows Server version.

Resolution

Build your container images by using base Windows images that include Windows Updates from March 2020 or later. For more information about Microsoft container compatibility, see Microsoft's documentation for the February 2020 Windows Server container incompatibility issue.

Image pull errors

Windows Server container images are often significantly larger than Linux images, which can lead to timeouts.

Symptoms

  • Error messages such as Failed to pull image or context cancelled.
  • Pods display the ErrImagePull status.

Cause

Windows Server container images, and the individual layers they are composed of, can be large. Their size can cause the kubelet agent to time out and fail when it downloads and extracts the container layers.

Resolution

To resolve these image pull failures, try the following solutions:

  • Increase node CPU: container extraction is executed in parallel across cores, so machine types with more cores reduce the overall pull time.
  • Optimize image layers: to improve Docker layer caching and make image pull retries more likely to succeed, break your application layers into smaller layers. For more information, see Images and layers in the Docker storage driver documentation.
  • Use manual pulls: connect to your Windows Server nodes and manually execute the docker pull command on container images before creating your Pods.

For more general advice, see Troubleshoot image pulls.

Image family reached end of life

GKE periodically deprecates older Windows Server image families when vendor support ends. This deprecation blocks the creation of node pools with those images.

Symptoms

When creating a node pool with a Windows image, you receive an error similar to the following:

WINDOWS_SAC image family for 1.18.20-gke.501 has reached end of life, newer
versions are still available.

Cause

The selected Windows Server image family is no longer supported in GKE.

Resolution

Choose a Windows image that is available and supported. You can find the support end date for GKE Windows node images by using the gcloud container get-server-config command as described in Mapping GKE and Windows versions.

Timeout during node pool creation

Initializing a large number of Windows Server nodes simultaneously can cause timeouts.

Symptoms

Node pool creation operations time out before completing.

Cause

Node pool creation can time out if you are creating a large number of nodes (for example, 500) and it's the first node pool in the cluster that uses a Windows Server image.

Resolution

Reduce the initial node count when creating the node pool. After the node pool is created, you can increase the number of nodes.

Windows nodes become NotReady with error: PLEG is not healthy

Rapidly scheduling multiple Windows containers on a single node can overwhelm the Pod Lifecycle Event Generator (PLEG).

Symptoms

  • Windows nodes enter a NotReady status.
  • Events or logs display a PLEG is not healthy error message.

Cause

A known Kubernetes issue occurs when multiple Pods are started very rapidly on a single Windows node.

Resolution

To recover from PLEG failures and prevent reoccurrence:

  • Restart the affected Windows Server node.
  • Limit Windows Pod creation to no more than one Pod every 30 seconds.

Inconsistent TerminationGracePeriod

Differences between Windows container shutdown timers and Kubernetes grace period settings can cause containers to be terminated unexpectedly.

Symptoms

Containers are force-terminated by Windows before the duration configured in the TerminationGracePeriodSeconds field expires.

Cause

The internal Windows system timeout for the container differs from the grace period specified in the Kubernetes Pod manifest.

Resolution

Modify the Windows container timeout by editing container-local registry keys at image build time. Align the TerminationGracePeriodSeconds field in your Pod manifest accordingly.

Network connectivity problems

Maximum Transmission Unit (MTU) size mismatches between Windows Server container networking and Google Cloud networks can cause dropped packets.

Symptoms

Applications running inside Windows Server containers experience network connectivity failures or dropped packets.

Cause

Windows Server container networking often assumes a network MTU of 1500, which is incompatible with Google Cloud's MTU of 1460.

Resolution

Configure both the container network interface MTU and the Windows Server node network interface MTU value to 1460 or lower. For more information, see known issues for Windows containers in the Compute Engine documentation.

Node startup issues

New Windows Server instances can fail to complete initialization scripts or register with the control plane.

Symptoms

Windows Server nodes fail to initialize or fail to join the cluster.

Cause

Errors during the node initialization prevent the node from starting or joining the cluster.

Resolution

To identify which startup errors could be causing the issue, review the node's serial port output:

gcloud compute instances get-serial-port-output NODE_NAME \
    --zone=COMPUTE_ZONE

Replace the following:

  • NODE_NAME: the name of the node.
  • COMPUTE_ZONE: the compute zone for the node.

Intermittently unreachable Services in Windows nodes with cluster running 1.24 or earlier

On clusters running version 1.24 or earlier, restarting the kube-proxy component creates temporary network routing delays while Host Network Service (HNS) Load Balancer rules are reprocessed.

Symptoms

Services are intermittently unreachable from Pods running on Windows nodes.

Cause

For GKE clusters running version 1.24 or earlier, if an event restarts the kube-proxy component on a Windows node—for example, node startup, node upgrade, or manual restart—the component must sync and re-create all HNS Load Balancer rules. If the cluster has a high number of these rules, there can be a significant delay in processing them, lasting around 30 seconds per rule. During this sync delay, Services are intermittently unreachable from Pods running on that node. For more information, see the original issue in GitHub.

Resolution

Upgrade your cluster control plane to version 1.25 or later. This behavior is substantially improved in newer versions, as detailed in the pull request in GitHub.

What's next