When your Pods in Google Kubernetes Engine (GKE) are stuck in a `Pending` state and
new nodes aren't being added, it often indicates an issue with the cluster
autoscaler's scale-up functionality. This issue can prevent your applications
from scaling to meet demand, delay deployments, and affect service availability.

Use this page to diagnose and resolve common problems that prevent the
cluster autoscaler from adding new nodes. Fixing these issues lets the
Kubernetes scheduler place your workloads promptly and helps your cluster adapt
to increasing load.

This information is important for Application developers, who need their
applications and services to schedule and run reliably, and for
Platform admins and operators, who are responsible for ensuring the cluster can
dynamically provision resources to meet workload requirements and maintain
service levels. For more information about the common roles and example tasks
that we reference in Google Cloud content, see
[Common GKE user roles and tasks](https://docs.cloud.google.com/kubernetes-engine/enterprise/docs/concepts/roles-tasks.md.txt).

## Understand when cluster autoscaler scales up your nodes

Before you proceed to the troubleshooting steps, it can be helpful to understand
when cluster autoscaler would try to scale up your nodes. Cluster autoscaler
only adds nodes when existing resources are insufficient.

Every 10 seconds, cluster autoscaler checks if there are any Pods that are
unschedulable. A Pod becomes unschedulable when the Kubernetes scheduler cannot
place it on any existing node due to insufficient resources, node constraints,
or unmet Pod requirements.

When cluster autoscaler finds unschedulable Pods, it evaluates if adding a node
would allow the Pod to get scheduled. If adding a node would let a Pod get
scheduled cluster autoscaler adds a new node to the managed instance group
(MIG). The Kubernetes scheduler can then schedule the Pod on the newly
provisioned node.

### Check if you have unschedulable Pods

To determine if your cluster needs to scale up, check for unscheduled Pods:

1. In the Google Cloud console, go to the **Workloads** page.

   [Go to Workloads](https://console.cloud.google.com/kubernetes/workload)
2. In the **Filter** field,
   enter `unschedulable` and press <kbd>Enter</kbd>.

   If there are any Pods listed, then you have unschedulable Pods. To
   troubleshoot unschedulable Pods, see
   [Error: Pod unschedulable](https://docs.cloud.google.com/kubernetes-engine/docs/troubleshooting/deployed-workloads.md.txt#PodUnschedulable).
   Resolving the underlying cause of unschedulable Pods can often enable
   cluster autoscaler to scale up. To identify and resolve errors that are
   specific to cluster autoscaler, explore the following sections.

   If there are no Pods listed, cluster autoscaler doesn't need to scale
   up and is working as expected.

#### Check if you previously had unschedulable Pods

If you're investigating what caused cluster autoscaler to fail in the past,
check for previously unschedulable Pods:

1. In the Google Cloud console, go to the **Logs Explorer** page.

   [Go to Logs Explorer](https://console.cloud.google.com/logs)
2. Specify a time range for the log entries that you want to view.

3. In the query pane, enter the following query:

   ```sql
   logName="projects/PROJECT_ID/logs/events"
   jsonPayload.source.component="default-scheduler"
   jsonPayload.reason="FailedScheduling"
   ```

   Replace `PROJECT_ID` with your project ID.
4. Click **Run query**.

   If there are any results listed, then you had unschedulable Pods in the time
   range that you specified.

## Check if the issue is caused by a limitation

After you've confirmed that you have unscheduled Pods, make sure your issue
with cluster autoscaler isn't caused by one of the [limitations for the cluster
autoscaler](https://docs.cloud.google.com/kubernetes-engine/docs/concepts/cluster-autoscaler#limitations).

## Analyze Cloud Monitoring metrics for scale-up issues

GKE cluster autoscaler exposes metrics to Cloud Monitoring that provide insights
into its decisions, especially in relation to ComputeClasses.
These metrics are available on GKE version 1.37 and later, and
include the following.

- **Pending Pods** : `kubernetes.io/autoscaler/cluster_pending_pods_per_ccc` shows Pods the autoscaler is trying to schedule.
- **Provisioning attempts** : `kubernetes.io/autoscaler/cluster_node_provisioning_attempts_count_per_ccc` tracks how often autoscaler tries to add nodes.
- **Provisioning failures** : `kubernetes.io/autoscaler/cluster_node_provisioning_failed_attempts_count_per_ccc` shows failed node creation attempts and reasons.

These metrics use the `k8s_entity` resource type, where the value of the
`resource.labels.entity_type` field is `ComputeClass`. The value of the
`resource.labels.entity_name` field indicates the ComputeClass name. If the
`entity_name` field has an empty string (`""`) as the value, the metric is for
cluster autoscaler decisions that aren't related to a ComputeClass.

Use Metrics Explorer to filter and group results by the `entity_name` field.
For example, use the `resource.labels.entity_name = ""` string to see activities
that aren't related to ComputeClasses, and metric labels (like `reason`) to
understand whether scale-up issues are specific to a ComputeClass.

For query examples, see [Monitor node provisioning](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/cluster-autoscaler-visibility.md.txt#monitor_provisioning)
and [Monitor pending Pod status](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/cluster-autoscaler-visibility.md.txt#monitor_pending_pod_status).

## View errors

You can often diagnose the cause of scale up issues by viewing error messages:

- If you've already seen an error message, see the [error
  messages table](https://docs.cloud.google.com/kubernetes-engine/docs/troubleshooting/cluster-autoscaler-scale-up#error-table) for advice on resolving the error.

- If you haven't seen a message yet, use one of the following options:

  - **Issues less than 72 hours old** : [View error notifications in
    the Google Cloud console](https://docs.cloud.google.com/kubernetes-engine/docs/troubleshooting/cluster-autoscaler-scale-up#view-notifications).
  - **Issues over 72 hours old** : [View errors in events in
    Cloud Logging](https://docs.cloud.google.com/kubernetes-engine/docs/troubleshooting/cluster-autoscaler-scale-up#view-events).

### View errors in notifications

If the issue you observed happened less than 72 hours ago, view notifications
about errors in the Google Cloud console. These notifications provide valuable
insights into why cluster autoscaler didn't scale up and offer advice on how to
resolve the error and view relevant logs for further investigation.

To view the notifications in the Google Cloud console, complete the following
steps:

1. In the Google Cloud console, go to the **Kubernetes clusters** page.

   [Go to Kubernetes clusters](https://console.cloud.google.com/kubernetes/list)
2. Review the **Notifications** column. The following notifications are
   associated with scale up issues:

   - `Can't scale up`
   - `Can't scale up pods`
   - `Can't scale up a node pool`
3. Click the relevant notification to see a pane with details about what caused
   the issue and recommended actions to resolve it.

4. Optional: To view the logs for this event, click **Logs** . This action takes
   you to Logs Explorer with a pre-populated query to help you further
   investigate the scaling event. To learn more about how scale up events work,
   see [View cluster autoscaler
   events](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/cluster-autoscaler-visibility).

If you're still experiencing issues after reviewing the advice in the
notification, consult the [error messages tables](https://docs.cloud.google.com/kubernetes-engine/docs/troubleshooting/cluster-autoscaler-scale-up.md.txt#error-table) for further
help.

### View errors in events

If the issue you observed happened more than 72 hours ago, view events in
Cloud Logging. When there has been an error, it's often recorded in an event.

To view cluster autoscaler logs in the Google Cloud console, complete the
following steps:

1. In the Google Cloud console, go to the **Kubernetes clusters** page.

   [Go to Kubernetes clusters](https://console.cloud.google.com/kubernetes/list/overview)
2. Select the name of the cluster that you want to investigate to view its
   **Cluster details** page.

3. On the **Cluster details** page, click the **Logs** tab.

4. On the **Logs** tab, click the **Autoscaler Logs** tab to view the logs.

5. Optional: To apply more advanced filters to narrow the results, click the
   button with the arrow on the right side of the page to view the logs in
   Logs Explorer.

To learn more about scale up events work, see [View cluster autoscaler
events](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/cluster-autoscaler-visibility). For one
example of how to use Cloud Logging, see the following [troubleshooting
example](https://docs.cloud.google.com/kubernetes-engine/docs/troubleshooting/cluster-autoscaler-scale-up#example-troubleshooting).

#### Example: Troubleshoot an issue over 72 hours old

The following example shows you how you might investigate and resolve an issue
with a cluster not scaling up.

**Scenario**: For the past hour, a Pod has been marked as unschedulable. Cluster
autoscaler did not provision any new nodes to schedule the Pod.

**Solution**:

1. Because the issue happened over 72 hours ago, you investigate the issue using Cloud Logging instead of looking at the notification messages.
2. In Cloud Logging, you find the logging details for cluster autoscaler events, as described in [View errors in events](https://docs.cloud.google.com/kubernetes-engine/docs/troubleshooting/cluster-autoscaler-scale-up.md.txt#view-events).
3. You search for `scaleUp` events that contain the Pod that you're investigating
   in the `triggeringPods` field. You could filter the log entries, including
   filtering by a particular JSON field value. Learn more in [Advanced logs
   queries](https://docs.cloud.google.com/logging/docs/view/advanced-queries).

4. You don't find any scale up events. However, if you did, you could try to
   find an [`EventResult`](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/cluster-autoscaler-visibility.md.txt#eventresult-event)
   that contains the same `eventId` as the `scaleUp` event. You could then look
   at the `errorMsg` field and consult the list of possible [scaleUp
   error messages](https://docs.cloud.google.com/kubernetes-engine/docs/troubleshooting/cluster-autoscaler-scale-up#scaleup-errors).

5. Because you didn't find any `scaleUp` events, you continue to search
   for `noScaleUp` events and review the following fields:

   - `unhandledPodGroups`: contains information about the Pod (or Pod's controller).
   - `reason`: provides global reasons indicating scaling up could be blocked.
   - `skippedMigs`: provides reasons why some MIGs might be skipped.
6. You find a `noScaleUp` event for your Pod, and all MIGs in the
   `rejectedMigs` field have the same reason message ID of
   `"no.scale.up.mig.failing.predicate"` with two parameters: `"NodeAffinity"`
   and `"node(s) did not match node selector"`.

**Resolution**:

After consulting the [list of error messages](https://docs.cloud.google.com/kubernetes-engine/docs/troubleshooting/cluster-autoscaler-scale-up.md.txt#error-table), you discover that
cluster autoscaler can't scale up a node pool because of a failing scheduling
predicate for the pending Pods. The parameters are the name of the failing
predicate and the reason why it failed.

To resolve the issue, you review the manifest of the Pod, and discover that it
has a node selector that doesn't match any MIG in the cluster. You delete the
selector from the manifest of the Pod and recreate the Pod. Cluster autoscaler
adds a new node and the Pod is scheduled.

## Resolve scale up errors

If you are using custom ComputeClasses and the scale-up issues are related to the specific node configurations defined in your ComputeClass, refer to [Troubleshoot custom ComputeClass issues](https://docs.cloud.google.com/kubernetes-engine/docs/troubleshooting/custom-computeclass.md.txt).

After you have identified your error, use the following tables to help you
understand what caused the error and how to resolve it.

### ScaleUp errors

You can find event error messages for `scaleUp` events in the corresponding
`eventResult` event, in the `resultInfo.results[].errorMsg` field.

When a scale-up operation fails because it exceeds a quota, it is a node
creation error that triggers a system backoff period, which can last up to 30
minutes. For more information, see [Backoff
periods](https://docs.cloud.google.com/kubernetes-engine/docs/concepts/cluster-autoscaler#backoff-periods).

| Message | Details | Parameters | Mitigation |
|---|---|---|---|
| `"scale.up.error.out.of.resources"` | Resource errors occur when you try to request new resources in a zone that cannot accommodate your request due to the current unavailability of a Compute Engine resource, such as GPUs or CPUs. | Failing MIG IDs. | Follow the [resource availability troubleshooting steps](https://docs.cloud.google.com/compute/docs/troubleshooting/troubleshooting-resource-availability.md.txt) in the Compute Engine documentation. |
| `"scale.up.error.quota.exceeded"` | The scaleUp event failed because some of the MIGs couldn't be increased, due to exceeded Compute Engine quota. | Failing MIG IDs. | Check the **Errors** tab of the MIG in the Google Cloud console to see what quota is being exceeded. After you know which quota is being exceeded, follow the instructions to [request a quota increase](https://docs.cloud.google.com/compute/quotas#requesting_additional_quota). |
| `"scale.up.error.waiting.for.instances.timeout"` | Scale up of managed instance group failed to scale up due to timeout. | Failing MIG IDs. | If the instances boot but don't register with the control plane, then follow the steps in [Find an instance that failed node registration](https://docs.cloud.google.com/kubernetes-engine/docs/troubleshooting/node-registration.md.txt#find-instance-failed-registration). For other causes, see [MIG instances failed to appear in time](https://docs.cloud.google.com/kubernetes-engine/docs/troubleshooting/cluster-autoscaler-scale-up.md.txt#instances-failed-to-appear). This error is usually transient. If the error persists, then [contact Cloud Customer Care](https://docs.cloud.google.com/kubernetes-engine/docs/getting-support.md.txt) for further investigation. |
| `"scale.up.error.ip.space.exhausted"` | Can't scale up because instances in some of the managed instance groups ran out of IPs. This means that the cluster doesn't have enough unallocated IP address space to use to add new nodes or Pods. | Failing MIG IDs. | Follow the troubleshooting steps in [Not enough free IP address space for Pods](https://docs.cloud.google.com/kubernetes-engine/docs/troubleshooting/vpc-clusters.md.txt#not_enough_space). |
| `"scale.up.error.service.account.deleted"` | Can't scale up because the service account was deleted. | Failing MIG IDs. | Try to [undelete the service account](https://docs.cloud.google.com/iam/docs/service-accounts-delete-undelete.md.txt#undeleting). If that procedure is unsuccessful, [contact Cloud Customer Care](https://docs.cloud.google.com/kubernetes-engine/docs/getting-support.md.txt) for further investigation. |

### Reasons for a noScaleUp event

A `noScaleUp` event is periodically emitted when there are unschedulable Pods
in the cluster and cluster autoscaler cannot scale the cluster up to schedule
the Pods. `noScaleUp` events are best-effort, and don't cover all possible cases.

#### NoScaleUp top-level reasons

Top-level reason messages for `noScaleUp` events appear in the
`noDecisionStatus.noScaleUp.reason` field. The message contains a top-level
reason for why cluster autoscaler cannot scale the cluster up.

| Message | Details | Mitigation |
|---|---|---|
| `"no.scale.up.in.backoff"` | No scale up because scaling up is in a backoff period (temporarily blocked). This message that can occur during scale up events with a large number of Pods. | This message should be transient. Check this error after a few minutes. If this message persists, [contact Cloud Customer Care](https://docs.cloud.google.com/kubernetes-engine/docs/getting-support.md.txt) for further investigation. |

#### NoScaleUp top-level node auto-provisioning reasons

Top-level node auto-provisioning reason messages for `noScaleUp` events appear
in the `noDecisionStatus.noScaleUp.napFailureReason` field. The message contains
a top-level reason for why cluster autoscaler cannot provision new node pools.

| Message | Details | Mitigation |
|---|---|---|
| `"no.scale.up.nap.disabled"` | Node auto provisioning couldn't scale up because node auto provisioning is not enabled at cluster level. If node auto-provisioning is disabled, new nodes won't be automatically provisioned if the pending Pod has requirements that can't be satisfied by any existing node pools. | Review the cluster configuration and consider [enabling node auto-provisioning](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/node-auto-provisioning.md.txt#enable). |

#### NoScaleUp MIG-level reasons

MIG-level reason messages for `noScaleUp` events appear in the
`noDecisionStatus.noScaleUp.skippedMigs[].reason` and
`noDecisionStatus.noScaleUp.unhandledPodGroups[].rejectedMigs[].reason` fields.
The message contains a reason why cluster autoscaler can't increase the size of
a particular MIG.

| Message | Details | Parameters | Mitigation |
|---|---|---|---|
| `"no.scale.up.mig.skipped"` | Cannot scale up a MIG because it was skipped during the simulation. | Reasons why the MIG was skipped (for example, missing a Pod requirement). | Review the parameters included in the error message and address why the MIG was skipped. |
| `"no.scale.up.mig.failing.predicate"` | Can't scale up a node pool because of a failing scheduling predicate for the pending Pods. | Name of the failing predicate and reasons why it failed. | Review the Pod requirements, such as [affinity rules](https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/FAQ.md#does-ca-respect-node-affinity-when-selecting-node-groups-to-scale-up), taints or tolerations, and resource requirements. |

#### NoScaleUp Pod-group-level node auto-provisioning reasons

Pod-group-level node auto-provisioning reason messages for `noScaleUp` events
appear in the
`noDecisionStatus.noScaleUp.unhandledPodGroups[].napFailureReasons[]` field. The
message contains a reason why cluster autoscaler cannot provision a new node
pool to schedule a particular Pod group.

| Message | Details | Parameters | Mitigation |
|---|---|---|---|
| `"no.scale.up.nap.pod.gpu.no.limit.defined"` | Node auto-provisioning couldn't provision any node group because a pending Pod has a GPU request, but [GPU resource limits are not defined at the cluster level](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/node-auto-provisioning.md.txt#gpu_limits). | Requested GPU type. | Review the pending Pod's GPU request, and update the cluster-level node auto-provisioning [configuration for GPU limits](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/node-auto-provisioning.md.txt#gpu_limits). |
| `"no.scale.up.nap.pod.gpu.type.not.supported"` | Node auto-provisioning did not provision any node group for the Pod because it has requests for an unknown GPU type. | Requested GPU type. | Check the pending Pod's configuration for the GPU type to ensure that it matches a [supported GPU type](https://docs.cloud.google.com/compute/docs/gpus.md.txt#nvidia_gpus_for_compute_workloads). |
| `"no.scale.up.nap.pod.zonal.resources.exceeded"` | Node auto-provisioning did not provision any node group for the Pod in this zone because doing so would either violate the [cluster-wide maximum resource limits](https://docs.cloud.google.com/kubernetes-engine/docs/concepts/node-auto-provisioning.md.txt#limits_for_clusters), exceed the available resources in the zone, or there is no machine type that could fit the request. | Name of the considered zone. | Review and update [cluster-wide maximum resource limits](https://docs.cloud.google.com/kubernetes-engine/docs/concepts/node-auto-provisioning.md.txt#limits_for_clusters), the Pod resource requests, or the [available zones for node auto-provisioning](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/node-auto-provisioning.md.txt#auto-provisioning_locations). |
| `"no.scale.up.nap.pod.zonal.failing.predicates"` | Node auto-provisioning did not provision any node group for the Pod in this zone because of failing predicates. | Name of the considered zone and reasons why predicates failed, for example: `node(s) didn't match Pod's node affinity/selector` or `node(s) didn't match pod anti-affinity rules`. | Review the failing predicate reasons in the log entry. Update the pending Pod's scheduling constraints (node selectors, node affinity rules, PersistentVolume requirements, tolerations, or topology spread constraints), or update the cluster's node auto-provisioning configuration. For step-by-step troubleshooting instructions, see [Node auto-provisioning fails scheduling predicates](https://docs.cloud.google.com/kubernetes-engine/docs/troubleshooting/cluster-autoscaler-scale-up.md.txt#nap-pod-zonal-failing-predicates). |

## Conduct further investigation

The following sections provide guidance on how to use Logs Explorer and
`gcpdiag` to gain additional insights into your errors.

### Investigate errors in Logs Explorer

To further investigate an error message, view the logs specific to that error:

1. In the Google Cloud console, go to the **Logs Explorer** page.

   [Go to Logs Explorer](https://console.cloud.google.com/logs)
2. In the [query
   pane](https://docs.cloud.google.com/logging/docs/view/logs-explorer-interface#logs-viewer-interface),
   enter the query that matches your error message prefix:

   - For `scale.up.error.*` messages:

     ```sql
     resource.type="k8s_cluster"
     log_id("container.googleapis.com/cluster-autoscaler-visibility")
     jsonPayload.resultInfo.results.errorMsg.messageId="ERROR_MESSAGE"
     ```

     Replace `ERROR_MESSAGE` with the message that you
     want to investigate, such as `scale.up.error.out.of.resources`.
   - For `no.scale.up.nap.pod.*` messages:

     ```sql
     resource.type="k8s_cluster"
     log_id("container.googleapis.com/cluster-autoscaler-visibility")
     jsonPayload.noDecisionStatus.noScaleUp.unhandledPodGroups.napFailureReasons.messageId="ERROR_MESSAGE"
     ```

     Replace `ERROR_MESSAGE` with the message that you
     want to investigate, such as
     `no.scale.up.nap.pod.zonal.failing.predicates`. For the top-level
     `no.scale.up.nap.disabled` message, query the
     `jsonPayload.noDecisionStatus.noScaleUp.napFailureReason.messageId`
     field instead.
   - For `no.scale.up.mig.*` messages:

     ```sql
     resource.type="k8s_cluster"
     log_id("container.googleapis.com/cluster-autoscaler-visibility")
     jsonPayload.noDecisionStatus.noScaleUp.unhandledPodGroups.rejectedMigs.reason.messageId="ERROR_MESSAGE"
     ```

     Replace `ERROR_MESSAGE` with the message that you
     want to investigate, such as `no.scale.up.mig.failing.predicate`. For
     the `no.scale.up.mig.skipped` message, query the
     `jsonPayload.noDecisionStatus.noScaleUp.skippedMigs.reason.messageId`
     field instead.
3. Click **Run query**.

### Debug some errors with gcpdiag

`gcpdiag` is an open source tool created with support from Google Cloud
technical engineers. It isn't an officially supported Google Cloud product.

If you've experienced one of the following error messages, you can use
[`gcpdiag`](https://gcpdiag.dev/runbook/diagnostic-trees/gke/cluster-autoscaler/#parameters)
to help troubleshoot the issue:

- `scale.up.error.out.of.resources`
- `scale.up.error.quota.exceeded`
- `scale.up.error.waiting.for.instances.timeout`
- `scale.up.error.ip.space.exhausted`
- `scale.up.error.service.account.deleted`

For a list and description of all `gcpdiag` tool flags, see the [`gcpdiag` usage
instructions](https://github.com/GoogleCloudPlatform/gcpdiag?tab=readme-ov-file#usage).

## Resolve complex scale-up errors

The following sections help you resolve errors that require multiple steps or
don't have an associated event message from the cluster autoscaler.

### Pod doesn't fit on node

**Symptoms**:

Your Pods remain in a `Pending` state with the `PodUnschedulable` error message,
or your cluster shows the `Pods unschedulable` error message, and cluster
autoscaler doesn't scale up to add nodes.

**Cause**:

Cluster autoscaler only adds nodes to a node pool if the machine type provides
enough resources, such as CPUs, GPUs, memory, and storage, for a pending Pod. If
a Pod's resource requests exceed the resources of a single node, cluster
autoscaler doesn't scale up because the Pod still can't fit on a new node.

Also, a node's allocatable resources are less than its total resources because
system components reserve a portion of those resources. For more information
about how GKE calculates allocatable resources, see [Node
allocatable
resources](https://docs.cloud.google.com/kubernetes-engine/docs/concepts/plan-node-sizes#node_allocatable_resources).

**Resolution**:

To check whether a Pod's resource requests exceed the resources provided by your
nodes and resolve the issue, complete the following steps. This example checks
CPU resources, but the same steps apply to GPUs, memory, and storage resources:

1. In the Google Cloud console, go to the **Workloads** page.

   [Go to Workloads](https://console.cloud.google.com/kubernetes/workload)
2. Click the `PodUnschedulable` error message.

3. In the **Details** pane, click the name of the Pod. If there are
   multiple Pods, start with the first Pod and repeat the following process for
   each Pod.

4. On the **Pod details** page, click the **YAML** tab.

5. Note each container's resource requests in the Pod to find the total
   resource requests. For example, in the following Pod configuration, the Pod
   needs 2 vCPUs:

   ```yaml
   resources:
     limits:
       cpu: "3"
     requests:
       cpu: "2"
   ```
6. View the node pool details from the cluster with the unscheduled Pod:

   1. In the Google Cloud console, go to the **Kubernetes clusters** page.

      [Go to Kubernetes clusters](https://console.cloud.google.com/kubernetes/list)
   2. Click the name of the cluster that has the `Pods unschedulable` error
      message.

   3. On the **Cluster details** page, click the **Nodes** tab.

   4. In the **Node pools** section, note the value in the **Machine type**
      column, for example, `n1-standard-1`.

7. Compare the resource request with the vCPUs provided by the machine type.
   For example, if a Pod requests 2 vCPUs, but the available nodes use the
   `n1-standard-1` machine type, the nodes only have 1 vCPU. With a
   configuration like this, cluster autoscaler doesn't trigger a scale-up
   because the Pod still can't fit on a new node. For more information
   about available machine types, see [Machine families resource and comparison
   guide](https://docs.cloud.google.com/compute/docs/machine-resource) in the Compute Engine
   documentation.

8. Resolve the mismatch based on your workload needs:

   - **If the Pod requires the requested resources**: create a node pool with a machine type that can support the Pod's resource requests.
   - **If the Pod's resource requests aren't accurate**: update the Pod's definition so that the Pod can fit on existing nodes.

### MIG instances failed to appear in time

**Symptoms**:

You see one of the following error messages:

- In the Google Cloud console, the cluster shows the following notification:

      Can't scale up because instances in managed instance groups hosting node pools failed to appear in time

- In Logging, the `eventResult` event for the scale-up
  contains the following error message:

      scale.up.error.waiting.for.instances.timeout

**Cause**:

Compute Engine can't create nodes in the MIG, or the new nodes fail to
register with the control plane before the scale-up operation times out. This
issue can happen for any of the following reasons:

- **Failed node registration**: Compute Engine creates the nodes, but they don't register with the control plane because of one or more firewall rules, a recent change to private services access or VPC Network Peering, or a missing Identity and Access Management (IAM) service account.
- **Exceeded project quotas**: new instances in the MIG don't appear because your project exceeds a quota for CPUs, IP addresses, or another resource.
- **Unavailable IP addresses**: the subnet doesn't have enough free IP addresses to create new nodes.
- **Invalid instance template**: the MIG uses an invalid instance template, such as a template with a missing image, which prevents scale-up.
- **Insufficient resources**: the zone doesn't have enough Compute Engine resources, such as GPUs or CPUs, to create a new node.
- **Inaccessible APIs** : a required API, such as `compute.googleapis.com`, isn't accessible because of a firewall rule or a VPC Service Controls perimeter.
- **Network latency**: significant network latency delays the creation or usability of a new node.

**Resolution**:

1. To identify the cause of the timeout, [list the instance errors in the
   MIG](https://docs.cloud.google.com/compute/docs/instance-groups/getting-info-about-migs#listing_instance_errors),
   and then check the result:

   - **The MIG doesn't list errors from the time of the failure** : Compute Engine created the nodes, but the nodes didn't register with the control plane before the timeout. In the next step, start with **Failed node registration** . If the nodes registered but took too long, then see **Network latency**.
   - **The MIG lists errors from the time of the failure**:
     Compute Engine couldn't create the nodes. Use the error code
     and error message to find the cause in the next step. Common error
     codes include the following:

     - `QUOTA_EXCEEDED`: exceeded project quotas
     - `IP_SPACE_EXHAUSTED`: unavailable IP addresses
     - `ZONE_RESOURCE_POOL_EXHAUSTED`: insufficient resources

     Some error codes have a `_WITH_DETAILS` suffix and include more
     information in the error message. If the error message reports a
     resource that doesn't exist, such as a source image, then see the
     **Invalid instance template** bullet point.
2. Use the resolution that corresponds to the cause of your issue:

   - **Failed node registration** : follow the steps in [Find an instance
     that failed node
     registration](https://docs.cloud.google.com/kubernetes-engine/docs/troubleshooting/node-registration#find-instance-failed-registration).
     You can also check for recent configuration changes:

     - **Firewall rules**: to check whether a firewall rule was recently
       created, updated, or deleted, run the following query in
       Logs Explorer:

       ```sql
       resource.type="gce_firewall_rule"
       log_id("cloudaudit.googleapis.com/activity")
       ```

       If a change blocks traffic between the nodes and the control
       plane or to Google APIs, then [update or replace the firewall
       rule](https://docs.cloud.google.com/firewall/docs/using-firewalls#updating_firewall_rules). For
       more information about the required connectivity, see
       [Prerequisites for successful node
       registration](https://docs.cloud.google.com/kubernetes-engine/docs/troubleshooting/node-registration#check-prereqs-for-node-registration).
     - **IAM service account** : for a missing service
       account, see the `scale.up.error.service.account.deleted` row in
       the [ScaleUp errors](https://docs.cloud.google.com/kubernetes-engine/docs/troubleshooting/cluster-autoscaler-scale-up.md.txt#scaleup-errors) table.

     - **Private services access** : to check for recent changes in audit
       logs, see [Private services access audit
       logging](https://docs.cloud.google.com/vpc/docs/private-services-access-audit-logging).

     - **VPC Network Peering**: to check whether peering was recently
       added, updated, or removed, run the following query in
       Logs Explorer for each peered project:

       ```sql
       log_id("cloudaudit.googleapis.com/activity")
       resource.type="gce_network"
       protoPayload.methodName:("compute.networks.addPeering" OR
       "compute.networks.updatePeering" OR
       "compute.networks.removePeering")
       ```

       If a peering change disrupted connectivity to the control plane,
       then [update the peering
       connection](https://docs.cloud.google.com/vpc/docs/using-vpc-peering#update-peer-connection).
       If the connection was deleted, then see [VPC Network Peering
       connection is accidentally
       deleted](https://docs.cloud.google.com/kubernetes-engine/docs/troubleshooting/network-isolation#vpc_peering_network).
   - **Exceeded project quotas** : see the `scale.up.error.quota.exceeded`
     row in the [ScaleUp errors](https://docs.cloud.google.com/kubernetes-engine/docs/troubleshooting/cluster-autoscaler-scale-up.md.txt#scaleup-errors) table.

   - **Unavailable IP addresses** : see the
     `scale.up.error.ip.space.exhausted` row in the [ScaleUp
     errors](https://docs.cloud.google.com/kubernetes-engine/docs/troubleshooting/cluster-autoscaler-scale-up#scaleup-errors) table.

   - **Invalid instance template** : see [The instance template is not
     valid](https://docs.cloud.google.com/compute/docs/troubleshooting/troubleshooting-migs#the_instance_template_is_not_valid)
     in the Compute Engine documentation.

   - **Insufficient resources** : see the `scale.up.error.out.of.resources`
     row in the [ScaleUp errors](https://docs.cloud.google.com/kubernetes-engine/docs/troubleshooting/cluster-autoscaler-scale-up.md.txt#scaleup-errors) table.

   - **Inaccessible APIs**: in Logs Explorer, run a query to check for
     activity related to firewall rules or VPC Service Controls perimeters:

     - To check whether a firewall rule was recently changed, run the
       audit log query in the **Firewall rules** bullet point of
       **Failed node registration** . If you enabled
       [Firewall Rules Logging](https://docs.cloud.google.com/firewall/docs/vpc-firewall-rules-logging-overview.md.txt)
       for your rules, then you can also view the connections that your
       rules denied by running the following query:

       ```sql
       resource.type="gce_subnetwork"
       log_id("compute.googleapis.com/firewall")
       jsonPayload.disposition="DENIED"
       ```

       If the logs show a firewall rule that blocks traffic to required
       Google APIs, then [update or replace the firewall
       rule](https://docs.cloud.google.com/firewall/docs/using-firewalls#updating_firewall_rules) to
       allow the traffic. For more information about networking
       requirements, see [Prerequisites for network connection to Google
       APIs and
       services](https://docs.cloud.google.com/kubernetes-engine/docs/troubleshooting/node-registration#network-connection-to-apis).
     - If you use VPC Service Controls, then select your organization in
       the Google Cloud console and run the following query in
       Logs Explorer to check whether a perimeter was recently added
       or modified:

       ```sql
       logName="organizations/ORGANIZATION_ID/logs/cloudaudit.googleapis.com%2Factivity"
       protoPayload.serviceName="accesscontextmanager.googleapis.com"
       ```

       Replace `ORGANIZATION_ID` with your
       organization ID.

       You can also add the following filters to find an update or
       creation of a VPC Service Controls perimeter:

       ```sql
       severity=NOTICE
       protoPayload.methodName=("google.identity.accesscontextmanager.v1.AccessContextManager.CreateServicePerimeter"
       OR "google.identity.accesscontextmanager.v1.AccessContextManager.UpdateServicePerimeter")
       ```

       If a new or modified perimeter blocks access to required Google
       APIs, then [retrieve VPC Service Controls errors from audit
       logs](https://docs.cloud.google.com/vpc-service-controls/docs/retrieve-troubleshoot-errors) to
       identify the blocked service and [update the service
       perimeter](https://docs.cloud.google.com/vpc-service-controls/docs/manage-service-perimeters#update)
       to allow access. For more information about using cluster
       autoscaler in a perimeter, see [Autoscaler not working in
       perimeters with accessible services and restricted services
       enabled](https://docs.cloud.google.com/vpc-service-controls/docs/troubleshoot-services-within-perimeter#autoscaler-not-working).
   - **Network latency** : transient network latency usually resolves on
     subsequent scale-up attempts. For persistent latency, see
     [Troubleshooting common networking
     issues](https://docs.cloud.google.com/compute/docs/troubleshooting/troubleshooting-networking) in
     the Compute Engine documentation.

### Node auto-provisioning fails scheduling predicates

**Symptoms**:

Your Pods remain in a `Pending` state and cluster autoscaler doesn't create a
new node pool:

- For Standard clusters with node auto-provisioning enabled,
  the Google Cloud console displays the following notification:

      Can't scale up because node auto-provisioning can't provision a node pool for a Pod with failing predicates

- In Logging, for both Autopilot and Standard
  clusters, the `noDecisionStatus` event in the
  `container.googleapis.com/cluster-autoscaler-visibility` log contains the
  `no.scale.up.nap.pod.zonal.failing.predicates` message ID in the
  `jsonPayload.noDecisionStatus.noScaleUp.unhandledPodGroups[].napFailureReasons[]`
  field.

**Cause**:

Node auto-provisioning can't create a node pool because the pending Pod has
scheduling constraints that no candidate node configuration in that zone can
satisfy. These constraints include node selectors, affinity rules, untolerated
taints, topology spread constraints, or storage requirements. This issue applies
to Autopilot clusters, Standard clusters with node
auto-provisioning enabled, and custom ComputeClasses with node pool
auto-creation.

**Resolution**:

To identify the failing predicate and resolve the issue, complete the following
steps:

1. In the Google Cloud console, go to the **Logs Explorer** page.

   [Go to Logs Explorer](https://console.cloud.google.com/logs)
2. In the [query
   pane](https://docs.cloud.google.com/logging/docs/view/logs-explorer-interface#logs-viewer-interface),
   view the failing predicate details by entering the following query:

   ```sql
   resource.type="k8s_cluster"
   resource.labels.cluster_name="CLUSTER_NAME"
   log_id("container.googleapis.com/cluster-autoscaler-visibility")
   jsonPayload.noDecisionStatus.noScaleUp.unhandledPodGroups.napFailureReasons.messageId="no.scale.up.nap.pod.zonal.failing.predicates"
   ```

   Replace `CLUSTER_NAME` with the name of your cluster.
3. Click **Run query**.

4. Expand the most recent log entry and inspect the
   `jsonPayload.noDecisionStatus.noScaleUp.unhandledPodGroups[]` array. For the
   affected `podGroup.samplePod` field, examine the `parameters` array inside
   each `napFailureReasons` entry:

   - The first element in the `parameters` array is the evaluated zone.
   - Each remaining element in the `parameters` array is a predicate failure reason.

   The following example log entry excerpt shows a Pod that failed the node
   affinity or selector predicate in the `us-central1-a` zone:

   ```json
   {
     "jsonPayload": {
       "noDecisionStatus": {
         "measureTime": "1706540000",
         "noScaleUp": {
           "unhandledPodGroups": [
             {
               "napFailureReasons": [
                 {
                   "messageId": "no.scale.up.nap.pod.zonal.failing.predicates",
                   "parameters": [
                     "us-central1-a",
                     "node(s) didn't match Pod's node affinity/selector"
                   ]
                 }
               ],
               "podGroup": {
                 "samplePod": {
                   "controller": {
                     "apiVersion": "apps/v1",
                     "kind": "ReplicaSet",
                     "name": "web-app-6d4cf56db6"
                   },
                   "name": "web-app-6d4cf56db6-x8k2p",
                   "namespace": "default"
                 },
                 "totalPodCount": 1
               }
             }
           ],
           "unhandledPodGroupsTotalCount": 1
         }
       }
     }
   }
   ```
5. Resolve the scheduling conflict that corresponds to the predicate failure
   reason in the `parameters` array, as described in the following table:

   | Predicate failure reason in log | Common causes | Resolution |
   |---|---|---|
   | `node(s) didn't match Pod's node affinity/selector` | - The Pod sets the `cloud.google.com/gke-nodepool` label to target a manually created node pool. - The Pod requests a zone outside the cluster's configured auto-provisioning locations. - The Pod requests an operating system image in the `cloud.google.com/gke-os-distribution` label that doesn't match the cluster's default node image. - The Pod selects custom node labels that aren't defined in the node auto-provisioning configuration or custom ComputeClass. | - If the Pod sets the `cloud.google.com/gke-nodepool` label, node auto-provisioning can't create a node pool for that Pod. Check whether the targeted node pool has reached its maximum node limit, or remove the `cloud.google.com/gke-nodepool` selector so that node auto-provisioning can create a node pool. - If the Pod requests a specific zone, confirm that the zone is included in the configured [auto-provisioning locations](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/node-auto-provisioning.md.txt#auto-provisioning_locations). - If the Pod sets the `cloud.google.com/gke-os-distribution` label, such as `ubuntu` or `cos_containerd`, confirm that the value matches the cluster's [default node image](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/node-auto-provisioning.md.txt#set-default). Node auto-provisioning creates node pools only with the default node image. Update the default node image for node auto-provisioning, or manually create an autoscaled node pool that uses the required node image. - If the Pod selects custom node labels, verify that those labels are defined in the node auto-provisioning configuration or custom ComputeClass. Alternatively, you can follow the [workload separation](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/workload-separation.md.txt) requirements by configuring a matching node selector and toleration. Otherwise, remove unrecognized label selectors or target a manually created node pool that has those labels. - If the Pod requests a GPU type or machine series that isn't available in the zone, cluster autoscaler reports the `no.scale.up.nap.pod.zonal.illegal.config` message ID instead. To confirm zone availability, see [GPU locations](https://docs.cloud.google.com/compute/docs/gpus/gpu-regions-zones) and [Regions and zones](https://docs.cloud.google.com/compute/docs/regions-zones.md.txt). |
   | - `node(s) didn't match PersistentVolume's node affinity` - `node(s) had volume node affinity conflict` - `pod has unbound immediate PersistentVolumeClaims` | - The Pod references a PersistentVolumeClaim bound to a zonal PersistentVolume in a different zone, such as when the StorageClass uses the `volumeBindingMode: Immediate` setting. - The Pod requests a `pd-standard`, `pd-balanced`, or `pd-ssd` volume with a machine series that doesn't support that disk type when GKE enforces [disk type topology](https://docs.cloud.google.com/kubernetes-engine/docs/concepts/hyperdisk.md.txt#supported_node_scheduling), such as in Autopilot clusters or when the StorageClass sets the `use-allowed-disk-topology: "true"` parameter. | - Match the Pod's `nodeSelector` or `nodeAffinity` zone to the zone of the bound PersistentVolume. To prevent volumes from provisioning in a conflicting zone, configure your StorageClass with the `volumeBindingMode: WaitForFirstConsumer` setting. - Use a disk type that the requested machine series supports. For example, C4 and N4 machine series support only Google Cloud Hyperdisk volumes, and C3 and M3 machine series don't support `pd-standard` volumes. For more information, see [Machine series support for Hyperdisk](https://docs.cloud.google.com/compute/docs/disks/hyperdisks.md.txt#machine-type-support) and [Machine series support for Persistent Disk](https://docs.cloud.google.com/compute/docs/disks/persistent-disks.md.txt#pd-machine-series). |
   | `node(s) had untolerated taint(s)`, or a message that starts with `node(s) had untolerated taint` | The Pod targets a node auto-provisioning configuration or ComputeClass that applies taints to provisioned nodes, such as GPU taints or workload separation taints, without a matching toleration in the `spec.tolerations` field. | For taints applied by your node auto-provisioning configuration or ComputeClass, add a matching toleration to the `spec.tolerations` field in the Pod manifest. |
   | - `node(s) didn't match pod topology spread constraints` - `node(s) didn't match pod affinity rules` - `node(s) didn't match pod anti-affinity rules` - `node(s) didn't satisfy existing pods anti-affinity rules` | - The Pod defines `topologySpreadConstraints` rules with the `whenUnsatisfiable: DoNotSchedule` setting, or `podAffinity` or `podAntiAffinity` rules with the `requiredDuringSchedulingIgnoredDuringExecution` setting, that can't be satisfied by adding a node in that zone. - A zonal PersistentVolume keeps the Pod in a single zone while a topology spread constraint or anti-affinity rule blocks scheduling in that zone because another matching Pod already runs there. | - Review the Pod's `spec.topologySpreadConstraints` and `spec.affinity` fields to confirm that enough zones or topology domains are available in the cluster's [auto-provisioning locations](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/node-auto-provisioning.md.txt#auto-provisioning_locations) to satisfy the constraint. - If a zonal PersistentVolume conflicts with a strict topology spread or anti-affinity rule, change the value of the `whenUnsatisfiable` field to `ScheduleAnyway`, increase the `maxSkew` value, or update the workload's storage or anti-affinity configuration. |

### Unhealthy clusters prevent scale-up

**Symptoms**:

Cluster autoscaler halts all scaling operations, and the
`cluster-autoscaler-status` ConfigMap contains an event with the type `Warning`
and `ClusterUnhealthy` listed as the reason.

To view the `cluster-autoscaler-status` ConfigMap, run the following command:

```bash
kubectl describe configmap cluster-autoscaler-status -n kube-system
```

**Cause**:

Cluster autoscaler doesn't scale up if it considers a cluster to be unhealthy.
Cluster autoscaler determines cluster health based on the ratio of healthy and
ready nodes, not on control plane health. If 45% of nodes in a cluster are
unhealthy or not ready, cluster autoscaler halts all operations.

Cluster autoscaler also considers a ready node to be `NotReady` when the node
has a taint with the `ignore-taint.cluster-autoscaler.kubernetes.io/` prefix.
Cluster autoscaler treats the node as `NotReady` for as long as that taint is
present.

**Resolution**:

To resolve this issue, decrease the number of unhealthy nodes by doing one or
both of the following actions:

- Investigate and repair unhealthy or `NotReady` nodes so that fewer than 45% of nodes in the cluster are unhealthy.
- If ready nodes have a taint with the `ignore-taint.cluster-autoscaler.kubernetes.io/` prefix, remove the taint.

## What's next

- Review the [Kubernetes cluster autoscaler
  FAQ](https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/FAQ.md).
- [Watch a YouTube video about troubleshooting and resolving scaling
  issues](https://www.youtube.com/watch?v=IdCjKUZah-Q).
- [Troubleshoot custom ComputeClass issues](https://docs.cloud.google.com/kubernetes-engine/docs/troubleshooting/custom-computeclass.md.txt).

- If you can't find a solution to your problem in the documentation, see
  [Get support](https://docs.cloud.google.com/kubernetes-engine/docs/getting-support.md.txt) for further help,
  including advice on the following topics:

  - Opening a support case by contacting [Cloud Customer Care](https://cloud.google.com/support-hub).
  - Getting support from the community by [asking questions on StackOverflow](http://stackoverflow.com/questions/tagged/google-kubernetes-engine) and using the `google-kubernetes-engine` tag to search for similar issues. You can also join the [`#kubernetes-engine` Slack channel](https://googlecloud-community.slack.com/messages/C0B9GKTKJ/) for more community support.
  - Opening issues or feature requests by using the [public issue tracker](https://docs.cloud.google.com/support/docs/issue-trackers.md.txt).