This document helps you troubleshoot issues where GKE workloads fail
to schedule on nodes defined by a [custom ComputeClass](https://docs.cloud.google.com/kubernetes-engine/docs/concepts/about-custom-compute-classes.md.txt), or when the
cluster autoscaler does not provision the expected machine types.

This document is for Application developers and Platform admins and operators who use custom ComputeClasses to manage workload scheduling and node pool auto-creation in GKE. For more information about the common roles and example tasks referenced in Google Cloud content, see [Common GKE user roles and tasks](https://docs.cloud.google.com/kubernetes-engine/enterprise/docs/concepts/roles-tasks.md.txt).

> [!NOTE]
> **Note:** This document applies to both Standard and Autopilot clusters. Because direct node access such as SSH is not supported in Autopilot clusters, you must adjust your troubleshooting steps for those environments.

## Key concepts

To help you troubleshoot, make sure you're familiar with the following GKE components and mechanisms:

- **Custom ComputeClass** : a GKE-specific resource that lets you define a prioritized list of node
  configurations for autoscaling.
  For more information, see
  [About custom ComputeClasses](https://docs.cloud.google.com/kubernetes-engine/docs/concepts/about-custom-compute-classes.md.txt).

- **Cluster autoscaler** : the component that automatically adds or removes nodes in your cluster based on
  workload demand.
  For more information, see
  [About GKE Cluster autoscaling](https://docs.cloud.google.com/kubernetes-engine/docs/concepts/cluster-autoscaler.md.txt).

- **Node pool auto-creation** : GKE cluster autoscaler automatically creates and
  manages node pools based on workload requirements.
  For more information, see
  [About node poo- auto-creation](https://docs.cloud.google.com/kubernetes-engine/docs/concepts/node-auto-provisioning.md.txt).

- **Fallback logic** : the mechanism where the cluster autoscaler attempts to provision nodes matching
  your highest priority rule first.
  For more information, see
  [Choose your fallback compute priorities](https://docs.cloud.google.com/kubernetes-engine/docs/concepts/about-custom-compute-classes.md.txt#choose-priorities).

## Troubleshoot by symptoms

This document organizes troubleshooting steps in sequence, from fundamental checks to more advanced configurations. For a more comprehensive diagnosis, we recommend performing these steps in order. Alternatively, if you are experiencing specific issues, refer to the relevant links in the following table:

| Symptom | Troubleshooting steps |
|---|---|
| Pod requesting a custom ComputeClass is stuck in the `Pending` state | - [Perform basic diagnostic checks](https://docs.cloud.google.com/kubernetes-engine/docs/troubleshooting/custom-computeclass.md.txt#perform-basic-checks) - [Check Pod scheduling constraints](https://docs.cloud.google.com/kubernetes-engine/docs/troubleshooting/custom-computeclass.md.txt#scheduling-configuration) - [Node pool configuration](https://docs.cloud.google.com/kubernetes-engine/docs/troubleshooting/custom-computeclass.md.txt#node-pool-config) (for Standard clusters) - [Analyze cluster autoscaler pending Pods metric](https://docs.cloud.google.com/kubernetes-engine/docs/troubleshooting/custom-computeclass.md.txt#analyze-ca-metric) |
| Cluster autoscaler is not creating new nodes that match the custom ComputeClass | - [Verify custom ComputeClass configurations](https://docs.cloud.google.com/kubernetes-engine/docs/troubleshooting/custom-computeclass.md.txt#computeclass-validation) - [Verify node pool auto-creation configuration](https://docs.cloud.google.com/kubernetes-engine/docs/troubleshooting/custom-computeclass.md.txt#nap-integration) - [Analyze cluster autoscaler pending Pods metric](https://docs.cloud.google.com/kubernetes-engine/docs/troubleshooting/custom-computeclass.md.txt#analyze-ca-metric) - [Analyze node provisioning failures](https://docs.cloud.google.com/kubernetes-engine/docs/troubleshooting/custom-computeclass.md.txt#analyze-provisioning-failures) |
| Cluster autoscaler creates default machine types instead of specialized types | [Analyze autoscaler fallback behavior](https://docs.cloud.google.com/kubernetes-engine/docs/troubleshooting/custom-computeclass.md.txt#fallback-analysis) |
| Output from the `kubectl describe computeclass` command shows an unhealthy status | [Verify custom ComputeClass configurations](https://docs.cloud.google.com/kubernetes-engine/docs/troubleshooting/custom-computeclass.md.txt#computeclass-validation) |
| Workloads are not migrating to higher-priority nodes | [Verify general cluster autoscaler health](https://docs.cloud.google.com/kubernetes-engine/docs/troubleshooting/custom-computeclass.md.txt#general-autoscaler-health) |
| GPU or Spot VMs workloads have scheduling issues | - [Check Pod scheduling constraints](https://docs.cloud.google.com/kubernetes-engine/docs/troubleshooting/custom-computeclass.md.txt#scheduling-configuration) - [Verify GPU or Spot VMs configuration](https://docs.cloud.google.com/kubernetes-engine/docs/troubleshooting/custom-computeclass.md.txt#advanced-config) - [Analyze autoscaler fallback behavior](https://docs.cloud.google.com/kubernetes-engine/docs/troubleshooting/custom-computeclass.md.txt#fallback-analysis) |
| Capacity buffers in a custom ComputeClass are rejected, empty, or don't provision capacity | - [The ComputeClass rejects or ignores the buffers field](https://docs.cloud.google.com/kubernetes-engine/docs/troubleshooting/custom-computeclass.md.txt#buffers-field-rejected) - [Check `BuffersConfigured` condition reasons](https://docs.cloud.google.com/kubernetes-engine/docs/troubleshooting/custom-computeclass.md.txt#buffersconfigured-condition) - [Buffers are ready but GKE doesn't provision capacity](https://docs.cloud.google.com/kubernetes-engine/docs/troubleshooting/custom-computeclass.md.txt#buffers-no-capacity) |

## Out-of-scope issues

This document doesn't address the following issues:

- General GKE networking issues, such as Pod-to-Pod connectivity or service load balancing.
- Issues related to Horizontal Pod Autoscaler (HPA) or Vertical Pod Autoscaler (VPA).
- Issues unrelated to the custom ComputeClass mechanism, such as application-level errors or PersistentVolume problems that aren't related to scheduling constraints.
- Quota or [resource unavailability](https://docs.cloud.google.com/compute/docs/troubleshooting/troubleshooting-resource-availability.md.txt) issues that aren't directly related to the ComputeClass fallback logic.

## Identify placeholder variables

To customize the commands in this document, enter your specific values into the `Variable` column. Edited values are automatically synchronized across all code blocks and commands.

| Variable | Description |
|---|---|
| <var class="edit" scope="PROJECT_ID" translate="no">PROJECT_ID</var> | Your Google Cloud project ID. |
| <var class="edit" scope="LOCATION" translate="no">LOCATION</var> | The Compute Engine region or zone where the cluster is located. |
| <var class="edit" scope="CLUSTER_NAME" translate="no">CLUSTER_NAME</var> | The name of your cluster. |
| <var class="edit" scope="NODE_POOL_NAME" translate="no">NODE_POOL_NAME</var> | The name of the node pool to inspect (if applicable for Standard clusters). |
| <var class="edit" scope="CUSTOM_COMPUTECLASS_NAME" translate="no">CUSTOM_COMPUTECLASS_NAME</var> | The name of the custom ComputeClass requested by the Pod. |
| <var class="edit" scope="NAMESPACE" translate="no">NAMESPACE</var> | The namespace of the Pod that is failing to schedule. |
| <var class="edit" scope="POD_NAME" translate="no">POD_NAME</var> | The name of the Pod that is failing to schedule. |
| <var class="edit" scope="BUFFER_NAME" translate="no">BUFFER_NAME</var> | The name of a CapacityBuffer that GKE created for the ComputeClass. |

## Before you begin


To get the permissions that
you need to perform the tasks in this document,

ask your administrator to grant you the
following IAM roles on your Google Cloud project:

- To access GKE clusters: Kubernetes Engine Cluster Viewer (`roles/container.viewer`).
- To view logs: Logs Viewer (`roles/logging.viewer`).
- To manage GKE clusters: Kubernetes Engine Admin (`roles/container.admin`).


For more information about granting roles, see [Manage access to projects, folders, and organizations](https://docs.cloud.google.com/iam/docs/granting-changing-revoking-access.md.txt).


You might also be able to get
the required permissions through [custom
roles](https://docs.cloud.google.com/iam/docs/creating-custom-roles) or other [predefined
roles](https://docs.cloud.google.com/iam/docs/roles-overview#predefined).

To configure `kubectl` to communicate with the cluster, run the following
command:

```bash
  gcloud container clusters get-credentials CLUSTER_NAME
      --location LOCATION
      --project PROJECT_ID
```

## Perform basic diagnostic checks

Verify that the core components are correctly configured and that the cluster
supports custom ComputeClasses.

### Verify Pod status and selector

Confirm that the Pod is in the `Pending` state and correctly requests the custom ComputeClass.

1. List Pods in the `Pending` state:

   ```bash
   kubectl get pods --all-namespaces -o wide | grep Pending
   ```
2. Inspect the Pod's specification for the `nodeSelector` field:

   ```bash
   kubectl get pod POD_NAME
       -n NAMESPACE
       -o jsonpath='{.spec.nodeSelector}'
   ```

#### Evaluate the result

- **Output shows the label:** the `nodeSelector` field is correctly configured with the `cloud.google.com/compute-class` label.
- **Output doesn't show the label:**
  - **Interpretation:** there might be an incorrect or missing `nodeSelector` field for the `cloud.google.com/compute-class` label in your workload's deployment configuration.
  - **Resolution:** modify your workload's YAML file, such as a Deployment or Job, to include the `nodeSelector` field in the `spec.template.spec` section.

### Verify cluster version compatibility

Custom ComputeClasses require GKE version 1.30.3-gke.1451000 or later. Verify that your cluster is running a version that supports custom ComputeClasses.

Check your cluster version:

```bash
gcloud container clusters describe CLUSTER_NAME
    --location LOCATION
    --format="value(currentMasterVersion)"
```

#### Evaluate the result

- **Version `1.30.3-gke.1451000` or later:** your cluster version supports custom ComputeClasses.
- **Version earlier than `1.30.3-gke.1451000`:**
  - **Interpretation:** your cluster has not been upgraded to a version that supports custom ComputeClasses.
  - **Resolution:** you must [upgrade your cluster](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/upgrading-a-cluster.md.txt) to use custom ComputeClasses.

### Verify provisioned nodes for the custom ComputeClass

To check whether there are nodes in the cluster that are associated with your
ComputeClass, list the nodes matching the ComputeClass label:

```bash
kubectl get nodes -l cloud.google.com/compute-class=CUSTOM_COMPUTECLASS_NAME
```

If the output is `No resources found`, then there are no nodes that are associated
with the ComputeClass. Verify that your workloads actually selected that ComputeClass
and check your autoscaling events for scale-up errors.

## Verify custom ComputeClass configurations

Misconfigurations in the custom `ComputeClass` resource can prevent Pods from scheduling or prevent GKE from provisioning nodes correctly.

### Check ComputeClass health and status

Verify that GKE reports the custom ComputeClass as healthy.

1. List all `ComputeClass` resources:

   ```bash
   kubectl get computeclass
   ```
2. Describe the specific `ComputeClass` resource:

   ```bash
   kubectl describe computeclass CUSTOM_COMPUTECLASS_NAME
   ```

#### Evaluate the result

- **Health status shows `True`:** the custom ComputeClass is healthy. The following is an example of a healthy custom ComputeClass:

      Status:
        Conditions:
          Last Transition Time:  2024-01-19T17:18:48Z
          Message:               CCC is healthy.
          Reason:                Health
          Status: True
          Type:                  Health

- **Health status shows `False`:**

  - **Interpretation:** the custom ComputeClass is unhealthy. Review the `Message` and `Reason` fields in the output to identify the issue.
  - **Resolution:** perform the action that corresponds to the `Reason` field in the output:
    - `NodePoolNotExist`: ensure the referenced node pool exists or update the ComputeClass to reference an existing node pool.
    - `ReservationUnusable`: check the configuration and usage of the referenced reservation.
    - `Invalid machine type`: update the ComputeClass to use a machine type that is supported in the cluster's region or zone.

### Validate the `unsatisfiable` policy

The `whenUnsatisfiable` field determines behavior when no priority rules can be
met.

Check the policy:

```bash
kubectl get computeclass CUSTOM_COMPUTECLASS_NAME -o yaml
```

Inspect the `spec.whenUnsatisfiable` field in the output. This field can have one of the
following values:

- `DoNotScaleUp`: Pods remain in the `Pending` state if no preferred nodes can be created.
- `ScaleUpAnyway`: Pods might run on default node types (like E2-series) if preferred nodes are unavailable.

#### Evaluate the result

The effect of the `whenUnsatisfiable` policy depends on its value:

- **If the value is `DoNotScaleUp`:**
  - **Interpretation:** this is expected behavior when no priority rules can be satisfied, possibly due to [resource unavailability](https://docs.cloud.google.com/kubernetes-engine/docs/troubleshooting/custom-computeclass.md.txt#check-unavailability) or [quota limits](https://docs.cloud.google.com/kubernetes-engine/docs/troubleshooting/custom-computeclass.md.txt#validate-quotas). If Pods must wait for specific hardware, this value is correct.
  - **Resolution:** if running the workload is more critical than running on specific hardware, change the policy to `ScaleUpAnyway`.
- **If the value is `ScaleUpAnyway`:**
  - **Interpretation:** this is expected behavior. GKE is falling back to default node types because preferred nodes are unavailable.
  - **Resolution:** if Pods must not run on default node types, change the policy to `DoNotScaleUp`.
- **Default behavior:** if you haven't specified a value for the `whenUnsatisfiable` field and are using a GKE version earlier than 1.33, the policy defaults to `ScaleUpAnyway`.

The following example shows how to update the policy by editing the `whenUnsatisfiable` field in your ComputeClass manifest:

```yaml
apiVersion: cloud.google.com/v1
kind: ComputeClass
metadata:
  name: CUSTOM_COMPUTECLASS_NAME
spec:
  # ... other fields
  whenUnsatisfiable: DoNotScaleUp # or ScaleUpAnyway
```

## Check Pod scheduling constraints

Ensure that your Pod specification is compatible with the attributes of nodes provisioned by the custom ComputeClass.

### Verify Pod resource requests

Check if the Pod's CPU, memory, and GPU requests can be satisfied by at least one of the machine types defined in the custom ComputeClass's `priorities` field.

1. Get Pod resource requests:

   ```bash
   kubectl get pod POD_NAME
       -n NAMESPACE
       -o jsonpath='{.spec.containers[*].resources.requests}'
   ```

   Inspect the output for `cpu`, `memory`, and GPU requests, such as `nvidia.com/gpu`.
2. Compare these requests with the machine types defined in the `priorities`
   field of the custom ComputeClass:

   ```bash
   kubectl get computeclass CUSTOM_COMPUTECLASS_NAME
       -o jsonpath='{.spec.priorities}'
   ```

   Inspect the output for the `machineType` or `machineFamily` fields. For each machine type in the custom ComputeClass, check its specifications in the [machine types documentation](https://docs.cloud.google.com/compute/docs/machine-types) and verify if its allocatable resources are greater than the Pod's requests.

#### Evaluate the result

- **Resources compatible:** Pod's resource requests are less than or equal to the allocatable resources of at least one machine type in the ComputeClass.
- **Resources exceed capacity:**

  - **Interpretation:** the Pod cannot be scheduled because no machine type in the ComputeClass provides sufficient CPU, memory, or GPU. This can also happen if `nodePoolAutoCreation` field is set to `true` and the Pod's memory request exceeds the limits of auto-created node pools.
  - **Resolution:** adjust either the Pod's resource requests or the custom ComputeClass's machine types:
    - Reduce Pod resource requests: if the resource requests are high, lower the `cpu`, `memory`, or `gpu` values in the workload's YAML file.
    - Change to larger machine types: if the Pod's requests are justified, modify the `spec.priorities` field in your custom ComputeClass manifest to include larger `machineType` or `machineFamily` options that can fulfill the Pod demands. For example:

  ```yaml
  spec:
    priorities:
    - machineType: n2d-highmem-96 # A larger machine type
      spot: true
    # ... other priorities
  ```

### Check for conflicting taints and tolerations

Nodes created for a custom ComputeClass have the
`cloud.google.com/compute-class=CUSTOM_COMPUTECLASS_NAME:NoSchedule` taint.
GKE adds this toleration to Pods automatically.

However, specialized hardware like GPUs have additional taints such as
the `nvidia.com/gpu=present:NoSchedule` taint. If your ComputeClass uses
nodes with specialized hardware, Pods *must* have a toleration for these
taints to schedule on those nodes.

Check the `tolerations` field for the Pod:

```bash
kubectl get pod POD_NAME
    -n NAMESPACE
    -o jsonpath='{.spec.tolerations}'
```

#### Evaluate the result

- **Correct tolerations:** taints and tolerations are correctly configured.
- **Missing tolerations:**

  - **Interpretation:** missing tolerations prevent the Pod from scheduling on nodes with specialized hardware taints. For example, if the ComputeClass uses GPU nodes, the Pod might be missing the `nvidia.com/gpu=present:NoSchedule` toleration. For GPU-specific requirements, see [Verify GPU configuration](https://docs.cloud.google.com/kubernetes-engine/docs/troubleshooting/custom-computeclass.md.txt#verify-gpu).
  - **Resolution:** in the `tolerations` field in your Pod specification, add the necessary tolerations to match the taints on the nodes defined by the ComputeClass. For example, for GPU nodes, add a toleration for the `nvidia.com/gpu=present:NoSchedule` taint as follows:

    ```yaml
    spec:
    template:
    spec:
    tolerations:
    - key: "nvidia.com/gpu"
    operator: "Exists"
    effect: "NoSchedule"
    # ... other tolerations and Pod spec
    ```

## Analyze the cluster autoscaler pending Pods metric

*Applicable GKE versions: 1.37 and later*

To understand why Pods requesting a specific ComputeClass are
pending, use the `kubernetes.io/autoscaler/cluster_pending_pods_per_ccc` metric
in Cloud Monitoring. This metric helps you pinpoint whether the issue is a
lack of suitable nodes for the ComputeClass, a problem with provisioning,
or something inherent to the Pod's configuration. This metric shows the number
of Pods the cluster autoscaler is processing, broken down by the
ComputeClass, the reason for being pending, and whether it's a system or
user Pod.

### Steps to query

To query the metric in the Google Cloud console, perform the following steps:

1. In the Google Cloud console, go to the Metrics explorer page.
2. Select the **Resource type:** `k8s_entity`.
3. Select the **Metric:** `kubernetes.io/autoscaler/cluster_pending_pods_per_ccc`.
4. Add a **Filter:** `entity_type = ComputeClass`.
5. Add a **Filter** for your specific ComputeClass: `entity_name = "COMPUTECLASS_NAME"`.
   - To see Pods that are not using any ComputeClass, filter with `entity_name = ""`.
6. **Group by** `metric.labels.reason` to see the autoscaler states for these Pods. You can also group Pods by using the `metric.labels.type` key label.

### Example PromQL query

To find the status of pending user Pods requesting the
"<var translate="no">COMPUTECLASS_NAME</var>" ComputeClass, use the following PromQL query:

```promql
sum by (reason) (
  kubernetes_io:autoscaler_cluster_pending_pods_per_ccc{
    monitored_resource="k8s_entity",
    entity_type="ComputeClass",
    entity_name="COMPUTECLASS_NAME",
    type="user_pod"
  }
)
```

### Metric reasons

The output of the `metric.labels.reason` key label might include the following:

- **`unable_to_provision`**: The cluster autoscaler is blocked.
  Pods in this state for a ComputeClass can't be scheduled because new nodes can't be
  created. Check cluster autoscaler logs in the control plane for details
  about stockouts, quotas, or misconfigurations within the ComputeClass definition.
  Also review cluster autoscaler visibility events.

- **`provisioning_in_progress`**: A scale-up has been triggered for nodes that should
  match the ComputeClass requirements. If Pods remain here, there might be delays in node
  creation or readiness.

- **`unprocessed`**: The cluster autoscaler hasn't yet evaluated these Pods. A
  persistent high number might indicate autoscaler lag.

- **`no_action_taken`** : The cluster autoscaler has decided not to scale for these Pods.
  This behavior might result from Pod constraints that can't be satisfied by
  the ComputeClass's node types, or other cluster-level limits. Examine the Pod
  specification (`kubectl describe pod`) and cluster autoscaler logs.

## Analyze node provisioning failures

*Applicable GKE versions: 1.37 and later*

To check the current health status of a ComputeClass, see
[Verify ComputeClass health](https://docs.cloud.google.com/kubernetes-engine/docs/troubleshooting/custom-computeclass.md.txt#check-health). If you observe that Pods associated
with a ComputeClass are in an `unable_to_provision` state, the cluster autoscaler
is not provisioning new nodes that match the requirements of the ComputeClass.
You can use the
`kubernetes.io/autoscaler/cluster_node_provisioning_failed_attempts_count_per_ccc`
Cloud Monitoring metric to diagnose the root cause.

1. In the Google Cloud console, go to **Monitoring** , then **Metrics Explorer**.
2. Select the **Resource type** as `k8s_entity`.
3. Select the **Metric** as `kubernetes.io/autoscaler/cluster_node_provisioning_failed_attempts_count_per_ccc`.
4. Add a **Filter** : `resource.labels.entity_type = ComputeClass`.
5. Add a **Filter** for your specific ComputeClass: `entity_name = "CUSTOM_COMPUTECLASS_NAME"`.
   - To see Pods *not* using any ComputeClass, filter with `entity_name = ""`.
6. In the **Group By** section, select `metric.labels.reason` and `resource.labels.entity_name`.
7. Choose an **Aggregator** , such as `rate` or `delta`.

The rate or change in failures is displayed, broken down by the reason.

### Interpret failure reasons

The `metric.labels.reason` field provides insight into why node creation is
failing. By identifying the primary reasons for failure, you can take corrective
actions.

The possible reasons include the following:

| Reason | Explanation | Potential corrective actions |
|---|---|---|
| `QUOTA_EXCEEDED` | A Google Cloud quota limit was reached (for example, CPUs, disks, or IP addresses). Review your project quotas. | [Request a quota increase](https://docs.cloud.google.com/docs/quota_detail/view_manage#requesting_higher_quota). |
| `RESOURCE_POOL_EXHAUSTED` | Compute Engine unavailability; insufficient capacity for the requested resources in the zone. | Try different machine types or zones. Modify the ComputeClass to include [fallback machine types or zones](https://docs.cloud.google.com/kubernetes-engine/docs/concepts/about-custom-compute-classes.md.txt#choose-priorities). Use [reservations](https://docs.cloud.google.com/compute/docs/instances/reservations-overview.md.txt). |
| `IP_SPACE_EXHAUSTED` | The node subnet has run out of available IP addresses. | [Expand the subnet's primary IP address range](https://docs.cloud.google.com/vpc/docs/using-vpc#expand-subnet). |
| `PERMISSIONS_ERROR` | The GKE service account lacks necessary IAM permissions to create resources (for example, instances, disks). | Grant required roles to the [GKE service account](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/hardening-your-cluster.md.txt#use_least_privilege). |
| `VM_EXTERNAL_IP_ACCESS_POLICY_CONSTRAINT` | An organization policy (`constraints/compute.vmExternalIpAccess`) prevents creating nodes with external IPs as configured. | Review and modify the [organization policy](https://docs.cloud.google.com/resource-manager/docs/organization-policy/using-constraints#boolean-constraint) or ComputeClass node configuration. |
| `INVALID_RESERVATION` | The configured Compute Engine reservation is invalid, misconfigured, or doesn't match the node request. | Check the [reservation settings](https://docs.cloud.google.com/compute/docs/instances/reservations-overview.md.txt) and ComputeClass definition. |
| `RESERVATION_NOT_FOUND` | The specified Compute Engine reservation name does not exist. | Verify the reservation name in the ComputeClass. |
| `RESERVATION_NOT_READY` | The targeted reservation is not yet active or is still being provisioned. | Wait for the reservation to become active. |
| `RESERVATION_CAPACITY_EXCEEDED` | The reservation is full and cannot accommodate the request. | Increase reservation size or use other machine types. |
| `RESERVATION_INCOMPATIBLE` | The node configuration (for example, machine type or zone) does not match the reservation's properties. | Align the ComputeClass node spec with the reservation's properties. |
| `AUTOMATIC_RESERVATIONS_NOT_AVAILABLE` | An issue occurred while trying to use automatic reservations. | Check the logs; you might need to contact [Cloud Customer Care](https://cloud.google.com/support). |
| `AUTOMATIC_RESERVATIONS_NO_CAPACITY` | Automatic reservations lack sufficient capacity. | Might need to switch to manual reservations or different machine types. |
| `UNSUPPORTED_TPU_CONFIGURATION` | The requested TPU configuration is invalid or not supported in the selected zone or project. | Check the [TPU documentation](https://docs.cloud.google.com/tpu/docs/types-zones) and the ComputeClass. |
| `GkePersistentOperationError` | A persistent error occurred within the GKE control plane during the node creation process. | [Contact Cloud Customer Care](https://cloud.google.com/support). |
| `OTHER` | A catch-all explanation for failures that don't fit into other categories. | For more details, review the cluster autoscaler logs on the control plane. |

## Node pool configuration (Standard clusters)

In GKE Standard clusters, manually created node pools must be
labeled and tainted to work with a custom ComputeClass.

### Verify node pool labels and taints

1. Identify node pools in your custom ComputeClass. If the custom ComputeClass uses the `nodePools` field, note the names of the listed node pools:

   ```bash
   kubectl get computeclass CUSTOM_COMPUTECLASS_NAME -o yaml
   ```
2. For each node pool that you identified, verify its configuration:

   ```bash
   gcloud container node-pools describe NODE_POOL_NAME
       --cluster CLUSTER_NAME
       --location LOCATION
       --format="yaml(config.labels, config.taints)"
   ```

#### Evaluate the result

- **Node pool correctly configured:** the node pool has the label `cloud.google.com/compute-class: CUSTOM_COMPUTECLASS_NAME` and the taint `cloud.google.com/compute-class=CUSTOM_COMPUTECLASS_NAME:NoSchedule`.
- **Node pool misconfigured:**

  - **Interpretation:** the node pool was not configured with the label and taint required to associate it with the custom ComputeClass.
  - **Resolution:** [update the node pool](https://docs.cloud.google.com/sdk/gcloud/reference/container/node-pools/update.md.txt) to add the label and taint:

    1. Add or update node label:

       ```bash
       gcloud container node-pools update NODE_POOL_NAME \
           --cluster=CLUSTER_NAME --location=LOCATION \
           --node-labels=cloud.google.com/compute-class=CUSTOM_COMPUTECLASS_NAME
       ```
    2. Add or update node taint:

       ```bash
       gcloud container node-pools update NODE_POOL_NAME \
           --cluster=CLUSTER_NAME --location=LOCATION \
           --node-taints=cloud.google.com/compute-class=CUSTOM_COMPUTECLASS_NAME:NoSchedule
       ```

## Verify node pool auto-creation configuration

For both Autopilot and Standard clusters with `nodePoolAutoCreation` set to `true`,
node pool auto-creation must be correctly configured.

### Verify node pool auto-creation is enabled

1. Check if the `nodePoolAutoCreation.enabled` field in the custom ComputeClass is set to `true`:

   ```bash
   kubectl get computeclass CUSTOM_COMPUTECLASS_NAME -o yaml
   ```
2. Check if node pool auto creation is enabled on the cluster:

   ```bash
   gcloud container clusters describe CLUSTER_NAME
       --location LOCATION
       --format="value(autoscaling.enableNodeAutoprovisioning)"
   ```

If either is disabled, node pool auto-creation won't create new node pools for your custom ComputeClass.

#### Evaluate the result

- **Node pool auto-creation enabled:** in the ComputeClass, the `nodePoolAutoCreation.enabled` field is set to `true`, and node auto-provisioning is enabled at the cluster-level.
- **Node pool auto-creation disabled:**

  - **Interpretation:** node pool auto-creation is disabled if the value of `nodePoolAutoCreation.enabled` field is `false` or missing in the ComputeClass, or if cluster-level node auto-provisioning is disabled.
  - **Resolution:** enable node pool auto-creation:

    1. Edit the custom ComputeClass YAML file to include `nodePoolAutoCreation: enabled: true`:

       ```yaml
       spec:
         # ... priorities
         nodePoolAutoCreation:
           enabled: true
       ```
    2. [Enable node pool auto-creation at cluster level and configure resource limits](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/node-auto-provisioning.md.txt#enable):

       ```bash
       gcloud container clusters update CLUSTER_NAME --location LOCATION \
         --enable-autoprovisioning \
         --autoprovisioning-min-cpu=MIN_CPU \
         --autoprovisioning-max-cpu=MAX_CPU \
         --autoprovisioning-min-memory=MIN_MEMORY \
         --autoprovisioning-max-memory=MAX_MEMORY
       ```

### Check resource limits for node pool auto-creation

Node pool auto-creation has cluster-wide limits for CPU and memory. If the cluster's current usage plus the resources of a new node exceeds these
limits, node pool auto-creation won't provision new nodes.

1. View the resource limits:

   ```bash
   gcloud container clusters describe CLUSTER_NAME
       --location LOCATION
   --format="value(autoscaling.resourceLimits)"
   ```

   The output lists the `resourceType`, `minimum`, and `maximum` fields for CPU and memory (in GB).
2. Review the machine types in your custom ComputeClass's priorities. Check their CPU and memory specifications in the [machine types documentation](https://docs.cloud.google.com/compute/docs/machine-types).

3. Determine the current total CPU and memory capacity of all nodes in the cluster. The sum of current capacity plus the resources of a potential new node must not exceed the maximum limit for node pool auto-creation.

#### Evaluate the result

- **Sufficient capacity:** the cluster has sufficient CPU and memory capacity within resource limits for node pool auto-creation to provision a new node.
- **Limits exceeded:**

  - **Interpretation:** node pool auto-creation can't provision new nodes because the cluster has reached CPU or memory limits, or limits were set too low for the machine types in the ComputeClass.
  - **Resolution:** increase the resource limits for node pool auto-creation:

    1. Determine new maximum limits that account for current usage and future growth, including the largest machine types in the custom ComputeClass.

    2. [Update resource limits for node pool auto-creation](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/node-auto-provisioning.md.txt#set_limits). You can set multiple resources in one command:

       ```bash
       gcloud container clusters update CLUSTER_NAME --location LOCATION \
         --set-nap-resource-limits resourceType=cpu,maximum=NEW_MAX_CPU \
         --set-nap-resource-limits resourceType=memory,maximum=NEW_MAX_GB
       ```

## Analyze autoscaler fallback behavior

This section helps you investigate external factors to understand why the cluster autoscaler might be skipping preferred options and using fallbacks, or failing to scale up.

Custom ComputeClasses use prioritized fallback logic. If a Pod isn't scheduling on nodes matching the highest-priority rule, it's often due to constraints like resource unavailability or project quotas. When GKE can't provision nodes matching a specific priority rule, for example, due to a `ZONE_RESOURCE_POOL_EXHAUSTED` or `QUOTA_EXCEEDED` error from Compute Engine, the cluster autoscaler immediately tries the next rule in the `priorities` list. There is no waiting period before GKE falls back to the next priority, except when using TPUs or the Flex Start provisioning model, which support a configurable delay.

### Check for resource unavailability

Verify if resources are unavailable in the specified zone by checking cluster autoscaler logs or Compute Engine Managed instance group (MIG) errors.

**Option 1: Check cluster autoscaler visibility events**

In the Google Cloud console, go to **Cloud Logging \> Logs Explorer** and run the following query to find autoscaler events that might indicate resource unavailability:

```sql
resource.type="k8s_cluster"
resource.labels.location="LOCATION"
resource.labels.cluster_name="CLUSTER_NAME"
log_id("container.googleapis.com/cluster-autoscaler-visibility")
jsonPayload.noScaleUpReason.messageId="no.scale.up.nap.resource.exhausted"
```

**Option 2: Check MIG errors**

You can check for MIG errors in the Google Cloud console or by using a Cloud Logging query.

- **Using Google Cloud console**:

  1. In the Google Cloud console, go to **Compute Engine** \> **Instance groups**.
  2. Find the MIG corresponding to the node pool that is failing to scale up.
  3. Click the MIG name and go to the **Errors** tab. Look for messages indicating resource exhaustion.
- **Using Cloud Logging query**:

  1. In the Google Cloud console, go to **Cloud Logging \> Logs explorer**.
  2. Run the following query to check for resource exhaustion errors from the MIG:

  ```sql
  resource.type="gce_instance"
  log_id("cloudaudit.googleapis.com/activity")
  protoPayload.status.message:("ZONE_RESOURCE_POOL_EXHAUSTED" OR "does not have enough resources available to fulfill the request" OR "resource pool exhausted" OR "does not exist in zone")
  ```

#### Evaluate the result

- **Resources are available:** if logs don't show the `ZONE_RESOURCE_POOL_EXHAUSTED` messages, resource unavailability is unlikely to be the cause of scale-up failure.
- **Resources are not available:**

  - **Interpretation:** node provisioning fails due to a temporary high demand for a specific machine type (especially Spot VMs or GPUs) in that zone, or because a Pod is constrained by PersistentVolume affinity to a zone that's experiencing resource unavailability.
  - **Resolution:** Resource unavailability is transient, but you can improve resilience by adding flexibility to your configuration:

    - **Diversify machine types** : ensure the `spec.priorities` field in the custom ComputeClass contains multiple machine types or families as fallbacks:

      ```yaml
      spec:
        priorities:
        - machineFamily: c3  # Highest priority
        - machineFamily: n2d # Fallback option
        - machineFamily: e2  # Lowest priority
      ```
    - **Use regional clusters**: if the cluster is zonal, it is vulnerable to resource unavailability in that single zone. Using regional clusters lets the cluster autoscaler attempt to provision nodes in other zones within the region where capacity might be available.

    - **Use Compute Engine reservations** : for critical workloads that cannot tolerate delays, [create Compute Engine reservations](https://docs.cloud.google.com/compute/docs/instances/reservations-overview.md.txt) to help ensure capacity for specific machine types.

### Verify project quotas

Confirm that your project has sufficient quota for the resources, such as CPUs, GPUs, and IP addresses required by the new nodes.

1. Check autoscaler logs for quota errors. Use Cloud Logging to search for quota-related error messages in the autoscaler visibility events:

   ```sql
   resource.type="k8s_cluster"
   resource.labels.location="LOCATION"
   resource.labels.cluster_name="CLUSTER_NAME"
   log_id("container.googleapis.com/cluster-autoscaler-visibility")
   jsonPayload.noScaleUpReason.messageId="no.scale.up.nap.quota.exceeded"
   ```

   Alternatively, use the following Cloud Logging query to check logs for quota-related errors from the MIG:

   ```sql
   resource.type="gce_instance"
   protoPayload.methodName:"compute.instances.insert"
   protoPayload.status.message:"QUOTA_EXCEEDED"
   severity=ERROR
   ```
2. Review quotas in Google Cloud console:

   1. In the Google Cloud console, go to **IAM \& Admin \> Quotas**.
   2. Filter for the **Compute Engine API** service.
   3. Check the usage for relevant metrics like CPUs, GPUs (all types), and in-use IP addresses for the region your GKE cluster is in. Verify that current usage is not at the limit.

#### Evaluate the result

- **Quota below limits:** if quota usage is below the quota limits and no `QUOTA_EXCEEDED` errors are found in the logs, quota limits won't block scale-up.
- **Quota exceeded:**
  - **Interpretation:** node provisioning fails due to insufficient quota for resources like CPUs, GPUs, IP addresses, or MIGs.
  - **Resolution:** if your project has reached a quota limit, [request a quota increase](https://docs.cloud.google.com/docs/quota_detail/view_manage#requesting_higher_quota).

## Advanced configurations

Configurations such as GPUs, Spot VMs, and Compute Engine reservations have their own specific requirements and potential points of failure that need to be checked.

### Verify GPU configuration

For custom ComputeClasses that provision GPU nodes, validate the GPU configuration in the custom ComputeClass and ensure the Pod has the mandatory `nvidia.com/gpu` toleration.

1. Check the custom ComputeClass's YAML for a `gpu` block within a priority rule:

   ```bash
   kubectl get computeclass CUSTOM_COMPUTECLASS_NAME -o yaml
   ```

   The `gpu` block should specify a `type` field and `count` field, for example:

   ```yaml
   priorities:
   - machineType: a2-highgpu-1g
     gpu:
   type: nvidia-tesla-a100
   count: 1
   ```
2. Inspect the Pod for GPU toleration. Any Pod that needs to be scheduled on a GPU node *must* have the `nvidia.com/gpu` toleration, even if the Pod doesn't request a GPU itself.

   ```bash
   kubectl get pod POD_NAME -n NAMESPACE -o jsonpath='{.spec.tolerations}'
   ```

   Check for the toleration under `spec.tolerations` field.

#### Evaluate the result

- **GPU configured correctly:** if the ComputeClass defines GPU `type` and `count`, and Pods include the `nvidia.com/gpu` toleration, the GPU configuration is correct. Following shows the required toleration:

  ```yaml
  tolerations:
  - key: "nvidia.com/gpu"
  operator: "Exists"
  effect: "NoSchedule"
  ```
- **GPU misconfigured:**

  - **Interpretation:** the Pod might be missing the required `nvidia.com/gpu` toleration, the ComputeClass might be unhealthy due to GPU field mismatches, or the GKE version might not handle GPU configuration correctly.
  - **Resolution:** perform one of the following actions:
    - Modify the workload's YAML file to include the mandatory GPU toleration and re-apply the YAML file.
    - Upgrade the GKE cluster. If the custom ComputeClass is unhealthy and the issue is related to GPU fields, check for known issues and upgrade to a patched GKE version, for example, 1.31.8-gke.1045000 or later.

### Verify Spot VMs configuration

If you use Spot VMs, ensure that `spot: true` setting is in the correct priority rules in your ComputeClass manifest. Also, ensure that you understand the cluster autoscaler's pricing logic.

Inspect the ComputeClass manifest:

```bash
kubectl get computeclass CUSTOM_COMPUTECLASS_NAME -o yaml
```

In the output, look for `spot: true` in `spec.priorities` field, for example:

```yaml
priorities:
- machineFamily: n2d
  spot: true
```

The cluster autoscaler might use pricing data from `us-central1` as a baseline when comparing the cost of different Spot VMs types, which can lead to seemingly non-optimal choices in other regions. This is a known behavior.

#### Evaluate the result

- **Spot VMs configured correctly:** if the `spot: true` field is specified and cluster autoscaler provisions Spot VMs, the configuration works as expected.
- **Spot VMs failing to schedule:**

  - **Interpretation:** Pods requiring Spot VMs might be failing to schedule on Spot VMs due to resource unavailability in the target zone, or because the cluster autoscaler might be choosing a different VM type based on its `us-central1` pricing model.
  - **Resolution:**

    - If you suspect resource unavailability, see [Check for resource unavailability](https://docs.cloud.google.com/kubernetes-engine/docs/troubleshooting/custom-computeclass.md.txt#check-unavailability).
    - To control the selection of Spot VMs, explicitly list `machineType` entries in your `priorities` field from cheapest to most expensive for your region. This approach gives you direct control over the fallback order. For example:

      ```yaml
      spec:
        priorities:
        - machineType: t2d-standard-48 # Cheapest in this region
          spot: true
        - machineType: n2d-standard-48 # Fallback Spot option
          spot: true
        - machineType: n2d-standard-48 # On-demand fallback
          spot: false
      ```

## General cluster autoscaler health

This section helps you check for issues that might not be directly related to the custom ComputeClass configuration but might be affecting its operation.

### Check for concurrent operations

Verify that no other cluster or node pool operations are in progress
concurrently. GKE has limits on concurrent operations,
which can block autoscaling.

List ongoing operations that are not in a `DONE` state:

```bash
gcloud container operations list \
    --location=LOCATION \
    --filter='targetLink~"/clusters/CLUSTER_NAME" AND status!=DONE'
```

If the command returns any operations, an action like a cluster upgrade, node pool creation, or another modification might be in progress. Autoscaling events might be blocked until this operation completes.

#### Evaluate the result

- **No concurrent operations:** if the `list` command returns an empty list, cluster autoscaler is not blocked by any operations.
- **Concurrent operations found:**

  - **Interpretation:** if the command lists operations with status `RUNNING` or `PENDING`, a concurrent operation, such as a cluster upgrade or node pool modification, might be in progress and blocking autoscaling.
  - **Resolution:** wait for the ongoing operation to complete. You can monitor the status using an operation ID by using:

    ```bash
    gcloud container operations wait OPERATION_ID --location LOCATION
    ```

    Replace `OPERATION_ID` with the ID from the `list` command's output. After the blocking operation completes, cluster autoscaler should resume normal function.

### Review active migration

If you observe workloads staying on lower-priority nodes when higher-priority nodes are available, verify whether active migration is enabled. If the `activeMigration.optimizeRulePriority` field is set to `false` or omitted in your ComputeClass, GKE won't automatically move workloads to higher-priority nodes when they become available.

1. To check Pod tolerations, review `spec.tolerations` field. If the Pod has tolerations that match taints on multiple node pools of different priorities, the scheduler might place it on a lower-priority node if it's available first.

   ```bash
   kubectl get pod POD_NAME -n NAMESPACE -o jsonpath='{.spec.tolerations[*]}{"\n"}'
   ```
2. To check if active migration is enabled, inspect the ComputeClass manifest for the `spec.activeMigration.optimizeRulePriority` field.

   ```bash
   kubectl get computeclass CUSTOM_COMPUTECLASS_NAME -o yaml
   ```

#### Evaluate the result

- **Active migration enabled:** if the `activeMigration.optimizeRulePriority` field is `true`, GKE attempts to move workloads to higher-priority nodes when they become available.
- **Active migration disabled or ineffective:**

  - **Interpretation:** if `activeMigration.optimizeRulePriority` field is `false` or omitted, or if Pod [tolerations](https://docs.cloud.google.com/kubernetes-engine/docs/troubleshooting/custom-computeclass.md.txt#check-taints) are too broad, workloads stay on lower-priority nodes when higher-priority nodes are available. This approach allows workloads to be scheduled on lower-priority nodes that become available first.
  - **Resolution:** if you want workloads to move to higher-priority nodes, perform one of the following actions:

    - Use more specific scheduling constraints like `nodeAffinity` to prefer higher-priority node pools.
    - Edit the ComputeClass manifest to set `activeMigration.optimizeRulePriority: true` and apply the YAML file:

      ```yaml
      spec:
        activeMigration:
          optimizeRulePriority: true
      ```

## Troubleshoot capacity buffer issues

If you add capacity buffers to a custom ComputeClass by using the `spec.buffers`
field, then you might encounter the following issues.

### The ComputeClass rejects or ignores the buffers field

When you apply a ComputeClass manifest that includes the `spec.buffers` field,
the `kubectl apply` command might fail with an error similar to the following:

    strict decoding error: unknown field "spec.buffers"

Alternatively, if you apply the manifest by using the `--validate=warn` flag,
then the command succeeds, but the ComputeClass has no `BuffersConfigured`
condition.

This issue might occur because your cluster runs a GKE version
earlier than 1.37.1-gke.3020000. To check your cluster version,
run the following command:

```bash
gcloud container clusters describe CLUSTER_NAME \
    --location=LOCATION \
    --format="value(currentMasterVersion)"
```

#### Evaluate the result

- **Version earlier than `1.37.1-gke.3020000`:**

  - **Interpretation:** the cluster runs a GKE version earlier than 1.37.1-gke.3020000, which is the minimum version for [capacity buffers in custom ComputeClasses](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/configure-capacity-buffers-compute-classes.md.txt), so the API server doesn't recognize the `spec.buffers` field.
  - **Resolution:**

    1. [Upgrade your cluster](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/upgrading-a-cluster.md.txt) to GKE version 1.37.1-gke.3020000 or later.
    2. Apply the ComputeClass manifest again.
    3. Verify that GKE saved the buffers:

       ```bash
       kubectl get computeclass CUSTOM_COMPUTECLASS_NAME \
           -o jsonpath='{.spec.buffers}{"\n"}'
       ```

       If the output lists your buffers, then GKE saved
       them.

### Check `BuffersConfigured` condition reasons

When you configure the `spec.buffers` field in a ComputeClass,
GKE adds the `BuffersConfigured` condition to the
`status.conditions` field.

Check the `BuffersConfigured` condition on the ComputeClass:

```bash
kubectl get computeclass CUSTOM_COMPUTECLASS_NAME \
    -o jsonpath='{.status.conditions[?(@.type=="BuffersConfigured")]}{"\n"}'
```

#### Evaluate the result

- **The `status` field is `True` and the `reason` field is `Ready`:** all capacity buffers in the ComputeClass are configured and ready for provisioning. If no buffer nodes appear after several minutes, then see [Buffers are ready but GKE doesn't provision capacity](https://docs.cloud.google.com/kubernetes-engine/docs/troubleshooting/custom-computeclass.md.txt#buffers-no-capacity).
- **The `status` field is `False`:**

  - **Interpretation:** at least one buffer in the ComputeClass is still initializing or failed validation. Review the `reason` and `message` fields in the output to identify the issue.
  - **Resolution:** perform the action that corresponds to the `reason`
    field in the output:

    - `Pending`: at least one CapacityBuffer is still initializing. Wait a few minutes and check the condition again.
    - `BufferIsEmpty`: the `cpu` value in the `spec.buffers.limits` field
      is less than `"2"`, or the `memory` value is less than `2Gi`, and
      the condition has a message similar to the following:

          Buffer "BUFFER_NAME": CapacityBuffer has zero replicas

      The limits must allow at least one buffer unit of 2 vCPUs and
      2 GiB of memory. Increase each value that you set in the
      `spec.buffers.limits` field to at least one buffer unit: `"2"` or
      more for the `cpu` limit, and `2Gi` or more for the `memory` limit.
      Apply the manifest again. For more information about buffer units,
      see [Configure capacity buffers in custom ComputeClasses](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/configure-capacity-buffers-compute-classes.md.txt).
    - `DuplicateProvisioningStrategy`: two or more entries in the
      `spec.buffers` field use the same provisioning strategy. Keep at
      most one active buffer (`buffer.x-k8s.io/active-capacity`) and one
      standby buffer (`buffer.gke.io/standby-capacity`) in the
      `spec.buffers` field, and then apply the manifest again.

    - `BufferProvisioningFailed`: more than one buffer in the ComputeClass
      failed validation or reconciliation. Run the
      `kubectl get computeclass
      CUSTOM_COMPUTECLASS_NAME -o yaml` command and
      inspect the `status.observedBuffers` field to view the condition for
      each individual buffer.

    - `ReconcileFailed`: GKE couldn't create, update, or
      delete the managed CapacityBuffers or PodTemplates in the
      `gke-managed-ccc` namespace. Wait for GKE to retry.
      If the error persists, then
      [contact Cloud Customer Care](https://docs.cloud.google.com/kubernetes-engine/docs/troubleshooting/custom-computeclass.md.txt#get-support).

    - Child CapacityBuffer failure reasons, such as `error`,
      `FailedToGetPodTemplate`, or `NotReadyForProvisioning`: a single
      buffer failed validation on the managed CapacityBuffer, such as when
      the ComputeClass uses the Pod-based billing model (`podFamily`
      priority rules). Ensure that the ComputeClass uses node-based
      billing and valid buffer limits.

### Buffers are ready but GKE doesn't provision capacity

If the `BuffersConfigured` condition has a `True` status and the `Ready` reason,
but no buffer nodes appear after several minutes, then check the provisioning
strategy and the scale-up events for the buffers:

1. Inspect the ComputeClass status for an unsupported provisioning strategy:

   ```bash
   kubectl get computeclass CUSTOM_COMPUTECLASS_NAME -o yaml
   ```
2. List the CapacityBuffers that GKE created for the
   ComputeClass:

   ```bash
   kubectl get capacitybuffers -n gke-managed-ccc \
       -l cloud.google.com/compute-class=CUSTOM_COMPUTECLASS_NAME
   ```
3. View the events for a buffer:

   ```bash
   kubectl describe capacitybuffer BUFFER_NAME \
       -n gke-managed-ccc
   ```

   Events expire after about an hour.

#### Evaluate the result

- **`CustomProvisioningStrategy` condition in the `status.observedBuffers`
  field:**

  - **Interpretation:** the `provisioningStrategy` field has an unsupported
    value, such as a misspelled strategy name, so the
    `status.observedBuffers` field includes a `CustomProvisioningStrategy`
    condition that has a message similar to the following:

        Buffer uses custom provisioning strategy "buffer.gke.io/standby"

    GKE treats unknown values as custom strategies and
    doesn't provision capacity.
  - **Resolution:** set the value of the `provisioningStrategy` field to
    `buffer.x-k8s.io/active-capacity` or `buffer.gke.io/standby-capacity`,
    and then apply the manifest again.

- **`NotTriggerScaleUp` or `StandbyBufferScaleUpFailed` events on the
  CapacityBuffer:**

  - **Interpretation:** the cluster autoscaler can't add nodes. For example, quota limits, node pool size limits, or a lack of available capacity can block a scale-up operation. For standby buffers, node pool auto-creation and node auto-provisioning might both be disabled, or the ComputeClass priority rules might require machine types that standby buffers don't support.
  - **Resolution:**
    - A `NotTriggerScaleUp` event describes why the cluster autoscaler can't add nodes. For more information about quota issues, see [Capacity buffer fails to provision due to quota limits](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/configure-capacity-buffer.md.txt#quota-exceeded). For standby buffers in a Standard cluster, also [verify that node pool auto-creation is enabled](https://docs.cloud.google.com/kubernetes-engine/docs/troubleshooting/custom-computeclass.md.txt#verify-nap-enabled).
    - A `StandbyBufferScaleUpFailed` warning event means that the priority rules require machine types that standby buffers don't support. Update the priority rules to use supported machine types. For more information about the machine types that standby buffers support, see the [requirements and limitations of capacity buffers](https://docs.cloud.google.com/kubernetes-engine/docs/concepts/capacity-buffer.md.txt#requirements_and_limitations).

## Get support

- If you can't find a solution to your problem in the documentation, see
  [Get support](https://docs.cloud.google.com/kubernetes-engine/docs/getting-support.md.txt) for further help,
  including advice on the following topics:

  - Opening a support case by contacting [Cloud Customer Care](https://cloud.google.com/support-hub).
  - Getting support from the community by [asking questions on StackOverflow](http://stackoverflow.com/questions/tagged/google-kubernetes-engine) and using the `google-kubernetes-engine` tag to search for similar issues. You can also join the [`#kubernetes-engine` Slack channel](https://googlecloud-community.slack.com/messages/C0B9GKTKJ/) for more community support.
  - Opening issues or feature requests by using the [public issue tracker](https://docs.cloud.google.com/support/docs/issue-trackers.md.txt).

## What's next

- [Configure capacity buffers in custom ComputeClasses](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/configure-capacity-buffers-compute-classes.md.txt)
- [Read best practices for ComputeClasses](https://docs.cloud.google.com/kubernetes-engine/docs/best-practices/computeclasses.md.txt)
- [Apply ComputeClasses to Pods by default](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/run-pods-default-compute-classes.md.txt)
- [Configure workload separation in GKE](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/node-taints)
- [Troubleshoot cluster autoscaler scale-up issues](https://docs.cloud.google.com/kubernetes-engine/docs/troubleshooting/cluster-autoscaler-scale-up.md.txt)