Capacity buffers improve the responsiveness and reliability of critical
workloads by proactively managing spare cluster capacity and suspended states of
pre-provisioned, pre-configured capacity by using CapacityBuffers. Using
capacity buffers lets you explicitly define a specific amount of unused node
capacity within your cluster. This spare capacity helps to reduce Pod scheduling
time.

When a high-priority workload needs to scale up, the workload can use the empty
capacity without waiting for node provisioning. This approach reduces latency
and avoids resource contention during sudden spikes in demand.

This page shows you how to configure fixed-replica, resource-limit, and
percentage-based capacity buffers, dedicate a buffer to a specific workload,
customize standby buffer behavior, and preload container images. If you use
custom ComputeClasses and don't need custom buffer unit sizes, replica- or
percentage-based sizing, or custom standby start and refresh times, then
[configure capacity buffers in custom ComputeClasses](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/configure-capacity-buffers-compute-classes.md.txt) instead.

## Before you begin


Before you start, make sure that you have performed the following tasks:

- Enable the Google Kubernetes Engine API.
[Enable Google Kubernetes Engine API](https://console.cloud.google.com/apis/enableflow?apiid=container.googleapis.com)
- To use the Google Cloud CLI for this task, [install](https://docs.cloud.google.com/sdk/docs/install) and then [initialize](https://docs.cloud.google.com/sdk/docs/initialize.md.txt) the gcloud CLI. If you previously installed the gcloud CLI, get the latest version by running the `gcloud components update` command. Earlier gcloud CLI versions might not support running the commands in this document.

  > [!NOTE]
  > **Note:** For existing gcloud CLI installations, make sure to set the `compute/region` [property](https://docs.cloud.google.com/sdk/docs/properties.md.txt#setting_properties). If you use primarily zonal clusters, set the `compute/zone` instead. By setting a default location, you can avoid errors in the gcloud CLI like the following: `One of [--zone, --region] must be supplied: Please specify location`. You might need to specify the location in certain commands if the location of your cluster differs from the default that you set.

<!-- -->

- Create or have access to a GKE cluster running version 1.37.1-gke.3020000 or later. Standby buffers on Autopilot clusters require [custom ComputeClasses](https://docs.cloud.google.com/kubernetes-engine/docs/concepts/about-custom-compute-classes.md.txt).
- Enable [node pool auto-creation](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/node-auto-provisioning.md.txt) on your Standard clusters. In Autopilot clusters, node pool auto-creation is already enabled. Node pool auto-creation is optional but recommended for active buffers and required for standby buffers.

## Create prerequisite Kubernetes objects

To configure a CapacityBuffer, you need a namespace that holds all of the
required objects (the CapacityBuffer itself, and additional resources like a
PodTemplate or workload). The PodTemplate and CapacityBuffer must be in
the same namespace. You can create a namespace or use an existing namespace,
including the `default` namespace.

Depending on which type of CapacityBuffer you're configuring, you also require
one of the following:

- **PodTemplate**: defines the resource requirements for a single unit of buffer capacity. The configuration specified in the CapacityBuffer object references the Pod template.
- **Workload**: an existing workload that you reference in the CapacityBuffer
  object. This guide uses a Deployment object as an example workload, but
  capacity buffers support any of the following resource types:

  - Deployment
  - ReplicaSet
  - StatefulSet
  - ReplicationController
  - Job
  - CustomResourceDefinitions (CRDs) that implement the `scale` subresource.

This section provides examples of these objects. If you already have a workload
that you want to configure with a capacity buffer, proceed to
[Apply a capacity buffer](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/configure-capacity-buffer.md.txt#apply-buffer).

To create an example Kubernetes workload, complete the following steps:

1. Save the following manifest as `namespace.yaml`:

   ```yaml
   apiVersion: v1
   kind: Namespace
   metadata:
     name: capacity-buffer-example
     labels:
       name: capacity-buffer-example
   ```

   This manifest creates a namespace called `capacity-buffer-example`.
2. Optional: to use capacity buffers with a custom ComputeClass,
   save the following manifest as `custom-compute-class.yaml`:

   ```yaml
   apiVersion: cloud.google.com/v1
   kind: ComputeClass
   metadata:
     name: ccc-example
   spec:
     # Buffers are also created according to these priorities
     priorities:
     - machineFamily: n4
     - machineFamily: n4d
     - machineFamily: c4
     - machineFamily: c4d
     nodePoolAutoCreation:
       enabled: true
   ```

   This manifest creates a custom `ComputeClass` that
   defines and controls the compute priorities for the nodes that GKE
   provisions. To learn more, see
   [custom ComputeClasses](https://docs.cloud.google.com/kubernetes-engine/docs/concepts/about-custom-compute-classes.md.txt).

   > [!CAUTION]
   > **Caution:** If you use a custom ComputeClass with standby buffers and combine [active migration to higher-priority nodes](https://docs.cloud.google.com/kubernetes-engine/docs/concepts/about-custom-compute-classes.md.txt#optimize-rule-priority) (`activeMigration.optimizeRulePriority: true`) with higher-priority Spot VMs rules, GKE gradually migrates standby nodes to Spot VMs. Because suspended Spot VMs can't convert to on-demand VMs when resuming, this configuration can prevent standby nodes from resuming during capacity constraints. For details, see [Standby buffer nodes fail to resume](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/configure-capacity-buffer.md.txt#standby-resume-fails).

3. Save the following manifest as `buffer-pod-template.yaml`:

   ```yaml
   apiVersion: v1
   kind: PodTemplate
   metadata:
     name: buffer-unit-template
     namespace: capacity-buffer-example # the namespace must be the same namespace as the CapacityBuffer
   template:
     spec:
       terminationGracePeriodSeconds: 0
       containers:
       - name: buffer-container
         image: registry.k8s.io/pause:3.9
         resources:
           requests:
             cpu: "1"
             memory: "1Gi"
           limits:
             cpu: "1"
             memory: "1Gi"
       # Optional: Using buffers with a custom ComputeClass /
       # controls the properties of the provisioned nodes.
       nodeSelector:
         cloud.google.com/compute-class: ccc-example
   ```

   This manifest creates a `PodTemplate` that defines the resource requirements
   for a single unit of buffer capacity (`1` CPU and `1Gi` Memory). This
   configuration specifies the size of capacity units that
   GKE provisions for the buffer. For example, with this
   PodTemplate, GKE won't consider nodes with less than 1 CPU
   and 1Gi of available resources as part of the buffer, if the cluster scales up.
4. Save the following manifest as `sample-workload-deployment.yaml`:

   ```yaml
   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: critical-workload-ref
     namespace: capacity-buffer-example # the namespace must be the same namespace as the CapacityBuffer
   spec:
     replicas: 10
     selector:
       matchLabels:
         app: critical-workload
     template:
       metadata:
         labels:
           app: critical-workload
       spec:
         containers:
         - name: busybox
           image: busybox
           command: ["sleep", "3600"]
           resources:
             requests:
               cpu: 100m
         # Optional: Using buffers with a custom ComputeClass /
         # controls the properties of the provisioned nodes.
         nodeSelector:
           cloud.google.com/compute-class: ccc-example
   ```

   This manifest creates a sample Deployment with 10 replicas, which is the
   reference object for the percentage-based buffer example
   in the next section.
5. Apply the manifests to your cluster:

   ```bash
   kubectl apply -f namespace.yaml -f custom-compute-class.yaml -f buffer-pod-template.yaml -f sample-workload-deployment.yaml
   ```
6. Verify that GKE created the objects:

   ```bash
   kubectl get podtemplate -n capacity-buffer-example
   kubectl get deployment critical-workload-ref -n capacity-buffer-example
   ```

   The output is similar to the following:

       NAME                   AGE
       buffer-unit-template   1m

       NAME                    READY   UP-TO-DATE   AVAILABLE   AGE
       critical-workload-ref   10/10   10           10          1m

## Apply a capacity buffer

This section provides examples of the different types of capacity buffers
that you can apply to your workloads.

> [!NOTE]
> **Note:** It might take up to 5 minutes for the capacity buffer to react to changes.

### Configure a fixed replicas buffer

Configuring a CapacityBuffer with fixed replicas specifies the exact number of
buffer units that you want based on a PodTemplate.

To create a buffer with fixed replicas, complete the following steps:

1. Save the following manifest as `cb-fixed-replicas.yaml`:

   ```yaml
   apiVersion: autoscaling.x-k8s.io/v1beta1
   kind: CapacityBuffer
   metadata:
     name: fixed-replica-buffer
     namespace: NAMESPACE
   spec:
     podTemplateRef:
       name: POD_TEMPLATE
     replicas: 3
     provisioningStrategy: "STRATEGY"
   ```

   Replace the following:
   - `NAMESPACE`: the name of your namespace, for example `capacity-buffer-example`.
   - `POD_TEMPLATE`: the PodTemplate that defines your resource requirements, for example `buffer-unit-template`.
   - `STRATEGY`: the provisioning strategy, either `"buffer.x-k8s.io/active-capacity"` (default) or `"buffer.gke.io/standby-capacity"`.

   This manifest creates a CapacityBuffer resource that references a
   PodTemplate to request a specific number of buffer units.
2. Apply the manifest:

   ```bash
   kubectl apply -f cb-fixed-replicas.yaml
   ```
3. Confirm that GKE applied the capacity buffer:

   ```bash
   kubectl get capacitybuffer fixed-replica-buffer -n NAMESPACE
   ```

   The `replicas` field
   in the status should show `3`, which reflects the number of replicas that you defined
   in the manifest. The `STATUS` field should show `ReadyForProvisioning`.

### Configure a resource limits buffer

You can use the `limits` field to define a maximum amount of resources that the
buffer should consume, calculated based on your PodTemplate size.

To create a resource limits buffer, complete the following steps:

1. Save the following manifest as `cb-resource-limits.yaml`:

   ```yaml
   apiVersion: autoscaling.x-k8s.io/v1beta1
   kind: CapacityBuffer
   metadata:
     name: resource-limit-buffer
     namespace: NAMESPACE
   spec:
     podTemplateRef:
       name: POD_TEMPLATE
     limits:
       cpu: "5"
       memory: "5Gi"
     provisioningStrategy: "STRATEGY"
   ```

   Replace the following:
   - `NAMESPACE`: the name of your namespace, for example `capacity-buffer-example`.
   - `POD_TEMPLATE`: the PodTemplate that defines your resource requirements, for example `buffer-unit-template`.
   - `STRATEGY`: the provisioning strategy, either `"buffer.x-k8s.io/active-capacity"` (default) or `"buffer.gke.io/standby-capacity"`.

   This manifest creates a CapacityBuffer resource with a total limit of 5 CPUs
   and 5 GiB Memory. If you're using the PodTemplate example from the previous
   step, you define each unit as `1` CPU and `1Gi` Memory, which should result in 5
   buffer units.
2. Apply the manifest:

   ```bash
   kubectl apply -f cb-resource-limits.yaml
   ```
3. Confirm that GKE applied the capacity buffer:

   ```bash
   kubectl get capacitybuffer resource-limit-buffer -n NAMESPACE
   ```

   Check the CapacityBuffer status. The `replicas` field should show a value
   derived from the limits that you defined. If you're using the PodTemplate
   example from the previous section, you should see `5` buffer units because this is
   the maximum number of units that fit within the defined limits.

### Configure a percentage-based buffer

Configuring a percentage-based buffer dynamically sizes the buffer based on a
percentage of an existing scalable workload. Percentage-based capacity buffers
are supported only for Kubernetes scalable objects that implement the
[scale subresource](https://kubernetes.io/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definitions/#scale-subresource), such as Deployments, StatefulSets, ReplicaSets, or Jobs. You
can't define a percentage-based buffer for Pod templates because they don't have
a `replicas` field.

In most cases, start with fixed replicas or resource limit strategies rather
than percentage-based buffers. Consider the following when deciding:

- Percentage-based buffers are less responsive to sudden scale-ups if the workload scales to low numbers or zero because the safety margin decreases in proportion to active Pods.
- Standby nodes incur compute charges during their initialization period before suspending. Dynamically scaling standby buffers alongside workload fluctuations wastes compute resources, so use fixed replicas or resource limits for standby buffers instead.
- Percentage-based buffers are mainly useful for active buffers on large deployments that never scale to low replica counts.

To keep a minimum buffer size when the referenced workload scales down, set the
`replicas` field together with the `percentage` field. GKE uses
whichever field results in more buffer units. To cap the buffer size, also set
the `limits` field.

For example, consider a buffer with `percentage: 20` and `replicas: 3` that
references a Deployment with 10 replicas. Because 20% of 10 replicas is 2, which
is less than the `replicas` value of 3, GKE provisions 3 buffer
units. If the Deployment scales to 20 replicas, the buffer grows to 4 units. For
more information, see the [CapacityBuffer reference documentation](https://docs.cloud.google.com/kubernetes-engine/docs/reference/crds/capacitybuffer.md.txt).

To create a percentage-based buffer, complete the following steps:

1. Save the following manifest as `cb-percentage-based.yaml`:

   ```yaml
   apiVersion: autoscaling.x-k8s.io/v1beta1
   kind: CapacityBuffer
   metadata:
     name: percentage-buffer
     namespace: NAMESPACE
   spec:
     scalableRef:
       apiGroup: apps
       kind: Deployment
       name: SCALABLE_RESOURCE_NAME
     percentage: 20
     provisioningStrategy: "STRATEGY"
   ```

   Replace the following:
   - `NAMESPACE`: the name of your namespace.
   - `SCALABLE_RESOURCE_NAME`: the name of your scalable resource, for example `critical-workload-ref`.
   - `STRATEGY`: the provisioning strategy, either `"buffer.x-k8s.io/active-capacity"` (default) or `"buffer.gke.io/standby-capacity"`.

   This manifest creates a CapacityBuffer resource that requests a buffer size
   equivalent to 20% of the referenced resource's replicas. If you're using the
   Deployment example from the previous section, the replica value is set to `10`.

   > [!NOTE]
   > **Note:** You can set the `percentage` field to a value greater than `100` to create a buffer larger than the referenced workload. GKE rounds calculated buffer replica counts up to the nearest whole number, with a minimum of `0` when the referenced workload has `0` replicas or `percentage` is `0`.

2. Apply the manifest:

   ```bash
   kubectl apply -f cb-percentage-based.yaml
   ```
3. Confirm that GKE applied the capacity buffer:

   ```bash
   kubectl get capacitybuffer percentage-buffer -n NAMESPACE
   ```

   Check the CapacityBuffer status. The `replicas` field should show a value
   from the percentage calculation. If you're using the Deployment example
   from the previous section, you should see `2` buffer units, which is 20% of the
   10 replicas defined in the Deployment.
4. Test the dynamic scaling by manually scaling the Deployment up to 20
   replicas:

   ```bash
   kubectl scale deployment critical-workload-ref -n NAMESPACE --replicas=20
   ```

   The CapacityBuffer controller reacts and automatically scales the buffer
   to 4 replicas.

## Dedicate a capacity buffer to a specific workload

By default, any pending Pod that matches a buffer node's scheduling
constraints can claim the buffer capacity. When multiple workloads share a
[custom ComputeClass](https://docs.cloud.google.com/kubernetes-engine/docs/concepts/about-custom-compute-classes.md.txt) or node pool, use [workload separation](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/workload-separation.md.txt) to
reserve a capacity buffer for a specific workload. This approach requires
node pool auto-creation, or a [dedicated node pool](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/isolate-workloads-dedicated-nodes.md.txt) for active buffers on
Standard clusters.

Avoid dedicating separate capacity buffers to many individual workloads. Each
dedicated buffer requires its own node pool, which increases the time to
provision and refill buffers. For recommended buffer counts, see
[Scalability](https://docs.cloud.google.com/kubernetes-engine/docs/concepts/capacity-buffer.md.txt#scalability).

To dedicate a capacity buffer to a specific workload, complete the following
steps:

1. In your workload manifest, such as `sample-workload-deployment.yaml` from
   [Create prerequisite Kubernetes objects](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/configure-capacity-buffer.md.txt#k8s-prereqs), add a custom toleration and
   node selector to the Pod specification:

   ```yaml
   spec:
     tolerations:
   - key: KEY
   operator: Equal
   value: "VALUE"
   effect: NoSchedule
   nodeSelector:
   KEY: "VALUE"
       cloud.google.com/compute-class: ccc-example
   ```

   Replace the following:
   - `KEY`: a custom key for the taint and node label, such as `group`. For restricted key prefixes, see [Limitations of workload separation](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/workload-separation.md.txt#limitations).
   - `VALUE`: a custom value that identifies the workload, such as `critical-workload`.

   On Autopilot clusters, keep a custom ComputeClass or hardware
   selector in the `nodeSelector` field so that the buffer uses node-based
   billing instead of the unsupported [Pod-based billing model](https://docs.cloud.google.com/kubernetes-engine/docs/concepts/about-autopilot-mode-standard-clusters.md.txt#pricing).
2. Configure your CapacityBuffer to use the same scheduling constraints:

   - If your CapacityBuffer uses the `scalableRef` field, reference the workload in the `spec.scalableRef` field. The buffer automatically inherits the workload's `tolerations` and `nodeSelector` fields.
   - If your CapacityBuffer uses the `podTemplateRef` field, add the same `tolerations` and `nodeSelector` fields to your PodTemplate manifest, such as `buffer-pod-template.yaml`.
3. Apply your manifests and create the CapacityBuffer. For instructions, see
   [Apply a capacity buffer](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/configure-capacity-buffer.md.txt#apply-buffer). If you
   [preload images on standby buffers](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/configure-capacity-buffer.md.txt#preload-images), also add the custom toleration to
   your preloading DaemonSet so that it can schedule on the dedicated buffer
   nodes. If the buffer uses a custom ComputeClass, the DaemonSet also needs a
   toleration for the ComputeClass taint:

   ```yaml
   tolerations:
   - key: "buffer.gke.io/standby-capacity-node-suspended"
     operator: "Exists"
   - key: KEY
   operator: Equal
   value: "VALUE"
   effect: NoSchedule
   - key: cloud.google.com/compute-class
   operator: Equal
   value: ccc-example
   effect: NoSchedule
   ```
4. After the cluster autoscaler provisions the buffer nodes, verify that the
   nodes have the dedicated node label and node taint:

   ```bash
   kubectl get nodes -l KEY=VALUE \
       -o custom-columns='NAME:.metadata.name,TAINTS:.spec.taints[*].key'
   ```

   Each node in the output lists the `KEY` taint
   key. If the buffer uses a custom ComputeClass, the nodes also list the
   `cloud.google.com/compute-class` taint key. Standby buffer nodes also have
   taints that GKE manages, such as
   `buffer.gke.io/standby-capacity-node-suspended` on suspended nodes. When
   your workload scales up, run
   `kubectl get pods -n NAMESPACE -o wide` and
   confirm that the `NODE` column lists the dedicated node names from this
   output.

## Customize standby buffer behavior

You can use annotations to customize how standby buffers start and refresh.
Add these annotations to the `metadata.annotations` field of your
CapacityBuffer resource:

- `buffer.gke.io/standby-capacity-init-time`: the amount of time a node remains active after creation before GKE suspends it. Specify a duration string such as `5m` or `1h`. The default is `5m`.
- `buffer.gke.io/standby-capacity-refresh-frequency`: how often
  GKE refreshes suspended nodes. Specify a duration string such
  as `12h` or `24h`, up to a maximum of `120h` (5 days). The default is `24h`.

  > [!WARNING]
  > **Warning:** Setting `buffer.gke.io/standby-capacity-refresh-frequency` to a duration greater than `120h` (5 days) isn't supported. In GKE version 1.37.1-gke.2337000 and later, the admission controller rejects values greater than `120h`. In earlier versions, values greater than `120h` cause an admission warning in `v1alpha1` and `v1beta1` and a validation error in `v1`.

The following example shows a manifest with these optional fields to
customize the behavior of standby buffers:

```yaml
apiVersion: autoscaling.x-k8s.io/v1beta1
kind: CapacityBuffer
metadata:
  name: customized-standby-buffer
  namespace: my-namespace
  annotations:
    buffer.gke.io/standby-capacity-init-time: "15m"
    buffer.gke.io/standby-capacity-refresh-frequency: "12h"
spec:
  podTemplateRef:
    name: buffer-unit-template
  replicas: 3
  provisioningStrategy: "buffer.gke.io/standby-capacity"
```

## Preload images on standby buffers

To speed up workload startup times when a standby node resumes, you can preload
container images by using a DaemonSet. The DaemonSet runs during the start-up
period before the node is suspended.

To preload images by using the DaemonSet, complete the following steps:

1. Save the following manifest as `image-puller-daemonset.yaml`:

   ```yaml
   apiVersion: apps/v1
   kind: DaemonSet
   metadata:
     name: image-prefetch-daemonset
     namespace: NAMESPACE
   spec:
     selector:
       matchLabels:
         name: image-prefetch
     template:
       metadata:
         labels:
           name: image-prefetch
       spec:
         tolerations:
         - key: "buffer.gke.io/standby-capacity-node-suspended"
           operator: "Exists"
         initContainers:
         - name: image-puller
           image: IMAGE_NAME
           command: ["sh", "-c", "true"]
         containers:
         - name: pause
           image: registry.k8s.io/pause:3.9
   ```

   Replace the following:
   - `NAMESPACE`: the namespace for the DaemonSet, for example `capacity-buffer-example`.
   - `IMAGE_NAME`: the name of the image to preload, for example `your-app-image:latest`.

   This manifest includes the `buffer.gke.io/standby-capacity-node-suspended`
   toleration so that the DaemonSet remains scheduled on standby nodes after
   GKE suspends them.
2. Apply the DaemonSet manifest to your cluster:

   ```bash
   kubectl apply -f image-puller-daemonset.yaml
   ```
3. Verify that the DaemonSet is created:

   ```bash
   kubectl get daemonset image-prefetch-daemonset -n NAMESPACE
   ```
4. Verify that your capacity buffer is created and ready for provisioning:

   ```bash
   kubectl get capacitybuffer CAPACITY_BUFFER_NAME -n NAMESPACE
   ```

   Check the status. The `STATUS` field should show `ReadyForProvisioning`.

## Monitor capacity buffer status and performance

You can monitor the status and health of your capacity buffers by using `kubectl`
commands and Cloud Monitoring metrics.

### Verify CapacityBuffer resource status

To check the health of your capacity buffers and verify that they are ready to
receive workloads, complete the following steps:

1. Get the status of all capacity buffers across the cluster:

   ```bash
   kubectl get capacitybuffer -A
   ```
2. Inspect the detailed status, conditions, and event logs of a specific buffer:

   ```bash
   kubectl describe capacitybuffer CAPACITY_BUFFER_NAME -n NAMESPACE
   ```

### Identify suspended standby buffer nodes

Standby buffer VMs are pre-provisioned, but kept in a suspended state to help
reduce costs. You can recognize these suspended nodes because they have
a custom condition. To audit suspended node instances, run the following
command:

```bash
kubectl get nodes -o custom-columns='NAME:.metadata.name,SUSPENDED:.status.conditions[?(@.type=="Suspended")].status'
```

A status of `True` indicates a standby VM is suspended. A status of `False` or
`<none>` indicates an active, running node.

### Monitor performance with Cloud Monitoring

To monitor the end-to-end impact of capacity buffers on scale-up times, use the
**Startup Latency** dashboards in the Google Cloud console. Use the **Pod Startup
Latency Distribution** and **Pod First Ready Latency** charts to track the
`kubernetes.io/pod/latencies/pod_first_ready` metric from Pod creation to
readiness. You can also query `Scheduled` events in Cloud Logging to measure
Pod scheduling latency. For instructions, see
[Monitor startup latency metrics](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/monitor-startup-latency-metrics.md.txt).

## Remove capacity buffers

If you no longer need a capacity buffer for your workloads, delete the
CapacityBuffer object. This removes the placeholder Pods and allows the
cluster autoscaler to scale down the nodes.

```bash
kubectl delete capacitybuffer CAPACITY_BUFFER_NAME -n NAMESPACE
```

Replace `CAPACITY_BUFFER_NAME` with the name of the
CapacityBuffer that you want to delete.

## Troubleshooting

The following section contains information on resolving common issues with
capacity buffers.

### Capacity buffer not ready due to billing model

If you create a CapacityBuffer for a workload that uses the
[Pod-based billing model](https://docs.cloud.google.com/kubernetes-engine/docs/concepts/about-autopilot-mode-standard-clusters.md.txt#pricing)
(pay-per-Pod), the capacity buffer won't be ready for provisioning.

To identify this issue, check the CapacityBuffer status:

```bash
kubectl describe capacitybuffer BUFFER_NAME -n NAMESPACE
```

Look for a condition of the type `ReadyForProvisioning` with a status of `False`.

To resolve this issue, ensure that your CapacityBuffer references a workload
or PodTemplate that is compatible with node-based billing.

### Buffer Pods aren't scheduled on nodes provisioned by custom ComputeClasses

If your CapacityBuffer configuration dependency (for example, a PodTemplate) selects a
custom ComputeClass, you might encounter an issue where buffer Pods aren't scheduled on
the newly provisioned nodes. This issue causes the cluster autoscaler to repeatedly create redundant
node pools and nodes while idle nodes are deleted.

This issue occurs if the custom ComputeClass provisions nodes with taints or
specific runtime requirements, and your buffer Pods (or the PodTemplate they rely on)
don't have matching tolerations or settings.

To resolve this issue, update your CapacityBuffer configuration to specify the
required tolerations, node selectors, or runtime classes. For example, if your
dependency selects a custom ComputeClass configured with GKE Sandbox,
update the dependency to do one of the following:

- Specify the `spec.runtimeClassName: gvisor` field.
- Include an explicit toleration for the `sandbox.gke.io/runtime=gvisor:NoSchedule` taint.

For more information, see the [custom ComputeClass documentation](https://docs.cloud.google.com/kubernetes-engine/docs/concepts/about-custom-compute-classes.md.txt).

### Standby buffer nodes fail to resume

Suspended standby buffer nodes might fail to resume for one of the following
reasons:

- Resuming a suspended standby node requires available Compute Engine capacity in the zone for the node's machine type and specific CPU platform generation, such as within the E2 or N2 machine series. If the zone has insufficient compute capacity for that exact configuration, the node fails to resume. For more information, see [Compute Engine suspend and resume limitations](https://docs.cloud.google.com/compute/docs/instances/suspend-resume-instance.md.txt#limitations).
- If your custom ComputeClass enables [active migration to higher-priority nodes](https://docs.cloud.google.com/kubernetes-engine/docs/concepts/about-custom-compute-classes.md.txt#optimize-rule-priority) (`activeMigration.optimizeRulePriority: true`) and places Spot VMs at a higher priority than on-demand VMs, GKE migrates on-demand standby nodes to Spot VMs over time. Because Spot VMs are more subject to capacity constraints and suspended Spot VMs can't convert to on-demand VMs when resuming, these constraints can prevent those standby nodes from resuming.
- Resuming the node exceeds your project's Compute Engine quotas. For more information, see [Capacity buffer fails to provision due to quota limits](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/configure-capacity-buffer.md.txt#quota-exceeded).

To identify why a standby node failed to resume, query
[Compute Engine audit logs](https://docs.cloud.google.com/compute/docs/logging/audit-logging.md.txt) in Logs Explorer for
`ZONE_RESOURCE_POOL_EXHAUSTED` or `QUOTA_EXCEEDED` errors on
`compute.instances.resume` operations.

To resolve or mitigate this issue, do one or more of the following:

- Use regional clusters so that the cluster autoscaler can provision or resume nodes across multiple zones.
- Configure multiple machine families in the `spec.priorities` list of your custom ComputeClass. If suspended nodes can't resume, the cluster autoscaler can fall back to creating new nodes based on a different priority rule. These new nodes take longer to become ready than resumed standby nodes. This configuration doesn't help suspended nodes resume, because a suspended node can resume only with its original machine type.
- Use a dedicated custom ComputeClass with on-demand VMs, or set the `activeMigration.optimizeRulePriority` field to `false` for the ComputeClass that your standby buffer uses.
- Pair your standby buffer with a small active buffer to absorb immediate traffic spikes.

### Capacity buffer fails to provision due to quota limits

A capacity buffer might provision fewer replicas than requested or fail to
scale up if it exceeds [Kubernetes ResourceQuotas](https://kubernetes.io/docs/concepts/policy/resource-quotas/) or
Compute Engine quotas.

To identify which quota is limiting the buffer, check one or more of the
following quotas:

- Check whether a Kubernetes ResourceQuota in the namespace is limiting the
  buffer:

  ```bash
  kubectl describe capacitybuffer \
      CAPACITY_BUFFER_NAME \
      -n NAMESPACE
  ```

  In the `status.conditions` output, look for a `LimitedByQuotas` condition
  with a status of `True`. Both active and standby buffer units, including
  units backed by suspended standby nodes, count toward namespace
  ResourceQuota limits as if the buffer Pods were running. When a buffer
  exceeds the namespace quota, GKE caps the `status.replicas`
  field to fit within the available quota.
- Check whether Compute Engine project quotas are blocking node
  provisioning. In Logs Explorer, search the
  `container.googleapis.com/cluster-autoscaler-visibility` log for
  `no.scale.up.nap.quota.exceeded` events, or search Compute Engine
  audit logs for `QUOTA_EXCEEDED` errors.

  When checking Compute Engine quotas for standby buffers, verify both
  of the following quotas in addition to vCPU and memory quotas:
  - Suspended standby nodes retain their internal and external IP addresses, which consumes the in-use IP address quota.
  - When suspending a standby VM, Compute Engine writes the node's memory to disk, which consumes `Persistent disk standard (GB)` [disk quota](https://docs.cloud.google.com/compute/resource-usage.md.txt#disk_quota) proportional to the node's memory size.

To resolve the quota limit, do one of the following:

- Increase the Kubernetes ResourceQuota in the buffer's namespace.
- [Request a Compute Engine quota adjustment](https://docs.cloud.google.com/docs/quotas/view-manage.md.txt#requesting_higher_quota).
- Reduce the `replicas`, `limits`, or `percentage` value in your CapacityBuffer manifest.

### DaemonSet Pods stop receiving data after a standby node resumes

After a suspended standby buffer node resumes, DaemonSet Pods on that node might
stop receiving data over long-lived connections that were opened before the node
was suspended. The Pods don't log any errors and don't reconnect.

This issue occurs when a DaemonSet Pod acts as a receive-only client, such as
for a streaming HTTP response, a WebSocket connection, server-sent events, or a
gRPC server-streaming call. While the standby node is suspended, the server or
an intermediate network device times out and closes the connection. When the
node resumes, the Pod's network stack still considers the connection open.
Because the Pod doesn't send any data, it never detects that the connection was
closed and waits for data indefinitely.

To resolve this issue, configure the client application in your DaemonSet Pod to
detect closed connections and reconnect by doing one of the following:

- Configure client-side TCP keepalive in your Pod's application (enabling keepalive only on the server doesn't notify the Pod). For more information about TCP keepalive parameters, see [Idle connections](https://docs.cloud.google.com/compute/docs/troubleshooting/general-tips.md.txt#idle-connections).
  1. Enable the `SO_KEEPALIVE` socket option in your client library or language runtime, such as `socket.setKeepAlive(true, initialDelay)` in Node.js or `net.Dialer.KeepAlive` in Go. Some runtimes disable TCP keepalive by default.
  2. Configure a short keepalive idle time (`TCP_KEEPIDLE`), such as a few seconds, instead of relying on the operating system default.
  3. Update your application to catch the connection error (such as a connection reset or timeout) when a keepalive probe fails and open a new connection.
- Configure application-level heartbeats or read timeouts in your client, such as WebSocket ping and pong frames or [gRPC keepalive](https://grpc.io/docs/guides/keepalive/) pings, and update your application to reconnect when a timeout occurs.

## What's next

- Learn how to [configure capacity buffers in custom ComputeClasses](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/configure-capacity-buffers-compute-classes.md.txt).
- Learn more about [capacity buffers](https://docs.cloud.google.com/kubernetes-engine/docs/concepts/capacity-buffer.md.txt).
- Refer to the [CapacityBuffer reference documentation](https://docs.cloud.google.com/kubernetes-engine/docs/reference/crds/capacitybuffer.md.txt).