Configure capacity buffers

Capacity buffers improve the responsiveness and reliability of critical workloads by proactively managing spare cluster capacity and suspended states of pre-provisioned, pre-configured capacity by using a Kubernetes CapacityBuffer CustomResourceDefinition (CRD). Using capacity buffers lets you explicitly define a specific amount of unused node capacity within your cluster. This reserved capacity helps to reduce Pod scheduling time.

When a high-priority workload needs to scale up quickly, the new workload can use the empty capacity immediately without waiting for node provisioning. This approach minimizes latency and avoids resource contention during sudden spikes in demand.

This page provides methods for configuring capacity buffers: a fixed replicas buffer, a resource limits buffer, and a percentage-based buffer.

Before you begin

Before you start, make sure that you have performed the following tasks:

  • Enable the Google Kubernetes Engine API.
    • Enable Google Kubernetes Engine API
  • To use the Google Cloud CLI for this task, install and then initialize the gcloud CLI. If you previously installed the gcloud CLI, get the latest version by running the gcloud components update command. Earlier gcloud CLI versions might not support running the commands in this document.
  • Create or have access to a GKE cluster that meets the following version requirements:
    • For active buffers, use version 1.35.2-gke.1842000 or later on Standard and Autopilot clusters.
    • For standby buffers, use version 1.36.0-gke.2253000 or later on Standard clusters, or version 1.36.0-gke.2853000 or later on Autopilot clusters with custom ComputeClasses.
  • Enable node auto-provisioning on your Standard clusters. In Autopilot clusters, node auto-provisioning is already enabled. Node auto-provisioning is optional but recommended for active buffers and required for standby buffers.

Create prerequisite Kubernetes objects

To configure a CapacityBuffer, you need a namespace that holds all of the required objects (the CapacityBuffer itself, and additional resources like a PodTemplate or workload). The PodTemplate and CapacityBuffer must be in the same namespace. You can create a namespace or use an existing namespace, including the default namespace.

Depending on which type of CapacityBuffer you're configuring, you also require one of the following:

  • PodTemplate: defines the resource requirements for a single unit of buffer capacity. The configuration specified in the CapacityBuffer object references the Pod template.

  • Workload: an existing workload that you reference in the CapacityBuffer object. This guide uses a Deployment object as an example workload, but capacity buffers support any of the following resource types:

    • Deployment
    • ReplicaSet
    • StatefulSet
    • ReplicationController
    • Job

    • CustomResourceDefinitions (CRDs) that implement the scale subresource.

This section provides examples of these objects. If you already have a workload that you want to configure with a capacity buffer, proceed to Apply a capacity buffer.

To create an example Kubernetes workload, complete the following steps:

  1. Save the following manifest as namespace.yaml:

    apiVersion: v1
kind: Namespace
metadata:
  name: capacity-buffer-example
  labels:
    name: capacity-buffer-example

    This manifest creates a namespace called capacity-buffer-example.

  2. Optional: to use capacity buffers with a custom ComputeClass, save the following manifest as custom-compute-class.yaml:

    apiVersion: cloud.google.com/v1
kind: ComputeClass
metadata:
  name: ccc-example
spec:
  # Buffers are also created according to these priorities
  priorities:
  - machineFamily: n4
  - machineFamily: n4d
  - machineFamily: c4
  - machineFamily: c4d
  nodePoolAutoCreation:
    enabled: true

    This manifest creates a custom ComputeClass that defines and controls the compute priorities for the nodes that GKE provisions. To learn more, see custom ComputeClasses.

  3. Save the following manifest as buffer-pod-template.yaml:

    apiVersion: v1
kind: PodTemplate
metadata:
  name: buffer-unit-template
  namespace: capacity-buffer-example # the namespace must be the same namespace as the CapacityBuffer
template:
  spec:
    terminationGracePeriodSeconds: 0
    containers:
    - name: buffer-container
      image: registry.k8s.io/pause:3.9
      resources:
        requests:
          cpu: "1"
          memory: "1Gi"
        limits:
          cpu: "1"
          memory: "1Gi"
    # Optional: Using buffers with a custom ComputeClass /
    # controls the properties of the provisioned nodes.
    nodeSelector:
      cloud.google.com/compute-class: ccc-example

    This manifest creates a PodTemplate that defines the resource requirements for a single unit of buffer capacity (1 CPU and 1Gi Memory). This configuration specifies the size of capacity units that GKE provisions for the buffer. For example, with this PodTemplate, GKE won't consider nodes with less than 1 CPU and 1Gi of available resources as part of the buffer, if the cluster scales up.

  4. Save the following manifest as sample-workload-deployment.yaml:

    apiVersion: apps/v1
kind: Deployment
metadata:
  name: critical-workload-ref
  namespace: capacity-buffer-example # the namespace must be the same namespace as the CapacityBuffer
spec:
  replicas: 10
  selector:
    matchLabels:
      app: critical-workload
  template:
    metadata:
      labels:
        app: critical-workload
    spec:
      containers:
      - name: busybox
        image: busybox
        command: ["sleep", "3600"]
        resources:
          requests:
            cpu: 100m
      # Optional: Using buffers with a custom ComputeClass /
      # controls the properties of the provisioned nodes.
      nodeSelector:
        cloud.google.com/compute-class: ccc-example

    This manifest creates a sample Deployment with 10 replicas, which is the reference object for the percentage-based buffer example in the next section.

  5. Apply the manifests to your cluster:

    kubectl apply -f namespace.yaml -f custom-compute-class.yaml -f buffer-pod-template.yaml -f sample-workload-deployment.yaml

  6. Verify that GKE created the objects:

    kubectl get podtemplate -n capacity-buffer-example
kubectl get deployment critical-workload-ref -n capacity-buffer-example

    The output is similar to the following:

    NAME                   AGE
buffer-unit-template   1m

NAME                    READY   UP-TO-DATE   AVAILABLE   AGE
critical-workload-ref   10/10   10           10          1m

Apply a capacity buffer

This section provides examples of the different types of capacity buffers that you can apply to your workloads.

Configure a fixed replicas buffer

Configuring a CapacityBuffer with fixed replicas specifies the exact number of buffer units that you want based on a PodTemplate.

To create a buffer with fixed replicas, complete the following steps:

  1. Save the following manifest as cb-fixed-replicas.yaml:

    apiVersion: autoscaling.x-k8s.io/v1beta1
kind: CapacityBuffer
metadata:
  name: fixed-replica-buffer
  namespace: NAMESPACE
spec:
  podTemplateRef:
    name: POD_TEMPLATE
  replicas: 3
  provisioningStrategy: "STRATEGY"

    Replace the following:

    • NAMESPACE: the name of your namespace, for example capacity-buffer-example.
    • POD_TEMPLATE: the PodTemplate that defines your resource requirements, for example buffer-unit-template.
    • STRATEGY: the provisioning strategy, either "buffer.x-k8s.io/active-capacity" (default) or "buffer.gke.io/standby-capacity".

    This manifest creates a CapacityBuffer resource that references a PodTemplate to request a specific number of buffer units.

  2. Apply the manifest:

    kubectl apply -f cb-fixed-replicas.yaml

  3. Confirm that GKE applied the capacity buffer:

    kubectl get capacitybuffer fixed-replica-buffer -n NAMESPACE

    The replicas field in the status should show 3, which reflects the number of replicas that you defined in the manifest. The STATUS field should show ReadyForProvisioning.

Configure a resource limits buffer

You can use the limits field to define a maximum amount of resources that the buffer should consume, calculated based on your PodTemplate size.

To create a resource limits buffer, complete the following steps:

  1. Save the following manifest as cb-resource-limits.yaml:

    apiVersion: autoscaling.x-k8s.io/v1beta1
kind: CapacityBuffer
metadata:
  name: resource-limit-buffer
  namespace: NAMESPACE
spec:
  podTemplateRef:
    name: POD_TEMPLATE
  limits:
    cpu: "5"
    memory: "5Gi"
  provisioningStrategy: "STRATEGY"

    Replace the following:

    • NAMESPACE: the name of your namespace, for example capacity-buffer-example.
    • POD_TEMPLATE: the PodTemplate that defines your resource requirements, for example buffer-unit-template.
    • STRATEGY: the provisioning strategy, either "buffer.x-k8s.io/active-capacity" (default) or "buffer.gke.io/standby-capacity".

    This manifest creates a CapacityBuffer resource with a total limit of 5 CPUs and 5 GiB Memory. If you're using the PodTemplate example from the previous step, you define each unit as 1 CPU and 1Gi Memory, which should result in 5 buffer units.

  2. Apply the manifest:

    kubectl apply -f cb-resource-limits.yaml

  3. Confirm that GKE applied the capacity buffer:

    kubectl get capacitybuffer resource-limit-buffer -n NAMESPACE

    Check the CapacityBuffer status. The replicas field should show a value derived from the limits that you defined. If you're using the PodTemplate example from the previous section, you should see 5 buffer units because this is the maximum number of units that fit within the defined limits.

Configure a percentage-based buffer

Configuring a percentage-based buffer dynamically sizes the buffer based on a percentage of an existing scalable workload. Percentage-based capacity buffers are supported only for Kubernetes scalable objects that implement the scale subresource, such as Deployments, StatefulSets, ReplicaSets, or Jobs. You can't define a percentage-based buffer for Pod templates because they don't have a replicas field.

In most cases, start with fixed replicas or resource limit strategies rather than percentage-based buffers. Consider the following when deciding:

  • Percentage-based buffers are less responsive to sudden scale-ups if the workload scales to low numbers or zero because the safety margin decreases in proportion to active Pods.
  • Standby nodes incur compute charges during their initialization period before suspending. Dynamically scaling standby buffers alongside workload fluctuations wastes compute resources, so use fixed replicas or resource limits for standby buffers instead.
  • Percentage-based buffers are mainly useful for active buffers on large deployments that never scale to low replica counts.

In GKE version 1.36.3-gke.1244000 and later, to keep a minimum buffer size when the referenced workload scales down, set the replicas field together with the percentage field. GKE uses whichever field results in more buffer units. To cap the buffer size, also set the limits field.

For example, consider a buffer with percentage: 20 and replicas: 3 that references a Deployment with 10 replicas. Because 20% of 10 replicas is 2, which is less than the replicas value of 3, GKE provisions 3 buffer units. If the Deployment scales to 20 replicas, the buffer grows to 4 units. For more information, see the CapacityBuffer reference documentation.

To create a percentage-based buffer, complete the following steps:

  1. Save the following manifest as cb-percentage-based.yaml:

    apiVersion: autoscaling.x-k8s.io/v1beta1
kind: CapacityBuffer
metadata:
  name: percentage-buffer
  namespace: NAMESPACE
spec:
  scalableRef:
    apiGroup: apps
    kind: Deployment
    name: SCALABLE_RESOURCE_NAME
  percentage: 20
  provisioningStrategy: "STRATEGY"

    Replace the following:

    • NAMESPACE: the name of your namespace.
    • SCALABLE_RESOURCE_NAME: the name of your scalable resource, for example critical-workload-ref.
    • STRATEGY: the provisioning strategy, either "buffer.x-k8s.io/active-capacity" (default) or "buffer.gke.io/standby-capacity".

    This manifest creates a CapacityBuffer resource that requests a buffer size equivalent to 20% of the referenced resource's replicas. If you're using the Deployment example from the previous section, the replica value is set to 10.

  2. Apply the manifest:

    kubectl apply -f cb-percentage-based.yaml

  3. Confirm that GKE applied the capacity buffer:

    kubectl get capacitybuffer percentage-buffer -n NAMESPACE

    Check the CapacityBuffer status. The replicas field should show a value from the percentage calculation. If you're using the Deployment example from the previous section, you should see 2 buffer units, which is 20% of the 10 replicas defined in the Deployment.

  4. Test the dynamic scaling by manually scaling the Deployment up to 20 replicas:

    kubectl scale deployment critical-workload-ref -n NAMESPACE --replicas=20

    The CapacityBuffer controller reacts and automatically scales the buffer to 4 replicas.

Customize standby buffer behavior

You can use annotations to customize how standby buffers start and refresh. Add these annotations to the metadata.annotations field of your CapacityBuffer resource:

  • buffer.gke.io/standby-capacity-init-time: the amount of time a node remains active after creation before GKE suspends it. Specify a duration string such as 5m or 1h. The default is 5m.

  • buffer.gke.io/standby-capacity-refresh-frequency: how often GKE refreshes suspended nodes. Specify a duration string such as 12h or 24h. The default is 24h.

The following example shows a manifest with these optional fields to customize the behavior of standby buffers:

apiVersion: autoscaling.x-k8s.io/v1beta1
kind: CapacityBuffer
metadata:
  name: customized-standby-buffer
  namespace: my-namespace
  annotations:
    buffer.gke.io/standby-capacity-init-time: "15m"
    buffer.gke.io/standby-capacity-refresh-frequency: "12h"
spec:
  podTemplateRef:
    name: buffer-unit-template
  replicas: 3
  provisioningStrategy: "buffer.gke.io/standby-capacity"

Preload images on standby buffers

To speed up workload startup times when a standby node resumes, you can preload container images by using a DaemonSet. The DaemonSet runs during the start-up period before the node is suspended.

To preload images by using the DaemonSet, complete the following steps:

  1. Save the following manifest as image-puller-daemonset.yaml:

    apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: image-prefetch-daemonset
  namespace: NAMESPACE
spec:
  selector:
    matchLabels:
      name: image-prefetch
  template:
    metadata:
      labels:
        name: image-prefetch
    spec:
      tolerations:
      - key: "buffer.gke.io/standby-capacity-node-suspended"
        operator: "Exists"
      initContainers:
      - name: image-puller
        image: IMAGE_NAME
        command: ["sh", "-c", "true"]
      containers:
      - name: pause
        image: registry.k8s.io/pause:3.9

    Replace the following:

    • NAMESPACE: the namespace for the DaemonSet, for example capacity-buffer-example.
    • IMAGE_NAME: the name of the image to preload, for example your-app-image:latest.

    This manifest includes the buffer.gke.io/standby-capacity-node-suspended toleration so that the DaemonSet remains scheduled on standby nodes after GKE suspends them.

  2. Apply the DaemonSet manifest to your cluster:

    kubectl apply -f image-puller-daemonset.yaml

  3. Verify that the DaemonSet is created:

    kubectl get daemonset image-prefetch-daemonset -n NAMESPACE

  4. Verify that your capacity buffer is created and ready for provisioning:

    kubectl get capacitybuffer CAPACITY_BUFFER_NAME -n NAMESPACE

    Check the status. The STATUS field should show ReadyForProvisioning.

Monitor capacity buffer status and performance

You can monitor the status and health of your capacity buffers by using kubectl commands and Cloud Monitoring metrics.

Verify CapacityBuffer resource status

To check the health of your capacity buffers and verify that they are ready to receive workloads, complete the following steps:

  1. Get the status of all capacity buffers across the cluster:

    kubectl get capacitybuffer -A

  2. Inspect the detailed status, conditions, and event logs of a specific buffer:

    kubectl describe capacitybuffer CAPACITY_BUFFER_NAME -n NAMESPACE

Identify suspended standby buffer nodes

Standby buffer VMs are pre-provisioned, but kept in a suspended state to help reduce costs. You can recognize these suspended nodes because they have a custom condition. To audit suspended node instances, run the following command:

kubectl get nodes -o custom-columns='NAME:.metadata.name,SUSPENDED:.status.conditions[?(@.type=="Suspended")].status'

A status of True indicates a standby VM is suspended. A status of False or <none> indicates an active, running node.

Monitor performance with Cloud Monitoring

To monitor the end-to-end impact of capacity buffers on scale-up times, use the Startup Latency dashboards in the Google Cloud console. Use the Pod Startup Latency Distribution and Pod First Ready Latency charts to track the kubernetes.io/pod/latencies/pod_first_ready metric from Pod creation to readiness. You can also query Scheduled events in Cloud Logging to measure Pod scheduling latency. For instructions, see Monitor startup latency metrics.

Remove capacity buffers

If you no longer need a capacity buffer for your workloads, delete the CapacityBuffer object. This removes the placeholder Pods and allows the cluster autoscaler to scale down the nodes.

kubectl delete capacitybuffer CAPACITY_BUFFER_NAME -n NAMESPACE

Replace CAPACITY_BUFFER_NAME with the name of the CapacityBuffer that you want to delete.

Troubleshooting

The following section contains information on resolving common issues with capacity buffers.

Capacity buffer not ready due to billing model

If you create a CapacityBuffer for a workload that uses the Pod-based billing model (pay-per-Pod), the capacity buffer won't be ready for provisioning.

To identify this issue, check the CapacityBuffer status:

kubectl describe capacitybuffer BUFFER_NAME -n NAMESPACE

Look for a condition of the type ReadyForProvisioning with a status of False.

To resolve this issue, ensure that your CapacityBuffer references a workload or PodTemplate that is compatible with node-based billing.

Permission errors for custom scalable resources

If you configure a CapacityBuffer to work with custom scalable objects (using the scalableRef field), the cluster autoscaler might fail to scale the buffer if it lacks the necessary permissions. This issue only occurs in GKE versions earlier than 1.36.0-gke.2459000.

To resolve this issue, manually grant the required permissions by creating a ClusterRole and ClusterRoleBinding, such as in the following example:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: custom-scale-getter
rules:
- apiGroups: ["api.example.com"]
  resources: ["customreplicatedresources/scale"]
  verbs: ["get"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: ca-custom-scale-getter
subjects:
- kind: User
  name: "system:cluster-autoscaler"
  namespace: kube-system
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: custom-scale-getter

For more information about configuring RBAC, see the Kubernetes RBAC documentation.

Buffer Pods aren't scheduled on nodes provisioned by custom ComputeClasses

If your CapacityBuffer configuration dependency (for example, a PodTemplate) selects a custom ComputeClass, you might encounter an issue where buffer Pods aren't scheduled on the newly provisioned nodes. This issue causes the cluster autoscaler to repeatedly create redundant node pools and nodes while idle nodes are deleted.

This issue occurs if the custom ComputeClass provisions nodes with taints or specific runtime requirements, and your buffer Pods (or the PodTemplate they rely on) don't have matching tolerations or settings.

To resolve this issue, update your CapacityBuffer configuration to specify the required tolerations, node selectors, or runtime classes. For example, if your dependency selects a custom ComputeClass configured with GKE Sandbox, update the dependency to do one of the following:

  • Specify the spec.runtimeClassName: gvisor field.
  • Include an explicit toleration for the sandbox.gke.io/runtime=gvisor:NoSchedule taint.

For more information, see the custom ComputeClass documentation.

Standby buffer nodes fail to resume

Suspended standby buffer nodes might fail to resume for one of the following reasons:

  • Resuming a suspended standby node requires available Compute Engine capacity in the zone for the node's machine type and specific CPU platform generation, such as within the E2 or N2 machine series. If the zone has insufficient compute capacity for that exact configuration, the node fails to resume. For more information, see Compute Engine suspend and resume limitations.
  • If your custom ComputeClass enables active migration to higher-priority nodes (activeMigration.optimizeRulePriority: true) and places Spot VMs at a higher priority than on-demand VMs, GKE migrates on-demand standby nodes to Spot VMs over time. Because Spot VMs are more subject to capacity constraints and suspended Spot VMs can't convert to on-demand VMs when resuming, these constraints can prevent those standby nodes from resuming.
  • Resuming the node exceeds your project's Compute Engine quotas. For more information, see Capacity buffer fails to provision due to quota limits.

To identify why a standby node failed to resume, query Compute Engine audit logs in Logs Explorer for ZONE_RESOURCE_POOL_EXHAUSTED or QUOTA_EXCEEDED errors on compute.instances.resume operations.

To resolve or mitigate this issue, do one or more of the following:

  • Use regional clusters so that the cluster autoscaler can provision or resume nodes across multiple zones.
  • Configure multiple machine families in the spec.priorities list of your custom ComputeClass. If suspended nodes can't resume, the cluster autoscaler can fall back to creating new nodes based on a different priority rule. These new nodes take longer to become ready than resumed standby nodes. This configuration doesn't help suspended nodes resume, because a suspended node can resume only with its original machine type.
  • Use a dedicated custom ComputeClass with on-demand VMs, or set the activeMigration.optimizeRulePriority field to false for the ComputeClass that your standby buffer uses.
  • Pair your standby buffer with a small active buffer to absorb immediate traffic spikes.

Capacity buffer fails to provision due to quota limits

A capacity buffer might provision fewer replicas than requested or fail to scale up if it exceeds Kubernetes ResourceQuotas or Compute Engine quotas.

To identify which quota is limiting the buffer, check one or more of the following quotas:

  • Check whether a Kubernetes ResourceQuota in the namespace is limiting the buffer:

    kubectl describe capacitybuffer \
    CAPACITY_BUFFER_NAME \
    -n NAMESPACE

    In the status.conditions output, look for a LimitedByQuotas condition with a status of True. Both active and standby buffer units, including units backed by suspended standby nodes, count toward namespace ResourceQuota limits as if the buffer Pods were running. When a buffer exceeds the namespace quota, GKE caps the status.replicas field to fit within the available quota.

  • Check whether Compute Engine project quotas are blocking node provisioning. In Logs Explorer, search the container.googleapis.com/cluster-autoscaler-visibility log for no.scale.up.nap.quota.exceeded events, or search Compute Engine audit logs for QUOTA_EXCEEDED errors.

    When checking Compute Engine quotas for standby buffers, verify both of the following quotas in addition to vCPU and memory quotas:

    • Suspended standby nodes retain their internal and external IP addresses, which consumes the in-use IP address quota.
    • When suspending a standby VM, Compute Engine writes the node's memory to disk, which consumes Persistent disk standard (GB) disk quota proportional to the node's memory size.

To resolve the quota limit, do one of the following:

What's next