> [!WARNING]
>
> **Preview**
>
>
> This feature is
>
> subject to the "Pre-GA Offerings Terms" in the General Service Terms section of the
> [Service Specific
> Terms](https://docs.cloud.google.com/terms/service-terms#1).
>
> Pre-GA features are available "as is" and might have limited support.
>
> For more information, see the
> [launch stage descriptions](https://cloud.google.com/products/#product-launch-stages).

This document explains how to design resilient Google Kubernetes Engine (GKE) clusters
and workload scheduling strategies that help you obtain
resources, such as GPUs, TPUs, and high-performance CPUs. By leveraging
Gemini in
Google Cloud and [Compute Advisor](https://docs.cloud.google.com/kubernetes-engine/docs/concepts/design-with-gemini.md.txt) (Preview), you can
prevent pending Pods and improve reliable scheduling for AI workloads.

This document is intended for Cloud architects and
Platform admins and operators who manage GKE infrastructure and
want to optimize capacity planning and scheduling.

## Overview of flexibility, efficiency, and availability of compute capacity on GKE

Kubernetes scheduling relies on declared resource requests. When scheduling
large-scale accelerators, like GPUs or TPUs, strict resource requests
can lead to clusters failing to scale up if specific hardware is unavailable.
You can optimize capacity availability on GKE by using the
following features:

- **Node pool auto-creation:** dynamic, multi-family node pool creation.
- **Workload-level fallbacks:** tolerations and node selectors configured to accept alternative hardware.
- **Geographic and regional scheduling:** GKE multi-zonal and multi-regional capabilities.
- **Reservation management:** consuming pre-purchased capacity before requesting on-demand resources.

## Best practices for flexibility, efficiency, and availability of compute capacity on GKE

This section provides recommendations to increase your capacity availability
when scheduling workloads on GKE. These best practices cover
strategies such as designing flexible hardware requirements, configuring node
pool auto-creation, and leveraging geographic distribution to adapt to resource
constraints. [Node pool auto-creation](https://docs.cloud.google.com/kubernetes-engine/docs/concepts/node-auto-provisioning.md.txt)
automatically manages node pools based on Pod specifications. Rather than
pre-defining node pools, specify resource requirements in your Pod manifests,
and let GKE dynamically create nodes.

You can also discover and implement the following best practices by using
Compute Advisor (Preview). For more information, see
[Use Compute Advisor](https://docs.cloud.google.com/kubernetes-engine/docs/concepts/design-with-gemini-obtainability.md.txt#example-prompts).

### Options for increasing the availability of compute capacity

To optimize capacity availability, you can instruct GKE to avoid
binding workloads to a single static machine family or accelerator type. The
following are examples of how you can configure flexible node pools and Pod
affinity rules for different workload classes. The specific alternatives you
choose depend on your application's resource demands:

- General-purpose CPU node pools:

  - **Primary example:** N4 (Intel-based general-purpose).
  - **Sample alternatives:** N4D (AMD EPYC general-purpose), C4 or C3 (Intel general-purpose), or C4D or C3D (AMD EPYC general-purpose).
  - **Implementation pattern:** configure Pod specs with a custom ComputeClass or node affinity rules that permit scheduling across multiple machine family labels, for example, `cloud.google.com/machine-family` in `["n4", "n4d", "c4", "c4d"]`. This configuration lets GKE provision whichever pool has available capacity.
- GPU node pools:

  - **Primary example:** A3 Series (NVIDIA H100 GPUs) or A4 Series (NVIDIA B200 GPUs).
  - **Sample alternatives:** G2 Series (NVIDIA L4 GPUs) or alternative accelerator tiers.
  - **Implementation pattern:** configure separate node pools or ComputeClasses for different GPU tiers. For inference or serving workloads that can run across accelerator generations, allow Pods to fall back to G2 (L4) or alternative GPU pools if primary accelerator provisioning is constrained.
- TPU node pools:

  - **Primary example:** TPU Ironwood (TPU7x).
  - **Sample alternatives:** TPU v6 (Trillium) or TPU v5 (v5e or v5p).
  - **Implementation pattern:** TPU slice provisioning can be highly constrained. Design training workloads with framework-level flexibility (for example, JAX or PyTorch configurations supporting variable slice topologies) to deploy on v6 or v5 slices when TPU Ironwood (TPU7x) capacity is unavailable.

The following table summarizes the primary selections and hardware alternatives
for different types of workloads:

| Workload type | Primary selection example | Sample alternatives | Architectural considerations |
|---|---|---|---|
| **System or core workloads** | N4 | N4D, C4, C4D, C3, C3D | Spans modern Intel and AMD hardware pools for node pool auto-creation. |
| **GPU inference and processing** | A3 (H100) or A4 (B200) | G2 (L4) | Flexibly targets lower-cost or higher-availability GPU node pools. |
| **TPU model training** | TPU Ironwood (TPU7x) | TPU v6, TPU v5 (v5e or v5p) | Utilizes flexible topologies for slice-based scheduling. |

### Implement node pool auto-creation and ComputeClasses

To optimize capacity availability, combine node pool auto-creation with
[ComputeClasses](https://docs.cloud.google.com/kubernetes-engine/docs/concepts/about-compute-classes.md.txt). The
following list includes best practices:

- **Define ComputeClasses:** create ComputeClass resources that specify a
  prioritized list of machine families, GPU types, or provisioning models.
  GKE attempts to provision nodes by using the highest-priority
  configuration available in the class. A prioritized fallback list
  significantly helps you obtain the resources that your
  workloads require. For example, to prevent GKE from falling
  back to general-purpose machines when a preferred specialized accelerator
  is unavailable, add the `whenUnsatisfiable: DoNotScaleUp` setting to your ComputeClass
  configuration. For more information, see
  [Control autoscaled node attributes with custom ComputeClasses](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/node-attributes-compute-classes.md.txt).

- **Enable active migration (`optimizeRulePriority: true`) for stateless or
  disruption-tolerant workloads:** configure
  [active migration](https://docs.cloud.google.com/kubernetes-engine/docs/concepts/about-custom-compute-classes.md.txt#active-migration)
  in your ComputeClass so GKE automatically cordons, drains,
  and replaces fallback nodes with your higher-priority machine series when
  capacity recovers in that location.

- **Use dynamic StorageClasses for cross-generation fallback:** in
  GKE version 1.35.3-gke.1290000 and later, when a
  ComputeClass spans Hyperdisk-only series (such as C4, C4D,
  and N4) and Persistent Disk series (such as N2 and N2D), configure
  stateful workloads so that GKE dynamically selects a
  compatible disk type. To do so, use a StorageClass with
  [automated disk type selection](https://docs.cloud.google.com/kubernetes-engine/docs/concepts/hyperdisk.md.txt#automated_disk_type_selection)
  (`parameters.type: dynamic`) and the
  `volumeBindingMode: WaitForFirstConsumer` setting.

- **Reference ComputeClasses in Pod specs:** in your workload Pod
  specifications, use the `cloud.google.com/compute-class` label to target
  your custom ComputeClass instead of a specific machine family or GPU type. For
  more information, see [Request a ComputeClass in a workload](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/node-attributes-compute-classes.md.txt#request-compute-class-workload).

- **Define multiple tolerations:** if you don't use ComputeClasses, use node
  affinity rules in your Pod specifications that permit a range of machine
  families (such as `cloud.google.com/machine-family` in `["n4", "n4d", "c4", "c4d"]`).
  For more information, see [Configure node pool auto-creation](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/node-auto-provisioning.md.txt).

### Geographic and multi-region flexibility on GKE

To increase capacity availability, deploy GKE clusters that span
multiple zones, or run multi-cluster architectures:

- **Multi-zonal clusters:** ensure node pools are configured to
  auto-scale across all available zones in a region.

- **Multi-region cluster federation:** for large asynchronous jobs (such as
  offline batch inference or distributed training), deploy a multi-cluster
  orchestrator (like Kueue or Cluster Director) to queue workloads globally
  and dispatch them to a region with available capacity.

- **Pods running on Spot VMs:** run interruptible workloads on Spot VMs by adding tolerations
  and instructing GKE to distribute excess capacity in different zones.

### Additional best practices for flexibility, efficiency, and availability of compute capacity in GKE

Beyond hardware diversification, incorporate these GKE
best practices to optimize cluster scaling success:

- **Use capacity buffers:** proactively reserve spare node capacity so that high-priority AI workloads can schedule immediately without waiting for new node provisioning. You can use declarative [capacity buffers](https://docs.cloud.google.com/kubernetes-engine/docs/concepts/capacity-buffer.md.txt) with the active buffer strategy for GPU and TPU workloads so that the cluster autoscaler automatically maintains spare capacity.
- **Use Flex-start with queued provisioning:** for large batch and AI model training jobs, use Flex-start with queued provisioning, which integrates with Kueue and Dynamic Workload Scheduler. Flex-start with queued provisions all-or-nothing atomic node allocation, which prevents partial cluster scale-up failures. For more information, see [Run a large-scale workload with flex-start with queued provisioning](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/provisioningrequest.md.txt).
- **Enable image streaming and container image preloading:** for large AI container images (such as PyTorch or TensorFlow images exceeding 10 GB), enable GKE Image streaming or use secondary boot disks for image preloading. This configuration reduces node warm-up time, which lets newly provisioned nodes start running workloads in seconds. For more information, see [Use Image streaming to pull container images](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/image-streaming.md.txt) and [Use secondary boot disks to preload data or container images](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/data-container-image-preloading.md.txt).
- **Set cluster autoscaler location policy to `ANY`:** configure node pools (especially for Spot VMs or flex-start) with the `ANY` location policy. This setting instructs the cluster autoscaler to search for requested capacity across all specified zones. Cluster autoscaler finds capacity over balancing node counts. For more information, see [Cluster autoscaler overview](https://docs.cloud.google.com/kubernetes-engine/docs/concepts/cluster-autoscaler.md.txt).
- **Optimize accelerator utilization with GPU sharing:** for workloads that don't require a dedicated GPU, use GPU time-sharing, multi-instance GPUs (MIG), or NVIDIA MPS to allow multiple containers to share a single accelerator. This approach optimizes effective capacity across node pools. For more information, see [About GPU sharing strategies in
  GKE](https://docs.cloud.google.com/kubernetes-engine/docs/concepts/timesharing-gpus).

## Use Compute Advisor

> [!WARNING]
>
> **Preview**
>
>
> This feature is
>
> subject to the "Pre-GA Offerings Terms" in the General Service Terms section of the
> [Service Specific
> Terms](https://docs.cloud.google.com/terms/service-terms#1).
>
> Pre-GA features are available "as is" and might have limited support.
>
> For more information, see the
> [launch stage descriptions](https://cloud.google.com/products/#product-launch-stages).

Compute Advisor is an AI-powered interface in the
Google Cloud console, powered by
Gemini, that helps you design resilient and cost-effective
architectures for GKE.

Compute Advisor provides near-real-time availability guidance for
Flex-start VMs and Spot VMs, while verifying your organization's policies and resource
quotas before deployment. Compute Advisor doesn't provide
availability guidance for workloads that require on-demand resources.

To access Gemini in the Google Cloud console, complete the
following steps:

1. In the Google Cloud console, go to the **Overview** page.

   [Go to Overview](https://console.cloud.google.com/kubernetes/overview)
2. In the **Design your infrastructure with Compute Advisor**
   section, submit a prompt. Gemini starts to generate a
   response.

   > [!CAUTION]
   > **Caution:** Generative AI can make mistakes. Validate all recommendations and configurations before you take any actions on a resource that Gemini recommends you to create, modify, or delete.

3. To generate architectural recommendations, run one of the following
   example prompts in Compute Advisor. When you click the
   **Run prompt in Compute Advisor** buttons, the
   Google Cloud console might take longer than 15 seconds to load:

- **General accelerator strategy**:

  *Use case:* use this prompt to analyze regional capacity signals and receive
  recommendations on machine types, zones, and fallback scheduling strategies
  when designing cluster configurations.

  ```bash
  Configure a GKE cluster to improve chances of obtaining scarce GPU or TPU capacity.
  ```

  [Run prompt in Compute Advisor](https://console.cloud.google.com/kubernetes/compute-advisor?advisor_ref=gke_obtainability_strategy)
- **Geographic flexibility and fallbacks**:

  *Use case:* use this prompt when designing multi-cluster architectures or
  global job queuing systems (for example, with Kueue) to shift workload
  execution across regions depending on resource availability.

  ```bash
  Configure multi-region fallbacks and geographic scheduling on GKE to increase GPU availability.
  ```

  [Run prompt in Compute Advisor](https://console.cloud.google.com/kubernetes/compute-advisor?advisor_ref=gke_geographic_flexibility)
- **Priority reservation consumption**:

  *Use case:* use this prompt to generate YAML configuration patterns for Pod
  affinity and autoscaling rules that prioritize reservation capacity.

  ```bash
  Configure GKE autoscaling rules and Pod specs to prioritize consuming active reservations before scaling into on-demand pools.
  ```

  [Run prompt in Compute Advisor](https://console.cloud.google.com/kubernetes/compute-advisor?advisor_ref=gke_reservation_consumption)
- **ComputeClasses for fallback prioritization**:

  *Use case:* use this prompt to generate the YAML manifest for a ComputeClass
  CustomResourceDefinition that prioritizes high-performance GPUs but
  includes lower-tier fallbacks to ensure workload scheduling.

  ```bash
  Define a ComputeClass manifest for GKE to prioritize A3 GPU nodes with automatic fallbacks to G2 (L4) GPUs.
  ```

  [Run prompt in Compute Advisor](https://console.cloud.google.com/kubernetes/compute-advisor?advisor_ref=gke_compute_class_fallbacks)
- **Node pool auto-creation for diversification**:

  *Use case:* Use this prompt to write the YAML manifest for GKE
  cluster autoscaler resource limits and Pod affinity rules that enable NAP to
  provision alternative GPU or CPU nodes automatically.

  ```bash
  Configure GKE node pool auto-creation to diversify machine families and prevent pending pods when regional accelerator capacity is constrained.
  ```

  [Run prompt in Compute Advisor](https://console.cloud.google.com/kubernetes/compute-advisor?advisor_ref=gke_nap_diversification)

## What's next

- [Design and optimize GKE clusters with Compute Advisor](https://docs.cloud.google.com/kubernetes-engine/docs/concepts/design-with-gemini.md.txt)
- [Configure node pool auto-creation](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/node-auto-provisioning.md.txt)
- [Troubleshoot horizontal Pod autoscaling](https://docs.cloud.google.com/kubernetes-engine/docs/troubleshooting/horizontal-pod-autoscaling.md.txt)