Design for resource obtainability on GKE with Gemini

Autopilot Standard

This document explains how to design resilient Google Kubernetes Engine (GKE) clusters and workload scheduling strategies that help you obtain resources, such as GPUs, TPUs, and high-performance CPUs. By leveraging Gemini in Google Cloud and Compute Advisor (Preview), you can prevent pending Pods and improve reliable scheduling for AI workloads.

This document is intended for Cloud architects and Platform admins and operators who manage GKE infrastructure and want to optimize capacity planning and scheduling.

Overview of resource obtainability on GKE

Kubernetes scheduling relies on declared resource requests. When scheduling large-scale accelerators, like GPUs or TPUs, strict resource requests can lead to clusters failing to scale up if specific hardware is unavailable. You can optimize capacity availability on GKE by using the following features:

Node pool auto-creation: dynamic, multi-family node pool creation.
Workload-level fallbacks: tolerations and node selectors configured to accept alternative hardware.
Geographic and regional scheduling: GKE multi-zonal and multi-regional capabilities.
Reservation management: consuming pre-purchased capacity before requesting on-demand resources.

Best practices for resource obtainability on GKE

This section provides recommendations to increase your capacity availability when scheduling workloads on GKE. These best practices cover strategies such as designing flexible hardware requirements, configuring node pool auto-creation, and leveraging geographic distribution to adapt to resource constraints. Node pool auto-creation automatically manages node pools based on Pod specifications. Rather than pre-defining node pools, specify resource requirements in your Pod manifests, and let GKE dynamically create nodes.

You can also discover and implement the following best practices by using Compute Advisor. For more information, see Use Compute Advisor.

Hardware flexibility options

To optimize capacity availability, you can instruct GKE to avoid binding workloads to a single static machine family or accelerator type. The following are examples of how you can configure flexible node pools and Pod affinity rules for different workload classes. The specific alternatives you choose depend on your application's resource demands:

General-purpose CPU node pools:
- Primary example: N2 (Intel-based general-purpose).
- Sample alternatives: N2D (AMD EPYC), C2 or C2D (Compute-optimized), or E2 (Cost-optimized).
- Implementation pattern: configure Pod specs with node affinity or tolerations that permit scheduling across multiple machine family labels, for example, cloud.google.com/machine-family in ["n2", "n2d", "c2d"]. This configuration lets GKE provision whichever pool has available capacity.
GPU node pools:
- Primary example: A2 Series (NVIDIA A100 GPUs).
- Sample alternatives: L4 (Universal AI/ML) or T4 (Inference).
- Implementation pattern: Configure separate node pools or ComputeClasses for different GPU tiers. For workloads that can run without A100-specific features, allow Pods to fall back to L4 or T4 pools if A2 provisioning is constrained.
TPU node pools:
- Primary example: TPU Ironwood (TPU7x).
- Sample alternatives: TPU v6 (Trillium) or TPU v5 (v5e or v5p).
- Implementation pattern: TPU slice provisioning can be highly constrained. Design training workloads with framework-level flexibility (for example, JAX or PyTorch configurations supporting variable slice topologies) to deploy on v6 or v5 slices when TPU Ironwood (TPU7x) capacity is unavailable.

The following table summarizes the primary selections and hardware alternatives for different types of workloads:

Workload type	Primary selection example	Sample alternatives	Architectural considerations
System or core workloads	N2	N2D, C2D, E2	Spans Intel and AMD hardware pools for node pool auto-creation.
GPU inference and processing	A2 (A100)	L4, T4	Flexibly targets lower-cost or higher-availability GPU node pools.
TPU model training	TPU Ironwood (TPU7x)	TPU v6, TPU v5 (v5e or v5p)	Utilizes flexible topologies for slice-based scheduling.

Implement node pool auto-creation and ComputeClasses

To optimize capacity availability, combine node pool auto-creation with ComputeClasses. The following list includes best practices:

Define ComputeClasses: create ComputeClass resources that specify a prioritized list of machine families, GPU types, or provisioning models. GKE attempts to provision nodes by using the highest-priority configuration available in the class. A prioritized fallback list significantly helps you obtain the resources that your workloads require. For example, to prevent GKE from falling back to general-purpose machines when a preferred specialized accelerator is unavailable, add the whenUnsatisfiable: DoNotScaleUp setting to your ComputeClass configuration. For more information, see Control autoscaled node attributes with custom ComputeClasses.
Reference ComputeClasses in Pod specs: in your workload Pod specifications, use the cloud.google.com/compute-class label to target your custom ComputeClass instead of a specific machine family or GPU type. For more information, see Request a ComputeClass in a workload.
Define multiple tolerations: if you don't use ComputeClasses, use node affinity rules in your Pod specifications that permit a range of machine families (such as cloud.google.com/machine-family in ["n2", "n2d"]). For more information, see Configure node pool auto-creation.

Geographic and multi-region flexibility on GKE

To increase capacity availability, deploy GKE clusters that span multiple zones, or run multi-cluster architectures:

Multi-zonal clusters: ensure node pools are configured to auto-scale across all available zones in a region.
Multi-region cluster federation: for large asynchronous jobs (such as offline batch inference or distributed training), deploy a multi-cluster orchestrator (like Kueue or Cluster Director) to queue workloads globally and dispatch them to a region with available capacity.
Pods running on Spot VMs: run interruptible workloads on Spot VMs by adding tolerations and instructing GKE to distribute excess capacity in different zones.

Additional best practices for GKE obtainability

Beyond hardware diversification, incorporate these GKE best practices to optimize cluster scaling success:

Implement Pod overprovisioning (capacity buffers): deploy low-priority "pause" Pods that reserve node capacity in advance. When high-priority AI workloads are submitted and regional capacity is constrained, Kubernetes immediately preempts the pause Pods, allowing containers to start up without waiting for new node provisioning. For more information, see About capacity buffers.
Use Flex-start with queued provisioning: for large batch and AI model training jobs, use Flex-start with queued provisioning, which integrates with Kueue and Dynamic Workload Scheduler. Flex-start with queued provisions all-or-nothing atomic node allocation, which prevents partial cluster scale-up failures. For more information, see Run a large-scale workload with flex-start with queued provisioning.
Enable image streaming and container image preloading: for large AI container images (such as PyTorch or TensorFlow images exceeding 10 GB), enable GKE Image streaming or use secondary boot disks for image preloading. This configuration reduces node warm-up time, which lets newly provisioned nodes start running workloads in seconds. For more information, see Use Image streaming to pull container images and Use secondary boot disks to preload data or container images.
Set cluster autoscaler location policy to ANY: configure node pools (especially for Spot VMs or flex-start) with the ANY location policy. This setting instructs the cluster autoscaler to search for requested capacity across all specified zones. Cluster autoscaler finds capacity over balancing node counts. For more information, see Cluster autoscaler overview.
Optimize accelerator utilization with GPU sharing: for workloads that don't require a dedicated GPU, use GPU time-sharing, multi-instance GPUs (MIG), or NVIDIA MPS to allow multiple containers to share a single accelerator. This approach optimizes effective capacity across node pools. For more information, see About GPU sharing strategies in GKE.

Use Compute Advisor

Compute Advisor is an AI-powered interface in the Google Cloud console, powered by Gemini, that helps you design resilient architectures for GKE. Compute Advisor provides near-real-time Flex-start VMs and Spot VMs availability guidance, while verifying your organization policies and resource quotas before deployment. Compute Advisor doesn't provide availability guidance for workloads that require on-demand resources.

To access Gemini in the Google Cloud console, complete the following steps:

In the Google Cloud console, go to the Overview page.

Go to Overview
In the Design your infrastructure with Compute Advisor section, submit a prompt. Gemini starts to generate a response.

Caution: Generative AI can make mistakes. Before taking any actions on a resource that Gemini recommends you to create, modify, or delete, validate all recommendations and configurations.
To generate architectural recommendations, run one of the following example prompts in Compute Advisor. When you click the Run prompt in Compute Advisor buttons, the Google Cloud console might take longer than 15 seconds to load:
- General accelerator strategy:
  
  Use case: use this prompt to analyze regional capacity signals and receive recommendations on machine types, zones, and fallback scheduling strategies when designing cluster configurations.
```
Configure a GKE cluster to improve chances of obtaining scarce GPU or TPU capacity.
```
  Run prompt in Compute Advisor
- Geographic flexibility and fallbacks:
  
  Use case: use this prompt when designing multi-cluster architectures or global job queuing systems (for example, with Kueue) to shift workload execution across regions depending on resource availability.
```
Configure multi-region fallbacks and geographic scheduling on GKE to increase GPU availability.
```
  Run prompt in Compute Advisor
- Priority reservation consumption:
  
  Use case: use this prompt to generate YAML configuration patterns for Pod affinity and autoscaling rules that prioritize reservation capacity.
```
Configure GKE autoscaling rules and Pod specs to prioritize consuming active reservations before scaling into on-demand pools.
```
  Run prompt in Compute Advisor
- ComputeClasses for fallback prioritization:
  
  Use case: use this prompt to generate the YAML manifest for a ComputeClass CustomResourceDefinition that prioritizes high-performance GPUs but includes lower-tier fallbacks to ensure workload scheduling.
```
Define a ComputeClass manifest for GKE to prioritize A2 GPU nodes with automatic fallbacks to L4 or T4 GPUs.
```
  Run prompt in Compute Advisor
- Node pool auto-creation for diversification:
  
  Use case: Use this prompt to write the YAML manifest for GKE cluster autoscaler resource limits and Pod affinity rules that enable NAP to provision alternative GPU or CPU nodes automatically.
```
Configure GKE node pool auto-creation to diversify machine families and prevent pending pods when regional accelerator capacity is constrained.
```
  Run prompt in Compute Advisor