Demand scaling overview

This document provides an overview of how to scale compute capacity and obtain hardware resources across Google Cloud to maintain performance, control costs, and avoid downtime. To handle variable demand, you can autoscale Compute Engine instances, orchestrate containers in Google Kubernetes Engine (GKE) or workloads in clusters, reserve capacity in advance, and distribute traffic across resources.

The recommendations in this document apply to all available compute options, except Cloud Run and Batch, which manage and scale compute infrastructure for you. For more information about the compute options that you can use in Google Cloud before you scale, see Choose compute options.

Compute infrastructure scaling options

To scale your compute capacity and improve resource obtainability, choose from the features that your compute service supports. You can combine multiple features—such as reservations, autoscalers, and load balancers—to match your workload requirements. If you want to evaluate these features side by side based on their purpose, supported compute options, triggers, and use cases, then see Compare scaling options in this document.

Each compute service scales capacity and helps you obtain resources by using the following options:

  • All compute options: across compute services except Cloud Run for reservations, you can have a high level of assurance for hardware availability in advance and route user requests across healthy backends. You can use the following cross-product options:

    • Reservations: request capacity in a specific zone for immediate use or for a future date. Reservations aren't an autoscaler, but they provide a high level of assurance that hardware such as graphics processing units (GPUs), tensor processing units (TPUs), and Local SSD disks is available when your workloads scale out.

    • Traffic distribution with Cloud Load Balancing: distribute incoming requests across healthy compute instances, node pools, or containers, and route traffic to new resources as capacity scales.

  • Compute Engine: you scale capacity when you create virtual machine (VM) or bare metal instances, or when you use managed instance groups (MIGs). A MIG manages a group of compute instances as a single entity and can add or remove capacity based on load, schedules, or hardware availability. For workloads that require hardware isolation, you can also autoscale sole-tenant node groups on dedicated physical servers. You can use the following options:

    • MIG autoscaling: add or remove compute instances in a MIG based on CPU usage, load balancer capacity, Cloud Monitoring metrics, schedules, or predictive forecasts.

    • MIG instance flexibility: rank multiple machine types in a MIG to improve resource obtainability.

    • Bulk mode and resize requests in MIGs: add all requested compute instances in a MIG at the same time when full capacity becomes available, including for GPUs, TPUs, and H4D instances with Flex-start VMs.

    • Standby pools in MIGs: keep a standby pool of pre-configured suspended or stopped compute instances in a MIG to speed up scale-out. When traffic or CPU usage increases, the MIG resumes or starts the compute instances instead of creating new ones.

    • Bulk compute instance creation: create multiple compute instances in a single API request with automatic zone and machine type selection.

    • Sole-tenant node group autoscaling: add or remove dedicated physical sole-tenant nodes based on guest compute instance demand to maintain hardware isolation and optimize bring-your-own-license costs.

    • Custom orchestrator: call the Compute Engine API from your own or open-source schedulers to create, resize, and delete compute instances based on custom job queues or business rules.

  • Cluster Director: Google deploys managed clusters for artificial intelligence (AI), machine learning (ML), and high performance computing (HPC) workloads that use either Slurm orchestration or managed compute orchestration backed by zonal MIGs. In Slurm clusters, the Slurm scheduler creates dynamic compute nodes when queued jobs exceed static baseline nodes and powers them down after five minutes of idle time. In managed compute clusters, you run workloads over SSH or integrate your own scheduler, and scale capacity when you adjust MIG target sizes. You can use the following options:

    • Cluster orchestration: autoscale dynamic compute nodes within managed Slurm partitions based on job queue demand, or provision managed compute clusters backed by zonal MIGs to run workloads over SSH or integrate your own scheduler.

    • Live cluster modification: update an active cluster to raise static or maximum dynamic node counts, resize or add MIGs, add compute resource configurations, or add partitions and nodesets without disruption to active jobs.

    • Capacity consumption options: obtain cluster compute nodes through future reservations for capacity blocks, future reservations in calendar mode, Flex-start VMs, Spot VMs, or on-demand compute instances.

  • Cluster Toolkit: you define self-managed AI, ML, and HPC clusters as code with YAML blueprints and Terraform modules orchestrated by Slurm or GKE. Capacity scales based on Slurm job queues or Kubernetes Pod demand, or when you edit and redeploy your cluster blueprint. You can use the following options:

    • Slurm dynamic nodesets: configure the slurm-nodeset-dynamic module so that Slurm creates and deletes ephemeral compute instances based on queued jobs.

    • GKE node pool autoscaling: set minimum and maximum node counts in your blueprint with the gke-node-pool module so that GKE adds or removes nodes as Pod demand changes.

    • Live blueprint reconfiguration: update and redeploy blueprints to add or resize partitions and static nodes in an active Slurm cluster or to resize node pools in a deployed GKE cluster.

    • Multi-reservation Slurm partitions: group multiple reservation-backed nodesets into one Slurm partition so that large jobs span capacity across separate reservations.

  • GKE: capacity scales across two layers. Workload autoscalers adjust Pod replica counts or container CPU and memory requests as demand changes. Infrastructure autoscalers add or remove underlying nodes and node pools. Google manages node capacity for you in Autopilot mode, whereas you configure cluster autoscaler, node pool auto-creation, and custom ComputeClass resources in Standard mode. Startup accelerators also reduce the time that new Pods need before they serve traffic. You can use the following options:

    • Node and cluster capacity: adjust node counts, create node pools, and obtain hardware with the following features:

      • Custom ComputeClasses: define prioritized hardware fallback rules, Spot VMs or Flex-start preferences, reservation consumption, active migration to higher-priority hardware, and workload-level node pool auto-creation.

      • Cluster autoscaler: add or remove nodes in existing Standard mode node pools when Pods wait for resources or when nodes are underutilized.

      • Node pool auto-creation: create and delete node pools in parallel based on the CPU, memory, GPU, TPU, or zone requirements of pending Pods.

      • Flex-start provisioning: obtain GPUs, TPUs, and H4D nodes for up to seven days either node by node or all at once through queued provisioning, and replace expiring nodes through ComputeClasses.

      • Capacity buffers: pre-provision spare active or suspended standby nodes, including active buffers that align with ComputeClass priorities, so that pending Pods schedule without a wait for new nodes.

      • Container-optimized compute platform: speed up node allocation and Pod placement with Balanced and Scale-Out ComputeClasses for Autopilot clusters and Autopilot mode workloads in Standard clusters.

      • Sole-tenant nodes in GKE: run Standard mode node pools on dedicated physical servers to meet isolation or license requirements.

    • Pod and workload capacity: adjust Pod replicas, container sizes, and batch queues with the following features:

      • Horizontal Pod autoscaling (HPA): add or remove Pod replicas based on CPU usage, memory usage, or custom metrics.

      • Vertical Pod autoscaling (VPA): adjust container CPU and memory requests and limits based on historical usage.

      • Multidimensional Pod autoscaling (MPA): combine HPA on CPU and VPA on memory for the same workload at the same time.

      • Custom and external metrics: trigger HPA scale decisions from application metrics or external services such as Pub/Sub and Cloud Load Balancing.

      • Scale to zero: scale idle Pods and node pools down to zero replicas and wake them when demand resumes in HPA custom and external metrics or in Kubernetes Event-driven Autoscaling (KEDA) triggers.

      • Kueue job queues: queue batch and AI workloads, share resource quotas across teams, and coordinate simultaneous node allocation.

    • Startup acceleration: shorten container initialization and model load times during scale-out with the following features:

      • Cloud Storage FUSE with parallel streaming: mount Cloud Storage buckets as local file systems with parallel reads and file caches.

      • CPU startup boost: grant temporary extra CPU to containers while they initialize, and then return CPU requests to baseline values.

      • Image streaming: stream container image blocks on demand from Artifact Registry so that Pods start without waiting for the entire image download.

      • Secondary boot disks: attach preloaded Persistent Disk volumes with container images or model data to nodes in parallel during node boot.

      • In-place Pod resize: update CPU and memory allocations on active containers without Pod restarts.

      • Pod snapshots: restore Pod replicas from saved memory and file system checkpoints instead of a cold boot.

      • Run:ai Model Streamer: stream model weights in safetensors format from Cloud Storage or persistent volumes in parallel into GPU or TPU memory.

      • Volume Populator: preload Google Cloud Hyperdisk ML volumes with model weights and datasets from Cloud Storage before Pods start.

  • Google Cloud VMware Engine: Google manages dedicated VMware private cloud clusters where capacity expands or shrinks as physical ESXi hosts join or leave the cluster. To add or remove ESXi hosts when sustained CPU, memory, or vSAN storage usage crosses your configured thresholds, use autoscale policies.

Compare scaling options

To select the scaling or resource obtainability option that matches your operational requirements, compare features in the following table:

Scaling option Primary purpose Supported compute options Compatible scaling options Scaling trigger Use cases
Reservations Reserve zone capacity for immediate use or for a future date with a high level of availability assurance All compute options, except Cloud Run All scaling options Manual or calendar reservation request; not an autoscaler
  • Planned growth and peak scale events
  • Large migrations
  • Backup and disaster recovery
  • High-demand resources, such as GPUs and TPUs
Traffic distribution with Cloud Load Balancing Distribute incoming traffic across healthy compute instances, node pools, or containers All compute options All scaling options Incoming user requests and backend health checks
  • High-traffic web applications and APIs
  • Multi-zone and multi-region availability
  • Zero-downtime scale-out
MIG autoscaling Add or remove compute instances in a MIG for variable workload demand Compute Engine Reservations, MIG instance flexibility, standby pools in MIGs, sole-tenant node group autoscaling, custom orchestrator CPU usage, load balancer capacity, Cloud Monitoring metrics, schedules, or predictive forecasts
  • Web services and APIs
  • Parallel batch workloads
  • Multi-zone high availability
  • Stateless microservices
MIG instance flexibility Rank multiple machine types in a MIG to improve resource obtainability Compute Engine MIG autoscaling, reservations, Spot VMs When primary machine types lack capacity during MIG instance creation or scale-out
  • Higher compute instance obtainability
  • Cost and performance balance across machine series
  • Stateless web and batch workloads
Bulk mode and resize requests in MIGs Add all requested compute instances in a MIG at the same time when full capacity becomes available Compute Engine and Cluster Director Reservations, live cluster modification, custom orchestrator Target size set in bulk mode or creation of a resize request, fulfilled when full hardware capacity is available
  • Distributed AI/ML model training
  • Multi-node jobs that require all compute instances at the same time
  • Time-flexible GPU, TPU, and H4D workloads
Standby pools in MIGs Keep a standby pool of pre-configured suspended or stopped compute instances ready to speed up MIG scale-out Compute Engine MIG autoscaling, reservations Manual resize or MIG scale-out demand that resumes or starts pooled compute instances
  • Applications with slow operating system or runtime initialization
  • Sudden traffic spikes
  • Lower standby cost than active idle compute instances
Bulk compute instance creation Create multiple compute instances in one API request with automatic zone and machine selection Compute Engine Reservations, Spot VMs, custom orchestrator Single API or Google Cloud CLI bulk insert request
  • One-time batch capacity bursts
  • Custom cluster deployments
  • Multi-instance hardware allocation
Sole-tenant node group autoscaling Autoscale dedicated physical servers for isolated workloads Compute Engine and GKE Reservations, MIG autoscaling, custom orchestrator, cluster autoscaler Resource demand of guest compute instances
  • Security and regulatory compliance
  • Per-core or per-socket license optimization
  • Hardware performance isolation
Custom orchestrator Control compute instance creation, MIG target size, and deletion from your own scheduler, such as Ray, SkyPilot, or HTCondor Compute Engine and Cluster Director Reservations, MIG autoscaling, bulk mode and resize requests in MIGs, bulk compute instance creation, sole-tenant node group autoscaling, live cluster modification Custom business metrics or external scheduler queues
  • Proprietary scaling logic
  • Legacy on-premises scheduler migration
  • Fine-grained job lifecycle control
Slurm orchestration and dynamic nodes Autoscale dynamic compute nodes in Slurm partitions for batch, AI, and HPC jobs Cluster Director and Cluster Toolkit Reservations, Flex-start VMs, Spot VMs, live cluster modification, live blueprint reconfiguration Queued Slurm jobs and node idle timeouts
  • HPC workloads
  • Large-scale AI/ML training
  • Batch simulation queues
Live cluster modification Add or resize partitions, nodesets, MIGs, or static and dynamic node limits on an active cluster Cluster Director Slurm orchestration and dynamic nodes, bulk mode and resize requests in MIGs, custom orchestrator, reservations Cluster update request on an active cluster
  • Baseline cluster expansion without job disruption
  • GPU, TPU, or CPU nodeset and MIG additions on active clusters
Live blueprint reconfiguration Update and redeploy blueprints to add or resize Slurm partitions, static nodes, or GKE node pools on an active cluster Cluster Toolkit Slurm orchestration and dynamic nodes, cluster autoscaler, reservations Blueprint update and redeployment on an active cluster
  • Infrastructure-as-code cluster expansion without job disruption
  • Multi-reservation Slurm partitions
Custom ComputeClasses Define prioritized hardware fallback rules, capacity consumption, and node migration GKE in Standard and Autopilot modes Cluster autoscaler, node pool auto-creation, Flex-start provisioning in GKE, reservations, capacity buffers Pending Pods that match a ComputeClass selector or availability of higher-priority hardware
  • High obtainability across GPU, TPU, or CPU series
  • Fallback from reservations to Flex-start or Spot VMs
  • Active migration back to preferred hardware
Cluster autoscaler Add or remove nodes in existing node pools GKE in Standard mode, and Cluster Toolkit GKE blueprints Custom ComputeClasses, reservations, sole-tenant node group autoscaling, HPA, VPA, capacity buffers, scale to zero Pending Pods or node underutilization
  • Containerized workloads with variable traffic
  • Sudden spikes in demand
Node pool auto-creation Create and delete node pools in parallel to match Pod hardware requirements GKE: managed in Autopilot mode; configurable in Standard mode Custom ComputeClasses, reservations, Flex-start provisioning in GKE, HPA, VPA, capacity buffers Pending Pods that require machine series, accelerators, or zones absent from existing node pools
  • Diverse GPU, TPU, or CPU workload demands
  • Multi-tenant clusters that create multiple node pools in parallel
Flex-start provisioning in GKE Obtain GPUs, TPUs, and H4D nodes as capacity opens, per node or all at once with queued provisioning GKE in Standard and Autopilot modes Custom ComputeClasses, node pool auto-creation, cluster autoscaler, Kueue job queues Pending Pods or queued ProvisioningRequest objects when hardware capacity opens
  • AI/ML model training and fine-tuning
  • Batch inference workloads
  • Simultaneous multi-node accelerator allocation
Capacity buffers Pre-provision spare active or suspended standby nodes for fast Pod placement and obtainability GKE in Standard and Autopilot modes Custom ComputeClasses, cluster autoscaler, node pool auto-creation, HPA, reservations Consumption of buffered node capacity by scheduled Pods
  • Latency-sensitive applications
  • AI inference and agent workloads
  • Sudden traffic spikes
Container-optimized compute platform Speed up node readiness and Pod placement with managed infrastructure and Balanced or Scale-Out ComputeClasses GKE: Autopilot clusters, and Autopilot mode workloads in Standard clusters Node pool auto-creation, HPA, VPA, capacity buffers, reservations Pending Pods that require additional compute capacity
  • Rapid cluster scale-out
  • Workloads with variable resource demand
  • Managed infrastructure operations
HPA Add or remove Pod replicas to match workload demand GKE in Standard and Autopilot modes Cluster autoscaler, node pool auto-creation, MPA, custom and external metrics, scale to zero CPU usage, memory usage, or custom and external metrics
  • Stateless applications
  • Web services and APIs
  • Variable request volume
VPA Adjust container CPU and memory requests and limits based on usage GKE in Standard and Autopilot modes Cluster autoscaler, node pool auto-creation, MPA, in-place Pod resize, reservations Historical container CPU and memory usage
  • Stateful applications
  • Workloads with uncertain resource needs
  • Container resource optimization
MPA Scale Pod replicas on CPU and container memory requests at the same time GKE in Standard and Autopilot modes Cluster autoscaler, node pool auto-creation, custom and external metrics Simultaneous CPU demand shifts and container memory usage changes
  • Stateless services with variable traffic and changing memory needs
Custom and external metrics Scale Pod replicas based on application or external service signals GKE in Standard and Autopilot modes HPA, MPA, scale to zero, cluster autoscaler, node pool auto-creation Application metrics or external queue and traffic metrics
  • Queue-based workers, such as Pub/Sub consumers
  • Latency-bound web APIs
  • AI inference throughput targets
Scale to zero Scale idle Pods and node pools down to zero and wake them on demand or events GKE in Standard and Autopilot modes HPA, custom and external metrics, cluster autoscaler, node pool auto-creation Absence of demand, or new demand in HPA custom and external metrics or KEDA triggers
  • Intermittent or bursty workloads
  • Event-driven batch jobs
  • Zero idle compute cost
Kueue job queues Queue batch jobs, enforce team quotas, and coordinate simultaneous node allocation GKE in Standard and Autopilot modes Flex-start provisioning in GKE, Custom ComputeClasses, cluster autoscaler, node pool auto-creation Admitted batch jobs when cluster quota and hardware capacity open
  • Multi-team GPU and TPU quota management
  • Queued AI training and batch jobs
Cloud Storage FUSE with parallel streaming Mount Cloud Storage buckets as file systems with parallel reads and caches GKE in Standard and Autopilot modes HPA, cluster autoscaler, node pool auto-creation, Pod snapshots Pod startup and sequential file reads from mounted buckets
  • High-throughput AI training and inference from Cloud Storage
  • Multi-node checkpoint access
CPU startup boost Grant temporary extra CPU to containers during initialization GKE in Standard and Autopilot modes HPA, VPA, cluster autoscaler, node pool auto-creation Container startup and initialization phase
  • Applications with high CPU needs at startup, such as Java Virtual Machine warmup
  • Latency-sensitive API scale-out
Image streaming Stream container image blocks on demand from Artifact Registry during Pod startup GKE in Standard and Autopilot modes HPA, cluster autoscaler, node pool auto-creation, capacity buffers Container image pull from Artifact Registry when a Pod starts
  • Large AI/ML container images
  • Fast microservice scale-out
Secondary boot disks Attach preloaded Persistent Disk volumes with container images or model data at node boot GKE in Standard and Autopilot modes Custom ComputeClasses, HPA, cluster autoscaler, node pool auto-creation, capacity buffers Node boot in a node pool configured with secondary boot disk images
  • Preloaded model weights or datasets on nodes
  • Base container image layer caching
In-place Pod resize Update active container CPU and memory allocations without Pod restarts GKE in Standard and Autopilot modes VPA, cluster autoscaler, node pool auto-creation Updated container CPU or memory requests and limits
  • Stateful or long-lived applications
  • Vertical scale without connection drops or state loss
Pod snapshots Restore Pod replicas from saved memory and file system checkpoints GKE in Standard and Autopilot modes HPA, cluster autoscaler, node pool auto-creation, capacity buffers Pod replica creation from a saved snapshot
  • AI inference servers with long model load times
  • Applications with slow runtime warmup
  • Rapid scale-out during demand spikes
Run:ai Model Streamer Stream model tensors in parallel into GPU or TPU memory GKE in Standard and Autopilot modes HPA, cluster autoscaler, node pool auto-creation, Volume Populator AI inference server startup and model load requests
  • Large language model inference with vLLM
  • Fast GPU and TPU replica scale-out
Volume Populator Preload Hyperdisk ML volumes with data from Cloud Storage GKE in Standard and Autopilot modes HPA, cluster autoscaler, node pool auto-creation, Pod snapshots PersistentVolumeClaim creation that references a GCPDataSource resource
  • Preloaded AI/ML model weights
  • Shared read-only dataset access across multiple nodes
Autoscale policies Add or remove ESXi hosts in a VMware private cloud cluster VMware Engine Cloud Load Balancing Sustained CPU, memory, or vSAN storage usage
  • Migrated VMware clusters with variable load
  • Storage-intensive or compute-intensive vSphere workloads

Best practices for scalability

To design infrastructure that scales without bottlenecks or capacity shortages, apply the best practices in this section across compute capacity, network design, and application architecture.

Plan compute capacity and obtainability

To prepare your infrastructure for demand growth and improve hardware obtainability, apply the following practices:

  • Estimate your workload needs: before you configure autoscalers, measure your baseline usage and peak fluctuations. To inspect historical usage, forecast demand, and set accurate minimum and maximum compute instance limits, use Capacity Planner.

  • Design for resource obtainability: when you request scarce hardware such as GPUs or TPUs, optimize compute provisioning across the following three dimensions:

    • Location flexibility: distribute workloads across zones with regional MIGs.

    • Machine type flexibility: allow multiple machine series through MIG instance flexibility or GKE custom ComputeClasses.

    • Time flexibility: use Flex-start provisioning when jobs can wait for capacity.

  • Use reservations for high capacity assurance: for baseline traffic, peak scale events, or scarce GPUs and TPUs, reserve capacity in advance. Reservations provide a high level of assurance that resources are available in your target zone when your workloads scale out. If a single Slurm job exceeds one reservation, then span Cluster Toolkit partitions across multiple reservations.

  • Pre-warm spare capacity for latency-sensitive workloads: if your application can't wait for new compute instances to provision during sudden traffic spikes, then maintain ready-to-use spare capacity. Configure GKE capacity buffers to pre-provision active or standby nodes, or use standby pools in Compute Engine MIGs to resume suspended compute instances without full operating system boot delays.

Design networks for scale

To move data between scaled resources without delays or address exhaustion, apply the following network practices:

  • Allocate sufficient IP addresses: verify that your subnets have enough free IP addresses for the maximum instance or node count in your autoscaler limits. In GKE, also check secondary IP ranges for Pods and Services. If a subnet runs out of IP addresses, then autoscalers fail to create new compute instances or Pods.

  • Choose high-throughput network interfaces: for workloads that transfer large datasets or model weights across nodes, select high-bandwidth network interface controllers such as Google Virtual Network Interface Card or remote direct memory access. These interfaces reduce CPU overhead and increase bandwidth between scaled compute instances.

  • Optimize instance placement: for multi-node AI training or HPC jobs, group compute instances on physical servers close together with compact placement policies or topology-aware placement in Cluster Director and Cluster Toolkit. Close physical placement reduces network hops and latency across scaled nodes. For fault-tolerant web services, spread compute instances across zones instead so that a single zone outage doesn't disrupt traffic.

Design applications for scale

To let your applications scale out and scale in without user disruption, apply the following architecture practices:

  • Keep application tiers stateless: store user session data, checkpoints, and shared files in external storage or databases rather than on local instance disks. This design lets autoscalers add or remove compute instances and Pods at any time without data loss.

  • Decouple components with queues and microservices: split monolithic workloads into independent services connected by load balancers or message queues such as Pub/Sub. This architecture lets you scale each service based on its own demand—including down to zero with event-driven autoscalers—and absorb traffic bursts without overload on downstream backends.

  • Shorten container startup and model load times: reduce the delay between resource creation and traffic readiness so that scaled capacity serves requests right away. In GKE, stream or preload container images and use CPU startup boost or Pod snapshots to speed up runtime initialization. For AI workloads, stream model weights from Cloud Storage FUSE, Run:ai Model Streamer, or pre-populated volumes.