Best practices for ComputeClasses

Platform engineers can use custom ComputeClasses to declaratively configure node settings and fallback priorities that Google Kubernetes Engine (GKE) uses to create nodes during autoscaling. You can create ComputeClasses based on specific strategies and workload requirements. This document provides best practices for designing and implementing ComputeClasses in your clusters. You should already be familiar with custom ComputeClasses. For a consolidated overview of all GKE best practices, see Best practices for GKE.

ComputeClass design

The following sections provide best practices for designing and implementing ComputeClasses in your clusters based on goals like maximizing obtainability and performance. ComputeClasses work with both manually created and auto-created node pools.

Design each ComputeClass based on a strategy

Design each ComputeClass to meet a specific goal for your workloads, teams, or organization. Use the fallback behavior of ComputeClasses and the ability to select both manually created and auto-created node pools to prioritize certain outcomes, such as reducing manual overhead or improving scheduling performance. The following sections describe common strategies.

Improve obtainability and reduce manual overhead

To delegate node pool creation to GKE, use only auto-created node pools in your ComputeClass. The autoscaler configures nodes based on hardware availability, Pod resource requirements, and zonal capacity. This strategy eliminates the need to manually create and tune node pools and can reduce the costs that are associated with idle, unused node capacity.

Improve scheduling performance and fine-tune nodes

To fine-tune your highest-priority nodes and reduce scheduling latency, use a mix of manually created and auto-created node pools in your ComputeClass. This hybrid strategy reduces how often Pods wait for GKE to create new node pools. Because your highest-priority node pools are manually created, you can fine-tune the hardware to meet the exact requirements of your Pods.

The hybrid strategy involves the following types of node pools in order of priority in the ComputeClass:

  1. Manually created node pools: these node pools have the exact specifications that you want most of your Pods to run on. Configure these node pools with specific node labels, node taints, capacity reservations, or special configurations such as kubelet parameters. Create these node pools with as many nodes as you estimate your Pods will need. In your ComputeClass, assign the highest priority to these node pools.
  2. Auto-created node pools: as a fallback measure, use the ComputeClass to request additional node pools that are still optimized for your Pods. Assign a lower priority to these auto-created node pools than for your manually created node pools.

The following example ComputeClass uses this hybrid strategy:

apiVersion: cloud.google.com/v1
kind: ComputeClass
metadata:
  name: hybrid-class
spec:
  nodePoolAutoCreation:
    enabled: true
  priorities:
  - nodepools: ['manual-pool1']
  - machineFamily: n4
    minCores: 16
    minMemoryGb: 64
  whenUnsatisfiable: DoNotScaleUp

When you deploy a workload that uses this ComputeClass, GKE places Pods on available nodes in manual-pool1. GKE creates new node pools only when the manually created node pool has no available capacity. The scheduling latency decreases when the number of existing nodes in the manually created node pool increases, because GKE doesn't need to create new nodes as frequently.

Explicitly define the last-resort scaling behavior

The whenUnsatisfiable field controls what happens if GKE can't meet the requirements of any of the priority rules in a ComputeClass. To avoid unexpected behavior after a version upgrade, explicitly specify a value for this field in every ComputeClass. Setting a value helps ComputeClass users to know what to expect when they select that ComputeClass in a workload. The recommended value for this field depends on the type of workload, as follows:

  • General-purpose workloads: if your workloads can run on any machine series, then specify a value of ScaleUpAnyway. If nodes that match a priority rule in the ComputeClass aren't available, GKE scales up nodes that use the default machine series of the cluster.
  • Workloads that need specialized hardware: for accelerators or high performance computing workloads that depend on specific hardware, such as GPUs or certain Compute Engine machine series, specify a value of DoNotScaleUp. If nodes that match a priority rule in the ComputeClass aren't available, the Pods remain in a Pending state until resources become available. This approach prevents Pods from running on incompatible hardware.

For more information, see Define scaling behavior when no priority rules apply.

Set a cluster-level default ComputeClass for most workloads

If most of your workloads have the same hardware requirements, then configure a default ComputeClass for the cluster. GKE applies the default ComputeClass to any workloads that don't explicitly select a ComputeClass. By setting a default ComputeClass, application operators don't need to change node selectors or manually request specific node pools and hardware in individual Pods. If you set a cluster-level default ComputeClass, then don't add node labels and taints for other ComputeClasses to existing node pools in the cluster. During scheduling for the cluster-level default ComputeClass, GKE ignores any node pools that have node labels or node taints for other ComputeClasses.

Set default ComputeClasses for namespaces to separate tenants

In addition to a cluster-level default ComputeClass, you can set a default ComputeClass for specific namespaces. If you have multi-tenant environments or you want to separate workloads that run on specialized hardware, then configure default ComputeClasses for those namespaces. To prevent system Pods from running on specialized hardware like GPU nodes, add a general-purpose ComputeClass as the default ComputeClass for system namespaces.

Run low-interaction workloads in Autopilot mode

If you have workloads that don't require manual interaction or management, then run those workloads in Autopilot mode by using ComputeClasses. You can enable Autopilot mode in any ComputeClass even when you have a Standard cluster. GKE runs the workloads that select an Autopilot ComputeClass on fully managed nodes that implement the security, scaling, and billing features of GKE Autopilot. For more information, see About Autopilot mode workloads in GKE Standard.

Stateful workloads

The following sections provide best practices for reducing disruptions or unexpected behavior in stateful workloads that rely on persistent data.

Disable active migration

Active migration automatically moves Pods to new nodes that have a higher priority in the ComputeClass or have the capacity to run unscheduled DaemonSet Pods. During active migration, GKE terminates Pods on existing nodes and creates new Pods on higher-priority nodes. If you have workloads that rely on data in local persistent storage, moving the Pods to new nodes might cause disruptions because the Pods lose access to persistent data. To avoid this issue, disable active migration for ComputeClasses that are intended for stateful workloads.

Improve scheduling reliability by using StorageClasses

Use StorageClasses to improve scheduling reliability for stateful workloads in the following ways:

  • Create volumes only after Pod creation: if you use dynamic volume provisioning, then specify a value of WaitForFirstConsumer in the volumeBindingMode field in a StorageClass. This volume binding mode prevents the creation of a PersistentVolume until after GKE creates a Pod that uses the corresponding PersistentVolumeClaim. GKE provisions the PersistentVolume in the same zone as the node that runs the Pod.
  • Use topology-aware StorageClasses: if your ComputeClass spans multiple generations of a machine series (for example, C4 and C3), then use a StorageClass that has automatic disk type selection enabled and that schedules only on nodes that support your specified disk types. You can use the built-in dynamic-rwo StorageClass or a custom StorageClass. Your stateful workloads can then run on multiple Compute Engine instance generations, because the cluster autoscaler dynamically chooses a compatible disk type.

Obtainability

The following sections provide best practices for improving obtainability in ComputeClasses, so that your Pods spend less time in a Pending state.

Request machine series instead of machine types

You can request a Compute Engine machine series or specific machine types in ComputeClass priority rules. Unless you have a strict dependency on a specific machine type, select a machine series by using the machineFamily field. During a scaling operation, GKE can create nodes that use any viable machine type in that machine series, which improves the likelihood of your Pods running on your most preferred node configuration.

Use capacity reservations for in-demand hardware

If your workloads rely on in-demand hardware, such as TPUs or high-performance GPUs, then create Compute Engine capacity reservations for the hardware and consume those reservations in your ComputeClasses. Capacity reservations improve the likelihood of hardware being available in your region or zone, which helps you to increase resource obtainability. To consume a reservation in a ComputeClass without affecting fallback behavior, use the Specific or AnyThenFail reservation affinity. If you use the AnyBestEffort or Automatic affinity and there's no available reserved capacity, then Compute Engine might bypass the ComputeClass priority rules and fall back to on-demand hardware. For more information, see Consuming reserved zonal resources.

Don't consume new reservations for at least one hour

The cluster autoscaler stores information about capacity reservations in a cache. When you create a new capacity reservation, the autoscaler might take up to one hour to discover that reservation. After you create a reservation, wait for at least one hour before you consume that reservation in a workload. If you deploy a workload that uses the reservation before the autoscaler stores the reservation in the cache, then the autoscaling operation might fail.

Security

The following sections provide best practices for improving the security of the ComputeClasses in your clusters. These measures are important because ComputeClasses can be used to create and configure nodes that use hardware that's expensive or has limited availability. Intentional or accidental misuse might result in workload disruptions, unplanned resource usage charges, and quota exhaustion.

Restrict API access to ComputeClass configurations

Workloads can use ComputeClasses to create nodes that run specialized hardware, including GPUs and TPUs. Restrict the access to create, modify, and delete ComputeClasses to the same principals who can create, modify, and delete nodes in your clusters. To control access to ComputeClasses, use RBAC policies.

Restrict ComputeClass availability by namespace

GKE customers often separate different teams or workload types by Kubernetes namespace. ComputeClasses are a cluster-scoped resource, which means that any workload in any namespace can select any ComputeClass by default. To avoid intentional or accidental misuse, use ValidatingAdmissionPolicies to control the set of ComputeClasses that workloads in each namespace can select. For example, you might prevent Pods in your web frontend namespace from being able to select ComputeClasses that create accelerators. Verify that your ValidatingAdmissionPolicies check for the following common configurations:

  • Check all selection fields: a workload can select a ComputeClass by using the nodeSelector, nodeAffinity, or tolerations field in the Pod specification. To avoid unintended ComputeClass selection, check all of these fields in your ValidatingAdmissionPolicy expressions.
  • Check for wildcard toleration bypasses: explicitly block or validate wildcard tolerations (for example, the operator: Exists toleration with no key). These wildcard selectors can encompass most node taints, including ComputeClass taints.
  • Check all workload controllers: configure the policy's matchConstraints to cover all workload controller resources (such as Deployment, StatefulSet, DaemonSet, Job, and CronJob). Don't scope your checks to only Pod resources.

For more information, see Restrict access to modify and select ComputeClasses.

Reliability

The following sections provide best practices for improving the reliability of autoscaling and Pod migration for ComputeClasses, which reduces the risk of disruptions or stuck Pods.

Prevent the use of conflicting node selectors

Node selectors in your Pods affect where GKE places those Pods and, in Autopilot mode or with node pool auto-creation, might trigger the creation of new node pools in your cluster. If you have Pods that select a ComputeClass and use node selectors to request nodes that conflict with the ComputeClass configuration, then GKE might not schedule the Pods at all.

For example, consider a ComputeClass that requests only on-demand instances. If a Pod selects that ComputeClass and selects Spot VMs in a node selector, GKE can't schedule the Pod because the ComputeClass and node selector conflict with each other. To avoid this issue, use methods like ValidatingAdmissionPolicies to prevent Pods that select ComputeClasses from also selecting system node labels. For more information, see Node selectors for system node labels.

Test all changes to active migration and autoscaling settings

The active migration and autoscaling settings in a ComputeClass directly affect how frequently GKE terminates your Pods to perform tasks like moving Pods to more preferred hardware and consolidating underutilized nodes. Modifications to these settings in an existing ComputeClass might result in unexpected workload disruptions. Before you apply any modifications to these settings in existing ComputeClasses, test the changes in a staging environment. You can also use annotations to protect critical workloads from eviction during scaling.

Test ComputeClass CRD updates before cluster upgrades

GKE regularly updates the ComputeClass CustomResourceDefinition (CRD) to add fields, modify field behavior, and fix issues. Field additions and modifications typically become effective in specific GKE versions. Before you upgrade your production clusters to new minor versions or patch versions, check whether changes to the CRD cause workload issues by using the following guidelines:

Use PodDisruptionBudgets to improve workload availability

ComputeClass operations that cause Pod evictions, such as active migration, respect any configured PodDisruptionBudgets. For example, you can configure an inference Deployment to have a PodDisruptionBudget that requires more than 70% of the Pods to be available. During active migration, if a Pod eviction violates that budget, then GKE doesn't evict the Pod. Specify PodDisruptionBudgets for workloads like the following:

  • Stateless workloads, such as inference Deployments.
  • Replicated stateful workloads, such as highly-available database applications.

Don't rely on PodDisruptionBudgets to protect workloads that need to run to completion, have only one instance, or rely on local persistent data. Specify a budget that balances between workload availability and allows functions like upgrades to complete.

Protect critical workloads from eviction

If you have workloads where each Pod must run to completion before being terminated, then add the cluster-autoscaler.kubernetes.io/safe-to-evict: "false" annotation to the Pod specification. This annotation prevents GKE from evicting Pods during autoscaling operations. Use this annotation to protect Pods that can't tolerate disruptions, such as single-instance stateful workloads and long-running batch Jobs.

Summary of best practices

This document has the following best practices for ComputeClasses:

What's next