Set up Standard clusters to use Autopilot by default

You can configure Google Kubernetes Engine (GKE) Standard clusters to function similarly to Autopilot mode clusters by enabling cluster settings like the default ComputeClass, Autopilot cluster security policies, and cluster scale-to-zero.

This document is intended for platform administrators and security engineers who want to get all of the benefits of Autopilot mode in a Standard cluster with the flexibility to modify or disable individual settings. If you don't require this flexibility, then create a dedicated Autopilot cluster instead.

You should already be familiar with the following concepts:

Autopilot cluster settings

To configure a Standard cluster to function like an Autopilot cluster, you configure all of the following settings:

  • Autopilot ComputeClass as cluster default: set one of the built-in Autopilot ComputeClasses. Any Pods that don't select a ComputeClass run by using the specified default ComputeClass. The built-in ComputeClasses all use Autopilot mode and the Pod-based billing model.
  • Autoscale the cluster to zero nodes: configure the cluster autoscaler to scale the cluster to zero nodes when there are no active user workloads in the cluster. You can optionally adjust the delay period before the autoscaler scales the cluster to zero nodes.
  • Zero initial nodes: during cluster creation, prevent the automatic creation of a default node pool. For existing clusters, delete any manually created node pools.
  • Autopilot cluster security policies: enforce all of the Autopilot security policies. These policies help to protect system workloads and identities.

You can modify your cluster to change any of these settings independently of the other settings. For example, if you need to manually create a node pool, then you can disable the corresponding Autopilot cluster security policy without changing any of the other settings.

Flexibility and security considerations

When you create an Autopilot cluster, the cluster mode is permanent. Cluster settings like scaling to zero nodes, preventing the manual creation of node pools, and the various security constraints are immutable. However, when you configure a Standard cluster to use all of the settings of an Autopilot cluster, the the settings are mutable. Anyone who has access to modify cluster configurations can change or disable individual settings.

To enforce the use of these settings across an organization, folder, or project, organization administrators can use a custom organization policy constraint. For more information, see the Enforce Autopilot settings with custom organization policies section of this document.

Limitations

If you configure Confidential GKE Nodes at the cluster level, then GKE ignores your command to use an Autopilot ComputeClass to as the cluster default.

Before you begin

Before you start, make sure that you have performed the following tasks:

  • Enable the Google Kubernetes Engine API.
  • Enable Google Kubernetes Engine API
  • To use the Google Cloud CLI for this task, install and then initialize the gcloud CLI. If you previously installed the gcloud CLI, get the latest version by running the gcloud components update command. Earlier gcloud CLI versions might not support running the commands in this document.
  • Ensure that you have a Standard cluster that meets all of the requirements.

Requirements

To use all of the Autopilot capabilities in a Standard cluster, your cluster must meet the following requirements:

  • The cluster must run GKE version 1.37.1-gke.3020000 or later. The version requirements for individual settings are as follows:

    • To set a built-in Autopilot ComputeClass as the cluster default, the cluster must run GKE version 1.37.1-gke.1380000 or later.
    • To create a cluster that has zero initial nodes, the cluster must run GKE version 1.37.1-gke.3020000 or later.
    • To configure the cluster autoscaler to scale the cluster to zero nodes, the cluster must run GKE version 1.37.1-gke.3020000 or later.

    To upgrade your cluster to a compatible version, see Manually upgrade a cluster's control plane or node pools.

  • The cluster must meet all of the requirements for using Autopilot ComputeClasses. To check whether your cluster meets these requirements, see Run workloads in Autopilot mode in Standard clusters.

Required roles and permissions

To get the permissions that you need to create or update clusters to use Autopilot mode, ask your administrator to grant you the following IAM roles on your Google Cloud project:

For more information about granting roles, see Manage access to projects, folders, and organizations.

You might also be able to get the required permissions through custom roles or other predefined roles.

Create a Standard cluster that uses Autopilot mode

To create a new Standard cluster that uses all of the Autopilot settings, run the following command:

gcloud container clusters create CLUSTER_NAME \
    --location=CONTROL_PLANE_LOCATION \
    --num-nodes=0 \
    --default-compute-class=BUILT_IN_COMPUTECLASS_NAME \
    --enable-cluster-scale-to-zero \
    --autopilot-cluster-policies=no-system-mutation,no-system-impersonation,no-unsafe-webhooks,no-standard-node-pools

Replace the following:

  • CLUSTER_NAME: a name for your new cluster.
  • CONTROL_PLANE_LOCATION: a region for your cluster control plane, such as us-central1.
  • BUILT_IN_COMPUTECLASS_NAME: the built-in ComputeClass to use as the cluster default, such as autopilot.

Optionally, you can adjust the delay period before the cluster autoscaler scales the cluster to zero nodes by using the --cluster-scale-to-zero-delay flag:

gcloud container clusters create CLUSTER_NAME \
    --location=CONTROL_PLANE_LOCATION \
    --num-nodes=0 \
    --default-compute-class=BUILT_IN_COMPUTECLASS_NAME \
    --enable-cluster-scale-to-zero \
    --cluster-scale-to-zero-delay=SCALE_TO_ZERO_DELAY \
    --autopilot-cluster-policies=no-system-mutation,no-system-impersonation,no-unsafe-webhooks,no-standard-node-pools

Replace SCALE_TO_ZERO_DELAY with the delay duration. This value must specify an integer and a unit of time, such as 0s, 1m, or 3h. The maximum delay is 3 hours. If you don't specify the --cluster-scale-to-zero-delay flag, the default delay is 3 hours.

If your cluster creation command fails and displays an error message, check the error message for more information. The most common errors are as follows:

  • Your cluster uses a version that doesn't support one of the specified flags. To resolve this error, update your cluster to the minimum required version as specified in the Requirements section.
  • You specified node pool configuration flags (such as the --machine-type flag) in your cluster creation command. Because your new cluster has no initial node pool, you can't specify node pool configuration flags. To resolve this error, remove the node pool configuration flags from your command.
  • You specified the name of a custom ComputeClass for the --default-compute-class flag. To resolve this error, specify one of the built-in Autopilot ComputeClasses.

Update an existing Standard cluster to use Autopilot mode

You can modify an existing Standard cluster to use Autopilot mode. This process involves potentially disruptive changes to your workloads, such as deleting manually created node pools. Perform these steps during a scheduled maintenance period.

  1. Verify that your cluster meets all of the Requirements in this document.

  2. Review your cluster and make any required changes to avoid conflicts with Autopilot cluster policies. For more information about the potential changes, see Supported policies.

  3. When you set an Autopilot ComputeClass as the cluster default, incoming Pods that don't select a different ComputeClass become Autopilot workloads. Check any workloads that don't select a ComputeClass for the following properties:

    • Verify that your workloads comply with the Autopilot security constraints, or configure an allowlist for privileged workloads.
    • Find and remove cloud.google.com/gke-nodepool node selectors or node affinity rules from the Pods.
    • Check whether any namespaces have the cloud.google.com/default-compute-class or cloud.google.com/default-compute-class-non-daemonset label. Namespace-level default ComputeClasses take precedence over the cluster-level default ComputeClass.
  4. Check whether the cluster has a custom ComputeClass that's named default:

    kubectl get computeclass default -o yaml
    

    If the output is blank, then skip to the next step. If the default ComputeClass exists, then the output is similar to the following:

    apiVersion: cloud.google.com/v1
    kind: ComputeClass
    metadata:
      annotations:
        cloud.google.com/gke-managed-default-compute-class: "true"
      name: default
    # Multiple lines are omitted here.
    

    In the output, if the cloud.google.com/gke-managed-default-compute-class: "true" annotation is in the metadata.annotations field, then skip to the next step. If this annotation doesn't exist, then delete the ComputeClass:

    kubectl delete computeclass default
    
  5. Find and delete any existing manually created node pools in the cluster.

  6. Enable the Autopilot settings for the cluster:

    1. Set the cluster default ComputeClass to a built-in ComputeClass:

      gcloud container clusters update CLUSTER_NAME \
          --location=CONTROL_PLANE_LOCATION \
          --default-compute-class=BUILT_IN_COMPUTECLASS_NAME
      

      Replace the following:

      • CLUSTER_NAME: the name of your existing Standard cluster.
      • CONTROL_PLANE_LOCATION: the region of your cluster control plane.
      • BUILT_IN_COMPUTECLASS_NAME: the built-in Autopilot ComputeClass to use as the cluster default, such as autopilot.
      1. Enable cluster autoscaling and let the autoscaler scale the cluster to zero nodes:

        gcloud container clusters update CLUSTER_NAME \
            --location=CONTROL_PLANE_LOCATION \
            --enable-autoscaling \
            --enable-cluster-scale-to-zero
        

        Optionally, you can adjust the delay period before the cluster autoscaler scales the cluster to zero nodes by using the --cluster-scale-to-zero-delay flag:

        gcloud container clusters update CLUSTER_NAME \
            --location=CONTROL_PLANE_LOCATION \
            --enable-autoscaling \
            --enable-cluster-scale-to-zero \
            --cluster-scale-to-zero-delay=SCALE_TO_ZERO_DELAY
        

        Replace SCALE_TO_ZERO_DELAY with the delay duration. This value must specify an integer and a unit of time, such as 0s, 1m, or 3h. The maximum delay is 3 hours. If you don't specify the --cluster-scale-to-zero-delay flag, then the default delay is 3 hours.

      2. Enable the Autopilot cluster policies:

        gcloud container clusters update CLUSTER_NAME \
            --location=CONTROL_PLANE_LOCATION \
            --autopilot-cluster-policies=no-system-mutation,no-system-impersonation,no-unsafe-webhooks,no-standard-node-pools
        

    If any of your cluster update commands fail and display an error message, check the error message for more information. The most common errors are as follows:

    • Your cluster uses a version that doesn't support one of the specified flags. To resolve this error, upgrade your cluster to the minimum required version as specified in the Requirements section.
    • You specified the name of a custom ComputeClass for the --default-compute-class flag. To resolve this error, specify one of the built-in Autopilot ComputeClasses.
    • Your cluster has manually created node pools, so GKE can't enforce the no-standard-node-pools policy. To resolve this error, delete the manually created node pools.

Verify that your cluster uses Autopilot mode

After you configure your new or existing cluster, verify that the Autopilot settings are active:

  1. Verify that the default ComputeClass, scale-to-zero, and cluster policy settings are enabled on the cluster:

    gcloud container clusters describe CLUSTER_NAME \
        --location=CONTROL_PLANE_LOCATION \
        --format="yaml(autoscaling.defaultComputeClassConfig,autoscaling.scaleToZeroConfig,autopilot.clusterPolicyConfig)"
    

    The output is similar to the following:

    autoscaling:
      defaultComputeClassConfig:
        computeClass: autopilot
        enabled: true
      scaleToZeroConfig:
        enabled: true
        delay: 3h
    autopilot:
      clusterPolicyConfig:
        noManualNodePools: true
        noSystemImpersonation: true
        noSystemMutation: true
        noUnsafeWebhooks: true
    
  2. Verify that GKE created the managed default ComputeClass in your cluster:

    kubectl get computeclass default -o yaml
    

    The output is similar to the following:

    If the output is blank, then skip to the next step. If the default ComputeClass exists, then the output is similar to the following:

    apiVersion: cloud.google.com/v1
    kind: ComputeClass
    metadata:
      annotations:
        cloud.google.com/gke-managed-default-compute-class: "true"
      name: default
    # Multiple lines are omitted here.
    spec:
      autopilot:
        enabled: true
    # Multiple lines are omitted here.
    

    In this output, the cloud.google.com/gke-managed-default-compute-class: "true" annotation indicates that GKE created the ComputeClass.

  3. When you deploy a Pod that doesn't select a different ComputeClass, verify that the Pod runs on an Autopilot-managed node:

    kubectl get pods -o wide
    

    In the output, nodes that are managed by Autopilot have the gk3- prefix in the NODE column.

Modify or disable individual Autopilot settings

Because each Autopilot capability in a Standard cluster is modular, you can customize or disable individual settings at any time. If you're a security administrator, you can use custom organization policy constraints to enforce these settings. For more information, see Enforce Autopilot settings with custom organization policies.

Enforce Autopilot settings with custom organization policies

Because Autopilot settings in Standard clusters can be modified or disabled by users who have cluster update permissions, organization administrators can use custom organization policy constraints in Organization Policy Service to enforce these settings across an organization, folder, or project. The following example constraint denies cluster creation or update operations for clusters that don't enforce all of the Autopilot settings:

name: organizations/ORGANIZATION_ID/customConstraints/custom.enforceAutopilotSettings
resourceTypes:
- container.googleapis.com/Cluster
methodTypes:
- CREATE
condition: >-
  "resource.ClusterPolicyConfig.no_system_mutation != true ||
  resource.ClusterPolicyConfig.no_system_impersonation != true ||
  resource.ClusterPolicyConfig.no_unsafe_webhooks != true ||
  resource.ClusterPolicyConfig.no_standard_node_pools != true ||
  resource.ScaleToZeroConfig.enabled != true ||
  !(resource.DefaultComputeClassConfig.computeClass in [
    'autopilot', 'autopilot-spot', 'autopilot-arm', 'autopilot-arm-spot'
  ])"
actionType: DENY
displayName: Enforce Autopilot cluster policies in Standard clusters
description: All new clusters must use all Autopilot settings by default.

Replace ORGANIZATION_ID with your organization ID.

What's next