You can configure Google Kubernetes Engine (GKE) Standard clusters to function similarly to Autopilot mode clusters by enabling cluster settings like the default ComputeClass, Autopilot cluster security policies, and cluster scale-to-zero.
This document is intended for platform administrators and security engineers who want to get all of the benefits of Autopilot mode in a Standard cluster with the flexibility to modify or disable individual settings. If you don't require this flexibility, then create a dedicated Autopilot cluster instead.
You should already be familiar with the following concepts:
Autopilot cluster settings
To configure a Standard cluster to function like an Autopilot cluster, you configure all of the following settings:
- Autopilot ComputeClass as cluster default: set one of the built-in Autopilot ComputeClasses. Any Pods that don't select a ComputeClass run by using the specified default ComputeClass. The built-in ComputeClasses all use Autopilot mode and the Pod-based billing model.
- Autoscale the cluster to zero nodes: configure the cluster autoscaler to scale the cluster to zero nodes when there are no active user workloads in the cluster. You can optionally adjust the delay period before the autoscaler scales the cluster to zero nodes.
- Zero initial nodes: during cluster creation, prevent the automatic creation of a default node pool. For existing clusters, delete any manually created node pools.
- Autopilot cluster security policies: enforce all of the Autopilot security policies. These policies help to protect system workloads and identities.
You can modify your cluster to change any of these settings independently of the other settings. For example, if you need to manually create a node pool, then you can disable the corresponding Autopilot cluster security policy without changing any of the other settings.
Flexibility and security considerations
When you create an Autopilot cluster, the cluster mode is permanent. Cluster settings like scaling to zero nodes, preventing the manual creation of node pools, and the various security constraints are immutable. However, when you configure a Standard cluster to use all of the settings of an Autopilot cluster, the the settings are mutable. Anyone who has access to modify cluster configurations can change or disable individual settings.
To enforce the use of these settings across an organization, folder, or project, organization administrators can use a custom organization policy constraint. For more information, see the Enforce Autopilot settings with custom organization policies section of this document.
Limitations
If you configure Confidential GKE Nodes at the cluster level, then GKE ignores your command to use an Autopilot ComputeClass to as the cluster default.
Before you begin
Before you start, make sure that you have performed the following tasks:
- Enable the Google Kubernetes Engine API. Enable Google Kubernetes Engine API
- To use the Google Cloud CLI for this task,
install and then
initialize the
gcloud CLI. If you previously installed the gcloud CLI, get the latest
version by running the
gcloud components updatecommand. Earlier gcloud CLI versions might not support running the commands in this document.
- Ensure that you have a Standard cluster that meets all of the requirements.
Requirements
To use all of the Autopilot capabilities in a Standard cluster, your cluster must meet the following requirements:
The cluster must run GKE version 1.37.1-gke.3020000 or later. The version requirements for individual settings are as follows:
- To set a built-in Autopilot ComputeClass as the cluster default, the cluster must run GKE version 1.37.1-gke.1380000 or later.
- To create a cluster that has zero initial nodes, the cluster must run GKE version 1.37.1-gke.3020000 or later.
- To configure the cluster autoscaler to scale the cluster to zero nodes, the cluster must run GKE version 1.37.1-gke.3020000 or later.
To upgrade your cluster to a compatible version, see Manually upgrade a cluster's control plane or node pools.
The cluster must meet all of the requirements for using Autopilot ComputeClasses. To check whether your cluster meets these requirements, see Run workloads in Autopilot mode in Standard clusters.
Required roles and permissions
To get the permissions that you need to create or update clusters to use Autopilot mode, ask your administrator to grant you the following IAM roles on your Google Cloud project:
-
Create and update clusters and node pools:
Kubernetes Engine Cluster Admin (
roles/container.clusterAdmin) -
Manage ComputeClasses and workloads in the cluster:
Kubernetes Engine Developer (
roles/container.developer)
For more information about granting roles, see Manage access to projects, folders, and organizations.
You might also be able to get the required permissions through custom roles or other predefined roles.
Create a Standard cluster that uses Autopilot mode
To create a new Standard cluster that uses all of the Autopilot settings, run the following command:
gcloud container clusters create CLUSTER_NAME \
--location=CONTROL_PLANE_LOCATION \
--num-nodes=0 \
--default-compute-class=BUILT_IN_COMPUTECLASS_NAME \
--enable-cluster-scale-to-zero \
--autopilot-cluster-policies=no-system-mutation,no-system-impersonation,no-unsafe-webhooks,no-standard-node-pools
Replace the following:
CLUSTER_NAME: a name for your new cluster.CONTROL_PLANE_LOCATION: a region for your cluster control plane, such asus-central1.BUILT_IN_COMPUTECLASS_NAME: the built-in ComputeClass to use as the cluster default, such asautopilot.
Optionally, you can adjust the delay period before the cluster autoscaler
scales the cluster to zero nodes by using the --cluster-scale-to-zero-delay
flag:
gcloud container clusters create CLUSTER_NAME \
--location=CONTROL_PLANE_LOCATION \
--num-nodes=0 \
--default-compute-class=BUILT_IN_COMPUTECLASS_NAME \
--enable-cluster-scale-to-zero \
--cluster-scale-to-zero-delay=SCALE_TO_ZERO_DELAY \
--autopilot-cluster-policies=no-system-mutation,no-system-impersonation,no-unsafe-webhooks,no-standard-node-pools
Replace SCALE_TO_ZERO_DELAY with the delay duration.
This value must specify an integer and a unit of time, such as 0s, 1m, or
3h. The maximum delay is 3 hours. If you don't specify the
--cluster-scale-to-zero-delay flag, the default delay is 3 hours.
If your cluster creation command fails and displays an error message, check the error message for more information. The most common errors are as follows:
- Your cluster uses a version that doesn't support one of the specified flags. To resolve this error, update your cluster to the minimum required version as specified in the Requirements section.
- You specified node pool configuration flags (such as the
--machine-typeflag) in your cluster creation command. Because your new cluster has no initial node pool, you can't specify node pool configuration flags. To resolve this error, remove the node pool configuration flags from your command. - You specified the name of a custom ComputeClass for the
--default-compute-classflag. To resolve this error, specify one of the built-in Autopilot ComputeClasses.
Update an existing Standard cluster to use Autopilot mode
You can modify an existing Standard cluster to use Autopilot mode. This process involves potentially disruptive changes to your workloads, such as deleting manually created node pools. Perform these steps during a scheduled maintenance period.
Verify that your cluster meets all of the Requirements in this document.
Review your cluster and make any required changes to avoid conflicts with Autopilot cluster policies. For more information about the potential changes, see Supported policies.
When you set an Autopilot ComputeClass as the cluster default, incoming Pods that don't select a different ComputeClass become Autopilot workloads. Check any workloads that don't select a ComputeClass for the following properties:
- Verify that your workloads comply with the Autopilot security constraints, or configure an allowlist for privileged workloads.
- Find and remove
cloud.google.com/gke-nodepoolnode selectors or node affinity rules from the Pods. - Check whether any namespaces have the
cloud.google.com/default-compute-classorcloud.google.com/default-compute-class-non-daemonsetlabel. Namespace-level default ComputeClasses take precedence over the cluster-level default ComputeClass.
Check whether the cluster has a custom ComputeClass that's named
default:kubectl get computeclass default -o yamlIf the output is blank, then skip to the next step. If the
defaultComputeClass exists, then the output is similar to the following:apiVersion: cloud.google.com/v1 kind: ComputeClass metadata: annotations: cloud.google.com/gke-managed-default-compute-class: "true" name: default # Multiple lines are omitted here.In the output, if the
cloud.google.com/gke-managed-default-compute-class: "true"annotation is in themetadata.annotationsfield, then skip to the next step. If this annotation doesn't exist, then delete the ComputeClass:kubectl delete computeclass defaultFind and delete any existing manually created node pools in the cluster.
Enable the Autopilot settings for the cluster:
Set the cluster default ComputeClass to a built-in ComputeClass:
gcloud container clusters update CLUSTER_NAME \ --location=CONTROL_PLANE_LOCATION \ --default-compute-class=BUILT_IN_COMPUTECLASS_NAMEReplace the following:
CLUSTER_NAME: the name of your existing Standard cluster.CONTROL_PLANE_LOCATION: the region of your cluster control plane.BUILT_IN_COMPUTECLASS_NAME: the built-in Autopilot ComputeClass to use as the cluster default, such asautopilot.
Enable cluster autoscaling and let the autoscaler scale the cluster to zero nodes:
gcloud container clusters update CLUSTER_NAME \ --location=CONTROL_PLANE_LOCATION \ --enable-autoscaling \ --enable-cluster-scale-to-zeroOptionally, you can adjust the delay period before the cluster autoscaler scales the cluster to zero nodes by using the
--cluster-scale-to-zero-delayflag:gcloud container clusters update CLUSTER_NAME \ --location=CONTROL_PLANE_LOCATION \ --enable-autoscaling \ --enable-cluster-scale-to-zero \ --cluster-scale-to-zero-delay=SCALE_TO_ZERO_DELAYReplace
SCALE_TO_ZERO_DELAYwith the delay duration. This value must specify an integer and a unit of time, such as0s,1m, or3h. The maximum delay is 3 hours. If you don't specify the--cluster-scale-to-zero-delayflag, then the default delay is 3 hours.Enable the Autopilot cluster policies:
gcloud container clusters update CLUSTER_NAME \ --location=CONTROL_PLANE_LOCATION \ --autopilot-cluster-policies=no-system-mutation,no-system-impersonation,no-unsafe-webhooks,no-standard-node-pools
If any of your cluster update commands fail and display an error message, check the error message for more information. The most common errors are as follows:
- Your cluster uses a version that doesn't support one of the specified flags. To resolve this error, upgrade your cluster to the minimum required version as specified in the Requirements section.
- You specified the name of a custom ComputeClass for the
--default-compute-classflag. To resolve this error, specify one of the built-in Autopilot ComputeClasses. - Your cluster has manually created node pools, so GKE can't
enforce the
no-standard-node-poolspolicy. To resolve this error, delete the manually created node pools.
Verify that your cluster uses Autopilot mode
After you configure your new or existing cluster, verify that the Autopilot settings are active:
Verify that the default ComputeClass, scale-to-zero, and cluster policy settings are enabled on the cluster:
gcloud container clusters describe CLUSTER_NAME \ --location=CONTROL_PLANE_LOCATION \ --format="yaml(autoscaling.defaultComputeClassConfig,autoscaling.scaleToZeroConfig,autopilot.clusterPolicyConfig)"The output is similar to the following:
autoscaling: defaultComputeClassConfig: computeClass: autopilot enabled: true scaleToZeroConfig: enabled: true delay: 3h autopilot: clusterPolicyConfig: noManualNodePools: true noSystemImpersonation: true noSystemMutation: true noUnsafeWebhooks: trueVerify that GKE created the managed
defaultComputeClass in your cluster:kubectl get computeclass default -o yamlThe output is similar to the following:
If the output is blank, then skip to the next step. If the
defaultComputeClass exists, then the output is similar to the following:apiVersion: cloud.google.com/v1 kind: ComputeClass metadata: annotations: cloud.google.com/gke-managed-default-compute-class: "true" name: default # Multiple lines are omitted here. spec: autopilot: enabled: true # Multiple lines are omitted here.In this output, the
cloud.google.com/gke-managed-default-compute-class: "true"annotation indicates that GKE created the ComputeClass.When you deploy a Pod that doesn't select a different ComputeClass, verify that the Pod runs on an Autopilot-managed node:
kubectl get pods -o wideIn the output, nodes that are managed by Autopilot have the
gk3-prefix in theNODEcolumn.
Modify or disable individual Autopilot settings
Because each Autopilot capability in a Standard cluster is modular, you can customize or disable individual settings at any time. If you're a security administrator, you can use custom organization policy constraints to enforce these settings. For more information, see Enforce Autopilot settings with custom organization policies.
- Disable specific Autopilot policies in your cluster, such as
no-standard-node-pools. - Disable the cluster-level default ComputeClass. Alternatively, you can
set a custom ComputeClass as the cluster default by creating a ComputeClass
that's named
default. Prevent the cluster from autoscaling to zero nodes:
gcloud container clusters update CLUSTER_NAME \ --location=CONTROL_PLANE_LOCATION \ --no-enable-cluster-scale-to-zero
Enforce Autopilot settings with custom organization policies
Because Autopilot settings in Standard clusters can be modified or disabled by users who have cluster update permissions, organization administrators can use custom organization policy constraints in Organization Policy Service to enforce these settings across an organization, folder, or project. The following example constraint denies cluster creation or update operations for clusters that don't enforce all of the Autopilot settings:
name: organizations/ORGANIZATION_ID/customConstraints/custom.enforceAutopilotSettings
resourceTypes:
- container.googleapis.com/Cluster
methodTypes:
- CREATE
condition: >-
"resource.ClusterPolicyConfig.no_system_mutation != true ||
resource.ClusterPolicyConfig.no_system_impersonation != true ||
resource.ClusterPolicyConfig.no_unsafe_webhooks != true ||
resource.ClusterPolicyConfig.no_standard_node_pools != true ||
resource.ScaleToZeroConfig.enabled != true ||
!(resource.DefaultComputeClassConfig.computeClass in [
'autopilot', 'autopilot-spot', 'autopilot-arm', 'autopilot-arm-spot'
])"
actionType: DENY
displayName: Enforce Autopilot cluster policies in Standard clusters
description: All new clusters must use all Autopilot settings by default.
Replace ORGANIZATION_ID with your organization ID.
What's next
- Learn more about Autopilot workloads in GKE Standard
- Run workloads in Autopilot mode in Standard clusters
- Apply ComputeClasses to Pods by default
- Enforce Autopilot security policies in Standard clusters
- Autoscale a cluster