This document shows you how to disable the Google Kubernetes Engine (GKE)-managed cluster autoscaler and deploy a self-hosted cluster autoscaler in your Standard cluster by using Helm. Deploying a self-hosted cluster autoscaler lets you customize autoscaling flags, test experimental parameters, or run custom builds from the open source GKE cluster autoscaler repository.
This document is intended for platform administrators who manage cluster scaling and infrastructure and require fine-grained control over autoscaler parameters beyond what the GKE-managed cluster autoscaler provides.
Requirements and limitations
- A self-hosted cluster autoscaler is supported only on GKE Standard clusters that run version 1.37.2-gke.1042000 or later.
- A self-hosted cluster autoscaler isn't compatible with Autopilot clusters, Standard clusters that contain Autopilot-managed node pools, or Standard clusters that use Autopilot mode for ComputeClasses.
- Your cluster must have at least one running worker node at all times to host the self-hosted cluster autoscaler Pod. A cluster that runs a self-hosted cluster autoscaler can't scale down to zero total nodes.
- You are responsible for deploying, updating, and monitoring the self-hosted cluster autoscaler workload. Cloud Customer Care doesn't troubleshoot self-hosted or custom-built cluster autoscaler deployments. To receive support from Customer Care for autoscaling issues, switch back to the GKE-managed cluster autoscaler.
- If GKE-managed autoscaling is disabled,
GKE doesn't emit control plane
cluster autoscaler visibility events
to the
container.googleapis.com/cluster-autoscaler-visibilitylog stream. Instead, view scaling logs and metrics directly from the self-hosted cluster autoscaler Pod.
Before you begin
Before you start, make sure that you have performed the following tasks:
- Enable the Google Kubernetes Engine API. Enable Google Kubernetes Engine API
- To use the Google Cloud CLI for this task,
install and then
initialize the
gcloud CLI. If you previously installed the gcloud CLI, get the latest
version by running the
gcloud components updatecommand. Earlier gcloud CLI versions might not support running the commands in this document.
- Verify that you have an existing Standard cluster that runs version 1.37.2-gke.1042000 or later, with Workload Identity Federation for GKE enabled on the cluster and on at least one node pool. You can also create a new cluster in the next section.
- Install Helm version 4.0.0 or later, Git, Go, and Docker.
- Verify that you have a Docker repository in Artifact Registry or another container registry that your cluster can pull from to store your self-hosted cluster autoscaler container image. For more information, see Store Docker container images in Artifact Registry.
- Configure
kubectlto communicate with your cluster. For more information, see Installkubectland configure cluster access.
Required roles
To get the permissions that you need to disable managed autoscaling, configure Identity and Access Management (IAM) resources, and deploy the self-hosted cluster autoscaler, ask your administrator to grant you the following IAM roles on your project:
-
Manage GKE clusters:
Kubernetes Engine Admin (
roles/container.admin) -
Create custom IAM roles:
Role Administrator (
roles/iam.roleAdmin) -
Create and manage IAM service accounts:
Service Account Admin (
roles/iam.serviceAccountAdmin) -
Grant IAM roles on the project:
Project IAM Admin (
roles/resourcemanager.projectIamAdmin)
For more information about granting roles, see Manage access to projects, folders, and organizations.
You might also be able to get the required permissions through custom roles or other predefined roles.
Disable the GKE-managed cluster autoscaler
Before you deploy a self-hosted cluster autoscaler, you must disable the GKE-managed cluster autoscaler in the control plane to prevent race conditions between the two autoscaler instances. Disabling the GKE-managed cluster autoscaler stops the control plane from scaling nodes. However, GKE preserves your existing node pool autoscaling configurations and ComputeClasses so that the self-hosted cluster autoscaler can act on them.
To disable the GKE-managed cluster autoscaler, follow these steps, complete one of the following steps:
To disable the GKE-managed cluster autoscaler on an existing Standard cluster, run the following command:
gcloud beta container clusters update CLUSTER_NAME \ --location=CONTROL_PLANE_LOCATION \ --disable-managed-autoscalingIf Workload Identity Federation for GKE isn't already enabled on the cluster and on at least one node pool to host the self-hosted cluster autoscaler Pod, run the following commands:
gcloud container clusters update CLUSTER_NAME \ --location=CONTROL_PLANE_LOCATION \ --workload-pool=PROJECT_ID.svc.id.goog gcloud container node-pools update NODEPOOL_NAME \ --cluster=CLUSTER_NAME \ --location=CONTROL_PLANE_LOCATION \ --workload-metadata=GKE_METADATA ```
To create a new Standard cluster with the GKE-managed cluster autoscaler already disabled, run the following command:
gcloud beta container clusters create CLUSTER_NAME \ --location=CONTROL_PLANE_LOCATION \ --cluster-version=CLUSTER_VERSION \ --workload-pool=PROJECT_ID.svc.id.goog \ --disable-managed-autoscaling
Replace the following:
CLUSTER_NAME: the name of your Standard cluster.CONTROL_PLANE_LOCATION: the Compute Engine region or zone of the control plane of your cluster. Provide a region for regional clusters, or a zone for zonal clusters.PROJECT_ID: your Google Cloud project ID.NODEPOOL_NAME: the name of an existing node pool to host the self-hosted cluster autoscaler Pod.CLUSTER_VERSION: the GKE version for the new cluster, which must be 1.37.2-gke.1042000 or later.
To verify that the GKE-managed cluster autoscaler is disabled on your cluster, run the following command:
gcloud beta container clusters describe CLUSTER_NAME \
--location=CONTROL_PLANE_LOCATION \
--format="value(autoscaling.managedAutoscalingDisabled)"
The output is True.
Configure IAM permissions for the self-hosted cluster autoscaler
The self-hosted cluster autoscaler requires IAM permissions to manage Compute Engine instance groups and read cluster metadata in your project.
Grant the autoscaler these permissions by creating a least-privilege custom IAM role and binding it to the autoscaler's Kubernetes ServiceAccount with Workload Identity Federation for GKE:
Retrieve your project number:
gcloud projects describe PROJECT_ID \ --format="value(projectNumber)"Record the output value as
PROJECT_NUMBERto use in later steps.Save the following custom IAM role definition as a
gcp-role.yamlfile:Create the
selfHostedClusterAutoscalercustom IAM role in your project:gcloud iam roles create selfHostedClusterAutoscaler \ --project=PROJECT_ID \ --file=gcp-role.yamlCreate an IAM service account named
selfhosted-autoscaler:gcloud iam service-accounts create selfhosted-autoscaler \ --project=PROJECT_IDGrant the custom IAM role to the
selfhosted-autoscalerIAM service account:gcloud projects add-iam-policy-binding PROJECT_ID \ --member="serviceAccount:selfhosted-autoscaler@PROJECT_ID.iam.gserviceaccount.com" \ --role="projects/PROJECT_ID/roles/selfHostedClusterAutoscaler"If your cluster uses node auto-provisioning or custom ComputeClasses to create new node pools, grant the Service Account User role (
roles/iam.serviceAccountUser) to theselfhosted-autoscalerservice account. Run the following command for each IAM service account that your new nodes use:gcloud iam service-accounts add-iam-policy-binding NODE_SA_EMAIL \ --project=PROJECT_ID \ --role="roles/iam.serviceAccountUser" \ --member="serviceAccount:selfhosted-autoscaler@PROJECT_ID.iam.gserviceaccount.com"Replace
NODE_SA_EMAILwith the email address of the IAM service account that your nodes use. Grant this role on the node auto-provisioning default service account and on any service account that you specify in a ComputeClass. Unless you configure a custom service account, node auto-provisioning uses the Compute Engine default service account,PROJECT_NUMBER-compute@developer.gserviceaccount.com.Allow the
selfhosted-autoscalerKubernetes ServiceAccount in thekube-systemnamespace impersonate the IAM service account:gcloud iam service-accounts add-iam-policy-binding selfhosted-autoscaler@PROJECT_ID.iam.gserviceaccount.com \ --project=PROJECT_ID \ --role="roles/iam.workloadIdentityUser" \ --member="serviceAccount:PROJECT_ID.svc.id.goog[kube-system/selfhosted-autoscaler]"
Deploy the self-hosted cluster autoscaler by using Helm
Deploy the self-hosted cluster autoscaler by using the Helm chart in the
charts/cluster-autoscaler directory
of the open source GKE cluster autoscaler repository:
Retrieve your cluster ID:
gcloud container clusters describe CLUSTER_NAME \ --location=CONTROL_PLANE_LOCATION \ --project=PROJECT_ID \ --format="value(id)"Record the output value as
CLUSTER_IDfor the Helm configuration file.Clone the GKE cluster autoscaler repository:
git clone https://github.com/GoogleCloudPlatform/cluster-autoscaler.gitBuild and push the cluster autoscaler container image to your container registry:
cd cluster-autoscaler make release \ REGISTRY=REGISTRY_PATH \ TAG=IMAGE_TAGReplace the following:
REGISTRY_PATH: the base path of your container registry, such asLOCATION-docker.pkg.dev/PROJECT_ID/REPOSITORY. Themake releasecommand builds and pushes the image toREGISTRY_PATH/cluster-autoscaler:IMAGE_TAG.IMAGE_TAG: the tag for your container image, such as37.234.0.
For more information about building the container image, see the GKE cluster autoscaler repository.
Navigate to the Helm chart directory:
cd charts/cluster-autoscalerCreate a
my-values.yamlfile to configure your cluster identity, Workload Identity Federation for GKE service account, and autoscaler parameters:image: repository: "REGISTRY_PATH/cluster-autoscaler" tag: "IMAGE_TAG" cluster: name: "CLUSTER_NAME" hash: "CLUSTER_ID" projectNumber: "PROJECT_NUMBER" location: "CONTROL_PLANE_LOCATION" regional: IS_REGIONAL nodeSelector: iam.gke.io/gke-metadata-server-enabled: "true" serviceAccount: name: "selfhosted-autoscaler" gcpWorkloadIdentity: "selfhosted-autoscaler@PROJECT_ID.iam.gserviceaccount.com" config: autoscalingProfile: AUTOSCALING_PROFILEReplace the following:
REGISTRY_PATH: the base path of your container registry from an earlier step, such asLOCATION-docker.pkg.dev/PROJECT_ID/REPOSITORY.IMAGE_TAG: the container image tag that you pushed in the previous step, such as37.234.0. If you omit theimage.tagfield, then the Helm chart defaults to theappVersionvalue that you set when packaging the chart.CLUSTER_ID: the cluster ID that you retrieved in an earlier step.PROJECT_NUMBER: the project number that you retrieved in an earlier step.IS_REGIONAL:trueif your cluster is a regional cluster, orfalseif your cluster is a zonal cluster.AUTOSCALING_PROFILE: the autoscaling profile to apply. Specify thedefaultprofile for balanced scaling or theoptimizeUtilizationprofile to scale down underutilized nodes more aggressively. For a complete list of configurable flags and parameters, see the defaultvalues.yamlfile in GitHub.
Package the Helm chart with the cluster autoscaler component version that you want to run, and install the chart in the
kube-systemnamespace:helm package \ --version=CA_VERSION \ --app-version=CA_VERSION . helm upgrade --install selfhosted-ca \ cluster-autoscaler-CA_VERSION.tgz \ --namespace=kube-system \ -f my-values.yamlReplace
CA_VERSIONwith the cluster autoscaler component version in the<minor>.<ca-version>.<patch>format, such as37.234.0. The chart uses this version to determine which autoscaler experiments to enable from thevalues.yamlfile.
Verify the self-hosted cluster autoscaler deployment
After you install the Helm chart, verify that the self-hosted cluster autoscaler Pod is running and actively monitoring your cluster:
Verify that the self-hosted cluster autoscaler Pod is in the
Runningstate:kubectl get pods -n kube-system \ -l app.kubernetes.io/instance=selfhosted-caThe output is similar to the following:
NAME READY STATUS RESTARTS AGE selfhosted-ca-cluster-autoscaler-6d4b7c9f8b-x9v2k 1/1 Running 0 2mInspect the logs from the self-hosted cluster autoscaler Pod to confirm that it starts without authentication errors:
kubectl logs -n kube-system \ -l app.kubernetes.io/instance=selfhosted-ca \ --tail=50Check the
cluster-autoscaler-statusConfigMap to view the current scaling status that the autoscaler reports:kubectl get configmap cluster-autoscaler-status \ -n kube-system -o yaml
Switch back to the GKE-managed cluster autoscaler
To return to the GKE-managed cluster autoscaler, remove your self-hosted deployment before re-enabling managed autoscaling in the control plane:
Uninstall the self-hosted cluster autoscaler Helm release:
helm uninstall selfhosted-ca --namespace=kube-systemRe-enable the GKE-managed cluster autoscaler on your cluster:
gcloud beta container clusters update CLUSTER_NAME \ --location=CONTROL_PLANE_LOCATION \ --enable-managed-autoscalingVerify that the GKE-managed cluster autoscaler is enabled:
gcloud beta container clusters describe CLUSTER_NAME \ --location=CONTROL_PLANE_LOCATION \ --format="value(autoscaling.managedAutoscalingDisabled)"The output is empty or
False.
What's next
- Learn more about how the cluster autoscaler works.
- Learn how to configure cluster autoscaling for node pools.
- Explore the open source GKE cluster autoscaler repository and Helm chart.