Deploy a self-hosted cluster autoscaler

This document shows you how to disable the Google Kubernetes Engine (GKE)-managed cluster autoscaler and deploy a self-hosted cluster autoscaler in your Standard cluster by using Helm. Deploying a self-hosted cluster autoscaler lets you customize autoscaling flags, test experimental parameters, or run custom builds from the open source GKE cluster autoscaler repository.

This document is intended for platform administrators who manage cluster scaling and infrastructure and require fine-grained control over autoscaler parameters beyond what the GKE-managed cluster autoscaler provides.

Requirements and limitations

  • A self-hosted cluster autoscaler is supported only on GKE Standard clusters that run version 1.37.2-gke.1042000 or later.
  • A self-hosted cluster autoscaler isn't compatible with Autopilot clusters, Standard clusters that contain Autopilot-managed node pools, or Standard clusters that use Autopilot mode for ComputeClasses.
  • Your cluster must have at least one running worker node at all times to host the self-hosted cluster autoscaler Pod. A cluster that runs a self-hosted cluster autoscaler can't scale down to zero total nodes.
  • You are responsible for deploying, updating, and monitoring the self-hosted cluster autoscaler workload. Cloud Customer Care doesn't troubleshoot self-hosted or custom-built cluster autoscaler deployments. To receive support from Customer Care for autoscaling issues, switch back to the GKE-managed cluster autoscaler.
  • If GKE-managed autoscaling is disabled, GKE doesn't emit control plane cluster autoscaler visibility events to the container.googleapis.com/cluster-autoscaler-visibility log stream. Instead, view scaling logs and metrics directly from the self-hosted cluster autoscaler Pod.

Before you begin

Before you start, make sure that you have performed the following tasks:

  • Enable the Google Kubernetes Engine API.
  • Enable Google Kubernetes Engine API
  • To use the Google Cloud CLI for this task, install and then initialize the gcloud CLI. If you previously installed the gcloud CLI, get the latest version by running the gcloud components update command. Earlier gcloud CLI versions might not support running the commands in this document.

Required roles

To get the permissions that you need to disable managed autoscaling, configure Identity and Access Management (IAM) resources, and deploy the self-hosted cluster autoscaler, ask your administrator to grant you the following IAM roles on your project:

For more information about granting roles, see Manage access to projects, folders, and organizations.

You might also be able to get the required permissions through custom roles or other predefined roles.

Disable the GKE-managed cluster autoscaler

Before you deploy a self-hosted cluster autoscaler, you must disable the GKE-managed cluster autoscaler in the control plane to prevent race conditions between the two autoscaler instances. Disabling the GKE-managed cluster autoscaler stops the control plane from scaling nodes. However, GKE preserves your existing node pool autoscaling configurations and ComputeClasses so that the self-hosted cluster autoscaler can act on them.

To disable the GKE-managed cluster autoscaler, follow these steps, complete one of the following steps:

  • To disable the GKE-managed cluster autoscaler on an existing Standard cluster, run the following command:

    gcloud beta container clusters update CLUSTER_NAME \
        --location=CONTROL_PLANE_LOCATION \
        --disable-managed-autoscaling
    

    If Workload Identity Federation for GKE isn't already enabled on the cluster and on at least one node pool to host the self-hosted cluster autoscaler Pod, run the following commands:

    gcloud container clusters update CLUSTER_NAME \
        --location=CONTROL_PLANE_LOCATION \
        --workload-pool=PROJECT_ID.svc.id.goog
    gcloud container node-pools update NODEPOOL_NAME \
        --cluster=CLUSTER_NAME \
        --location=CONTROL_PLANE_LOCATION \
        --workload-metadata=GKE_METADATA
        ```
    
  • To create a new Standard cluster with the GKE-managed cluster autoscaler already disabled, run the following command:

    gcloud beta container clusters create CLUSTER_NAME \
        --location=CONTROL_PLANE_LOCATION \
        --cluster-version=CLUSTER_VERSION \
        --workload-pool=PROJECT_ID.svc.id.goog \
        --disable-managed-autoscaling
    

Replace the following:

  • CLUSTER_NAME: the name of your Standard cluster.
  • CONTROL_PLANE_LOCATION: the Compute Engine region or zone of the control plane of your cluster. Provide a region for regional clusters, or a zone for zonal clusters.
  • PROJECT_ID: your Google Cloud project ID.
  • NODEPOOL_NAME: the name of an existing node pool to host the self-hosted cluster autoscaler Pod.
  • CLUSTER_VERSION: the GKE version for the new cluster, which must be 1.37.2-gke.1042000 or later.

To verify that the GKE-managed cluster autoscaler is disabled on your cluster, run the following command:

gcloud beta container clusters describe CLUSTER_NAME \
    --location=CONTROL_PLANE_LOCATION \
    --format="value(autoscaling.managedAutoscalingDisabled)"

The output is True.

Configure IAM permissions for the self-hosted cluster autoscaler

The self-hosted cluster autoscaler requires IAM permissions to manage Compute Engine instance groups and read cluster metadata in your project.

Grant the autoscaler these permissions by creating a least-privilege custom IAM role and binding it to the autoscaler's Kubernetes ServiceAccount with Workload Identity Federation for GKE:

  1. Retrieve your project number:

    gcloud projects describe PROJECT_ID \
        --format="value(projectNumber)"
    

    Record the output value as PROJECT_NUMBER to use in later steps.

  2. Save the following custom IAM role definition as a gcp-role.yaml file:

    title: Self Hosted Cluster Autoscaler Role
    description: Least privilege role for self-hosted CA to manage GKE clusters
    stage: BETA
    includedPermissions:
    - compute.acceleratorTypes.list
    - compute.instanceGroupManagers.get
    - compute.instanceGroupManagers.list
    - compute.instanceGroupManagers.update
    - compute.instanceTemplates.get
    - compute.instances.delete
    - compute.instances.get
    - compute.instances.list
    - compute.instances.setMetadata
    - compute.machineTypes.get
    - compute.machineTypes.list
    - compute.regions.get
    - compute.regions.list
    - compute.reservations.list
    - compute.resourcePolicies.list
    - compute.zoneOperations.get
    - compute.zones.get
    - compute.zones.list
    - container.clusters.get
    - container.clusters.update
    - container.operations.get
  3. Create the selfHostedClusterAutoscaler custom IAM role in your project:

    gcloud iam roles create selfHostedClusterAutoscaler \
        --project=PROJECT_ID \
        --file=gcp-role.yaml
    
  4. Create an IAM service account named selfhosted-autoscaler:

    gcloud iam service-accounts create selfhosted-autoscaler \
        --project=PROJECT_ID
    
  5. Grant the custom IAM role to the selfhosted-autoscaler IAM service account:

    gcloud projects add-iam-policy-binding PROJECT_ID \
        --member="serviceAccount:selfhosted-autoscaler@PROJECT_ID.iam.gserviceaccount.com" \
        --role="projects/PROJECT_ID/roles/selfHostedClusterAutoscaler"
    
  6. If your cluster uses node auto-provisioning or custom ComputeClasses to create new node pools, grant the Service Account User role (roles/iam.serviceAccountUser) to the selfhosted-autoscaler service account. Run the following command for each IAM service account that your new nodes use:

    gcloud iam service-accounts add-iam-policy-binding NODE_SA_EMAIL \
        --project=PROJECT_ID \
        --role="roles/iam.serviceAccountUser" \
        --member="serviceAccount:selfhosted-autoscaler@PROJECT_ID.iam.gserviceaccount.com"
    

    Replace NODE_SA_EMAIL with the email address of the IAM service account that your nodes use. Grant this role on the node auto-provisioning default service account and on any service account that you specify in a ComputeClass. Unless you configure a custom service account, node auto-provisioning uses the Compute Engine default service account, PROJECT_NUMBER-compute@developer.gserviceaccount.com.

  7. Allow the selfhosted-autoscaler Kubernetes ServiceAccount in the kube-system namespace impersonate the IAM service account:

    gcloud iam service-accounts add-iam-policy-binding selfhosted-autoscaler@PROJECT_ID.iam.gserviceaccount.com \
        --project=PROJECT_ID \
        --role="roles/iam.workloadIdentityUser" \
        --member="serviceAccount:PROJECT_ID.svc.id.goog[kube-system/selfhosted-autoscaler]"
    

Deploy the self-hosted cluster autoscaler by using Helm

Deploy the self-hosted cluster autoscaler by using the Helm chart in the charts/cluster-autoscaler directory of the open source GKE cluster autoscaler repository:

  1. Retrieve your cluster ID:

    gcloud container clusters describe CLUSTER_NAME \
        --location=CONTROL_PLANE_LOCATION \
        --project=PROJECT_ID \
        --format="value(id)"
    

    Record the output value as CLUSTER_ID for the Helm configuration file.

  2. Clone the GKE cluster autoscaler repository:

    git clone https://github.com/GoogleCloudPlatform/cluster-autoscaler.git
    
  3. Build and push the cluster autoscaler container image to your container registry:

    cd cluster-autoscaler
    make release \
        REGISTRY=REGISTRY_PATH \
        TAG=IMAGE_TAG
    

    Replace the following:

    • REGISTRY_PATH: the base path of your container registry, such as LOCATION-docker.pkg.dev/PROJECT_ID/REPOSITORY. The make release command builds and pushes the image to REGISTRY_PATH/cluster-autoscaler:IMAGE_TAG.
    • IMAGE_TAG: the tag for your container image, such as 37.234.0.

    For more information about building the container image, see the GKE cluster autoscaler repository.

  4. Navigate to the Helm chart directory:

    cd charts/cluster-autoscaler
    
  5. Create a my-values.yaml file to configure your cluster identity, Workload Identity Federation for GKE service account, and autoscaler parameters:

    image:
      repository: "REGISTRY_PATH/cluster-autoscaler"
      tag: "IMAGE_TAG"
    
    cluster:
      name: "CLUSTER_NAME"
      hash: "CLUSTER_ID"
      projectNumber: "PROJECT_NUMBER"
      location: "CONTROL_PLANE_LOCATION"
      regional: IS_REGIONAL
    
    nodeSelector:
      iam.gke.io/gke-metadata-server-enabled: "true"
    
    serviceAccount:
      name: "selfhosted-autoscaler"
      gcpWorkloadIdentity: "selfhosted-autoscaler@PROJECT_ID.iam.gserviceaccount.com"
    
    config:
      autoscalingProfile: AUTOSCALING_PROFILE
    

    Replace the following:

    • REGISTRY_PATH: the base path of your container registry from an earlier step, such as LOCATION-docker.pkg.dev/PROJECT_ID/REPOSITORY.
    • IMAGE_TAG: the container image tag that you pushed in the previous step, such as 37.234.0. If you omit the image.tag field, then the Helm chart defaults to the appVersion value that you set when packaging the chart.
    • CLUSTER_ID: the cluster ID that you retrieved in an earlier step.
    • PROJECT_NUMBER: the project number that you retrieved in an earlier step.
    • IS_REGIONAL: true if your cluster is a regional cluster, or false if your cluster is a zonal cluster.
    • AUTOSCALING_PROFILE: the autoscaling profile to apply. Specify the default profile for balanced scaling or the optimizeUtilization profile to scale down underutilized nodes more aggressively. For a complete list of configurable flags and parameters, see the default values.yaml file in GitHub.
  6. Package the Helm chart with the cluster autoscaler component version that you want to run, and install the chart in the kube-system namespace:

    helm package \
        --version=CA_VERSION \
        --app-version=CA_VERSION .
    helm upgrade --install selfhosted-ca \
        cluster-autoscaler-CA_VERSION.tgz \
        --namespace=kube-system \
        -f my-values.yaml
    

    Replace CA_VERSION with the cluster autoscaler component version in the <minor>.<ca-version>.<patch> format, such as 37.234.0. The chart uses this version to determine which autoscaler experiments to enable from the values.yaml file.

Verify the self-hosted cluster autoscaler deployment

After you install the Helm chart, verify that the self-hosted cluster autoscaler Pod is running and actively monitoring your cluster:

  1. Verify that the self-hosted cluster autoscaler Pod is in the Running state:

    kubectl get pods -n kube-system \
        -l app.kubernetes.io/instance=selfhosted-ca
    

    The output is similar to the following:

    NAME                                                READY   STATUS    RESTARTS   AGE
    selfhosted-ca-cluster-autoscaler-6d4b7c9f8b-x9v2k   1/1     Running   0          2m
    
  2. Inspect the logs from the self-hosted cluster autoscaler Pod to confirm that it starts without authentication errors:

    kubectl logs -n kube-system \
        -l app.kubernetes.io/instance=selfhosted-ca \
        --tail=50
    
  3. Check the cluster-autoscaler-status ConfigMap to view the current scaling status that the autoscaler reports:

    kubectl get configmap cluster-autoscaler-status \
        -n kube-system -o yaml
    

Switch back to the GKE-managed cluster autoscaler

To return to the GKE-managed cluster autoscaler, remove your self-hosted deployment before re-enabling managed autoscaling in the control plane:

  1. Uninstall the self-hosted cluster autoscaler Helm release:

    helm uninstall selfhosted-ca --namespace=kube-system
    
  2. Re-enable the GKE-managed cluster autoscaler on your cluster:

    gcloud beta container clusters update CLUSTER_NAME \
        --location=CONTROL_PLANE_LOCATION \
        --enable-managed-autoscaling
    
  3. Verify that the GKE-managed cluster autoscaler is enabled:

    gcloud beta container clusters describe CLUSTER_NAME \
        --location=CONTROL_PLANE_LOCATION \
        --format="value(autoscaling.managedAutoscalingDisabled)"
    

    The output is empty or False.

What's next