Customize containerd configuration in GKE nodes

Autopilot Standard

This page shows you how to customize the configuration of the containerd container runtime on your Google Kubernetes Engine (GKE) nodes. Before reading this document, ensure that you're familiar with what a container runtime is and why you'd want to customize it.

About containerd configuration in GKE

You can manually configure a set of options in the containerd runtime on GKE nodes that run an operating system like Container-Optimized OS. Customizing the runtime lets you configure special requirements like access to private image registries. To set these options, you create a YAML file called a runtime configuration file and pass the file to GKE when you create or update a cluster.

This method of customizing containerd lets you avoid deploying privileged DaemonSets, which are a security risk. When you provide GKE with a runtime configuration file, GKE recreates your nodes and updates the containerd config.toml file on every node with your configuration. The configuration persists through node termination, upgrades, and recreations.

The runtime configuration file only lets you configure options in containerd. GKE also supports configuring specific kubelet options and low-level Linux kernel options by using a separate file called a node system configuration file. For more details, see Customizing node system configuration.

Limitations

You can't use a runtime configuration file to change containerd settings in Windows node images.

Available containerd configuration options

The following sections describe the options that you can configure using a runtime configuration file.

Note: When setting containerd registries, we recommend using registryHosts configuration, which covers more registry fields, rather than privateRegistryAccessConfig. And registryHosts overrides any matching FQDNs in privateRegistryAccessConfig.

privateRegistryAccessConfig

Access private image registries with private credentials that you store in Secret Manager.

This option is available with GKE versions 1.27.3-gke.1700 or later for Container-Optimized OS node images, and 1.33 or later for Ubuntu node images.

For instructions, see Access private registries with private CA certificates.

privateRegistryAccessConfig:
  enabled: true
  certificateAuthorityDomainConfig:
  - gcpSecretManagerCertificateConfig:
    secretURI: "SECRET_LOCATION"
  fqdns:
  - "FQDN1"
  - "FQDN2"

This configuration has the following fields:

• enabled: a boolean to enable private registry configuration. If you set enabled: false, delete any other fields in the privateRegistryAccessConfig item.
• certificateAuthorityDomainConfig: contains up to five certificate and FQDN definitions.
• gcpSecretManagerCertificateConfig: contains a certificate stored in Secret Manager and an array of FQDNs.
• secretURI: the location of the certificate in Secret Manager. privateRegistryAccessConfig supports global secrets only in secretURI.
• fqdns: a list of fully-qualified domain names of private registries. You can also use IPv4 addresses, but we recommend using the FQDN.

registryHosts

Configure advanced settings for containerd registries, such as capabilities, custom headers, and certificates. This corresponds to containerd's hosts.toml.

This option is available for GKE versions 1.34.1-gke.2980000 or later.

registryHosts:
- server: "REGISTRY_SERVER_FQDN"
  hosts:
  - host: "MIRROR_FQDN"
    capabilities:
    - "HOST_CAPABILITY_PULL"
    - "HOST_CAPABILITY_RESOLVE"
    - "HOST_CAPABILITY_PUSH"
    overridePath: false
    dialTimeout: "30s"
    header:
    - key: "HEADER_KEY"
      value:
      - "HEADER_VALUE_1"
      - "HEADER_VALUE_2"
    ca:
    - gcpSecretManagerSecretUri: "projects/PROJECT_ID_OR_NUMBER/secrets/CA_SECRET/versions/VERSION"
    client:
    - cert:
        gcpSecretManagerSecretUri: "projects/PROJECT_ID_OR_NUMBER/secrets/CLIENT_CERT_SECRET/versions/VERSION"
      key:
        gcpSecretManagerSecretUri: "projects/PROJECT_ID_OR_NUMBER/secrets/CLIENT_KEY_SECRET/versions/VERSION"

This configuration has the following fields:

• server: The hostname of the registry server (for example, example.com). This is used to name the configuration file on the node. This should be fully-qualified domain name or IPv4 address.
• hosts: A list of host-specific configurations for the server.
• host: The hostname of a registry mirror (for example, mirror.example.com). This should be fully-qualified domain names or IPv4 addresses.
• capabilities: A list of capabilities specifying what operations a host is capable of performing. Supported values are any of the following:
  • HOST_CAPABILITY_PULL: represents the capability to fetch manifests and blobs by digest.
  • HOST_CAPABILITY_RESOLVE: represents the capability to fetch manifests by name.
  • HOST_CAPABILITY_PUSH: represents the capability to push blobs and manifests.
  If unspecified, all capabilities are enabled.
• overridePath: A boolean. If true, indicates that the host's API root endpoint is defined in the URL path rather than by the API specification. This can be used with non-compliant OCI registries which are missing the /v2 prefix. Defaults to false.
• dialTimeout: A duration string (for example, "30s") specifying the maximum duration allowed for a connection attempt to complete. Maximum allowed value is "180s". If not set, containerd sets a default of "30s". The value should be a decimal number of seconds with an s suffix.
• header: A list of custom HTTP headers to send to the registry host.
  • key: The header key.
  • value: A list of header values.
• ca: A list of certificate configurations for the registry host's Certificate Authority (CA). To create secret for certificates and configure required permissions, see instructions in Store your CA public keys in Secret Manager.
  • gcpSecretManagerSecretUri: The URI of a secret in Secret Manager containing the CA certificate. It supports both global secrets and regional secrets. The formats are the following, for the respective secret types:
    • Global secret format: projects/PROJECT_ID_OR_NUMBER/secrets/SECRET_NAME/versions/VERSION.
    • Regional secret format: projects/PROJECT_ID_OR_NUMBER/locations/REGION/secrets/SECRET_NAME/versions/VERSION.
• client: A list of client certificate and key pairs for authenticating to the registry host. To create secret for certificates and configure required permissions, see instructions in Store your CA public keys in Secret Manager.
  • cert: The client certificate configuration.
  • gcpSecretManagerSecretUri: The URI of a secret in Secret Manager containing the client certificate. It supports both global secrets and regional secrets.
  • key: The client private key configuration.
  • gcpSecretManagerSecretUri: The URI of a secret in Secret Manager containing the client key. It supports both global secrets and regional secrets.

writableCgroups

Mount the /sys/fs/cgroup file system in read-write mode so that workloads can manage the resources of their child processes.

This option is available for GKE versions 1.34.1-gke.2541000 or later.

For more information, see Configure writable cgroups for containers.

writableCgroups:
  enabled: true

Apply containerd configuration to new clusters

This section shows you how to apply a containerd configuration file when you create a new GKE cluster.

Run the following command to create Autopilot clusters:

gcloud container clusters create-auto CLUSTER_NAME \
    --location=LOCATION \
    --scopes="cloud-platform" \
    --containerd-config-from-file="PATH_TO_CONFIG_FILE"

Replace the following:

• CLUSTER_NAME: the name of your new cluster.
• LOCATION: the Compute Engine location of your new cluster.
• PATH_TO_CONFIG_FILE: the path to the configuration file that you created, like ~/containerd-configuration.yaml.

You can enable the containerd configuration on new Standard clusters by running the gcloud container clusters create command with the same options.

Apply containerd configuration to new node pools

You can apply the containerd configuration to a new GKE node pool with following command:

gcloud container node-pools create NODE_POOL_NAME \
    --cluster=CLUSTER_NAME \
    --location=LOCATION \
    --scopes="cloud-platform" \
    --containerd-config-from-file="PATH_TO_CONFIG_FILE"

Replace the following:

• NODE_POOL_NAME: the name of your new node pool.
• CLUSTER_NAME: the name of the existing cluster.
• LOCATION: the Compute Engine location of your new node pool.
• PATH_TO_CONFIG_FILE: the path to the configuration file that you created, like ~/containerd-configuration.yaml.

Apply containerd configuration to existing clusters

This section shows you how to apply a containerd configuration to existing clusters and nodes.

Note: Updating a cluster with a containerd configuration recreates the nodes to apply the configuration. This might cause temporary disruptions in running workloads. GKE only recreates nodes if automatic upgrades are enabled. This recreation respects any maintenance windows that you set. In Standard clusters that don't have automatic upgrades enabled, you must manually recreate your nodes to apply the configuration.

Check access scopes

Existing clusters must have the cloud-platform access scope to use this feature. This section shows you how to check your access scopes and update an existing cluster with a new or modified private registry configuration file.

For details about the default access scopes in new clusters, see Access scopes in GKE.

Check Autopilot access scopes

Run the following command:

gcloud container clusters describe CLUSTER_NAME \
    --location=LOCATION \
    --flatten=nodeConfig \
    --format='csv[delimiter="\\n",no-heading](oauthScopes)'

If your cluster doesn't have the https://www.googleapis.com/auth/cloud-platform access scope, create a new cluster with this access scope.

Check Standard access scopes

To check your Standard cluster access scopes, check a node pool:

gcloud container node-pools describe NODE_POOL_NAME \
    --cluster=CLUSTER_NAME \
    --location=LOCATION \
    --format='value[delimiter="\\n"](config.oauthScopes)'

Replace NODE_POOL_NAME with the name of the node pool.

If your cluster doesn't have the https://www.googleapis.com/auth/cloud-platform access scope, create a new node pool with the cloud-platform access scope and delete your existing node pool.

Update the cluster to use your configuration file

Run the following command:

gcloud container clusters update CLUSTER_NAME \
    --location=LOCATION \
    --containerd-config-from-file="PATH_TO_CONFIG_FILE"

Recreate nodes in Standard clusters

If your Standard cluster doesn't use automatic upgrades, you must manually recreate your node pools to apply the new configuration. To trigger a manual node recreation, upgrade your cluster to the same GKE version that it already uses.

gcloud container clusters upgrade CLUSTER_NAME \
    --location=LOCATION \
    --cluster-version=VERSION

Replace VERSION with the same GKE patch version that the cluster already uses.

Disable containerd configuration options

Disable privateRegistryAccessConfig

1. Update your configuration file to specify enabled: false in privateRegistryAccessConfig and delete any other fields in the item, like in the following example:

   privateRegistryAccessConfig:
     enabled: false

2. Apply the updated configuration file to your cluster. For instructions, see Apply containerd configuration to existing clusters.

Disable registryHosts

1. Update your configuration file to specify an empty array in the registryHosts item, like in the following example:

   registryHosts: []

2. Apply the updated configuration file to your cluster. For instructions, see Apply containerd configuration to existing clusters.