About privileged workload admission in Autopilot mode

Autopilot

You can control which privileged workloads can run in Google Kubernetes Engine (GKE) Autopilot mode by using allowlists and policies. This document describes the default constraints that Autopilot enforces on workloads, the types of privileged workloads that you can run in Autopilot mode, and the methods of controlling privileged workload admission.

This document is for the following people:

Platform admins and operators who want to run privileged GKE workloads from various sources in Autopilot mode.
Identity and account admins who want to improve the security posture of your organization by blocking all privileged workloads except for specific workloads.

About privileged workloads in Autopilot mode

To improve your security posture, Autopilot mode clusters and nodes enforce specific security constraints by default. For example, Autopilot rejects most Pods that set the spec.securityContext.privileged field to true in a container. Privileged workloads in Autopilot are workloads that don't meet these default constraints. By default, Autopilot nodes in a Google Cloud organization reject privileged workloads.

Specific privileged workloads can run in Autopilot mode only if the cluster has allowlists that match those workloads. When a cluster admin creates or modifies a cluster, they can specify paths to allowlists that the cluster can run. By default, every Google Cloud organization supports allowlists from Autopilot partners and verified open source projects.

Organization administrators can modify this default behavior in the following ways by using Organization Policy Service:

Allow all workloads from GKE Autopilot partners and verified open source workloads. This is the default behavior in all Google Cloud organizations.
Allow specific partner workloads or verified open source workloads.
Allow only specific customer-owned privileged workloads.
Deny all privileged workloads.

This deny all, and then allow some approach is a security best practice that lets you control exactly what runs with elevated privileges in your GKE clusters.

You might want to control privileged workloads in Autopilot mode in situations like the following:

You run workloads in Autopilot mode in Standard clusters and you want privileged workloads to run on those nodes.
You want to allow only a specific set of privileged workloads to run in your organization, folder, or project.

Sources of privileged Autopilot workloads

The following table describes the types of privileged workloads that you can run in Autopilot:

Privileged workload types
Autopilot partner workloads	A subset of Google Cloud partners provide privileged workloads for Autopilot mode. GKE verifies these partner workloads. These workloads are available to any customer. By default, any customer can install the corresponding allowlists in any Autopilot or Standard cluster. For more information about the available partner workloads, see Autopilot partners.
Open source workloads	Specific privileged open source workloads have allowlists that let you run those workloads in Autopilot mode. GKE verifies these workloads. By default, any customer can install the corresponding allowlists in any Autopilot or Standard cluster. For more information about the available open source workloads, see Run privileged open source workloads on GKE Autopilot.
Customer-owned workloads	If you're an eligible customer who has privileged workloads that you want to run in Autopilot nodes, you can create allowlists for those workloads. This capability is disabled by default for all Google Cloud organizations. You use Organization Policy Service to add allowlist paths for specific workloads, and then update clusters to install those allowlists. Note: Running your own privileged workloads in Autopilot is available only to eligible Google Cloud customers. To check whether you're eligible, contact Cloud Customer Care. For more information, see the Customer-owned privileged workloads section.

Privileged workload types

Autopilot partner workloads

A subset of Google Cloud partners provide privileged workloads for Autopilot mode. GKE verifies these partner workloads. These workloads are available to any customer. By default, any customer can install the corresponding allowlists in any Autopilot or Standard cluster.

For more information about the available partner workloads, see Autopilot partners.

Open source workloads

Specific privileged open source workloads have allowlists that let you run those workloads in Autopilot mode. GKE verifies these workloads. By default, any customer can install the corresponding allowlists in any Autopilot or Standard cluster.

For more information about the available open source workloads, see Run privileged open source workloads on GKE Autopilot.

Customer-owned workloads

If you're an eligible customer who has privileged workloads that you want to run in Autopilot nodes, you can create allowlists for those workloads. This capability is disabled by default for all Google Cloud organizations. You use Organization Policy Service to add allowlist paths for specific workloads, and then update clusters to install those allowlists.

For more information, see the Customer-owned privileged workloads section.

How privileged workload admission control works

To allow specific privileged workloads to run in your Autopilot nodes, you install allowlists that correspond to each workload. These allowlists are WorkloadAllowlist Kubernetes custom resources. GKE validates the workload details against any WorkloadAllowlists in the cluster, and allows admission only if the details match. By default, every Autopilot or Standard cluster in a Google Cloud organization lets you install WorkloadAllowlists for Autopilot partner workloads and for open source workloads.

At a high level, running privileged workloads in Autopilot nodes involves these steps:

The organization administrator verifies that the container.managed.autopilotPrivilegedAdmission managed Organization Policy constraint allows the installation of allowlists from a specific source. For more information, see the Organization Policy managed constraint for allowlists section.
For customer-owned privileged workloads, the platform administrator creates allowlists that match the privileged workloads. For more information, see the Customer-owned privileged workloads section.
The cluster admin configures an Autopilot or Standard cluster to allow installation of allowlists from a specific source. For more information, see the Cluster configuration section.
The cluster admin creates an AllowlistSynchronizer that references the path to the allowlist. The AllowlistSynchronizer installs the allowlist and keeps the allowlist up to date. For more information, see the Allowlist installation section.

After these steps are completed, an app operator can deploy the privileged workload in the cluster.

Organization Policy managed constraint for allowlists

By default, every Google Cloud organization enforces an organization policy that's based on the container.managed.autopilotPrivilegedAdmission managed constraint.

This managed constraint has the following parameters:

allowAnyGKEPath: a boolean value that lets cluster admins configure clusters for allowlist installation from paths that have the gke:// prefix. The following values are supported:
- true: let cluster admins configure clusters with the GKE default allowlist sources, with any allowlist path that starts with gke://, or with an empty string (which prevents all allowlists). This is the default value for the allowAnyGKEPath parameter.
- false: require that cluster admins configure clusters with an empty string for allowlist sources or with any allowlist paths from the allowPaths parameter.
allowPaths: a list of approved allowlist sources that cluster admins can specify values from when they configure a cluster.

As an organization administrator, you can update the parameters of this policy to control which privileged workloads can run in your Google Cloud organization. For example, the following organization policy configuration lets cluster admins install allowlists only for a specific Google Cloud partner workload:

name: organizations/ORGANIZATION_ID/policies/container.managed.autopilotPrivilegedAdmission
spec:
  rules:
  - enforce: true
    parameters:
      allowAnyGKEPath: ALLOW_GKE_PATHS
      allowPaths:
      - PATH1
      - PATH2
      - PATH3

Replace the following:

ALLOW_GKE_PATHS: whether to allow any GKE-approved allowlists. Specify one of the following values:
- True: allow cluster configuration with any GKE partner workload or verified open source workloads. This is the default value.
- False: allow cluster configuration with only the paths in the allowPaths field.
Caution: If you specify a value of False in the allowAnyGKEPath parameter and you don't add any paths in the allowPaths parameter, allowlists from any source, including GKE partners, are disabled for all of the clusters in your organization, folder, or project. Unless you're sure that none of your clusters rely on these partner workloads, keep the default value of True.
PATH1, PATH2, ...: a list of paths to approved allowlist sources that cluster admins can use when they configure clusters. By default, this parameter is empty. If you specify paths for this parameter, cluster admins must specify zero or more of those paths when they create or update a cluster.

For more information about controlling privileged workload admission in organization policies, see Restrict privileged GKE workloads in organizations.

Customer-owned privileged workloads

As a GKE customer, you might have your own specialized workloads that require more privileges in the cluster than the default Autopilot security constraints allow. In GKE version 1.35 and later, you can exempt these workloads from the default Autopilot constraints by creating allowlists.

An allowlist is a YAML file that defines a WorkloadAllowlist Kubernetes custom resource. The specification of a WorkloadAllowlist matches various fields in the Kubernetes Pod specification. When you define a WorkloadAllowlist, you match the values in the WorkloadAllowlist to corresponding fields in the specification of your privileged workload. You store the YAML files in a Cloud Storage bucket.

To install these customer-owned allowlists in a cluster, the following steps must happen:

Your organization administrator adds the Cloud Storage paths to an organization policy. For more information, see the About the Organization Policy managed constraint section.
You add the Cloud Storage paths to a cluster configuration. For more information, see the Cluster configuration section.

For more information about creating your own WorkloadAllowlists, see Create allowlists for privileged Autopilot workloads.

Cluster configuration

To run a privileged workload in Autopilot mode, a cluster admin adds one or more approved allowlist paths to the cluster configuration. After you add the paths to the cluster, app operators can install the corresponding allowlists and privileged workloads in the cluster.

By default, you can install allowlists from Autopilot partners and approved open source projects. This default set of approved sources for allowlists might change because of an organization policy, as described in the preceding section. If an organization policy controls the approved allowlist paths, you must coordinate with your organization administrator to identify those paths. You can then specify one or more of the approved paths in the cluster configuration.

For more information about configuring clusters, see Run privileged workloads in Autopilot.

Allowlist installation

To install an allowlist from an approved source in your cluster, you use a Kubernetes custom resource named the AllowlistSynchronizer. You define and create AllowlistSynchronizers similarly to how you create other Kubernetes resources like Pods and Deployments. Each AllowlistSynchronizer has a list of paths to allowlists that you want to install in the cluster.

When you create an AllowlistSynchronizer, GKE installs allowlists from these paths and keeps the allowlists up-to-date. Every 10 minutes, GKE checks for changes to allowlist files. If an update exists, the synchronizer installs the updated allowlist in the cluster.

After GKE installs an allowlist based on an AllowlistSynchronizer, you can create the corresponding privileged workload in the cluster.

For more information about installing allowlists in a cluster, see Run privileged workloads in Autopilot.

Allowlist paths

When you configure allowlist support in your organization policy or clusters, you specify paths to one or more WorkloadAllowlists that you've approved for installation. The syntax for these paths depends on who created the WorkloadAllowlist, as follows:

Specific Autopilot partners and approved open source projects:
```
gke://REPOSITORY_PATH/SELECTOR
```
Replace the following:
- REPOSITORY_PATH: the path to the directory that contains allowlists for the workload, such as Grafana/alloy. For more information about the allowlist paths for these workloads, see the following pages:
  - Autopilot partner workloads
  - Privileged open source workloads with Autopilot support
- SELECTOR: the name of a specific file or the * character, which selects every file in that directory.
For example, the following path selects any allowlist in the Grafana/alloy directory:
```
gke://Grafana/alloy/*
```
Customer-owned privileged workloads:
```
gs://DIRECTORY_PATH/SELECTOR
```
Replace DIRECTORY_PATH with the path to the directory in Cloud Storage that contains your allowlist files. You can specify multiple paths.

You can specify one or more of the preceding paths in your organization policy or cluster configuration. For example, you can add GKE partner workloads and customer-owned workloads to the same configuration. You can also use the * character to select all allowlists. For example, gke://* selects any GKE-approved allowlist from any source.

The following table describes the configuration that's needed in organization policies or clusters to support certain objectives:

Objective	Organization policy configuration	Cluster configuration
Allow all GKE-approved WorkloadAllowlists	Do the following: Set the `allowAnyGKEPath` parameter to the default value of `True`. Keep the `allowPaths` parameter empty.	Do one of the following: Don't configure allowlists for your cluster. By default, all clusters allow the installation of GKE-approved WorkloadAllowlists. Specify only `gke://*` as the approved path when you configure allowlists for your cluster.
Allow only specific GKE-approved WorkloadAllowlists	Do the following: Set the `allowAnyGKEPath` parameter to `False`. Specify paths to specific GKE-approved WorkloadAllowlists in the `allowPaths` parameter.	Specify paths to specific GKE-approved WorkloadAllowlists when you create or update a cluster. An organization policy might restrict which GKE-approved paths you can specify.
Allow only customer-managed WorkloadAllowlists	Do the following: Set the `allowAnyGKEPath` parameter to `False`. Specify paths to customer-managed WorkloadAllowlists in the `allowPaths` parameter.	Specify paths to customer-managed WorkloadAllowlists when you create or update a cluster. The paths that you specify must be in the `allowPaths` parameter of an organization policy.
Prevent the use of WorkloadAllowlists	Do the following: Set the `allowAnyGKEPath` parameter to `False`. Keep the `allowPaths` parameter empty.	Set an empty string `""` as the approved path when you create or update a cluster.