You can use Google Kubernetes Engine (GKE) [ComputeClasses](https://docs.cloud.google.com/kubernetes-engine/docs/concepts/about-compute-classes.md.txt) to create nodes that
meet various workload requirements. Pods can use ComputeClasses to create
hardware. Depending on the ComputeClass, that hardware might be in-demand,
limited in availability, or relatively expensive for your enterprise. This
tutorial shows you how to use Kubernetes RBAC and
[ValidatingAdmissionPolicies](https://kubernetes.io/docs/reference/access-authn-authz/validating-admission-policy/) to restrict the creation, deletion,
modification, and selection of ComputeClasses.

These restrictions can help you to prevent situations like the following:

- Pods that don't need specialized hardware like GPUs or TPUs select ComputeClasses that request that hardware.
- An unauthorized entity creates or modifies ComputeClasses to get access to hardware that's reserved for critical workloads.

This document uses an example ValidatingAdmissionPolicy that blocks common types
of misuse, such as wildcard ComputeClass tolerations or the selection of
specific disallowed ComputeClasses. You can modify the ValidatingAdmissionPolicy
to meet requirements that are specific to your organization, such as blocking
workloads from certain teams from using ComputeClasses.

## Objectives

- Use a ValidatingAdmissionPolicy to limit the ComputeClasses that Pods can select in each namespace to an approved list.
- Use role-based access control (RBAC) to restrict who can create, modify, or delete ComputeClasses in your clusters.

## Costs


In this document, you use the following billable components of Google Cloud:


- [Kubernetes Engine](https://cloud.google.com/kubernetes-engine/pricing)
- [Google Workspace](https://workspace.google.com/pricing) or [Cloud Identity](https://cloud.google.com/identity/pricing)


To generate a cost estimate based on your projected usage,
use the [pricing calculator](https://docs.cloud.google.com/products/calculator).
New Google Cloud users might be eligible for a [free trial](https://docs.cloud.google.com/free).

<br />

When you finish the tasks that are described in this document, you can avoid
continued billing by deleting the resources that you created. For more information, see
[Clean up](https://docs.cloud.google.com/kubernetes-engine/docs/tutorials/restrict-computeclass-usage-admission.md.txt#clean-up).

## Before you begin

1. If you use a local shell, install the `kubectl` component:

   ```bash
   gcloud components install kubectl
   ```
2. Ensure that you have a GKE cluster that uses version 1.30 or later. You can also [create an Autopilot cluster](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/creating-an-autopilot-cluster.md.txt#set-version) for this tutorial.
3. Configure your cluster for Google Groups for RBAC:
   1. Set up the `gke-security-groups` group in your domain.
   2. Enable Google Groups for RBAC in your cluster.

   For more information, see
   [Set up your Google Groups](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/google-groups-rbac.md.txt#setup-group).

### Required roles


To get the permissions that
you need to create RBAC policies and Kubernetes objects in your cluster,

ask your administrator to grant you the
Kubernetes Engine Admin (`role/container.admin`)
IAM role on your project.


For more information about granting roles, see [Manage access to projects, folders, and organizations](https://docs.cloud.google.com/iam/docs/granting-changing-revoking-access.md.txt).


You might also be able to get
the required permissions through [custom
roles](https://docs.cloud.google.com/iam/docs/creating-custom-roles) or other [predefined
roles](https://docs.cloud.google.com/iam/docs/roles-overview#predefined).

## Prepare the environment

To prepare your cluster for the creation and verification tasks in this
tutorial, follow these steps:

1. Connect to your cluster:

   ```bash
   gcloud container clusters get-credentials CLUSTER_NAME \
       --location=CONTROL_PLANE_LOCATION
   ```

   Replace the following:
   - `CLUSTER_NAME`: the name of your cluster.
   - `CONTROL_PLANE_LOCATION`: the [region or
     zone](https://docs.cloud.google.com/compute/docs/regions-zones#available) of the cluster control plane, such as `us-central1` or `us-central1-a`.
2. Create a namespace to use for this tutorial. You can also use any existing
   namespace in the cluster.

   ```bash
   kubectl create namespace computeclass-vap-tutorial
   ```
3. Create an RBAC policy that grants access to manage
   ValidatingAdmissionPolicies and ValidatingAdmissionPolicyBindings:

   1. Save the following manifest as `validatingadmissionpolicy-editor.yaml`:

      ```yaml
      apiVersion: rbac.authorization.k8s.io/v1
      kind: ClusterRole
      metadata:
        name: validatingadmissionpolicy-editor
      rules:
      - apiGroups: ["admissionregistration.k8s.io"]
        resources: ["validatingadmissionpolicies","validatingadmissionpolicybindings"]
        verbs: ["get", "list", "create", "update", "patch", "delete"]
      ---
      apiVersion: rbac.authorization.k8s.io/v1
      kind: ClusterRoleBinding
      metadata:
        name: edit-validatingadmissionpolicies
      subjects:
      - kind: User
        name: USER_ACCOUNT
        apiGroup: rbac.authorization.k8s.io
      roleRef:
        apiGroup: rbac.authorization.k8s.io
        kind: ClusterRole
        name: validatingadmissionpolicy-editor
      ```

      Replace
      `USER_ACCOUNT` with the email address of
      your user account.
   2. Create the RBAC policy:

      ```bash
      kubectl apply -f validatingadmissionpolicy-editor.yaml
      ```
4. Create a ComputeClass in your cluster. If the cluster already has a
   ComputeClass, skip this step.

   1. Save the following manifest as `access-computeclass.yaml`:

      ```yaml
      apiVersion: cloud.google.com/v1
      kind: ComputeClass
      metadata:
        name: access-restriction-class
      spec:
        priorities:
        - machineFamily: e2
        nodePoolAutoCreation:
          enabled: true
        whenUnsatisfiable: ScaleUpAnyway
      ```
   2. Create the ComputeClass:

      ```bash
      kubectl apply -f access-computeclass.yaml
      ```

## Create a ValidatingAdmissionPolicy

To restrict the ComputeClasses that Pods can select, you create a
ValidatingAdmissionPolicy and enforce it in a specific namespace. You use the
ValidatingAdmissionPolicy specification to control how Pods can select
ComputeClasses. The following steps show you how to create and enforce this
policy:

1. Save the following ValidatingAdmissionPolicy manifest as
   `restrict-computeclass-usage-vap.yaml`:

   <br />

   ```yaml
   apiVersion: admissionregistration.k8s.io/v1
   kind: ValidatingAdmissionPolicy
   metadata:
     name: restrict-computeclass-usage
   spec:
     failurePolicy: Fail # If an internal error occurs, deny the request.
     variables:
     # Check whether the admission request is for a Deployment.
     - name: isDeployment
       expression: "object.kind == 'Deployment'"
     # Get the Pod specification from the admission request by reading the
     # spec.template.spec field for Deployments or the spec field for static Pods.
     - name: podSpec
       expression: "variables.isDeployment ? object.spec.template.spec : object.spec"
     # Check whether a node selector or an affinity rule that explicitly requests a
     # disallowed ComputeClass.
     - name: hasForbiddenNodeSelectorOrAffinity
       expression: >-
         (has(variables.podSpec.nodeSelector) &&
         variables.podSpec.nodeSelector['cloud.google.com/compute-class'] == 'COMPUTECLASS_NAME') ||
         (has(variables.podSpec.affinity) &&
         has(variables.podSpec.affinity.nodeAffinity) &&
         has(variables.podSpec.affinity.nodeAffinity.requiredDuringSchedulingIgnoredDuringExecution) &&
         variables.podSpec.affinity.nodeAffinity.requiredDuringSchedulingIgnoredDuringExecution.nodeSelectorTerms.exists(term, has(term.matchExpressions) &&
           term.matchExpressions.exists(
             exp, exp.key == 'cloud.google.com/compute-class' &&
             exp.operator == 'In' && exp.values.exists(v, v == 'COMPUTECLASS_NAME')
           )
         ))

     # Check whether the Pod has a toleration for the taint that corresponds to a
     # disallowed ComputeClass.
     - name: hasForbiddenComputeClassToleration
       expression: >-
         has(variables.podSpec.tolerations) &&
         variables.podSpec.tolerations.exists(t, 
           t.key == 'cloud.google.com/compute-class' &&
           (t.operator == 'Exists' || (t.operator == 'Equal' && t.value == 'COMPUTECLASS_NAME')) &&
           has(t.effect) && t.effect == 'NoSchedule')

     # Check whether the Pod has a toleration that could match any taint.
     - name: hasWildcardToleration
       expression: >-
         has(variables.podSpec.tolerations) &&
         variables.podSpec.tolerations.exists(t, 
           (t.operator == 'Exists' && !(has(t.key) && t.key != ''))
         )

     # Trigger the ValidatingAdmissionPolicy when Pods or Deployments are created or
     # updated.
     matchConstraints:
       resourceRules:
       - apiGroups: [""]
         apiVersions: ["v1"]
         operations: ["CREATE", "UPDATE"]
         resources: ["pods"]
       - apiGroups: ["apps"]
         apiVersions: ["v1"]
         operations: ["CREATE", "UPDATE"]
         resources: ["deployments"]

     # Validate whether any of the expressions in the variables section evaluate to
     # true.
     validations:
       - expression: >-
           !(variables.hasForbiddenNodeSelectorOrAffinity ||
             variables.hasForbiddenComputeClassToleration ||
             variables.hasWildcardToleration)
         message: >-
           Pods and Deployments in this namespace cannot request ComputeClass
           COMPUTECLASS_NAME or use wildcard
           tolerations.
   ```

   Replace `COMPUTECLASS_NAME` with the name of a
   ComputeClass that you want to prevent Pods from selecting.

   This ValidatingAdmissionPolicy has the following properties:
   - Triggers for static Pods or Pods in Deployments.
   - Finds Pods that do any of the following:

     - Use a node selector or a node affinity rule to explicitly select a specific disallowed ComputeClass.
     - Use a [toleration](https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/) that corresponds to a node taint for a specific disallowed ComputeClass.
     - Use a toleration that matches all node taints in the cluster.

   This ValidatingAdmissionPolicy is an example. You can use expressions to
   trigger the policy based on your own conditions, such as preventing Pods
   that have specific labels from selecting certain ComputeClasses.
2. Create the ValidatingAdmissionPolicy:

   ```bash
   kubectl apply -f restrict-computeclass-usage-vap.yaml
   ```
3. Save the following ValidatingAdmissionPolicyBinding as
   `restrict-computeclass-usage-binding.yaml`:

   ```yaml
   apiVersion: admissionregistration.k8s.io/v1
   kind: ValidatingAdmissionPolicyBinding
   metadata:
     name: restrict-computeclass-usage-binding
   spec:
     policyName: restrict-computeclass-usage
     validationActions: ["Deny","Audit"]
     matchResources:
       namespaceSelector:
         matchLabels:
           kubernetes.io/metadata.name: computeclass-vap-tutorial # Replace with the name of any namespace in the cluster.
   ```

   This ValidatingAdmissionPolicyBinding enforces the ValidatingAdmissionPolicy
   from the previous step in the `computeclass-vap-tutorial` namespace. The
   values in the `validationActions` field reject Pods or Deployments that
   violate the ValidatingAdmissionPolicy and add an entry to the Kubernetes
   audit log.
4. Create the ValidatingAdmissionPolicyBinding:

   ```bash
   kubectl apply -f restrict-computeclass-usage-binding.yaml
   ```

## Configure RBAC policies

In addition to using a ValidatingAdmissionPolicy to prevent workloads from using
certain ComputeClasses, use RBAC to prevent unauthorized principals from
creating and modifying the ComputeClasses in your cluster. The following steps
show you how to grant access to ComputeClasses to a specific group of users by
using Google Groups for RBAC:

1. Create a group for ComputeClass editors:

   1. In the Google Admin console, go to the **Groups** page.

      [Go to Groups](https://admin.google.com/ac/groups)
   2. Click **Create group**.

   3. On the **Group details** page, do the following:

      1. In the **Group name** field, specify `computeclass-editors`.
      2. In the **Group email** field, specify `computeclass-editors`.
      3. Select the **Security** checkbox.
      4. Click **Next**.
   4. On the **Access type** page, in the **Group members** column, verify
      that the **Who can view members** checkbox is selected.

   5. Click **Create group**.

   6. In the navigation menu, click **Directory** \> **Groups**.

   7. Hover over the **gke-security-groups** row, and then click
      **Add members**.

   8. In the **Add members to gke-security-groups** dialog, specify
      `computeclass-editors` and select that group from the results.

   9. Click **Add to group**.

2. Add the email addresses of authorized users to the `computeclass-editors`
   group.

3. Save the following ClusterRole as `computeclass-editor-clusterrole.yaml`:

   ```yaml
   apiVersion: rbac.authorization.k8s.io/v1
   kind: ClusterRole
   metadata:
     name: computeclass-editor
   rules:
   -   apiGroups: ["cloud.google.com"]
     resources: ["computeclasses"]
     verbs: ["create","update"]
   ```
4. Create the ClusterRole:

   ```bash
   kubectl apply -f computeclass-editor-clusterrole.yaml
   ```
5. Save the following ClusterRoleBinding as
   `computeclass-editor-role-binding.yaml`:

   ```yaml
   apiVersion: rbac.authorization.k8s.io/v1
   kind: ClusterRoleBinding
   metadata:
     name: computeclass-editor-role-binding
   subjects:
   - kind: Group
     name: computeclass-editors@GROUP_DOMAIN
     apiGroup: rbac.authorization.k8s.io
   roleRef:
     kind: ClusterRole
     name: computeclass-editor
     apiGroup: rbac.authorization.k8s.io # Required for role references
   ```

   Replace `GROUP_DOMAIN` with the domain name of
   the `computeclass-editors` group.
6. Create the ClusterRoleBinding:

   ```bash
   kubectl apply -f computeclass-editor-role-binding.yaml
   ```

## Verify your restrictions

The following sections show you how to verify that the restrictions that you
configured in the preceding sections work as expected. If you configure your
own restrictions in ValidatingAdmissionPolicies or RBAC policies, then test the
restrictions by creating a workload that violates those policies.

### Verify the ValidatingAdmissionPolicy

1. Save any of the following manifests, each of which violates a different
   condition of the ValidatingAdmissionPolicy:

   - Deployment that uses a node selector to select a disallowed
     ComputeClass:

     ```yaml
     apiVersion: apps/v1
     kind: Deployment
     metadata:
       name: disallowed-computeclass-deployment
       namespace: computeclass-vap-tutorial
     spec:
       replicas: 1
       selector:
         matchLabels:
           app: forbidden-selector-app
       template:
         metadata:
           labels:
             app: forbidden-selector-app
         spec:
           containers:
           - name: my-app-container
             image: nginx:latest
           # The node selector triggers the policy.
           nodeSelector:
             cloud.google.com/compute-class: COMPUTECLASS_NAME
     ```
   - Deployment that has a toleration for a disallowed ComputeClass:

     ```yaml
     apiVersion: apps/v1
     kind: Deployment
     metadata:
       name: disallowed-computeclass-deployment-toleration
       namespace: computeclass-vap-tutorial
     spec:
       replicas: 1
       selector:
         matchLabels:
           app: forbidden-selector-app
       template:
         metadata:
           labels:
             app: forbidden-selector-app
         spec:
           containers:
           - name: my-app-container
             image: nginx:latest
           tolerations:
           - key: cloud.google.com/compute-class
             operator: Equal
             value: COMPUTECLASS_NAME
             effect: NoSchedule
     ```
   - Deployment that has a node affinity rule for a disallowed ComputeClass:

     ```yaml
     apiVersion: apps/v1
     kind: Deployment
     metadata:
       name: disallowed-computeclass-deployment-toleration
       namespace: computeclass-vap-tutorial
     spec:
       replicas: 1
       selector:
         matchLabels:
           app: forbidden-selector-app
       template:
         metadata:
           labels:
             app: forbidden-selector-app
         spec:
           containers:
           - name: my-app-container
             image: nginx:latest
           affinity:
             nodeAffinity:
               requiredDuringSchedulingIgnoredDuringExecution:
                 nodeSelectorTerms:
                 - matchExpressions:
                   - key: cloud.google.com/compute-class
                     operator: In
                     values:
                     - COMPUTECLASS_NAME
     ```
   - Pod that has a toleration for any node taint in the cluster:

     ```yaml
     apiVersion: v1
     kind: Pod
     metadata:
       name: disallowed-computeclass-deployment-toleration
       namespace: computeclass-vap-tutorial
     spec:
       containers:
       - name: my-app-container
         image: nginx:latest
       tolerations:
       - operator: Exists
         effect: NoSchedule
     ```
2. Create the test workload:

   ```bash
   kubectl apply -f PATH_TO_WORKLOAD_MANIFEST
   ```

   The output is similar to the following:

       Error from server (BadRequest): admission webhook "validation-policy.kubernetes.io" denied the request: Pods and Deployments in this namespace cannot request ComputeClass COMPUTECLASS_NAME or use wildcard tolerations.

### Verify the RBAC configuration

To verify your RBAC configuration, follow these steps:

1. Authenticate to the cluster by using the credentials of the a user who is a
   member of the `computeclass-editors` group.

2. Check whether the user can create a ComputeClass:

   ```bash
   kubectl auth can-i create computeclasses.cloud.google.com \
       --as=MEMBER_USER
   ```

   Replace `MEMBER_USER` with the email address
   of a user who is a member of the group.

   The output is `yes`.
3. Authenticate to the cluster by using the credentials of the a user who isn't
   a member of the `computeclass-editors` group.

4. Check whether the user can create a ComputeClass:

   ```bash
   kubectl auth can-i create computeclasses.cloud.google.com \
       --as=NON_MEMBER_USER
   ```

   Replace `NON_MEMBER_USER` with the email address
   of a user who isn't a member of the group.

   The output is `no`.

If the output of the `kubectl auth can-i` command is `yes` for a user who isn't
a member of the group, verify the following:

- The `computeclass-editors` group is a member of the `gke-security-groups` group.
- The email address of the group in your ClusterRoleBinding is the exact email address of the group, such as `computeclass-editors@example.com`.

## Clean up


To avoid incurring charges to your Google Cloud account for the resources used in this
tutorial, either delete the project that contains the resources, or keep the project and
delete the individual resources.

If you deploy the ValidatingAdmissionPolicy and RBAC policies in this tutorial
in a production cluster, then GKE blocks incoming Pods and API
requests that violate the policies. To keep these policies active, skip this
section. The following sections show you how to delete the project or individual
resources if you followed this tutorial in a learning or testing environment.

### Delete the project

> [!CAUTION]
> **Caution** : Deleting a project has the following effects:
>
> - **Everything in the project is deleted.** If you used an existing project for the tasks in this document, when you delete it, you also delete any other work you've done in the project.
> - **Custom project IDs are lost.** When you created this project, you might have created a custom project ID that you want to use in the future. To preserve the URLs that use the project ID, such as an `appspot.com` URL, delete selected resources inside the project instead of deleting the whole project.
>
>
> If you plan to explore multiple architectures, tutorials, or quickstarts, reusing projects
> can help you avoid exceeding project quota limits.

1. Delete a Google Cloud project:

```
gcloud projects delete PROJECT_ID
```

<br />

### Delete individual resources

1. Delete the ComputeClass that you created:

   ```bash
   kubectl delete computeclass access-restriction-class
   ```
2. Delete the ValidatingAdmissionPolicy and the
   ValidatingAdmissionPolicyBinding:

   ```bash
   kubectl delete validatingadmissionpolicy restrict-computeclass-usage
   kubectl delete validatingadmissionpolicybinding restrict-computeclass-usage-binding
   ```
3. Delete the ClusterRoles and the ClusterRoleBindings:

   ```bash
   kubectl delete clusterroles \
       computeclass-editor validatingadmissionpolicy-editor
   kubectl delete clusterrolebindings \
       computeclass-editor-role-binding edit-validatingadmissionpolicies
   ```
4. Delete the example namespace:

   ```bash
   kubectl delete namespace computeclass-vap-tutorial
   ```

## What's next

- [Review best practices for ComputeClasses](https://docs.cloud.google.com/kubernetes-engine/docs/best-practices/computeclasses.md.txt)
- [Review best practices for cluster security](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/hardening-your-cluster.md.txt)
- Explore reference architectures, diagrams, and best practices about Google Cloud. Take a look at our [Cloud Architecture Center](https://docs.cloud.google.com/architecture).