This topic describes how to enable workload identity for your
GKE on AWS workloads to control their access to AWS resources.

For information about using workload identity with Google Cloud
Identity and Access Management (IAM) accounts to control access to GCP resources, see
[Using workload identity with Google Cloud](https://docs.cloud.google.com/kubernetes-engine/multi-cloud/docs/aws/previous-generation/how-to/workload-identity-gcp.md.txt).

## Overview

Workload identity uses AWS IAM permissions to control access to cloud
resources. With workload identity, you can assign different IAM roles to
each workload. This fine-grained control of permissions lets you follow the
principle of
[least privilege](https://en.wikipedia.org/wiki/Principle_of_least_privilege).
Without workload identity, you must assign AWS IAM roles to your
GKE on AWS nodes, giving all workloads on the node the same
permissions as the node itself.

To enable workload identity for your cluster, complete the following
steps, which are grouped by the administrative roles that perform them.

**Cluster administrator**

1. Create a Cloud Storage bucket to store [OIDC discovery data](https://docs.cloud.google.com/kubernetes-engine/multi-cloud/docs/aws/previous-generation/how-to/workload-identity-aws.md.txt#oidc-discovery).
2. Create an Identity and Access Management role to read from that bucket.
3. Create a user cluster with [workload identity enabled](https://docs.cloud.google.com/kubernetes-engine/multi-cloud/docs/aws/previous-generation/how-to/workload-identity-aws.md.txt#create-cluster).
4. Create a [webhook on your cluster](https://docs.cloud.google.com/kubernetes-engine/multi-cloud/docs/aws/previous-generation/how-to/workload-identity-aws.md.txt#create_the_workload_identity_webhook) that applies workload identity credentials to Pods on creation. If you don't want to use the webhook, you can manually set environment variables in your pods.
5. Configure the [AWS OIDC provider](https://docs.cloud.google.com/kubernetes-engine/multi-cloud/docs/aws/previous-generation/how-to/workload-identity-aws.md.txt#configure-aws-oidc).
6. Create [AWS IAM roles and
   policies](https://docs.cloud.google.com/kubernetes-engine/multi-cloud/docs/aws/previous-generation/how-to/workload-identity-aws#creating_aws_iam_roles_and_policies).

**Cluster administrator or developer**

6. Create [Kubernetes
   service accounts](https://docs.cloud.google.com/kubernetes-engine/multi-cloud/docs/aws/previous-generation/how-to/workload-identity-aws#creating_kubernetes_service_accounts_for_workloads), and bind AWS policies to them.

**Developer**

7. [Apply credentials to your Pods](https://docs.cloud.google.com/kubernetes-engine/multi-cloud/docs/aws/previous-generation/how-to/workload-identity-aws.md.txt#apply_credentials_with_the_webhook).

<br />

## Prerequisites

To complete the steps in this document, you must have the following setup:

- An GKE on AWS [management service](https://docs.cloud.google.com/kubernetes-engine/multi-cloud/docs/aws/previous-generation/how-to/installing-management.md.txt).
- User clusters that are running a Kubernetes version greater than 1.17.9.

  > [!NOTE]
  > **Note:** If you created a Kubernetes 1.17 or 1.18 cluster, you must [Upgrade the webhook](https://docs.cloud.google.com/kubernetes-engine/multi-cloud/docs/aws/previous-generation/how-to/workload-identity-aws.md.txt#upgrading) before upgrading to Kubernetes 1.19.

- The following permissions and tools.

### Permissions

To create a cluster with workload identity enabled, you need the following
permissions:

**Google Cloud**

- Create a publicly readable Cloud Storage bucket with uniform bucket-level access enabled.
- Grant `management-sa@PROJECT_NAME.iam.gserviceaccount.com` read/write permissions to the bucket.

**AWS**

- Create an AWS OIDC provider
- Create AWS IAM roles

### Tools

On your local machine, we recommend having the
[`jq` tool](https://stedolan.github.io/jq/download/) installed.

## Creating the OIDC discovery bucket

This section is for [cluster administrators](https://docs.cloud.google.com/kubernetes-engine/multi-cloud/docs/aws/previous-generation/how-to/oidc.md.txt#personas).

Your user cluster needs to store the OIDC discovery data in a publicly
accessible Cloud Storage bucket. The bucket includes OIDC discovery
configuration and public keys. AWS uses the contents to authenticate requests
from your user clusters.

Your bucket must have the following attributes:

- Be [publicly readable](https://docs.cloud.google.com/storage/docs/access-control/making-data-public.md.txt).
- Have [uniform bucket-level access](https://docs.cloud.google.com/storage/docs/uniform-bucket-level-access.md.txt) enabled.

If you don't have a bucket with these attributes, create one by using the
following `gcloud storage` commands:

```bash
BUCKET=BUCKET_NAME
gcloud storage buckets create gs://${BUCKET} --uniform-bucket-level-access
gcloud storage buckets add-iam-policy-binding gs://${BUCKET} \
    --member=allUsers --role=roles/storage.objectViewer
```

Replace `BUCKET_NAME` with the name of your new bucket.

### Grant the management service account permissions

The Identity and Access Management service account for the GKE on AWS management
service needs permissions to read and write objects into this bucket.

1. Grant your management service account permissions by using the following
   command.

   ```bash
   MANAGEMENT_SA=management-sa@PROJECT_NAME.iam.gserviceaccount.com
   gcloud storage buckets add-iam-policy-binding gs://${BUCKET} \
       --member=serviceAccount:${MANAGEMENT_SA} \
       --role=roles/storage.admin
   ```

   Replace `PROJECT_NAME` with your Google Cloud project.
2. Create a new IAM role with permissions to manage this bucket.
   To create the role, first save the role definition to a file, then create
   the role and bind the role to your management service account.

   To complete these steps, run the following commands:

       cat << EOF >  anthos-oidc-role.yaml
       title: anthosAwsOidcStorageAdmin
       description: permissions to manage the OIDC buckets
       stage: GA
       includedPermissions:
       - storage.buckets.get
       EOF

       gcloud iam roles create anthosAwsOidcStorageAdmin --project=PROJECT_NAME \
         --file=anthos-oidc-role.yaml

       gcloud projects add-iam-policy-binding \
         PROJECT_NAME \
         --member=serviceAccount:${MANAGEMENT_SA} \
         --role=projects/PROJECT_NAME/roles/anthosAwsOidcStorageAdmin

   Replace `PROJECT_NAME` with your Google Cloud project.

   The Google Cloud CLI confirms that the policy binding is created.

## Creating a user cluster

This section is for [cluster administrators](https://docs.cloud.google.com/kubernetes-engine/multi-cloud/docs/aws/previous-generation/how-to/oidc.md.txt#personas).

### Create a user cluster with workload identity enabled

Create a user cluster that contains details about your OIDC discovery bucket. You
set this information in your `AWSCluster`'s
`spec.controlPlane.workloadIdentity.oidcDiscoveryGCSBucket` field.

In this example, you create a cluster manually from `AWSCluster` and
`AWSNodePool` CRDs.

1. Change to the directory with your GKE on AWS configuration.
   You created this directory when
   [Installing the management service](https://docs.cloud.google.com/kubernetes-engine/multi-cloud/docs/aws/previous-generation/how-to/installing-management.md.txt).

   ```sh
   cd anthos-aws
   ```

   <br />

2. From your `anthos-aws` directory, use
   `anthos-gke` to switch context to your management service.

   ```sh
   cd anthos-aws
   anthos-gke aws management get-credentials
   ```

   > [!NOTE]
   > **Note:** `anthos-gke` uses the credentials you specified when you ran `aws configure`.

   <br />

3. Open a text editor and copy the following `AWSCluster` definition into a file
   named `custom-cluster.yaml`.

   ```yaml
   apiVersion: multicloud.cluster.gke.io/v1
   kind: AWSCluster
   metadata:
     name: CLUSTER_NAME
   spec:
     region: AWS_REGION
     networking:
       vpcID: VPC_ID
       podAddressCIDRBlocks: POD_ADDRESS_CIDR_BLOCKS
       serviceAddressCIDRBlocks: SERVICE_ADDRESS_CIDR_BLOCKS
       ServiceLoadBalancerSubnetIDs: SERVICE_LOAD_BALANCER_SUBNETS
     controlPlane:
       version:  CLUSTER_VERSION # Latest version is 1.25.5-gke.2100
       instanceType: AWS_INSTANCE_TYPE
       keyName: SSH_KEY_NAME
       subnetIDs:
       - CONTROL_PLANE_SUBNET_IDS
       securityGroupIDs:
       - CONTROL_PLANE_SECURITY_GROUPS
       iamInstanceProfile: CONTROL_PLANE_IAM_ROLE
       rootVolume:
         sizeGiB: ROOT_VOLUME_SIZE
         volumeType: ROOT_VOLUME_TYPE # Optional
         iops: ROOT_VOLUME_IOPS # Optional
         kmsKeyARN: ROOT_VOLUME_KEY # Optional
       etcd:
         mainVolume:
           sizeGiB: ETCD_VOLUME_SIZE
           volumeType: ETCD_VOLUME_TYPE # Optional
           iops: ETCD_VOLUME_IOPS # Optional
           kmsKeyARN: ETCD_VOLUME_KEY # Optional
       databaseEncryption:
         kmsKeyARN: ARN_OF_KMS_KEY
       hub: # Optional
         membershipName: ANTHOS_CONNECT_NAME
       cloudOperations: # Optional
         projectID: YOUR_PROJECT
         location: GCP_REGION
         enableLogging: ENABLE_LOGGING
         enableMonitoring: ENABLE_MONITORING
       workloadIdentity: # Optional
         oidcDiscoveryGCSBucket: WORKLOAD_IDENTITY_BUCKET
   ```

   Replace the following:
   - <var translate="no">CLUSTER_NAME</var>: the name of your cluster.
   - <var translate="no">AWS_REGION</var>: the AWS region where your cluster runs.

     > [!NOTE]
     > **Note:** You must create your user cluster in the same region as your management service.

   - <var translate="no">VPC_ID</var>: the ID of the VPC where your cluster runs.

   - <var translate="no">POD_ADDRESS_CIDR_BLOCKS</var>: the range of IPv4 addresses used by the
     cluster's pods. Currently only a single range is supported. The range
     must not overlap with any subnets reachable from your network. It is
     safe to use the same range across multiple different AWSCluster objects.
     For example, `10.2.0.0/16`.

   - <var translate="no">SERVICE_ADDRESS_CIDR_BLOCKS</var>: the range of IPv4 addresses used
     by the cluster's services. Currently only a single range is supported.
     The range must not overlap with any subnets reachable from your network.
     It is safe to use the same range across multiple different AWSCluster
     objects. For example, `10.1.0.0/16`.

   - <var translate="no">SERVICE_LOAD_BALANCER_SUBNETS</var>: the subnet IDs where
     GKE on AWS can create public or private load balancers.

   - <var translate="no">CLUSTER_VERSION</var>: a
     [Kubernetes version](https://docs.cloud.google.com/kubernetes-engine/multi-cloud/docs/aws/previous-generation/concepts/versioning-and-upgrades.md.txt#versions)
     supported by GKE on AWS. The most recent version is
     1.25.5-gke.2100.

   - <var translate="no">AWS_INSTANCE_TYPE</var>: a
     [supported EC2 instance type](https://docs.cloud.google.com/kubernetes-engine/multi-cloud/docs/aws/previous-generation/reference/instance-types.md.txt).

   - <var translate="no">SSH_KEY_NAME</var>: an
     [AWS EC2 key pair](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-key-pairs.html#prepare-key-pair).

   - <var translate="no">CONTROL_PLANE_SUBNET_IDS</var>: the subnet IDs in the AZs
     where your control plane instances run.

   - <var translate="no">CONTROL_PLANE_SECURITY_GROUPS</var>: a securityGroupID created during
     the management service installation.
     You can customize this by adding any securityGroupIDs required to connect
     to the control plane.

   - <var translate="no">CONTROL_PLANE_IAM_PROFILE</var>: name of the AWS EC2 instance
     profile assigned to control plane replicas.

   - <var translate="no">ROOT_VOLUME_SIZE</var>: the size, in gibibyte (GiB), of your
     control plane root volumes.

   - <var translate="no">ROOT_VOLUME_TYPE</var> with the
     [EBS volume type](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-volume-types.html).
     For example, `gp3`.

   - <var translate="no">ROOT_VOLUME_IOPS</var> with the amount of provisioned IO operations per
     second (IOPS) for the volume. Only valid when `volumeType` is `GP3`. For
     more information, see
     [General Purpose SSD volumes (gp3)](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-volume-types.html#gp3-ebs-volume-type).

   - <var translate="no">ROOT_VOLUME_KEY</var> with the Amazon Resource Name of the
     [AWS KMS key](https://docs.cloud.google.com/kubernetes-engine/multi-cloud/docs/aws/previous-generation/how-to/prerequisites.md.txt#creating_a_kms_key) that
     encrypts your control plane instance root volumes.

   - <var translate="no">ETCD_VOLUME_SIZE</var>: the size of volumes used by etcd.

   - <var translate="no">ETCD_VOLUME_TYPE</var> with the
     [EBS volume type](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-volume-types.html).
     For example, `gp3`.

   - <var translate="no">ETCD_VOLUME_IOPS</var> with the amount of provisioned IO operations per
     second (IOPS) for the volume. Only valid when `volumeType` is `gp3`. For
     more information, see
     [General Purpose SSD volumes (gp3)](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-volume-types.html#gp3-ebs-volume-type).

   - <var translate="no">ETCD_VOLUME_KEY</var> with the Amazon Resource Name of the
     [AWS KMS key](https://docs.cloud.google.com/kubernetes-engine/multi-cloud/docs/aws/previous-generation/how-to/prerequisites.md.txt#creating_a_kms_key) that
     encrypts your control plane etcd data volumes.

   - <var translate="no">ARN_OF_KMS_KEY</var>: the AWS KMS key used to encrypt cluster
     Secrets.

   - <var translate="no">ANTHOS_CONNECT_NAME</var>: the
     [Connect](https://docs.cloud.google.com/anthos/multicluster-management/connect/overview)
     membership name used to register your cluster. The membership name
     must be unique. For example,
     `projects/YOUR_PROJECT/locations/global/memberships/CLUSTER_NAME`,
     where `YOUR_PROJECT` is your Google Cloud project
     and `CLUSTER_NAME` is a unique name in your project.
     This field is optional.

   - <var translate="no">YOUR_PROJECT</var>: your project ID.

   - <var translate="no">GCP_REGION</var>: the Google Cloud region where you want to store logs.
     Choose a region that is near the AWS region. For more information, see
     [Global Locations - Regions \& Zones](https://cloud.google.com/about/locations)
     --- for example, `us-central1`.

   - <var translate="no">ENABLE_LOGGING</var>: `true` or `false`, whether Cloud Logging is enabled on control plane nodes.

   - <var translate="no">ENABLE_MONITORING</var>: `true` or `false`, whether Cloud Monitoring is enabled on control plane nodes.

   - <var translate="no">WORKLOAD_IDENTITY_BUCKET</var>: the Cloud Storage bucket
     name containing your
     [workload identity](https://docs.cloud.google.com/kubernetes-engine/multi-cloud/docs/aws/previous-generation/how-to/workload-identity-cluster)
     discovery information. This field is optional.

4. Create one or more AWSNodePools for your cluster. Open a text
   editor and copy the following AWSCluster definition into a file
   named `custom-nodepools.yaml`.

   ```yaml
   apiVersion: multicloud.cluster.gke.io/v1
   kind: AWSNodePool
   metadata:
     name: NODE_POOL_NAME
   spec:
     clusterName: AWSCLUSTER_NAME
     version:  CLUSTER_VERSION # latest version is 1.25.5-gke.2100
     region: AWS_REGION
     subnetID: AWS_SUBNET_ID
     minNodeCount: MINIMUM_NODE_COUNT
     maxNodeCount: MAXIMUM_NODE_COUNT
     maxPodsPerNode: MAXIMUM_PODS_PER_NODE_COUNT
     instanceType: AWS_NODE_TYPE
     keyName: KMS_KEY_PAIR_NAME
     iamInstanceProfile: NODE_IAM_PROFILE
     proxySecretName: PROXY_SECRET_NAME
     rootVolume:
       sizeGiB: ROOT_VOLUME_SIZE
       volumeType: VOLUME_TYPE # Optional
       iops: IOPS # Optional
       kmsKeyARN: NODE_VOLUME_KEY # Optional 
   ```

   Replace the following:
   - <var translate="no">NODE_POOL_NAME</var>: a unique name for your AWSNodePool.
   - <var translate="no">AWSCLUSTER_NAME</var>: your AWSCluster's name. For example, `staging-cluster`.
   - <var translate="no">CLUSTER_VERSION</var>: a supported GKE on AWS Kubernetes [version](https://docs.cloud.google.com/kubernetes-engine/multi-cloud/docs/aws/previous-generation/concepts/versioning-and-upgrades.md.txt).
   - <var translate="no">AWS_REGION</var>: the same AWS region as your AWSCluster.
   - <var translate="no">AWS_SUBNET_ID</var>: an AWS subnet in the same region as your AWSCluster.
   - <var translate="no">MINIMUM_NODE_COUNT</var>: the minimum number of nodes in the node pool. See [Scaling user clusters](https://docs.cloud.google.com/kubernetes-engine/multi-cloud/docs/aws/previous-generation/how-to/scaling-user-cluster.md.txt) for more information.
   - <var translate="no">MAXIMUM_NODE_COUNT</var>: the maximum number of nodes in the node pool.
   - <var translate="no">MAXIMUM_PODS_PER_NODE_COUNT</var>: the maximum number of pods that GKE on AWS can allocate to a node.
   - <var translate="no">AWS_NODE_TYPE</var>: an [AWS EC2 instance type](https://docs.cloud.google.com/kubernetes-engine/multi-cloud/docs/aws/previous-generation/reference/instance-types.md.txt).
   - <var translate="no">KMS_KEY_PAIR_NAME</var>: the AWS KMS key pair assigned to each node pool worker.
   - <var translate="no">NODE_IAM_PROFILE</var>: the name of the AWS EC2 instance profile assigned to nodes in the pool.
   - <var translate="no">ROOT_VOLUME_SIZE</var>: the size, in gibibyte (GiB), of your control plane root volumes.
   - <var translate="no">VOLUME_TYPE</var>: the node's AWS [EBS volume type](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-volume-types.html). For example, `gp3`.
   - <var translate="no">IOPS</var>: the amount of provisioned IO operations per second (IOPS) for volumes. Only valid when `volumeType` is `gp3`.
   - <var translate="no">NODE_VOLUME_KEY</var>: the ARN of the AWS KMS key used to encrypt the volume. For more information, see [Using a customer managed CMK to encrypt volumes](https://docs.cloud.google.com/kubernetes-engine/multi-cloud/docs/aws/previous-generation/how-to/using-cmk.md.txt).
5. Apply the manifests to your management service.

   ```bash
   env HTTPS_PROXY=http://localhost:8118 \
     kubectl apply -f custom-cluster.yaml
   env HTTPS_PROXY=http://localhost:8118 \
     kubectl apply -f custom-nodepools.yaml
   ```

   > [!NOTE]
   > **Note:** Your cluster takes several minutes to start.

### Create a kubeconfig

While your user cluster starts, you can create a `kubeconfig` context for your
new user cluster. You use the context to authenticate to a user or management
cluster.

1. Use `anthos-gke aws clusters get-credentials` to generate a
   [`kubeconfig`](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/cluster-access-for-kubectl.md.txt#kubeconfig)
   for your user cluster in `~/.kube/config`.

   ```bash
   env HTTPS_PROXY=http://localhost:8118 \
     anthos-gke aws clusters get-credentials CLUSTER_NAME
   ```

   Replace <var translate="no">CLUSTER_NAME</var> with your cluster's name. For example,
   `cluster-0`.
2. Use `kubectl` to authenticate to your new user cluster.

   ```bash
   env HTTPS_PROXY=http://localhost:8118 \
     kubectl cluster-info
   ```

   If your cluster is ready, the output includes the URLs for Kubernetes
   components within your cluster.

## Viewing your cluster's status

The management service provisions AWS resources when you apply an
`AWSCluster` or `AWSNodePool`.

1. From your `anthos-aws` directory, use
   `anthos-gke` to switch context to your management service.

   ```sh
   cd anthos-aws
   anthos-gke aws management get-credentials
   ```

   > [!NOTE]
   > **Note:** `anthos-gke` uses the credentials you specified when you ran `aws configure`.

   <br />

2. To list your clusters, use `kubectl get AWSClusters`.

   ```bash
   env HTTPS_PROXY=http://localhost:8118 \
     kubectl get AWSClusters
   ```

   The output includes each cluster's name, state, age, version, and endpoint.

   For example, the following output includes only one `AWSCluster` named `cluster-0`:

       NAME        STATE          AGE     VERSION         ENDPOINT
       cluster-0   Provisioning   2m41s   1.25.5-gke.2100   gke-xyz.elb.us-east-1.amazonaws.com

### View your cluster's events

To see recent
[Kubernetes Events](https://www.bluematador.com/blog/kubernetes-events-explained)
from your user cluster, use `kubectl get events`.

1. From your `anthos-aws` directory, use
   `anthos-gke` to switch context to your management service.

   ```sh
   cd anthos-aws
   anthos-gke aws management get-credentials
   ```

   > [!NOTE]
   > **Note:** `anthos-gke` uses the credentials you specified when you ran `aws configure`.

   <br />

2. Run `kubectl get events`.

   ```bash
   env HTTPS_PROXY=http://localhost:8118 \
     kubectl get events
   ```

The output includes information, warning, and errors related to from your
management service.

## Creating the workload identity webhook

> [!IMPORTANT]
> **Important:** The workload identity webhook is not supported on cluster versions 1.22 and higher. After you upgrade to 1.22, your applications that depend on the webhook might not be able to receive credentials.

This section is for [cluster administrators](https://docs.cloud.google.com/kubernetes-engine/multi-cloud/docs/aws/previous-generation/how-to/oidc.md.txt#personas).

To provide workload identity credentials to your workloads with no additional
configuration, you can optionally create a webhook on your user clusters. This
webhook intercepts Pod creation requests and then makes the following AWS IAM
information available as environment variables to the Pod:

- `AWS_ROLE_ARN`: the Amazon Resource Name (ARN) of the IAM role
- `aws-iam-token`: the token exchanged for AWS IAM credentials
- `AWS_WEB_IDENTITY_TOKEN_FILE`: the path where the token is stored

With these variables, your workloads can call the AWS command-line tool or
SDK can access the resources granted to the AWS role.

Creating the webhook is optional. If you decide not to create the webhook, you
need to set the environment variables listed previously in the Pod. For
information about not using a webhook, see
[Applying credentials without the webhook](https://docs.cloud.google.com/kubernetes-engine/multi-cloud/docs/aws/previous-generation/how-to/workload-identity-aws.md.txt#deploy-without-webhook).

### Create YAML files for the webhook

To deploy the webhook, perform the following steps:

1. From your `anthos-aws` directory, use
   `anthos-gke` to switch context to your management service.

   ```sh
   cd anthos-aws
   anthos-gke aws management get-credentials
   ```

   > [!NOTE]
   > **Note:** `anthos-gke` uses the credentials you specified when you ran `aws configure`.

   <br />

2. Get the user cluster name with `kubectl`:

   ```bash
   env HTTPS_PROXY=http://localhost:8118 \
     kubectl get awscluster
   ```

   `kubectl` lists all your user clusters. Choose the user cluster you created
   with [workload identity enabled](https://docs.cloud.google.com/kubernetes-engine/multi-cloud/docs/aws/previous-generation/how-to/workload-identity-aws.md.txt#create-cluster).
3. Set the cluster's name in an environment variable.

   ```bash
   CLUSTER_NAME=CLUSTER_NAME
   ```

   Replace `CLUSTER_NAME` with the name of your cluster. For
   example, `cluster-0`.
4. Set environment variables for the workload identity Pod image and namespace.

   ```bash
   IDENTITY_IMAGE=amazon/amazon-eks-pod-identity-webhook:ed8c41f

   WEBHOOK_NAMESPACE=workload-identity-webhook
   ```
5. Generate the webhook YAML manifest in a file named `aws-webhook.yaml` by
   performing the following steps:

   ```bash
   env HTTPS_PROXY=http://localhost:8118 \
     anthos-gke aws clusters get-credentials ${CLUSTER_NAME}

   CLUSTER_CA=$(env HTTPS_PROXY=http://localhost:8118 \
     kubectl config view --raw -o json  | jq -r '.clusters[] | select(.name == "'$(kubectl config current-context)'") | .cluster."certificate-authority-data"')

   cat << EOF > aws-webhook.yaml
   apiVersion: v1
   kind: Namespace
   metadata:
     name: ${WEBHOOK_NAMESPACE}
   ---
   apiVersion: v1
   kind: ServiceAccount
   metadata:
     name: pod-identity-webhook
     namespace: ${WEBHOOK_NAMESPACE}
   ---
   apiVersion: rbac.authorization.k8s.io/v1
   kind: Role
   metadata:
     name: pod-identity-webhook
     namespace: ${WEBHOOK_NAMESPACE}
   rules:
     - apiGroups: ['']
       resources: ['secrets']
       verbs: ['create']
     - apiGroups: ['']
       resources: ['secrets']
       verbs: ['get', 'update', 'patch']
       resourceNames:
         - pod-identity-webhook
   ---
   apiVersion: rbac.authorization.k8s.io/v1
   kind: RoleBinding
   metadata:
     name: pod-identity-webhook
     namespace: ${WEBHOOK_NAMESPACE}
   roleRef:
     apiGroup: rbac.authorization.k8s.io
     kind: Role
     name: pod-identity-webhook
   subjects:
     - kind: ServiceAccount
       name: pod-identity-webhook
       namespace: ${WEBHOOK_NAMESPACE}
   ---
   apiVersion: rbac.authorization.k8s.io/v1
   kind: ClusterRole
   metadata:
     name: pod-identity-webhook
   rules:
     - apiGroups: ['']
       resources: ['serviceaccounts']
       verbs: ['get', 'watch',  'list']
     - apiGroups:  ['certificates.k8s.io']
       resources: ['certificatesigningrequests']
       verbs:  ['create', 'get', 'list', 'watch']
   ---
   apiVersion: rbac.authorization.k8s.io/v1
   kind: ClusterRoleBinding
   metadata:
     name: pod-identity-webhook
   roleRef:
     apiGroup: rbac.authorization.k8s.io
     kind: ClusterRole
     name: pod-identity-webhook
   subjects:
     - kind: ServiceAccount
       name: pod-identity-webhook
       namespace: ${WEBHOOK_NAMESPACE}
   ---
   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: pod-identity-webhook
     namespace: ${WEBHOOK_NAMESPACE}
   spec:
     replicas: 1
     selector:
       matchLabels:
         app: pod-identity-webhook
     template:
       metadata:
         labels:
           app: pod-identity-webhook
       spec:
         serviceAccountName: pod-identity-webhook
         containers:
           - name: pod-identity-webhook
             image: ${IDENTITY_IMAGE}
             imagePullPolicy: Always
             command:
               - /webhook
               - --in-cluster
               - --namespace=${WEBHOOK_NAMESPACE}
               - --service-name=pod-identity-webhook
               - --tls-secret=pod-identity-webhook
               - --annotation-prefix=eks.amazonaws.com
               - --token-audience=sts.amazonaws.com
               - --logtostderr
             volumeMounts:
               - name: webhook-certs
                 mountPath: /var/run/app/certs
                 readOnly: false
         volumes:
           - name: webhook-certs
             emptyDir: {}
   ---
   apiVersion: v1
   kind: Service
   metadata:
     name: pod-identity-webhook
     namespace: ${WEBHOOK_NAMESPACE}
     annotations:
       prometheus.io/port: '443'
       prometheus.io/scheme: https
       prometheus.io/scrape: 'true'
   spec:
     ports:
       - port: 443
         targetPort: 443
     selector:
       app: pod-identity-webhook
   ---
   apiVersion: admissionregistration.k8s.io/v1
   kind: MutatingWebhookConfiguration
   metadata:
     name: pod-identity-webhook
     namespace: ${WEBHOOK_NAMESPACE}
   webhooks:
     - name: pod-identity-webhook.amazonaws.com
       failurePolicy: Ignore
       sideEffects: 'None'
       admissionReviewVersions: ['v1beta1']
       clientConfig:
         service:
           name: pod-identity-webhook
           namespace: ${WEBHOOK_NAMESPACE}
           path: /mutate
         caBundle: ${CLUSTER_CA}
       rules:
         - operations: ['CREATE']
           apiGroups: ['']
           apiVersions: ['v1']
           resources: ['pods']
   EOF
   ```

   The contents of `aws-webhook.yaml` are ready to apply to your cluster.

### Apply the webhook to your user cluster

To apply the webhook to your user cluster, perform the following steps.

1. Apply the `aws-webhook.yaml` file to your user cluster.

       env HTTPS_PROXY=http://localhost:8118 \
         kubectl apply -f aws-webhook.yaml

2. When you apply the manifest, the webhook Pod generates Kubernetes
   certificate signing requests (CSR). Approve all requests from
   `system:serviceaccount:${WEBHOOK_NAMESPACE}:pod-identity-webhook`
   with `kubectl certificate approve`.

       env HTTPS_PROXY=http://localhost:8118 \
         kubectl certificate approve $(env HTTPS_PROXY=http://localhost:8118 \ &&\
         kubectl get csr -o \
           jsonpath="{.items[?(@.spec.username==\"system:serviceaccount:${WEBHOOK_NAMESPACE}:pod-identity-webhook\")].metadata.name}")

3. Verify that there are no remaining unapproved CSRs.

   Use `kubectl get csr` to check that all CSRs from the requestor
   `system:serviceaccount:${WEBHOOK_NAMESPACE}:pod-identity-webhook` are
   approved:

       env HTTPS_PROXY=http://localhost:8118 \
         kubectl get csr

   Response:

       NAME        AGE   REQUESTOR                                            CONDITION
       csr-mxrt8   10s   system:serviceaccount:default:pod-identity-webhook   Approved,Issued

## Configuring the AWS OIDC provider

This section is for [cluster administrators](https://docs.cloud.google.com/kubernetes-engine/multi-cloud/docs/aws/previous-generation/how-to/oidc.md.txt#personas).

To create an
[OIDC provider at AWS](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_providers_create_oidc_verify-thumbprint.html),
AWS requires an intermediate Certificate Authority (CA)
or server certificate thumbprint. Your OIDC discovery credentials are stored on
`storage.googleapis.com`, with a certificate signed by an intermediate CA named
[`GTS CA 1C3`](https://pki.goog/repository/). The SHA-1 thumbprint
of its intermediate CA `GTS CA 1C3` is `08745487E891C19E3078C1F2A07E452950EF36F6`.

To register your OIDC discovery bucket as an OIDC provider with AWS, perform
the following steps:

1. From your `anthos-aws` directory, use
   `anthos-gke` to switch context to your management service.

   ```sh
   cd anthos-aws
   anthos-gke aws management get-credentials
   ```

   > [!NOTE]
   > **Note:** `anthos-gke` uses the credentials you specified when you ran `aws configure`.

   <br />

2. Save the OIDC issuer URL, issuer host path, and Cloud Storage thumbprint
   in environment variables.

       ISSUER_URL=$(env HTTPS_PROXY=http://localhost:8118 \
         kubectl get awscluster ${CLUSTER_NAME} -o jsonpath='{.status.workloadIdentityInfo.issuerURL}')
       ISSUER_HOSTPATH=${ISSUER_URL#"https://"}
       CA_THUMBPRINT=08745487E891C19E3078C1F2A07E452950EF36F6

3. Use the `aws` command-line tool to create an OIDC provider on AWS.

       aws iam create-open-id-connect-provider \
         --url ${ISSUER_URL} \
         --thumbprint-list ${CA_THUMBPRINT} \
         --client-id-list sts.amazonaws.com

### Update the thumbprint

If Google rotates the CA for `storage.googleapis.com`, run the
following commands:

1. Copy the updated certificate thumbprint, `08745487E891C19E3078C1F2A07E452950EF36F6`.

2. Follow the instructions for the
   [`aws iam update-open-id-connect-provider-thumbprint`](https://awscli.amazonaws.com/v2/documentation/api/latest/reference/iam/update-open-id-connect-provider-thumbprint.html)
   command. Use `storage.googleapis.com` as the target hostname and
   `08745487E891C19E3078C1F2A07E452950EF36F6` as the thumbprint.

## Creating AWS IAM roles and policies

This section is for [cluster administrators](https://docs.cloud.google.com/kubernetes-engine/multi-cloud/docs/aws/previous-generation/how-to/oidc.md.txt#personas).

Create an AWS IAM role to bind to a Kubernetes service account. The
IAM role has permissions for `sts:AssumeRoleWithWebIdentity`.

To create the role, perform the following steps:

1. Find or create an
   [AWS IAM policy](https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies.html)
   that grants the necessary permissions for your workloads.

   You need the policy's Amazon resource name (ARN)
   [AWS IAM policy](https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_identifiers.html#identifiers-arns).
   For example, `arn:aws:iam::aws:policy/AmazonS3ReadOnlyAccess`.
2. Set environment variables with your authentication information.

   ```bash
   KSA_NAME=KUBERNETES_SERVICE_ACCOUNT
   WORKLOAD_NAMESPACE=WORKLOAD_IDENTITY_NAMESPACE

   AWS_ROLE_NAME=AWS_ROLE_NAME
   AWS_POLICY=EXISTING_AWS_POLICY
   ```

   Replace the following:
   - <var translate="no">KUBERNETES_SERVICE_ACCOUNT</var>: the name of the new Kubernetes service account
   - <var translate="no">WORKLOAD_IDENTITY_NAMESPACE</var>: the name of the namespace where workloads run
   - <var translate="no">AWS_ROLE_NAME</var>: the name for a new AWS role for your workloads
   - <var translate="no">EXISTING_AWS_POLICY</var>: the Amazon resource name (ARN) of an existing [AWS IAM policy](https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_identifiers.html#identifiers-arns) For example, `arn:aws:iam::aws:policy/AmazonS3ReadOnlyAccess`.
3. From your `anthos-aws` directory, use
   `anthos-gke` to switch context to your management service.

   ```sh
   cd anthos-aws
   anthos-gke aws management get-credentials
   ```

   > [!NOTE]
   > **Note:** `anthos-gke` uses the credentials you specified when you ran `aws configure`.

   <br />

4. Create an AWS IAM policy that allows your user cluster to
   assume temporary security credentials with the
   [AWS Security Token Service](https://docs.aws.amazon.com/STS/latest/APIReference/welcome.html):

       CLUSTER_ID=$(env HTTPS_PROXY=http://localhost:8118 \
         kubectl get awscluster ${CLUSTER_NAME} -o jsonpath='{.status.clusterID}')

       # Get the ID Provider ARN
       PROVIDER_ARN=$(aws iam list-open-id-connect-providers  \
       | jq '.OpenIDConnectProviderList' \
       | jq ".[] | select(.Arn |  contains(\"${CLUSTER_ID}\"))"   \
       | jq  '.Arn' | tr -d '"')

       # Create AWS role and policy
       cat > irp-trust-policy.json << EOF
       {
         "Version": "2012-10-17",
         "Statement": [
           {
             "Effect": "Allow",
             "Principal": {
               "Federated": "${PROVIDER_ARN}"
             },
             "Action": "sts:AssumeRoleWithWebIdentity",
             "Condition": {
               "StringEquals": {
                 "${ISSUER_HOSTPATH}:sub": "system:serviceaccount:${WORKLOAD_NAMESPACE}:${KSA_NAME}"
               }
             }
           }
         ]
       }
       EOF

   > [!NOTE]
   > **Note:** The value `"Version": "2012-10-17"` is the AWS policy schema version. Don't change this value.

5. To create an AWS IAM role with this policy and attach your existing
   policy to the role, perform the following commands:

   ```bash
   aws iam create-role \
     --role-name ${AWS_ROLE_NAME} \
     --assume-role-policy-document file://irp-trust-policy.json
   aws iam update-assume-role-policy \
     --role-name ${AWS_ROLE_NAME} \
     --policy-document file://irp-trust-policy.json
   aws iam attach-role-policy \
     --role-name ${AWS_ROLE_NAME} \
     --policy-arn ${AWS_POLICY}
   ```

   The `aws` command-line tool confirms that the policy is attached to your
   role.

## Creating Kubernetes service accounts for workloads

This section is for [developers or cluster administrators](https://docs.cloud.google.com/kubernetes-engine/multi-cloud/docs/aws/previous-generation/how-to/oidc.md.txt#personas).

To create Kubernetes service accounts bound to the AWS IAM role that was
specified previously, perform the following steps:

1. From your `anthos-aws` directory, use
   `anthos-gke` to switch context to your user cluster.

   ```sh
   cd anthos-aws
   env HTTPS_PROXY=http://localhost:8118 \
     anthos-gke aws clusters get-credentials CLUSTER_NAME
   ```
   Replace <var translate="no">CLUSTER_NAME</var> with your user cluster name.

   <br />

2. Create the Kubernetes service account by running the following commands:

       S3_ROLE_ARN=$(aws iam get-role \
         --role-name AWS_ROLE_NAME \
         --query Role.Arn --output text)

       cat << EOF  > k8s-service-account.yaml
       apiVersion: v1
       kind: ServiceAccount
       metadata:
         name: ${KSA_NAME}
         namespace: WORKLOAD_IDENTITY_NAMESPACE
       EOF

       env HTTPS_PROXY=http://localhost:8118 \
       kubectl apply -f k8s-service-account.yaml

       env HTTPS_PROXY=http://localhost:8118 \
       kubectl annotate sa --namespace ${WORKLOAD_NAMESPACE} ${KSA_NAME} eks.amazonaws.com/role-arn=${S3_ROLE_ARN}

   Replace the following:
   - `AWS_ROLE_NAME`: the name of the AWS IAM role to apply to your workloads
   - `WORKLOAD_IDENTITY_NAMESPACE`: the name of the namespace where workloads run

## Applying credentials to your Pods

This section is for [developers](https://docs.cloud.google.com/kubernetes-engine/multi-cloud/docs/aws/previous-generation/how-to/oidc.md.txt#personas).

This section assumes that you have deployed the workload identity
[webhook](https://docs.cloud.google.com/kubernetes-engine/multi-cloud/docs/aws/previous-generation/how-to/workload-identity-aws.md.txt#create_the_workload_identity_webhook).
If you haven't deployed the webhook, skip to
[Applying credentials without the webhook](https://docs.cloud.google.com/kubernetes-engine/multi-cloud/docs/aws/previous-generation/how-to/workload-identity-aws.md.txt#deploy-without-webhook).

### Apply credentials with the webhook

This section describes how to configure your Pods to read credentials made
available by the webhook.

#### Add the service account to the Pod

To use workload identity with a workload, add the Kubernetes service account
to the following fields:

- For a Deployment: `spec.template.spec.serviceAccountName`
- For a Pod: `spec.serviceAccount`

The following Pod manifest launches a base CentOS image and
contains the `spec.serviceAccount` field.

```yaml
apiVersion: v1
kind: Pod
metadata:
  name: sample-centos-pod
  namespace: WORKLOAD_IDENTITY_NAMESPACE
spec:
  containers:
  - command:
    - /bin/bash
    - -ec
    - while :; do echo '.'; sleep 500 ; done
    image: amazon/aws-cli
    name: centos
  serviceAccount: KUBERNETES_SERVICE_ACCOUNT
```

Replace the following:

- `WORKLOAD_IDENTITY_NAMESPACE`: the name of the namespace where workloads run
- `KUBERNETES_SERVICE_ACCOUNT`: the name of the Kubernetes service account you created previously

#### Check if Pods have the environment variables set

To check if Pods have the environment variables set, run the following
command to get the Pod's information:

```bash
kubectl get pod --namespace WORKLOAD_IDENTITY_NAMESPACE POD_NAME -o yaml
```

Replace the following:

- `WORKLOAD_IDENTITY_NAMESPACE`: the name of the namespace where workloads run
- `POD_NAME`: the name of the Pod to check

The output contains the environment variable values in
`spec.containers.command.env` and the mount point for the AWS IAM token. An
example Pod manifest follows.

    apiVersion: v1
    kind: Pod
    metadata:
      ...
    spec:
      containers:
      - command:
        - /bin/bash
        - -ec
        - while :; do echo '.'; sleep 500 ; done
        env:
        - name: AWS_ROLE_ARN
          value: arn:aws:iam::1234567890:role/my-example-workload-role-1
        - name: AWS_WEB_IDENTITY_TOKEN_FILE
          value: /var/run/secrets/eks.amazonaws.com/serviceaccount/token
        image: amazon/aws-cli
        imagePullPolicy: IfNotPresent
        name: centos
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
          name: my-k8s-serviceaccount-token-d4nz4
          readOnly: true
        - mountPath: /var/run/secrets/eks.amazonaws.com/serviceaccount
          name: aws-iam-token
          readOnly: true
      serviceAccount: my-k8s-serviceaccount
      serviceAccountName: my-k8s-serviceaccount
      volumes:
      - name: aws-iam-token
        projected:
          defaultMode: 420
          sources:
          - serviceAccountToken:
              audience: sts.amazonaws.com
              expirationSeconds: 86400
              path: token
      - name: my-k8s-serviceaccount-token-d4nz4
        secret:
          defaultMode: 420
          secretName: my-k8s-serviceaccount-token-d4nz4
       ...
    status:
      ...

### Apply credentials without the webhook

If you do not deploy the workload identity
[webhook](https://docs.cloud.google.com/kubernetes-engine/multi-cloud/docs/aws/previous-generation/how-to/workload-identity-aws.md.txt#create_the_workload_identity_webhook), you need to do the following:

- [Check if the following environment variables are set](https://docs.cloud.google.com/kubernetes-engine/multi-cloud/docs/aws/previous-generation/how-to/workload-identity-aws.md.txt#check-environment-variables):

  - `AWS_ROLE_ARN`: the Amazon Resource Name (ARN) of the IAM role
  - `AWS_WEB_IDENTITY_TOKEN_FILE`: the path where the token is stored
- Create a mount point for the IAM token (`aws-iam-token`) and the
  [service account associated with the AWS IAM role](https://docs.cloud.google.com/kubernetes-engine/multi-cloud/docs/aws/previous-generation/how-to/prerequisites.md.txt#create-service-accounts)

#### Create a Pod with credentials for workload identity

To create a Pod that includes the necessary credentials for workload identity,
perform the following steps:

1. Copy the following Pod manifest into a file named
   `sample-pod-no-webhook.yaml`. The configuration launches a base CentOS image
   with the necessary credentials.

   ```yaml
   apiVersion: v1
   kind: Pod
   metadata:
     name: sample-centos-pod-no-webhook
     namespace: WORKLOAD_IDENTITY_NAMESPACE
   spec:
     containers:
     - command:
       - /bin/bash
       - -ec
       - while :; do echo '.'; sleep 500 ; done
       image: centos:7
       name: centos
       env:
       - name: AWS_ROLE_ARN
         value: IAM_ROLE_ARN
       - name: AWS_WEB_IDENTITY_TOKEN_FILE
         value: /var/run/secrets/eks.amazonaws.com/serviceaccount/token
       volumeMounts:
       - mountPath: /var/run/secrets/eks.amazonaws.com/serviceaccount
         name: aws-iam-token
         readOnly: true
     volumes:
     - name: aws-iam-token
       projected:
         defaultMode: 420
         sources:
         - serviceAccountToken:
             audience: sts.amazonaws.com
             expirationSeconds: 86400
             path: token
     serviceAccount: KUBERNETES_SERVICE_ACCOUNT
   ```

   Replace the following:
   - `WORKLOAD_IDENTITY_NAMESPACE`: the name of the namespace where workloads run.
   - `IAM_ROLE_ARN`: the ARN of the IAM role granted to the Pod. For example, `arn:aws:iam::aws:policy/AmazonS3ReadOnlyAccess`.
   - `KUBERNETES_SERVICE_ACCOUNT`: the name of the Kubernetes service account you created previously.
2. Apply the Pod manifest to your cluster by using `kubectl`:

   ```bash
   env HTTPS_PROXY=http://localhost:8118 \
     kubectl apply -f sample-pod-no-webhook.yaml
   ```

#### Check if Pods can access AWS resources

The following procedure describes how to check whether the Pod has received the
credentials necessary for workload identity to function.

To complete the steps, you need to have the following:

- `bash` shell access to the container; most production images don't have a
  shell available. The following example shows you how to use the Pod
  [specified in the preceding section](https://docs.cloud.google.com/kubernetes-engine/multi-cloud/docs/aws/previous-generation/how-to/workload-identity-aws.md.txt#deploy-without-webhook)
  to access AWS S3.

- Your Pod needs to have outbound access to the internet to download the AWS
  command-line interface.

To check if the Pod can access an S3 bucket, perform the following steps:

1. Use `kubectl exec` to launch an interactive bash shell on the Pod
   `sample-centos-pod-no-webhook`:

   ```bash
   env HTTPS_PROXY=http://localhost:8118 \
     kubectl exec -it --namespace ${WORKLOAD_NAMESPACE} sample-centos-pod-no-webhook -- bash
   ```

   Your terminal opens the bash shell on the Pod.
2. Check the AWS IAM permissions and credentials by using the `aws` tool:

   ```bash
   aws sts assume-role-with-web-identity \
    --role-arn ${AWS_ROLE_ARN} \
    --role-session-name mh9test \
    --web-identity-token file:///var/run/secrets/eks.amazonaws.com/serviceaccount/token \
    --duration-seconds 1000
   ```

   The `aws` tool prints credentials information similar to the following:

   ```json
   {
       "AssumedRoleUser": {
           "AssumedRoleId": "AROAR2ZZZLEXVSDCDJ37N:mh9test",
           "Arn": "arn:aws:sts::126285863215:assumed-role/my-example-workload-role-1/mh9test"
       },
       "Audience": "sts.amazonaws.com",
       "Provider": "arn:aws:iam::126285863215:oidc-provider/storage.googleapis.com/gke-issuer-cec6c353",
       "SubjectFromWebIdentityToken": "system:serviceaccount:default:my-s3-reader-ksa",
       "Credentials": {
           "SecretAccessKey": "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY",
           "SessionToken": "MY_TOKEN",
           "Expiration": "2020-08-14T22:46:36Z",
           "AccessKeyId": "AKIAIOSFODNN7EXAMPLE"
       }
   }
   ```

   If you see the following message, check that the bucket is publicly
   accessible: `An error occurred (InvalidIdentityToken) when calling the
   AssumeRoleWithWebIdentity operation: Couldn't retrieve verification key
   from your identity provider, please reference AssumeRoleWithWebIdentity
   documentation for requirements`

## Upgrading the webhook

If you created a Kubernetes 1.18 or lower cluster with workload identity enabled
and the workload identity webhook version `release-0.2.2-gke.0`, you must
upgrade the webhook before upgrading to Kubernetes 1.19.

To upgrade the webhook, perform the following steps:

1. Confirm the webhook is installed by running the following commands:

   ```bash
   env HTTPS_PROXY=http://localhost:8118 \
     kubectl get MutatingWebhookConfiguration
   ```

   If your cluster has the webhook deployed, the output includes the following:

       NAME                   WEBHOOKS   AGE
       pod-identity-webhook   1          11m

   If the webhook is not deployed on your cluster, you can skip the following
   steps.
2. If you saved the `aws-webhook.yaml` file, you can delete the manifest. If
   you don't have this file available, you can delete the webhook's components
   manually. Choose from file or components below.

   ### File

   If you still have the
   [`aws-webhook.yaml` file](https://docs.cloud.google.com/kubernetes-engine/multi-cloud/docs/aws/previous-generation/how-to/workload-identity-aws.md.txt#create_yaml_files_for_the_webhook), run the
   following command to delete the webhook:

   ```bash
   env HTTPS_PROXY=http://localhost:8118 \
     kubectl delete -f aws-webhook.yaml
   ```

   ### Components

   To delete the webhook's components manually, run the following commands:

   ```bash
   env HTTPS_PROXY=http://localhost:8118 \
      kubectl delete namespace WEBHOOK_NAMESPACE
   env HTTPS_PROXY=http://localhost:8118 \
      kubectl delete clusterrole pod-identity-webhook
   env HTTPS_PROXY=http://localhost:8118 \
      kubectl delete clusterrolebinding pod-identity-webhook
   env HTTPS_PROXY=http://localhost:8118 \
      kubectl delete mutatingwebhookconfiguration pod-identity-webhook
   ```

   Replace <var translate="no">WEBHOOK_NAMESPACE</var> with the namespace where you installed
   the workload identity webhook. For example---
   `workload-identity-webhook`.
3. Check if you have any remaining certificate signing requests (CSRs) by run the
   following command:

   ```bash
   env HTTPS_PROXY=http://localhost:8118 \
     kubectl get csr |grep pod-identity-webhook
   ```

   If the output is blank, skip to the next step. If there are any remaining
   CSRs, the `kubectl` command will list existing CSRs. To remove the CSRs, run
   the following command:

   ```bash
   env HTTPS_PROXY=http://localhost:8118 \
     kubectl delete csr $(kubectl get csr -o \
     jsonpath="{.items[?(@.spec.username==\"system:serviceaccount:WEBHOOK_NAMESPACE:pod-identity-webhook\")].metadata.name}")
   ```

   Replace <var translate="no">WEBHOOK_NAMESPACE</var> with the namespace where you installed
   the workload identity webhook. For example--- `workload-identity-webhook`.
4. Follow the steps in
   [Create the webhook](https://docs.cloud.google.com/kubernetes-engine/multi-cloud/docs/aws/previous-generation/how-to/workload-identity-aws.md.txt#create_the_workload_identity_webhook) to deploy the
   new webhook version.

   After you deploy the new webhook version, you need to restart the Pods that
   use the webhook. You can restart your Pods by
   [Upgrading a user cluster](https://docs.cloud.google.com/kubernetes-engine/multi-cloud/docs/aws/previous-generation/how-to/upgrading.md.txt#user-clusters).

## Cleaning up

This section shows you how to remove resources that you created earlier in
this document.

### Clean up the service account and its associated IAM role

To delete the service account and its associated IAM
role, perform the following steps:

1. Clean up the service account:

   ```bash
   env HTTPS_PROXY=http://localhost:8118 \
     kubectl delete sa KUBERNETES_SERVICE_ACCOUNT --namespace WORKLOAD_IDENTITY_NAMESPACE
   ```

   Replace the following:
   - `KUBERNETES_SERVICE_ACCOUNT`: the name of the new Kubernetes service account
   - `WORKLOAD_IDENTITY_NAMESPACE`: the name of the namespace where workloads run
2. Clean up the AWS IAM role. Choose from one of the following:

   - Delete the AWS IAM role with the
     [AWS console](https://console.aws.amazon.com/iam/home?providers#/roles).

   - Delete the role with the AWS command-line tool using the following
     commands:

     ```bash
     aws iam  detach-role-policy \
       --role-name=${AWS_ROLE_NAME} \
       --policy-arn=${AWS_POLICY}
     aws iam delete-role --role-name=${AWS_ROLE_NAME}
     ```

### Delete your user cluster

To delete your user cluster, perform the steps in
[Uninstalling GKE on AWS](https://docs.cloud.google.com/kubernetes-engine/multi-cloud/docs/aws/previous-generation/how-to/uninstalling.md.txt#deleting_user_clusters).

### Clean up the AWS OIDC provider

After the user cluster is deleted, unregister and delete the OIDC provider on
AWS by using either the following `bash` shell command or the AWS console.

1. From your `anthos-aws` directory, use
   `anthos-gke` to switch context to your management service.

   ```sh
   cd anthos-aws
   anthos-gke aws management get-credentials
   ```

   > [!NOTE]
   > **Note:** `anthos-gke` uses the credentials you specified when you ran `aws configure`.

   <br />

2. Delete the role with the AWS command-line tool with the following
   commands:

   ```bash
   CLUSTER_ID=$(env HTTPS_PROXY=http://localhost:8118 \
     kubectl get awscluster ${CLUSTER_NAME} -o jsonpath='{.status.clusterID}')

   PROVIDER_ARN=$(aws iam list-open-id-connect-providers  \
   | jq '.OpenIDConnectProviderList' \
   | jq ".[] | select(.Arn |  contains(\"${CLUSTER_ID}\"))"   \
   | jq  '.Arn' | tr -d '"')

   aws iam delete-open-id-connect-provider \
     --open-id-connect-provider-arn=${PROVIDER_ARN}
   ```

   You receive confirmation that the AWS OIDC provider is deleted.

## What's next

- Learn about AWS [IAM Roles for Service Accounts (IRSA)](https://aws.amazon.com/blogs/opensource/introducing-fine-grained-iam-roles-service-accounts/), which GKE on AWS uses for workload identity.
- Learn about [Using workload identity with Google Cloud](https://docs.cloud.google.com/kubernetes-engine/multi-cloud/docs/aws/previous-generation/how-to/workload-identity-gcp.md.txt).