This topic describes how to rotate security credentials for your
GKE on AWS management service and user clusters. For more
information on the security features of GKE on AWS, see
[Security](https://docs.cloud.google.com/kubernetes-engine/multi-cloud/docs/aws/previous-generation/concepts/security.md.txt).

## Before you begin

To complete the steps on this page, you must have access to the directory with
your GKE on AWS configuration.

## Management service certificates

This section describes how to rotate certificates for your management service.

### Management certificate authorities

This section explains how to rotate Certificate authority (CA) signing
certificates for GKE on AWS components.

#### Management API server root CA

To rotate the API server root CA, perform the following steps:


1. Change to the directory with your GKE on AWS configuration. You created this directory when [Installing the management service](https://docs.cloud.google.com/kubernetes-engine/multi-cloud/docs/aws/previous-generation/how-to/installing-management.md.txt).

   ```sh
   cd anthos-aws
   ```
2. Open your `anthos-gke.status.yaml` in a text editor.
3. Remove all the values under the key `certificateAuthority`. This includes `encryptedPrivateKey.kmsKeyARN`, `encryptedPrivateKey.value`, and `encryptedPrivateKey.certificate`.
4. Run `anthos-gke aws management init` to update the
   `anthos-gke.status.yaml` file.

   ```bash
    anthos-gke aws management init
   ```
5. Run `anthos-gke aws management apply` to update the management service.

   ```bash
    anthos-gke aws management apply
   ```

<br />

> [!NOTE]
> **Note:** Performing these steps invalidates existing management credentials. You must regenerate credentials.

1. From your `anthos-aws` directory, use `anthos-gke` to switch context to your management service.

   ```sh
   cd anthos-aws
   anthos-gke aws management get-credentials
   ```

   > [!NOTE]
   > **Note:** `anthos-gke` uses the credentials you specified when you ran `aws configure`.

#### Other management CAs

This section describes how you can rotate all of the following CAs:

- Authentication webhook CA
- Etcd CA
- Service Account Signer CA

You can rotate these CAs with one of the following methods:

- Remove the `certificateAuthority` section from your `anthos-gke.status.yaml`.


  1. Change to the directory with your GKE on AWS configuration. You created this directory when [Installing the management service](https://docs.cloud.google.com/kubernetes-engine/multi-cloud/docs/aws/previous-generation/how-to/installing-management.md.txt).

     ```sh
     cd anthos-aws
     ```
  2. Open your `anthos-gke.status.yaml` in a text editor.
  3. Remove all the values under the key `certificateAuthority`. This includes `encryptedPrivateKey.kmsKeyARN`, `encryptedPrivateKey.value`, and `encryptedPrivateKey.certificate`.
  4. Run `anthos-gke aws management init` to update the
     `anthos-gke.status.yaml` file.

     ```bash
      anthos-gke aws management init
     ```
  5. Run `anthos-gke aws management apply` to update the management service.

     ```bash
      anthos-gke aws management apply
     ```

  <br />

- If a new GKE on AWS version is available,
  [upgrade your GKE on AWS management service](https://docs.cloud.google.com/kubernetes-engine/multi-cloud/docs/aws/previous-generation/how-to/upgrading-management).

- Recreate the management service EC2 instance.


  1. From your `anthos-aws` directory, use `terraform` to get the ID of your management service.

     ```sh
     cd anthos-aws
     terraform output cluster_id
     ```
     The output includes your management service ID. In the example below, the ID is `gke-12345abc`.

         terraform output cluster_id
         gke-12345abc

  2. Open the [AWS EC2 Console](https://console.aws.amazon.com/ec2/v2/home).
  3. Click **Instances**
  4. Find the instance named `cluster-id-management-0`.
  5. Select **Actions -\> Instance state -\> terminate** to remove the instance. EC2 automatically creates a new instance with the same EBS volume.

  <br />

### Management TLS Client / Server keys and certificates

To rotate the TLS client / server keys and certificates for your management service,
you recreate your management service instance. To recreate the instance,
perform the following steps:


1. From your `anthos-aws` directory, use `terraform` to get the ID of your management service.

   ```sh
   cd anthos-aws
   terraform output cluster_id
   ```
   The output includes your management service ID. In the example below, the ID is `gke-12345abc`.

       terraform output cluster_id
       gke-12345abc

2. Open the [AWS EC2 Console](https://console.aws.amazon.com/ec2/v2/home).
3. Click **Instances**
4. Find the instance named `cluster-id-management-0`.
5. Select **Actions -\> Instance state -\> terminate** to remove the instance. EC2 automatically creates a new instance with the same EBS volume.

<br />

### Google Cloud service accounts

#### Management service service accounts

To rotate the
[Google Cloud service accounts](https://docs.cloud.google.com/kubernetes-engine/multi-cloud/docs/aws/previous-generation/how-to/prerequisites.md.txt#create-service-accounts)
for your management service, perform the following steps.

1. Create new service accounts and download service account keys following the steps in [Prerequisites](https://docs.cloud.google.com/kubernetes-engine/multi-cloud/docs/aws/previous-generation/how-to/prerequisites.md.txt#create-service-accounts)
2. Change to the directory with your GKE on AWS configuration. You created this directory when [Installing the management service](https://docs.cloud.google.com/kubernetes-engine/multi-cloud/docs/aws/previous-generation/how-to/installing-management.md.txt).

   ```sh
   cd anthos-aws
   ```
3. If you downloaded keys to a different path, open your `anthos-gke.yaml` file
   in a text editor. Change the value of
   `.spec.googleCloud.serviceAccountKeys.managementService`,
   `.status.googleCloud.serviceAccountKeys.connectAgent`, and
   `.spec.googleCloud.serviceAccountKeys.node`
   to the new paths.

   ```yaml
   apiVersion: multicloud.cluster.gke.io/v1
   kind: AWSManagementService
   metadata:
    name: management
   spec:
    googleCloud:
      serviceAccountKeys:
        managementService: MANAGEMENT_KEY_PATH
        connectAgent: CONNECT_KEY_PATH
        node: NODE_KEY_PATH
    ...
   ```
4. Run `anthos-gke aws management init` to update the
   `anthos-gke.status.yaml` file.

   ```bash
   anthos-gke aws management init
   ```
5. Run `anthos-gke aws management apply` to update the management service.

   ```bash
   anthos-gke aws management apply
   ```

#### User cluster service accounts

To apply these service accounts on your AWSClusters and
AWSNodePools, you must
[upgrade](https://docs.cloud.google.com/kubernetes-engine/multi-cloud/docs/aws/previous-generation/how-to/upgrading.md.txt#user-clusters) or
[delete](https://docs.cloud.google.com/kubernetes-engine/multi-cloud/docs/aws/previous-generation/how-to/uninstalling.md.txt#deleting_user_clusters) then
[re-create](https://docs.cloud.google.com/kubernetes-engine/multi-cloud/docs/aws/previous-generation/how-to/creating-user-cluster.md.txt) them.

## User cluster certificates

This section describes how to rotate certificates for your user clusters.

### User cluster CAs and SSH keys

Most CAs for your user clusters are created when the cluster is created.

When you [delete a user cluster](https://docs.cloud.google.com/kubernetes-engine/multi-cloud/docs/aws/previous-generation/how-to/uninstalling.md.txt),
GKE on AWS rotates the following certificates:

- API server root CA
- API front proxy CA
- etcd CA
- Kubernetes service account signer CA
- Control plane to node SSH key pairs

#### User cluster authentication webhook CA

To rotate the user cluster authentication webhook CA, you edit
your `anthos-gke.status.yaml` file and apply the changes.


1. Change to the directory with your GKE on AWS configuration. You created this directory when [Installing the management service](https://docs.cloud.google.com/kubernetes-engine/multi-cloud/docs/aws/previous-generation/how-to/installing-management.md.txt).

   ```sh
   cd anthos-aws
   ```
2. Open your `anthos-gke.status.yaml` in a text editor.
3. Remove all the values under the key `certificateAuthority`. This includes `encryptedPrivateKey.kmsKeyARN`, `encryptedPrivateKey.value`, and `encryptedPrivateKey.certificate`.
4. Run `anthos-gke aws management init` to update the
   `anthos-gke.status.yaml` file.

   ```bash
    anthos-gke aws management init
   ```
5. Run `anthos-gke aws management apply` to update the management service.

   ```bash
    anthos-gke aws management apply
   ```

<br />

## User cluster TLS key pairs and certificates

GKE on AWS generates TLS key pairs and certificates when creating
an instance. To rotate these pairs, recreate the instance by performing the
following steps for your control plane and node pools.

### Control plane

1. From your `anthos-aws` directory, use `anthos-gke` to switch context to your management service.

   ```sh
   cd anthos-aws
   anthos-gke aws management get-credentials
   ```

   > [!NOTE]
   > **Note:** `anthos-gke` uses the credentials you specified when you ran `aws configure`.

2. Use `kubectl` to get your control plane's AWS EC2 target group from your
   AWSCluster.

   ```bash
   env HTTPS_PROXY=http://localhost:8118 \
     kubectl get awscluster cluster-name \
     -o jsonpath='{.status.targetGroupName}{"\n"}'
   ```

   The output includes the name of your control plane's EC2 target group. For
   example, `gke-123456a7-controlplane`.
3. Open the
   [AWS EC2 Console](https://console.aws.amazon.com/ec2/v2/home).
   Choose **Target Groups** from the left pane.

4. Click the search bar and find your target group. Click the **Name** of
   your target group and then click **Targets**. The list of your control plane
   instances appears.

5. For each instance in the target group, perform the following steps:

   1. Click the instance's **Instance ID**. The AWS EC2 Instances console
      appears.

   2. Click on the **Instance ID**.

   3. Select **Actions -\> Instance state -\> terminate** to remove the instance.
      EC2 automatically creates a new instance with the same EBS volume.

   4. Return to the Target Groups page.

6. After you have terminated all the instances in the group, return to the
   Target Groups page.

7. In the **Registered Targets** section of the page, find the **Status**
   column. Each of your instances should have a **Status** of **Healthy** .
   If any of the instances are healthy, wait several minutes and
   click the Refresh
   () icon.

8. After all instances in the target group are healthy, proceed to the next step.

### Node pools

> [!IMPORTANT]
> **Important:** When you recreate an EC2 instance in a node pool, your workloads are restarted on new nodes. You might experience downtime during an upgrade while workloads restart on a new node. Additionally, if you are running critical workloads, recreate instances one at a time and wait for the replacement to be registered as **Healthy** before recreating others.

To rotate your node pool's TLS certificates:

1. From your `anthos-aws` directory, use `anthos-gke` to switch context to your management service.

   ```sh
   cd anthos-aws
   anthos-gke aws management get-credentials
   ```

   > [!NOTE]
   > **Note:** `anthos-gke` uses the credentials you specified when you ran `aws configure`.

2. Use `kubectl` to get your node pool's AWS EC2 target group from your
   AWSNodePool.

   ```bash
   env HTTPS_PROXY=http://localhost:8118 \
     kubectl get awsnodepool -o jsonpath='{.items[*].status.autoScalingGroupName}{"\n"}'
   ```

   The output includes the name of your node pool's EC2 target group. For
   example, `gke-123456a7-nodepool-abc123`.
3. Open the
   [AWS EC2 Console](https://console.aws.amazon.com/ec2/v2/home).
   Choose **Target Groups** from the left pane.

4. Click the search bar and find your target group. Click the **Name** of
   your target group and then click **Targets**. The list of your control plane
   instances appears.

5. For each instance in the target group, perform the following steps:

   1. Click the instance's **Instance ID**. The AWS EC2 Instances console
      appears.

   2. Click on the **Instance ID**.

   3. Select **Actions -\> Instance state -\> terminate** to remove the instance.
      EC2 automatically creates a new instance with the same EBS volume.

   4. Return to the Target Groups page.

6. After you have terminated all the instances in the group, return to the
   Target Groups page.

7. In the **Registered Targets** section of the page, find the **Status**
   column. Each of your instances should have a **Status** of **Healthy** .
   If any of the instances are healthy, wait several minutes and
   click the Refresh
   () icon.

8. After all instances in the target group are healthy, proceed to the next step.