You can rotate the certificate authority (CA) certificates in a
Google Distributed Cloud admin cluster to renew them before they expire. These CA
certificates are created with the admin cluster, and they issue the
certificates that Kubernetes system components use to authenticate their
communication. You run the rotation with `gkectl`, and then update your
kubeconfig files, which the rotation invalidates. This document applies only to
admin clusters. For user clusters, see
[Rotating user cluster CA certificates](https://docs.cloud.google.com/kubernetes-engine/distributed-cloud/vmware/docs/how-to/ca-rotation.md.txt).

## About admin cluster CA certificates

There are three CA certificates used by the Kubernetes system in an admin
cluster:

- The etcd CA certificate secures communication from the Kubernetes API server
  to the etcd replicas and also communication between etcd replicas. This
  certificate is self-signed.

- The cluster CA certificate secures communication between the Kubernetes API
  server and all internal Kubernetes API clients, for example, the kubelet, the
  controller manager, and the scheduler. This certificate is self-signed.

- The front-proxy CA certificate secures communication with
  [aggregated APIs](https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/apiserver-aggregation/).
  This certificate is self-signed.

You can use `gkectl` to trigger a certificate rotation. During a rotation,
`gkectl` replaces the core system CA certificates for the
admin cluster with newly generated certificates. Then it distributes the new
CA certificates, leaf certificates, and private keys to admin cluster system
components. The rotation happens incrementally, so that system components can
continue to communicate during the rotation. Note, however, that workloads and
nodes are restarted during the rotation.

Without rotation, CA certificates and control-plane certificates will expire
after a period of time from the date the cluster was created.

The expiration period depends on the cluster type and its creation version:

- Advanced clusters:
  - 10-year expiration
- Non-advanced clusters:
  - Created before version 1.5: 10-year expiration
  - Created between versions 1.5 to 1.16: 5-year expiration
  - Created after version 1.16: 30-year expiration

The control plane certificates are automatically rotated during a cluster
upgrade, but the CAs are not automatically rotated. This means a CA rotation
must be performed before the CA's expiration date, in addition to regular
version upgrades.

> [!WARNING]
> **Warning:** A CA certificate rotation revokes the old CA certificates at the end of the operation. This invalidates kubeconfig files that were based on an old certificate. Consequently, the credentials in the kubeconfig file stop working after a CA certificate rotation. This guide includes instructions on how to update your kubeconfig file. The same issue applies to authentication configuration files.

## Limitations

- Note the following limitation with advanced clusters:

  - Version 1.31: CA rotation isn't supported on [advanced clusters](https://docs.cloud.google.com/kubernetes-engine/distributed-cloud/vmware/docs/how-to/admin-cluster-configuration-file-latest.md.txt#enable-advanced-cluster-field).
  - Version 1.32 and higher: CA rotation is supported on advanced clusters but there are some minor differences noted where applicable in this document.
- CA certificate rotation is limited to the etcd, cluster, and front-proxy
  certificates mentioned previously.

- CA certificate rotation is limited to certificates issued automatically by
  Google Distributed Cloud. It does not update certificates issued manually by an
  administrator, even if those certificates are signed by the system CAs.

- CA certificate rotation restarts the Kubernetes API server, other
  control-plane processes, and each node in the admin cluster multiple times.
  Each stage of a rotation progresses similarly to a cluster upgrade. While the
  admin cluster and the user clusters managed by the admin cluster do remain
  operational during a certificate rotation, you should expect that workloads
  in the admin cluster will be restarted and rescheduled. You should also expect
  brief periods of downtime for the admin cluster control plane and user cluster
  control plane.

- You must update the admin cluster kubeconfig file in the middle of a
  certificate rotation and again after the rotation completes. This is because
  the old cluster certificate is revoked, and the credentials in the kubeconfig
  file will no longer work.

- Once initiated, a CA certificate rotation cannot be rolled back.

- A CA certificate rotation might take considerable time to complete, depending
  on the size of the cluster.

- The certificate rotation process can be resumed by re-running the same
  command if it is interrupted. However, you must ensure that there is only one
  rotation command running at a time.

## Start the rotation

To start the certificate rotation, run the following command:

```
gkectl update credentials certificate-authorities rotate \
    --admin-cluster \
    --config ADMIN_CLUSTER_CONFIG \
    --kubeconfig ADMIN_CLUSTER_KUBECONFIG
```

Replace the following:

- <var translate="no">ADMIN_CLUSTER_CONFIG</var>: the path of the admin cluster configuration
  file

- <var translate="no">ADMIN_CLUSTER_KUBECONFIG</var>: the path of the admin cluster kubeconfig
  file

The behavior of the command differs depending on whether advanced cluster
is enabled:

### Not enabled

The `gkectl update credentials certificate-authorities rotate` command starts
and performs the first half of the rotation. The command then pauses to let
you run the next command to update the kubeconfig file.

### Update the kubeconfig file

When the `gkectl update credentials certificate-authorities rotate` command
pauses, update the kubeconfig file for the admin cluster. This places a new
client certificate and a new CA certificate in the kubeconfig file. The old
client certificate is removed from the kubeconfig file, and the old CA
certificate remains in the kubeconfig file.

```
gkectl update credentials certificate-authorities update-kubeconfig \
  --admin-cluster \
  --config ADMIN_CLUSTER_CONFIG \
  --kubeconfig ADMIN_CLUSTER_KUBECONFIG
```

### Continue the rotation

Run the following command to perform the second half of the procedure. The
command doesn't proceed until `gkectl` verifies that the updated kubeconfig
file is in the current directory.

```
gkectl update credentials certificate-authorities rotate \
  --admin-cluster \
  --complete \
  --config ADMIN_CLUSTER_CONFIG \
  --kubeconfig ADMIN_CLUSTER_KUBECONFIG
```

When the rotation is complete, it reports the current CA version.

### Update the kubeconfig file again

After the second half of the rotation completes, update the kubeconfig file
again. This removes the old CA certificate from the kubeconfig file.

```
gkectl update credentials certificate-authorities update-kubeconfig \
  --admin-cluster \
  --config ADMIN_CLUSTER_CONFIG \
  --kubeconfig ADMIN_CLUSTER_KUBECONFIG
```

### Enabled

If advanced cluster is enabled, the `gkectl update credentials
certificate-authorities rotate` command is synchronous. The command outputs
status messages to the admin workstation as the CA rotation progresses.

After the CA is rotated successfully, the command exits and a new kubeconfig
file is automatically generated. The kubeconfig file that you specified in the
command is replaced with a new one. The command output is similar to the
following:

```
Beginning CA rotation with generated CA
...
Successfully rotated CA for admin cluster. The kubeconfig file
"/home/ubuntu/kubeconfig" has been updated.
Done rotating certificate-authorities
```

## Distribute the new kubeconfig file

Distribute the new admin cluster kubeconfig file to all cluster users.