You can rotate the core system certificate authorities (CAs) in a
Google Distributed Cloud user cluster to renew them before they expire. During a
rotation, the admin cluster replaces the user cluster's CAs with newly
generated CAs, and then distributes the certificates and key pairs that the new
CAs issue to system components. The rotation is incremental, so system
components can keep communicating, but workloads and nodes are restarted. To
rotate CAs in an admin cluster, see
[Rotating admin cluster CA certificates](https://docs.cloud.google.com/kubernetes-engine/distributed-cloud/vmware/docs/how-to/admin-ca-rotation.md.txt).

## About user cluster CAs

The admin cluster manages three self-signed system CAs for each user cluster.
Each CA issues certificates, called *leaf certificates*, to individual system
components, which use them to authenticate and encrypt their connections:

- The etcd CA secures communication from the API server to the etcd replicas and traffic between etcd replicas.
- The cluster CA secures communication between the API server and all internal Kubernetes API clients (kubelets, controllers, schedulers).
- The front-proxy CA secures communication with [aggregated APIs](https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/apiserver-aggregation/).

You might also use an
[org CA](https://docs.cloud.google.com/kubernetes-engine/distributed-cloud/vmware/docs/concepts/security.md.txt)
to sign the certificate configured by the `authentication.sni` option, which
serves the Kubernetes API to clients outside the cluster. You manage this CA
and generate the SNI certificate manually. User cluster CA rotation doesn't
affect the org CA or the SNI certificate.

## Limitations

> [!WARNING]
> **Warning:** A CA rotation revokes the old CAs at the end of the operation. This invalidates kubeconfig files and authentication configuration files that were issued and distributed based on the old CA. These credentials stop working after a CA rotation. The following instructions include details on how to download a new user cluster kubeconfig from the admin cluster and regenerate authentication configuration files after the CA rotation completes.

- Note the following limitation with advanced clusters:

  - Version 1.31: CA rotation isn't supported on [advanced clusters](https://docs.cloud.google.com/kubernetes-engine/distributed-cloud/vmware/docs/how-to/user-cluster-configuration-file-latest.md.txt#enable-advanced-cluster-field).
  - Version 1.32 and higher: CA rotation is supported on advanced clusters, but there are some minor differences noted where applicable in this document.
- CA certificate rotation is limited to the etcd, cluster, and
  front-proxy CAs mentioned previously.

- CA certificate rotation is limited to certificates issued automatically by
  Google Distributed Cloud. It doesn't update certificates issued manually by an
  administrator, even if those certificates are signed by the system CAs.

- A CA rotation restarts the API server, other control-plane processes, and
  each node in the cluster multiple times. Each stage of a CA rotation
  progresses similarly to a cluster upgrade. While the user cluster does remain
  operational during a CA rotation, you should expect workloads to be
  restarted and rescheduled. You should also expect brief periods of
  control-plane downtime if your user cluster does not have a
  [high-availability control plane](https://docs.cloud.google.com/kubernetes-engine/distributed-cloud/vmware/docs/how-to/user-cluster-configuration-file-latest.md.txt#network-controlplaneipblock-ips-section).

- You must update the user cluster kubeconfig file and authentication
  configuration files after a CA rotation. This is because the old cluster
  certificate is revoked, and the credentials in the kubeconfig file no
  longer work.

- After a CA rotation is started, it cannot be paused or rolled-back.

- A CA rotation might take considerable time to complete, depending on the size of
  the user cluster.

## Perform a CA rotation

1. Start the rotation:

   ```
   gkectl update credentials certificate-authorities rotate \
       --config USER_CLUSTER_CONFIG \
       --kubeconfig ADMIN_CLUSTER_KUBECONFIG
   ```

   Replace the following:
   - <var translate="no">USER_CLUSTER_CONFIG</var>: the path of the user cluster
     configuration file

   - <var translate="no">ADMIN_CLUSTER_KUBECONFIG</var>: the path of the admin cluster
     kubeconfig file

The behavior of the command differs depending on whether advanced cluster
is enabled:

### Not enabled

If advanced cluster isn't enabled on the cluster, the command is asynchronous
and starts the CA rotation and then exits. You don't need to watch the
output of the command for the entire duration of the CA rotation. Instead, you
can periodically check on the progress by running the
`gkectl update credentials certificate-authorities status` command.

If the CA rotation starts successfully, you see a message similar to
following:

```
successfully started the CA rotation with CAVersion 2, use gkectl update credentials certificate-authorities status command to view the current state of CA rotation
```

If a CA rotation is already in progress, you see an error message similar
to the following:

```
Exit with error:
admission webhook "vonpremusercluster.onprem.cluster.gke.io" denied the request: requests must not modify CAVersion when cluster is not ready: ready condition is not true: ClusterCreateOrUpdate: Creating or updating user cluster control plane workloads
```

To view the status of the rotation:

```
gkectl update credentials certificate-authorities status \
    --config USER_CLUSTER_CONFIG \
    --kubeconfig ADMIN_CLUSTER_KUBECONFIG
```

The preceding command reports the `CAVersion`, which is an integer the
system automatically increments to differentiate the CAs used before and
after a rotation. The command also reports a status (`True` or `False`)
that indicates whether the CA rotation is complete, and a message describing
which `CAVersion` is currently in use by each component of the system.

If the CA rotation has already completed, you see a message similar to
this:

```
State of CARotation with CAVersion 2 is -
status: True,
reason: CARotationCompleted,
message: Control plane has CA bundle [2], certs from CA 2, CA 2 is CSR signer. Data plane has CA bundle [2], CA 2 was CSR signer at last restart.
```

If the CA rotation is still in progress, you see a message similar to this:

```
State of CARotation with CAVersion 2 is -
status: False,
reason: CARotationProgressed,
message: Control plane has CA bundle [1 2], certs from CA 2, CA 1 is CSR signer. Data plane has CA bundle [1 2], CA 1 was CSR signer at last restart.
```

### Update user cluster credentials

On clusters that don't have advanced cluster enabled, you must get a new user
cluster kubeconfig file from the admin cluster. This is because the CA
rotation revokes the CA that the old kubeconfig file was based on.

Get a new kubeconfig file:

```
kubectl --kubeconfig ADMIN_CLUSTER_KUBECONFIG get secret admin \
  -n USER_CLUSTER_NAME -o jsonpath='{.data.admin\.conf}' \
  | base64 --decode > USER_CLUSTER_NAME-kubeconfig
```

### Enabled

If advanced cluster is enabled, the `gkectl update credentials
certificate-authorities rotate` command is synchronous. The command outputs
status messages to the admin workstation as the CA rotation progresses.

After the CA is rotated successfully, the command exits and a new kubeconfig
file is automatically generated. The command output provides the name of the
new kubeconfig file, and is similar to the following:

```
Beginning CA rotation with generated CA
...
Successfully rotated CA for user cluster "USER_CLUSTER_NAME". The
kubeconfig file "/home/ubuntu"/USER_CLUSTER_NAME-kubeconfig" has
been updated.
```

The new kubeconfig is located in the same directory as the admin cluster
kubeconfig that you specified in the command. The name of the new kubeconfig
is `USER_CLUSTER_NAME-kubeconfig`.

## Distribute the new kubeconfig file

Distribute the new kubeconfig file to everyone who uses a kubeconfig file to
interact with the cluster.

## Update authentication configuration files

After the CA rotation completes, authentication configuration files must be
updated and redistributed. For more information see
[Manage user identity](https://docs.cloud.google.com/kubernetes-engine/distributed-cloud/vmware/docs/concepts/authentication.md.txt).

## Control plane certificates rotation

Without rotation, both the user cluster CAs and control-plane certificates
expire ten years from the date the cluster was created (for versions before 1.36.100, these certificates expire after five years). The user cluster's
control-plane certificates are automatically rotated within ten hours of each
user cluster upgrade, but the CAs are not automatically rotated. This means a
CA rotation must be performed at least once every ten years (five years for versions before 1.36.100) in addition to
regular version upgrades.

To prevent a user cluster from becoming unavailable, control-plane certificates
are rotated within ten hours following a user cluster upgrade. When this
happens, a message appears in the user cluster's CA rotation status.

To view the last version a user cluster has been upgraded to when control-plane
certificates were rotated:

```
gkectl update credentials certificate-authorities status \
--config USER_CLUSTER_CONFIG \
--kubeconfig ADMIN_CLUSTER_KUBECONFIG
```

The information appears at the end of the `message` field within ten hours of an
upgrade. For example:

```
Last Leaf Certificates Rotation Version: 1.16.0-gke.0.
```

## Troubleshooting a CA rotation

The `gkectl diagnose` command supports checking the expected status of a
completed CA rotation against a user-cluster. For instructions on how to run
`gkectl diagnose` on a user cluster, see
[Diagnosing cluster issues](https://docs.cloud.google.com/kubernetes-engine/distributed-cloud/vmware/docs/troubleshooting/diagnose.md.txt#diagnose_user_cluster).
If you experience issues with a CA rotation, contact Google support and
provide the `gkectl diagnose` output.

> [!NOTE]
> **Note:** Because this command checks for the expected end state of a CA rotation, it is expected for it to report that some certificates have not yet been rotated when run during a CA rotation.