Google Distributed Cloud lets you rotate user cluster certificate authorities (CAs) to
update the certificates and private keys that authenticate and encrypt system
communication. Rotating your user cluster CA periodically and after any
potential security incident helps you maintain cluster security. Automated CA
rotation requires active, valid certificates.

If your cluster certificates have expired, you must instead
[renew expired cluster certificates manually](https://docs.cloud.google.com/kubernetes-engine/distributed-cloud/bare-metal/docs/troubleshooting/expired-certs.md.txt)
to restore cluster operations.

## How CA rotation works

When you run the `bmctl update credentials certificate-authorities rotate`
command, Google Distributed Cloud performs the following actions:

- The command creates and uploads new cluster certificate authorities (CAs)
  for the cluster CA, the etcd CA, and the front-proxy CA to the user cluster
  namespace in the admin cluster.

- The admin cluster controllers replace the user cluster certificate
  authorities with the newly generated ones.

- The admin cluster controllers distribute the new public CA certificates and
  leaf certificate key pairs to user cluster system components.

- The command also updates the `stackdriver-prometheus-etcd-scrape` Secret,
  which was created by Google Distributed Cloud during cluster creation.
  Prometheus requires this secret to collect etcd metrics.

## Before you begin

Before you rotate your cluster's certificate authority, plan according to the
following conditions and impacts:

- Ensure admin and user clusters are at version 1.9.0 or higher before
  starting the CA rotation.

- CA rotation is incremental, allowing system components to communicate during
  the rotation.

- A CA rotation restarts the API server, other control-plane processes, and
  each node in the cluster multiple times. Each stage of a CA rotation
  progresses similarly to a cluster upgrade. While the user cluster does
  remain operational during a CA rotation, you should expect that workloads
  will be restarted and rescheduled.

- If your user cluster doesn't have a [high-availability control plane](https://docs.cloud.google.com/kubernetes-engine/distributed-cloud/bare-metal/docs/reference/cluster-config-ref.md.txt#controlplane-nodepoolspec-nodes),
  expect brief periods of control-plane downtime during CA rotation.

- Cluster management operations aren't allowed during CA rotation.

- CA rotation duration depends on the size of your cluster. For example, CA
  rotation may take close to two hours to complete for a cluster with one
  control plane and 50 worker nodes.

## Limitations

The certificate authority rotation capability has the
following limitations:

- CA rotation doesn't update certificates issued manually by an administrator,
  even if the cluster CA signs the certificates. Update and redistribute any
  manually issued certificates after user cluster CA rotation is complete.

- Once started, CA rotation can't be paused or rolled back.

## Start a cluster CA rotation

By default, TLS certificates have a 1-year expiration period, while the
`clientconfig-operator` certificate has a 10-year expiration period (for versions before 1.36.100 the `clientconfig-operator` certificate expires after five years).
Google Distributed Cloud renews these certificates when you rotate certificate
authorities. Google Distributed Cloud also renews 1-year TLS certificates during
cluster upgrades. For versions before 1.36.100, we recommend that you upgrade your clusters regularly to keep
them secure, supported, and to prevent TLS certificates from expiring, and
perform a CA rotation at least once every five years.

Use the following command to start the CA rotation process:

```bash
bmctl update credentials certificate-authorities rotate --cluster CLUSTER_NAME \
    --kubeconfig KUBECONFIG
```

Replace the following:

- `CLUSTER_NAME`: the name of the cluster for which you want to rotate CAs.
- `KUBECONFIG`: the path to the admin cluster kubeconfig file. For self-managing clusters, this file is the cluster's kubeconfig file.

The `bmctl` command exits after the CA is rotated successfully and a new
kubeconfig file is generated. The standard path for the kubeconfig file is
`bmctl-workspace/CLUSTER_NAME/CLUSTER_NAME-kubeconfig`.

## Troubleshoot a cluster CA rotation

The `bmctl update credentials` command displays the progress of the CA rotation.
The associated `update-credentials.log` file is saved to the following
timestamped directory:

    bmctl-workspace/CLUSTER_NAME/log/update-credentials-TIMESTAMP