Manual certificate renewal lets you restore control plane operations in
Google Distributed Cloud when Transport Layer Security (TLS) certificates expire. You
can't manage workloads and cluster lifecycles until you renew your expired
TLS certificates. For more information about the impact of expired certificates,
see
[Certificate expiration](https://docs.cloud.google.com/kubernetes-engine/distributed-cloud/bare-metal/docs/troubleshooting/failure-mode-analysis.md.txt#certificate_expiration).

TLS certificates, including etcd certificates, have a default 1-year expiration
period and don't update automatically. To prevent certificates from expiring and
keep your clusters secure and supported, we recommend that you
[upgrade your clusters regularly](https://docs.cloud.google.com/kubernetes-engine/distributed-cloud/bare-metal/docs/how-to/upgrade.md.txt).
Google Distributed Cloud renews these certificates during cluster upgrades and
when you
[rotate certificate authorities](https://docs.cloud.google.com/kubernetes-engine/distributed-cloud/bare-metal/docs/how-to/ca-rotation.md.txt).
You can use this guidance to recover clusters if certificates lapse before an
upgrade. For related details, see
[Common GKE user roles and tasks](https://docs.cloud.google.com/kubernetes-engine/enterprise/docs/concepts/roles-tasks.md.txt).

## Errors caused by certificate expiration

If the TLS certificates on your cluster expire, the core controllers can't
establish TLS connections with the Kubernetes API server. This lack of
connectivity causes the following errors:

- **Unable to connect to the server: x509**

  When you use `kubectl` to get your cluster nodes, the response includes an
  error that your certificates have expired, similar to the following example
  output:

      Unable to connect to the server: x509: certificate has expired or is not yet valid

- **could not connect: x509** or **rejected connection**

  Expired certificates block access to the etcd cluster, as peers can't
  communicate with each other. The etcd logs might contain error entries like
  the following:

      W | rafthttp: health check for peer 6221a1d241bb2d0a could not connect: x509: certificate
      has expired or is not yet valid
      I | embed: rejected connection from "10.200.0.4:46108" (error "remote error: tls: bad
      certificate", ServerName "")

## Check certificate expiration times

To check certificate expiration times, perform the following steps on each
control plane node:

1. Sign in to one of the control plane node machines and run the following
   command:

   ```bash
   sudo kubeadm certs check-expiration
   ```

   The command output lists the certificates created by `kubeadm` for the
   control plane components and their expiry, as shown in the following example
   output:

       CERTIFICATE                EXPIRES                  RESIDUAL TIME   CERTIFICATE AUTHORITY   EXTERNALLY MANAGED
       admin.conf                 Nov 28, 2021 19:09 UTC   53m                                     no
       apiserver                  Nov 28, 2021 19:09 UTC   53m             ca                      no
       apiserver-etcd-client      Nov 28, 2021 19:09 UTC   53m             etcd-ca                 no
       apiserver-kubelet-client   Nov 28, 2021 19:09 UTC   53m             ca                      no
       controller-manager.conf    Nov 28, 2021 19:09 UTC   53m                                     no
       etcd-healthcheck-client    Nov 28, 2021 19:09 UTC   53m             etcd-ca                 no
       etcd-peer                  Nov 28, 2021 19:09 UTC   53m             etcd-ca                 no
       etcd-server                Nov 28, 2021 19:09 UTC   53m             etcd-ca                 no
       front-proxy-client         Nov 28, 2021 19:09 UTC   53m             front-proxy-ca          no
       scheduler.conf             Nov 28, 2021 19:09 UTC   53m                                     no

       CERTIFICATE AUTHORITY   EXPIRES                  RESIDUAL TIME   EXTERNALLY MANAGED
       ca                      Nov 26, 2031 18:06 UTC   9y              no
       etcd-ca                 Nov 26, 2031 18:06 UTC   9y              no
       front-proxy-ca          Nov 26, 2031 18:06 UTC   9y              no

2. Run the following command to check the expiration times for `kubelet`
   certificates:

   ```bash
   sudo openssl x509 -in /var/lib/kubelet/pki/kubelet-client-current.pem -text | grep Validity -A2
   sudo openssl x509 -in /var/lib/kubelet/pki/kubelet-server-current.pem -text | grep Validity -A2
   ```

   The response for each command looks like the following example output:

       Validity
           Not Before: Sep 17 22:27:53 2021 GMT
           Not After : Sep 17 22:33:16 2022 GMT

   If all control plane nodes have been bootstrapped at the same time, then the
   certificate expiry times are within minutes of each other. This timing
   relationship applies across all control plane nodes. You can verify
   expiry times by running the preceding commands on each control plane node.
3. Run the following command on the admin workstation to check the expiration
   time of the client certificate in the kubeconfig file of the cluster:

   ```bash
   grep 'client-certificate-data' KUBECONFIG_PATH | \
       awk '{print $2}' | base64 -d | openssl x509 -text | grep Validity -A2
   ```

   The response looks like this sample output:

       Validity
           Not Before: Sep 17 22:27:53 2021 GMT
           Not After : Sep 17 22:33:16 2022 GMT

4. Run the following command to look up the certificate expiration for the
   cluster kubeconfig in the admin cluster:

   ```bash
   kubectl get secret/CLUSTER_NAME-kubeconfig \
       -n CLUSTER_NAMESPACE \
       --kubeconfig ADMIN_KUBECONFIG \
       -o jsonpath='{.data.value}' | base64 --decode | grep client-certificate-data | \
       awk '{print $2}' | base64 -d | openssl x509 -text | grep Validity -A2
   ```

   Replace the following:
   - `ADMIN_KUBECONFIG`: the
     path of the admin cluster kubeconfig file.

   - `CLUSTER_NAME`:
     the name of the cluster that you're renewing certificates for.

   - `CLUSTER_NAMESPACE`:
     the namespace of the cluster that you're renewing certificates for.

   The response looks like this sample output:

       Validity
           Not Before: Sep 17 22:27:53 2021 GMT
           Not After : Sep 17 22:33:16 2022 GMT

   The kubeconfig certificate in the admin cluster and the certificate in the
   kubeconfig file on the admin workstation are the same. Therefore, the
   output for this command and the command from the previous step must match.

## Renew certificates manually

To manually renew TLS certificates for a cluster, use the instructions in the
following sections.

> [!IMPORTANT]
> **Key Point:** The following instructions renew both expired and non-expired certificates. In either case, the expiration period for the renewed TLS certificates on your cluster is reset to one year. We strongly recommend that you renew cluster certificates before they expire to avoid significant cluster downtime. The recommended method for renewing certificates before they expire is to [upgrade](https://docs.cloud.google.com/kubernetes-engine/distributed-cloud/bare-metal/docs/how-to/upgrade.md.txt) your clusters regularly.

### Renew certificates on each control plane node

Perform the following steps on each control plane node of the affected cluster:

> [!IMPORTANT]
> **Important:** Renew certificates on only one node at a time. Perform all steps before starting on a new control plane node.

1. Back up the `/etc/kubernetes` folder.

2. Run the following `kubeadm` command to renew all the certificates. The
   command renews the certificates using the existing Certificate Authorities
   (CAs) on the machine:

   ```bash
   sudo kubeadm certs renew all
   ```

   The command output is similar to the following example:

       certificate embedded in the kubeconfig file for the admin to use and for kubeadm itself renewed
       certificate for serving the Kubernetes API renewed
       certificate the apiserver uses to access etcd renewed
       certificate for the API server to connect to kubelet renewed
       certificate embedded in the kubeconfig file for the controller manager to use renewed
       certificate for liveness probes to healthcheck etcd renewed
       certificate for etcd nodes to communicate with each other renewed
       certificate for serving etcd renewed
       certificate for the front proxy client renewed
       certificate embedded in the kubeconfig file for the scheduler manager to use renewed

3. Verify that the certificates have a new expiry time by running the following
   command:

   ```bash
   sudo kubeadm certs check-expiration
   ```
4. Not all control plane components support dynamic certificate reload. To pick
   up the renewed certificates, the following steps restart the following
   containers: `kube-apiserver`, `kube-scheduler`, and
   `kube-controller-manager`.

   Repeat the following steps for each of the three containers:

   > [!NOTE]
   > **Note:** If your certificates are expired, we recommend that you perform the following step for the `kube-apiserver` container first. Then, stop the `kube-scheduler` and `kube-controller-manager` containers.

   1. Find the container ID for each container:

      ```bash
      sudo crictl ps | grep CONTAINER_NAME
      ```

      Replace `CONTAINER_NAME`
      with the name of the following containers: `kube-apiserver`,
      `kube-scheduler`, or `kube-controller-manager`.

      The response is similar to the following output:

          c331ade490cb6       28df10594cd92      26 hours ago       Running          kube-apiserver ...

      The container ID is the value in the first column.
   2. Stop each container:

      ```bash
      sudo crictl stop CONTAINER_ID
      ```

      Replace `CONTAINER_ID`
      with the container ID from the preceding step.

      When the stopped container exits, kubelet creates a new container in its
      place and deletes the stopped one. If you encounter an error, such as
      `context deadline exceeded` (error code `DeadlineExceeded`), rerun the
      command.

### Verify that connectivity is restored

The kubeadm certs should now be renewed on all control plane nodes. If
you're renewing expired certificates, perform the following step:

- To verify the connection with the Kubernetes API server, run the following
  `kubectl` command on any control plane node:

  ```bash
  kubectl get nodes --kubeconfig /etc/kubernetes/admin.conf
  ```

The response should return the list of nodes for the cluster. If your
certificates are properly renewed, no TLS or certificate errors are returned.

### Update the kubeconfig secret in the cluster

The following steps use the renewed certificates from the `admin.conf` file to
update the kubeconfig secret for your cluster. However, the contents of the
updated `admin.conf` file can't be used as-is. You must make a copy of the
`admin.conf` file with some needed edits first.

To update the new kubeconfig to the secret, perform the following steps on a
control plane node:

1. Use `sed` to replace `kubernetes` in the `admin.conf` file with the name of
   your cluster and write the changes to a new file, `kubeconfig_secret.conf`:

   ```bash
   sed "s/kubernetes/CLUSTER_NAME/g" \
       /etc/kubernetes/admin.conf > /etc/kubernetes/kubeconfig_secret.conf
   ```
2. Use `diff` to confirm that the `kubeconfig_secret.conf` file has been
   updated:

   ```bash
   diff /etc/kubernetes/admin.conf /etc/kubernetes/kubeconfig_secret.conf
   ```

   The response shows all of the places where the `kubeconfig_secret.conf` file
   is different from the updated `admin.conf` file. For example, if you
   performed the preceding step for a cluster named `demo-cluster`, the output
   would look similar to the following:

   ```output
   6c6
   <   name: kubernetes
   ---
   >   name: demo-cluster
   9,12c9,12
   <     cluster: kubernetes
   <     user: kubernetes-admin
   <   name: kubernetes-admin@kubernetes
   < current-context: kubernetes-admin@kubernetes
   ---
   >     cluster: demo-cluster
   >     user: demo-cluster-admin
   >   name: demo-cluster-admin@demo-cluster
   > current-context: demo-cluster-admin@demo-cluster
   16c16
   < - name: kubernetes-admin
   ---
   > - name: demo-cluster-admin
   ```
3. Run the following commands to update the kubeconfig secret in your cluster:

   ```bash
   CLUSTER_KUBECONFIG_BASE64=$(base64 /etc/kubernetes/kubeconfig_secret.conf -w 0)

   kubectl get secret/CLUSTER_NAME-kubeconfig \
       -n CLUSTER_NAMESPACE \
       --kubeconfig /etc/kubernetes/admin.conf -o json | jq \
       --arg conf "${CLUSTER_KUBECONFIG_BASE64}" '.data."value" |= $conf' | kubectl apply \
       --kubeconfig /etc/kubernetes/admin.conf -f -
   ```
4. Restart the cluster lifecycle controllers in the managing cluster so that
   they pick up the updated kubeconfig secret.

   The `anthos-cluster-operator` and `cap-controller-manager` deployments
   connect to the cluster with the credentials from the
   `CLUSTER_NAME-kubeconfig`
   secret. These controllers read the secret once and keep using the previous
   certificate until they restart. When that certificate reaches its original
   expiration date, the controllers can no longer reconcile the cluster.

   Restart the lifecycle controller deployments:

   ```bash
   kubectl rollout restart deployment -n kube-system \
       -l baremetal.cluster.gke.io/lifecycle-controller-component=true \
       --kubeconfig ADMIN_KUBECONFIG
   ```

   Replace
   `ADMIN_KUBECONFIG`
   with `/etc/kubernetes/admin.conf` on a control plane node of a
   self-managing cluster (admin, hybrid, or standalone), or with the path to
   the admin cluster kubeconfig file on the admin workstation if you're
   renewing a user cluster.

### Replace the cluster kubeconfig file

To replace the kubeconfig file for your cluster with one that has the renewed
certificates, use the following steps:

1. Copy the `admin.conf` file from one of the cluster control plane nodes to
   the admin workstation.

   As noted in prior sections, the `admin.conf` file is found in the
   `etc/kubernetes` directory on the cluster control plane nodes.
2. To create the new kubeconfig file, run the following `kubectl` command on
   the admin workstation:

   ```bash
   kubectl --kubeconfig ADMIN_CONF_PATH get secret/CLUSTER_NAME-kubeconfig  \
       -n "CLUSTER_NAMESPACE" -o jsonpath='{.data.value}'  | \
       base64 --decode > new_kubeconfig.conf
   ```

   Replace the following:
   - `ADMIN_CONF_PATH`: the path of the `admin.conf`
     file that has been copied to the admin workstation from a control plane
     node.

   - `CLUSTER_NAME`:
     the name of the cluster that you're renewing certificates for.

   - `CLUSTER_NAMESPACE`:
     the namespace of the cluster that you're renewing certificates for.

   The `new_kubeconfig.conf` file contains the updated certificate data.
3. Verify that the new kubeconfig works by running any `kubectl` command, using
   the new credentials:

   ```bash
   kubectl get nodes --kubeconfig new_kubeconfig.conf
   ```
4. Replace the contents of the old kubeconfig file saved in the
   cluster directory on the admin workstation with the contents of the new
   kubeconfig file `new-kubeconfig.conf`.

   By default, the path to the cluster configuration file is
   `bmctl-workspace/CLUSTER_NAME/CLUSTER_NAME-kubeconfig`.

### Verify kubelet certificates and restart DaemonSets

To finish the process of renewing cluster certificates manually, perform the
following steps for each control plane node:

1. Sign in to the control plane node and verify the kubelet client and
   serving certificate expiry time by running the following commands:

   Kubelet certificates are rotated automatically as long as the control plane
   is reachable. The period for automatic renewal of kubelet certificates is
   shorter than the expiration period for control plane component certificates.
   Therefore, it's likely that kubelet certificates have been renewed before:

   ```bash
   sudo openssl x509 -in /var/lib/kubelet/pki/kubelet-client-current.pem -text | grep Validity -A2
   sudo openssl x509 -in /var/lib/kubelet/pki/kubelet-server-current.pem -text | grep Validity -A2
   ```

   The output of either command looks something like the following example:

       Validity
           Not Before: Nov 28 18:04:57 2022 GMT
           Not After : Nov 28 19:04:57 2023 GMT

2. Use the following command to restart the `etcd-defrag` container:

   The `etcd-defrag` container uses the etcd healthcheck client certificate
   (`etcd/healthcheck-client.crt`) to talk to etcd. You must restart the
   `etcd-defrag` container to pick up the updated certificates.

   ```bash
   kubectl rollout restart daemonset etcd-defrag -n kube-system --kubeconfig KUBECONFIG_PATH
   ```
3. For clusters at version 1.36 and earlier, if the `ipam-controller-manager`
   DaemonSet exists, restart it.

   In these versions, `ipam-controller-manager` uses the client certificate
   that's embedded in the `/etc/kubernetes/controller-manager.conf` file on
   each control plane node. The controller reads this certificate only when it
   starts. If you don't restart the DaemonSet, the controller keeps using the
   previous certificate. When that certificate reaches its original expiration
   date, leader election fails with `Unauthorized` errors and IP address
   management for the cluster stops.
   1. Check whether the `ipam-controller-manager` DaemonSet exists in your
      cluster:

      ```bash
      kubectl get daemonset ipam-controller-manager -n kube-system --kubeconfig KUBECONFIG_PATH
      ```

      If the command returns `NotFound`, skip the next step.
   2. Restart the `ipam-controller-manager` DaemonSet:

      ```bash
      kubectl rollout restart daemonset ipam-controller-manager -n kube-system --kubeconfig KUBECONFIG_PATH
      ```

   If you renewed the certificates previously without restarting
   `ipam-controller-manager`, run only the restart command.

After you complete these manual steps to renew cluster certificates, verify that
all Pods are running properly and that no TLS errors are reported for control
plane containers.

## What's next

If you need additional assistance, reach out to

[Cloud Customer Care](https://cloud.google.com/support-hub).
You can also see
[Getting support](https://docs.cloud.google.com/kubernetes-engine/distributed-cloud/bare-metal/docs/getting-support.md.txt) for more information about support resources, including the following:

- [Requirements](https://docs.cloud.google.com/kubernetes-engine/distributed-cloud/bare-metal/docs/getting-support.md.txt#intro-support) for opening a support case.
- [Tools](https://docs.cloud.google.com/kubernetes-engine/distributed-cloud/bare-metal/docs/getting-support.md.txt#support-tools) to help you troubleshoot, such as your environment configuration, logs, and metrics.
- Supported [components](https://docs.cloud.google.com/kubernetes-engine/distributed-cloud/bare-metal/docs/getting-support.md.txt#what-we-support).