This document explains how to upgrade the Kubernetes version of a
Google Distributed Cloud (GDC) air-gapped Kubernetes cluster. You must plan routine
upgrades for your clusters to ensure they incorporate the latest Kubernetes
features and fixes.

This document is for audiences such as IT administrators, security engineers,
and network administrators within the platform administrator group, who are
responsible for managing Kubernetes resources within their organization. For
more information, see
[Audiences for GDC air-gapped documentation](https://docs.cloud.google.com/distributed-cloud/hosted/docs/latest/gdcag/resources/audiences.md.txt).

## Before you begin

To complete the tasks in this document, you must request the necessary
permissions and prepare your environment.

### Request IAM roles

You must have specific roles to get the permissions you need to upgrade your
Kubernetes cluster. You must request roles scoped to your project and
organization.

#### Organization roles

To list available upgrade versions for a Kubernetes cluster, ask your
IAM Org Admin to grant you the **User Cluster Admin**
(`user-cluster-admin`) role. This role isn't bound to a namespace.

If you manage a standard cluster and are unable to obtain the User Cluster Admin role from your platform administrator group, you must request they provide you with the list of available Kubernetes upgrade versions.

#### Project roles

To create, delete, edit, or view cluster upgrades, ask your IAM Project
Admin to grant you the **Standard Cluster Admin** (`standard-cluster-admin`)
role. This role is bound to your project namespace.

### Prepare your environment

To run commands against a
[Kubernetes cluster](https://docs.cloud.google.com/distributed-cloud/hosted/docs/latest/gdcag/platform/pa-user/clusters.md.txt#cluster-architecture)
using the API, make sure you have the following resources:

- [Download](https://docs.cloud.google.com/distributed-cloud/hosted/docs/latest/gdcag/resources/gdcloud-download.md.txt) and
  [install](https://docs.cloud.google.com/distributed-cloud/hosted/docs/latest/gdcag/resources/gdcloud-install.md.txt) the gdcloud CLI.

- [Install the kubectl CLI](https://docs.cloud.google.com/distributed-cloud/hosted/docs/latest/gdcag/get-started/kubectl.md.txt).

- Locate the Kubernetes cluster name, or ask a member of the platform
  administrator group what the cluster name is.

- Use the Kubernetes cluster name to replace
  `CLUSTER_NAME` in these instructions.

- [Sign in and generate](https://docs.cloud.google.com/distributed-cloud/hosted/docs/latest/gdcag/application/ao-user/iam/sign-in.md.txt#zonal-resources-kubeconfig)
  the kubeconfig file for the management API server.

- Use the kubeconfig path of the management API server to replace
  `MANAGEMENT_API_SERVER` in these instructions.

## Upgrade a Kubernetes cluster

Use the
[`UserClusterUpgradeRequest`](https://docs.cloud.google.com/distributed-cloud/hosted/docs/latest/gdcag/apis/service/cluster/v1/cluster-v1.md.txt#userclusterupgraderequest)
API to upgrade an existing Kubernetes cluster.

To upgrade a Kubernetes cluster using the API, follow these steps:

1. List the available Kubernetes upgrade versions by printing the
   `kubernetesVersion` value of the `UserClusterMetadata` objects:

   ```bash
   kubectl --kubeconfig MANAGEMENT_API_SERVER \
       get userclustermetadata -o=custom-columns='TARGET_VERSION:.spec.kubernetesVersion'
   ```

   Replace `MANAGEMENT_API_SERVER` with the path to the
   kubeconfig file of the management API server.

   The command returns a list of available Kubernetes versions:

       TARGET_VERSION
       1.30.1000-gke.85

2. Note the Kubernetes version from the previous output that you want to
   upgrade your cluster to. The `targetVersion` you specify in the upgrade
   request in the next step must be one of these available versions.

3. Request the cluster upgrade by creating a `UserClusterUpgradeRequest`
   resource:

   ```yaml
     kubectl --kubeconfig MANAGEMENT_API_SERVER apply -f - <<EOF
     apiVersion: cluster.gdc.goog/v1
     kind: UserClusterUpgradeRequest
     metadata:
       name: CLUSTER_NAME
       namespace: CLUSTER_NAMESPACE
     spec:
       clusterRef:
         name: CLUSTER_NAME
       targetVersion: TARGET_VERSION
     EOF
   ```

   Replace the following:
   - `MANAGEMENT_API_SERVER`: the path to the kubeconfig file of the management API server.
   - `CLUSTER_NAME`: the name of the Kubernetes cluster to upgrade, such as `user-vm-2`.
   - `CLUSTER_NAMESPACE`: the namespace of the Kubernetes cluster. For shared clusters, use the `platform` namespace. For standard clusters, use the project namespace of the cluster.
   - `TARGET_VERSION`: the target version for the upgrade identified in the previous step, such as `1.30.1000-gke.85`.

   The average upgrade time for a Kubernetes cluster takes approximately 20
   minutes.
4. Verify the cluster upgrade by monitoring the `UserClusterUpgradeRequest`
   resource:

   ```bash
   kubectl --kubeconfig MANAGEMENT_API_SERVER \
       describe userclusterupgraderequest CLUSTER_NAME \
       -n CLUSTER_NAMESPACE
   ```

   Replace the following:
   - `MANAGEMENT_API_SERVER`: the path to the kubeconfig file of the management API server.
   - `CLUSTER_NAME`: the name of the Kubernetes cluster to upgrade, such as `user-vm-2`.
   - `CLUSTER_NAMESPACE`: the namespace of the Kubernetes cluster. For shared clusters, use the `platform` namespace. For standard clusters, use the project namespace of the cluster.

   Inspect the `Spec` section of the output. It contains `Current Version` and
   `Target Version`. The upgrade is still in progress if the `Current Version`
   and the `Target Version` differ.

       Spec:
         ...
         Current Version: 1.29.500-gke.60
         Target Version: 1.30.1000-gke.85
       Status:
         Conditions:
         - Message: Upgrade is in progress
           Reason: UpgradeInProgress
           Status: "False"
           Type: Succeeded

   The upgrade is complete and successful when the `Succeeded` condition has a
   status of `True`, and the `Current Version` matches the `Target Version`.

   If the `Succeeded` condition has a status of `False`, the upgrade failed. If
   you encounter errors related to your Kubernetes cluster upgrade, contact the
   infrastructure operator group for further assistance.

## Verify cluster status after upgrade

After your Kubernetes cluster is upgraded, confirm that the cluster and its
components are healthy by inspecting the `Cluster` resource status:

1. View the `status.conditions` and `status.componentsConditions` fields of the
   `Cluster` resource:

   ```bash
   kubectl --kubeconfig MANAGEMENT_API_SERVER \
       get cluster CLUSTER_NAME \
       -n CLUSTER_NAMESPACE -o yaml
   ```

   Replace the following:
   - `MANAGEMENT_API_SERVER`: the path to the kubeconfig file of the management API server.
   - `CLUSTER_NAME`: the name of the Kubernetes cluster, such as `user-vm-2`.
   - `CLUSTER_NAMESPACE`: the namespace of the Kubernetes cluster. For shared clusters, use the `platform` namespace. For standard clusters, use the project namespace of the cluster.
2. In the output, verify the following in the `status` section:

   - In `status.conditions`, verify that the `Ready` condition has a `status` of `True`, and the `Reconciling` condition has a `status` of `False`.
   - In `status.componentsConditions`, verify that the `Ready` condition has a `status` of `True`, the `Reconciling` condition has a `status` of `False`, and no condition shows a `reason` of `ReconciliationError` or `Reconciling`.

   For Kubernetes clusters with three control plane nodes and three worker
   nodes, the upgrade takes approximately 20 minutes. If the cluster or its
   components remain in a `Reconciling` state after 20 minutes, or if any
   condition shows `ReconciliationError`, contact your infrastructure operator
   group for further assistance.

## What's next

- [Design workload separation](https://docs.cloud.google.com/distributed-cloud/hosted/docs/latest/gdcag/resources/workload-separation.md.txt)
- [Maintain Kubernetes clusters](https://docs.cloud.google.com/distributed-cloud/hosted/docs/latest/gdcag/platform-application/pa-ao-operations/cluster.md.txt)
- [Manage node pools](https://docs.cloud.google.com/distributed-cloud/hosted/docs/latest/gdcag/platform/pa-user/manage-node-pools.md.txt)