You can diagnose and resolve issues that occur during cluster updates in
Google Distributed Cloud, such as operation timeouts, configuration changes that are
rejected or ignored, and preflight check failures. For more information about
which settings you can update, see
[What can and cannot be updated in clusters](https://docs.cloud.google.com/kubernetes-engine/distributed-cloud/vmware/docs/how-to/updating-general.md.txt).

## Update timeout

The update timeout is dynamically calculated based on the resources to update.
However, the calculation isn't always accurate. When the update times out,
errors similar to the following are displayed:

- In the user cluster:

      Failed to update the cluster:...timed out waiting for the condition...

- In the admin cluster:

      Failed to update the admin cluster:...timed out waiting for the condition...

You can safely ignore this kind of timeout error, and you can retry the update
command. If you retry the command and it times out again with the same error
message, reach out to
[Cloud Customer Care](https://docs.cloud.google.com/kubernetes-engine/distributed-cloud/vmware/docs/getting-support.md.txt).

## Update contains multiple changes

The `gkectl update admin` and `gkectl update cluster` commands don't let you
update multiple settings in one command. If your configuration file changes
more than one setting, the command returns an error similar to the following
example:

    Update summary for cluster X:
        antiAffinityGroups: enabled to be set to true from false          &config.AAGSpec{
            -   Enabled: false,
            +   Enabled: true,
              }
        user master cpu to be set to 5 from 4          config.NodePoolProps{
                Role:        "master",
                MachineType: "standard-master",
            -   CPUs:        4,
            +   CPUs:        5,
                MemoryMB:    8192,
                Replicas:    3,
                ... // 2 identical fields
                Labels:         nil,
                NodeTaints:     nil,
            -   Vsphere:        nil,
            &config.NodePoolVsphereSpec{Datastore: "lifecycle-workloads1-datastore1"},
            +   Vsphere:        nil,
                BootDiskSizeGB: nil,
                OSImageType:    "",
                ... // 5 identical fields
              }

    Exit with error:
    Failed to update the cluster: the update contains multiple changes. Please
    update only one feature at a time

This error could happen for various reasons, including the following:

- A mistake or misconfiguration.
- You ran `gkectl upgrade` before with the configuration diff, and expected the changes to be applied.
  - `gkectl upgrade` doesn't apply any configuration diffs except the version bump.
- You edited the configuration for another feature update before, but forgot to run the `gkectl update` command.

If you encounter this error, review the diff in the error message and update
the required settings one by one with multiple `gkectl update` commands. To help
identify changes, you can use `gkectl get-config` to
[generate configuration files from a cluster](https://docs.cloud.google.com/kubernetes-engine/distributed-cloud/vmware/docs/how-to/get-configuration-file-from-cluster.md.txt)
and view the existing state and configuration.

## Unsupported changes

The `gkectl update cluster` and `gkectl update admin` commands ignore
unsupported changes, and display error messages similar to the following
examples:

    detected unsupported changes: (-current +desired)
        ...
    -   AdvancedNetworking:       &true,
    +   AdvancedNetworking:       &false,
        ...
    , which will be ignored

If you encounter this error, review the diff in the error message and
perform the following actions:

- If the change is unintended, edit the config YAML file and update with only the correct, intended changes.
  - In the previous example, if you didn't intend to disable `AdvancedNetworking`, set `advancedNetworking: true` in the config YAML file.
- If the change is intended, the error indicates that the change isn't supported. Perform one of the following actions:
  - Recreate the cluster if applicable.
  - Reach out to [Cloud Customer Care](https://docs.cloud.google.com/kubernetes-engine/distributed-cloud/vmware/docs/getting-support.md.txt).

## OS image doesn't exist

The `gkectl update cluster` and `gkectl update admin` commands might fail with
`OS Images` preflight check failures similar to the following examples:

- In the user cluster:

      - Validation Category: OS Images
          - [FAILURE] User cluster OS images exist: os images  [xxxx] don't exist,
          please run `gkectl prepare` to upload os images.

- In the admin cluster:

      - Validation Category: OS Images
          - [FAILURE] Admin cluster OS images exist: os images [xxxx] don't exist,
          please run `gkectl prepare` to upload os images.

These errors can happen if the OS image was removed unexpectedly in your
vCenter environment, such as by a periodic cleanup job.

To re-import the OS images, run the following `gkectl prepare` command:

```
gkectl prepare \
    --bundle-path /var/lib/gke/bundles/gke-onprem-vsphere-TARGET_VERSION.tgz \
    --kubeconfig ADMIN_CLUSTER_KUBECONFIG \
    --skip-upload-container-images
```

## Not enough datastore free space for new node pools

When you add new node pools, the `gkectl update cluster` command might fail with
`VSphere Datastore FreeSpace` preflight check errors similar to the following
example:

      - [FAILURE] VSphere Datastore FreeSpace: vCenter datastore: xxxx insufficient
      FreeSpace, requires at least xxx  GB

This failure indicates that the datastore doesn't have enough free space for
the new node pools. To provide enough space for the update to succeed, do one
of the following:

- Free up space from the datastore.
- Configure a different `nodePools[].vsphere.datastore` datastore for the node pool.

## User cluster update fails after you add IP addresses to a MetalLB IP address pool

On an
[advanced cluster](https://docs.cloud.google.com/kubernetes-engine/distributed-cloud/vmware/docs/concepts/advanced-clusters),
you can't change the IP addresses of an existing address pool in
`loadBalancer.metalLB.addressPools`. In version 1.33 and higher, all new
clusters are advanced clusters. If you add IP addresses to an existing address
pool and run `gkectl update cluster`, the new configuration is accepted and
saved, but the update fails later, and the user cluster stays in the `UPDATING`
state. The update fails with an error similar to the following:

```text
admission webhook "vcluster.kb.io" denied the request: Cluster.baremetal.cluster.gke.io "CLUSTER_NAME" is invalid: spec.loadBalancer.addressPools: Forbidden: Fields should be immutable. ... If you want to add new addresses for load balancer, please add a new address pool with new addresses instead of updating an existing address pool.
```

To add IP addresses, add a new address pool that contains the new IP addresses,
and keep the existing address pools unchanged. For more information, see
[Update MetalLB](https://docs.cloud.google.com/kubernetes-engine/distributed-cloud/vmware/docs/how-to/bundled-load-balance-metallb.md.txt#update-metallb).

If your user cluster is already stuck in the `UPDATING` state, don't remove the
added IP addresses from your user cluster configuration file, because removing
IP addresses from an existing address pool is also rejected. Instead, contact
[Cloud Customer Care](https://docs.cloud.google.com/kubernetes-engine/distributed-cloud/vmware/docs/getting-support.md.txt)
for help with recovering the cluster.

## What's next

If you need additional assistance, reach out to

[Cloud Customer Care](https://cloud.google.com/support-hub).

You can also see
[Getting support](https://docs.cloud.google.com/kubernetes-engine/distributed-cloud/vmware/docs/getting-support.md.txt) for more information about support resources, including the following:

- [Requirements](https://docs.cloud.google.com/kubernetes-engine/distributed-cloud/vmware/docs/getting-support.md.txt#support_requirements) for opening a support case.
- [Tools](https://docs.cloud.google.com/kubernetes-engine/distributed-cloud/vmware/docs/getting-support.md.txt#support_tools) to help you troubleshoot, such as logs and metrics.
- Supported [components](https://docs.cloud.google.com/kubernetes-engine/distributed-cloud/vmware/docs/getting-support.md.txt#whats_supported), [versions](https://docs.cloud.google.com/kubernetes-engine/distributed-cloud/vmware/docs/getting-support.md.txt#version-support), and [features](https://docs.cloud.google.com/kubernetes-engine/distributed-cloud/vmware/docs/getting-support.md.txt#supported_features) of Google Distributed Cloud for VMware (software only).