Occasionally, Google Distributed Cloud runs into conditions where you need to
start an upgrade over again, or fix an error condition to complete the upgrade
successfully. The following conditions may occur when upgrading clusters. Try
to complete the cluster upgrade as noted with the condition.

## Recover admin, hybrid, and standalone cluster upgrades

Admin, hybrid, and standalone clusters are all upgraded with the [`bmctl upgrade`
command](https://docs.cloud.google.com/kubernetes-engine/distributed-cloud/bare-metal/docs/how-to/upgrade).

If your upgrade doesn't complete successfully, check the following conditions
and try the upgrade again.

- `bmctl` fails to parse the config file due to a mistake. Fix the config file and rerun the command.
- `bmctl` fails to bootstrap the temporary cluster. Retry the command.
- the log files indicate that the upgrade failed a [preflight check](https://docs.cloud.google.com/kubernetes-engine/distributed-cloud/bare-metal/docs/installing/preflight.md.txt). Correct the preflight condition, or force the upgrade.
- the upgrade took too long (more than 30 minutes) or timed out. Retry the upgrade command.
- the upgrade stops during the initialization phase:
  - For version 1.35 and later, if you use RHEL 8 or Rocky Linux, check the node logs for Ansible execution errors indicating an unsupported Python version. Ansible requires Python 3.9 or newer on the target cluster. [Verify the installation of Python 3.9 or higher](https://docs.cloud.google.com/kubernetes-engine/distributed-cloud/bare-metal/docs/installing/configure-os/rhel.md.txt#update-python) on the failing node and retry the operation.
  - For version 1.35 and later, check that `cgroupsv2` is enabled. For more information on cgroupsv2, see [migrating to cgroup v2](https://kubernetes.io/docs/concepts/architecture/cgroups/#migrating-cgroupv2).

## Recover user cluster upgrades

User clusters are [upgraded from an associated admin cluster](https://docs.cloud.google.com/kubernetes-engine/distributed-cloud/bare-metal/docs/how-to/upgrade.md.txt)
with the `kubectl apply` command.

If your upgrade doesn't complete successfully, check the following conditions
and try the upgrade again.

- Preflight check failed. Check the preflight check log and fix the corresponding error.
- You may need to construct a new preflight custom resource [following these instructions](https://docs.cloud.google.com/kubernetes-engine/distributed-cloud/bare-metal/docs/installing/preflight.md.txt#re-enabling_preflight_checks) and apply it to the cluster. Check that the triggered preflight passes, and that the cluster reconciler picks up the latest passing state.

## What's next

If you need additional assistance, reach out to

[Cloud Customer Care](https://cloud.google.com/support-hub).
You can also see
[Getting support](https://docs.cloud.google.com/kubernetes-engine/distributed-cloud/bare-metal/docs/getting-support.md.txt) for more information about support resources, including the following:

- [Requirements](https://docs.cloud.google.com/kubernetes-engine/distributed-cloud/bare-metal/docs/getting-support.md.txt#intro-support) for opening a support case.
- [Tools](https://docs.cloud.google.com/kubernetes-engine/distributed-cloud/bare-metal/docs/getting-support.md.txt#support-tools) to help you troubleshoot, such as your environment configuration, logs, and metrics.
- Supported [components](https://docs.cloud.google.com/kubernetes-engine/distributed-cloud/bare-metal/docs/getting-support.md.txt#what-we-support).