<button class="button button-with-icon cloud-select-dropdown__button button-primary" aria-haspopup="true">Version 1.14 </button>

- [Latest](https://docs.cloud.google.com/anthos/clusters/docs/bare-metal/latest/how-to/maintenance-mode)
- [Version 1.16](https://docs.cloud.google.com/anthos/clusters/docs/bare-metal/1.16/how-to/maintenance-mode.md.txt)
- [Earlier versions (unsupported)](https://docs.cloud.google.com/anthos/clusters/docs/bare-metal/1.14/getting-support.md.txt#version-support)

<br />

When you need to repair or maintain nodes, you should first put the nodes into
maintenance mode. Putting nodes into maintenance mode safely drains their
pods/workloads and excludes the nodes from pod scheduling. In maintenance mode,
you can work on your nodes without a risk of disrupting pod traffic.

## How it works

Google Distributed Cloud provides a way to place nodes into maintenance mode. This
approach lets other cluster components correctly know that the node is in
maintenance mode. When you place a node in maintenance mode, no additional pods
can be scheduled on the node, and existing pods are stopped.

Instead of using maintenance mode, you can manually use Kubernetes commands such
as `kubectl cordon` and `kubectl drain` on a specific node. If you run
Google Distributed Cloud version 1.12.0 (`anthosBareMetalVersion: 1.12.0`) or
lower, see the known issue on
[Nodes uncordoned if you don't use the maintenance mode procedure](https://cloud.google.com/anthos/clusters/docs/bare-metal/1.12/troubleshooting/known-issues#nodes-uncordoned-if-you-dont-use-the-maintenance-mode-procedure).

When you use the maintenance mode process, Google Distributed Cloud does the
following:

- [Node taints](https://kubernetes.io/docs/concepts/configuration/taint-and-toleration/)

  are added to specified nodes to indicate that no pods can be scheduled or
  executed on the nodes.
- A 20-minute timeout is enforced to ensure nodes don't get stuck waiting for
  pods to stop. Pods might not stop if they are configured to
  [tolerate all taints](https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/)
  or they have
  [finalizers](https://kubernetes.io/docs/concepts/overview/working-with-objects/finalizers/).
  Google Distributed Cloud attempts to stop all pods, but if the timeout is
  exceeded, the node is put into maintenance mode. This timeout prevents
  running pods from blocking upgrades.

## Put a node into maintenance mode

Choose the nodes you want to put into maintenance mode by specifying IP ranges
for the selected nodes under `maintenanceBlocks` in your cluster configuration
file. The nodes you choose must be in a ready state, and functioning in the
cluster.

To put nodes into maintenance mode:

1. Edit the cluster configuration file to select the nodes you want to put into
   maintenance mode.

   You can edit the configuration file with an editor of your choice, or you
   can edit the cluster custom resource directly by running the following
   command:

   ```bash
   kubectl -n CLUSTER_NAMESPACE edit cluster CLUSTER_NAME
   ```

   Replace the following:
   - `CLUSTER_NAMESPACE`: the namespace of the cluster.
   - `CLUSTER_NAME`: the name of the cluster.
2. Add the `maintenanceBlocks` section to the cluster configuration file to
   specify either a single IP address, or an address range, for nodes you want
   to put into maintenance mode.

   The following sample shows how to select multiple nodes by specifying a
   range of IP addresses:

   ```yaml
   metadata:
     name: my-cluster
     namespace: cluster-my-cluster
   spec:
     maintenanceBlocks:
       cidrBlocks:
       - 172.16.128.1-172.16.128.64
   ```
3. Save and apply the updated cluster configuration.

   Google Distributed Cloud starts putting the nodes into maintenance mode.
4. Run the following command to get the status of the nodes in your cluster:

   ```bash
   kubectl get nodes --kubeconfig=KUBECONFIG
   ```

   The response is something like the following:

       NAME                       STATUS   ROLES           AGE     VERSION
       user-anthos-baremetal-01   Ready    control-plane   2d22h   v1.25.10-gke.2100
       user-anthos-baremetal-04   Ready    worker          2d22h   v1.25.10-gke.2100
       user-anthos-baremetal-05   Ready    worker          2d22h   v1.25.10-gke.2100
       user-anthos-baremetal-06   Ready    worker          2d22h   v1.25.10-gke.2100

   Note that the nodes are still schedulable, but taints keep any pods (without
   an appropriate toleration) from being scheduled on the node.
5. Run the following command to get the number of nodes in maintenance mode:

   ```bash
   kubectl get nodepools
   ```

   The response should look something like the following output:

       NAME   READY   RECONCILING   STALLED   UNDERMAINTENANCE   UNKNOWN
       np1    3       0             0         1                  0

   This `UNDERMAINTENANCE` column in this sample shows that one node is in
   maintenance mode.

   Google Distributed Cloud also adds the following taints to nodes when they are
   put into maintenance mode:
   - `baremetal.cluster.gke.io/maintenance:NoExecute`
   - `baremetal.cluster.gke.io/maintenance:NoSchedule`

## Remove a node from maintenance mode

To remove nodes from maintenance mode:

1. Edit the cluster configuration file to clear the nodes you want to remove
   from maintenance mode.

   You can edit the configuration file with an editor of your choice, or you
   can edit the cluster custom resource directly by running the following
   command:

   ```bash
   kubectl -n CLUSTER_NAMESPACE edit cluster CLUSTER_NAME
   ```

   Replace the following:
   - `CLUSTER_NAMESPACE`: the namespace of the cluster.
   - `CLUSTER_NAME`: the name of the cluster.
2. Either edit the IP addresses to remove specific nodes from maintenance mode
   or remove the `maintenanceBlocks` section remove all does from maintenance
   mode.

3. Save and apply the updated cluster configuration.

4. Use `kubectl` commands to check the status of your nodes.