This page describes how to shut down and reboot any one blade of Google Distributed Cloud (GDC) air-gapped appliance,
for example to move the device to maintain a blade.

## Before you begin

Ensure you stop all workloads before proceeding further. Google cannot guarantee what will happen if workloads are active during a
shutdown.

If you want to shut down all the blades, follow [Shut down the device](https://docs.cloud.google.com/distributed-cloud/hosted/docs/latest/appliance/admin/shutdown). When following these instructions, shut down *only* one blade and keep Google Distributed Cloud (GDC) air-gapped appliance running with two active blades.

### Prerequisites

1. You can execute this runbook on a laptop or workstation connected to Google Distributed Cloud (GDC) air-gapped appliance's network. Alternatively, you can connect a laptop or workstation to switch by following [Connect the device](https://docs.cloud.google.com/distributed-cloud/hosted/docs/latest/appliance/admin/connect-the-device).
2. Make sure you have the access to Kubeconfig for the root-admin cluster.
3. Set correct KUBECONFIG environment variable by running `export KUBECONFIG=<path to kubeconfig>`.

### Shut down the blade

1. Get the node information by running `kubectl get nodes -A`. Determine `NODE_NAME` of blade to shut down.

2. Pause BareMetalHost sync by running following command for the blade to be shut down:

       kubectl annotate bmhost -n gpc-system NODE_NAME "baremetalhost.metal3.io/paused=true" --overwrite

   Sample output of this command is:

       baremetalhost.metal3.io/**-**-bm** annotated

3. Cordon the target node:

       kubectl cordon NODE_NAME

   Sample output is:

       node/**-**-bm** cordoned

4. Drain the target node:

       kubectl drain NODE_NAME --delete-emptydir-data --grace-period 900 --ignore-daemonsets --disable-eviction

   Sample output:

       node/**-**-bm** already cordoned
       WARNING: ignoring DaemonSet-managed Pods: kube-system/anetd-krj2z, kube-system/etcd-defrag-xh469, kube-system/ipam-controller-manager-2f4dz, kube-system/istio-cni-node-cgqv4, kube-system/kube-proxy-5mwf2, kube-system/localpv-mn2jh, kube-system/metallb-speaker-6l7sv, mon-system/mon-node-exporter-backend-nd8mp, netapp-trident/netapp-trident-node-linux-rrlmd, obs-system/anthos-audit-logs-forwarder-tpfqv, obs-system/anthos-log-forwarder-npjh4, obs-system/kube-control-plane-metrics-proxy-wp8nh, obs-system/log-failure-detector-crbnv, obs-system/oplogs-forwarder-sqwvj, vm-system/macvtap-v9pgp, vm-system/virt-handler-86khx
       pod/grafana-0 deleted
       pod/capi-kubeadm-bootstrap-controller-manager-1.30.400-gke.136lvgtf deleted
       pod/grafana-0 deleted
       pod/grafana-proxy-server-86d8fc4758-mkc4f deleted
       .
       .
       .

5. Gracefully shutdown the target node:

6. Turn off `NODE_NAME` using iLO:

   - Retrieve the credentials to access the iLO:

     1. Get the username:

            kubectl get secret bmc-credentials-NODE_NAME -n gpc-system -o jsonpath="{.data.username}" | base64 --decode

     2. Get the password:

            kubectl get secret bmc-credentials-NODE_NAME -n gpc-system -o jsonpath="{.data.password}" | base64 --decode

     3. Retrieve the `BMC-IP` address for `NODE_NAME` from values in `BMC-IP` column:

            kubectl get servers -A

   - Visit the `BMC-IP` address obtained in the previous step and sign in by entering the username and password obtained.

   - Hover over first button on top row. It should display `Power: ON`. Click it. A drop-down menu will appear, click first item labelled `Momentary Press`. The button colour will change from Green to Orange, meaning node is shutting down. Wait for the button to change color to yellow, indicating the machine has powered off. This will take a few minutes.

7. Wait for 30 minutes for the reconciliation to complete.

## Reboot the blade

This section describes steps to boot up a blade which was shut down earlier.

### Prerequisites

You can execute this runbook on a laptop or workstation connected to Google Distributed Cloud (GDC) air-gapped appliance's network. Alternatively, you can connect a laptop or workstation to switch by following [Connect the device](https://docs.cloud.google.com/distributed-cloud/hosted/docs/latest/appliance/admin/connect-the-device). Make sure you have the access to Kubeconfig for the root-admin cluster. And set correct KUBECONFIG environment variable by running `export KUBECONFIG=<path to kubeconfig>`.

### Plan of action

1. Press the power button on the blade. Once the blade is powered up, wait for a few minutes for the control plane to connect. `kubectl` should be able to connect to control plane in under 30 minutes.

2. Determine name of the target node by running `kubectl get nodes -A`.

3. Uncordon the target node to enable scheduling:

       kubectl uncordon `NODE_NAME`

4. Resume sync of BareMetalHost for the target node:

       kubectl annotate bmhost -n gpc-system NODE_NAME "baremetalhost.metal3.io/paused=false" --overwrite

5. Wait for 30 minutes for the reconciliation to complete.