<button class="button button-with-icon cloud-select-dropdown__button button-primary" aria-haspopup="true">Version 1.9 (unsupported)</button>

- [Latest](https://docs.cloud.google.com/anthos/clusters/docs/bare-metal/latest/how-to/update-clusters)
- [Version 1.16](https://docs.cloud.google.com/anthos/clusters/docs/bare-metal/1.16/how-to/update-clusters.md.txt)
- [Earlier versions (unsupported)](https://docs.cloud.google.com/anthos/clusters/docs/bare-metal/1.9/getting-support.md.txt#version-support)

<br />

After you create a cluster with `bmctl`, you can update the custom resources of
that cluster. The configuration file is stored as
`bmctl-workspace/CLUSTER-NAME/CLUSTER-NAME.yaml` unless you specified a
different location.

## Add or remove nodes in a cluster

In Google Distributed Cloud, you add or remove nodes in a cluster by
editing the cluster's node pool definitions. You can use the `bmctl`
command to change these definitions.

There are three different kinds of node pools in Google Distributed Cloud: control
plane, load balancer, and worker node pools.

### View node status

You can view the status of nodes and their respective node pools with the
`kubectl get` command.

For example, the following command shows the status of the node pools in the
cluster namespace `my-cluster`:

```bash
kubectl -n my-cluster get nodepools.baremetal.cluster.gke.io
```

The system returns results similar to the following:

    NAME                    READY   RECONCILING   STALLED   UNDERMAINTENANCE   UNKNOWN
    my-cluster              3       0             0         0                  0
    my-cluster-lb           2       0             0         0                  0
    np1                     3       0             0         0                  0

If you need more information on diagnosing your clusters, see
[Create snapshots for diagnosing clusters](https://docs.cloud.google.com/anthos/clusters/docs/bare-metal/1.9/troubleshooting/bmctl-snapshot.md.txt).

### Change nodes

Most node changes are specified in the cluster config file, which is then
applied to the cluster. We recommend you use the cluster config file as the
primary source for updating your cluster. It is a best practice to store your
config file in a version control system to track changes for troubleshooting
purposes. For all cluster types, use the `bmctl update` command to update your
cluster with your node changes.

The Google Distributed Cloud cluster config file includes a header section with
credential information. The credential entries and the rest of the config file
are valid YAML, but the credential entries are not valid for the cluster
resource. Use `bmctl update credentials` for credential updates.

When you remove nodes from a cluster, they are first drained of any pods. Nodes
will not be removed from the cluster if pods can't be rescheduled on other
nodes. The `bmctl update` command will parse the cluster configuration file and
apply custom resources based on the parsed result.

Here's a sample configuration with two nodes:

```yaml
---
apiVersion: baremetal.cluster.gke.io/v1
kind: NodePool
metadata:
  name: nodepool1
  namespace: cluster-cluster1
spec:
  clusterName: cluster1
  nodes:
  - address: 172.18.0.5
  - address: 172.18.0.6
```

You can remove a node from the node pool by deleting its entry:

```yaml
---
apiVersion: baremetal.cluster.gke.io/v1
kind: NodePool
metadata:
  name: nodepool1
  namespace: cluster-cluster1
spec:
  clusterName: cluster1
  nodes:
  - address: 172.18.0.5
```

To update the cluster, run the following command for the self-managing clusters,
such as admin and standalone clusters:

```bash
bmctl update cluster -c CLUSTER_NAME \
    --kubeconfig=KUBECONFIG
```

> [!NOTE]
> **Note:** `--kubeconfig` is the alias of `--admin-kubeconfig`. Use `--kubeconfig` for self-managing clusters and use `--admin-kubeconfig` for user clusters.

After the `bmctl update` command is executed successfully, it will take a while
for `machine-init` or `machine-reset` pods to be done.

The following sections describe some important differences for updating specific
node types.

#### Control plane and load balancer nodes

The control plane and load balancer node pool specifications for
Google Distributed Cloud are special. These specifications declare and control
critical cluster resources. The canonical source for these resources is their
respective sections in the cluster config file:

- `spec.controlPlane.nodePoolSpec`
- `spec.LoadBalancer.nodePoolSpec`

You add or remove control plane or load balancer nodes by editing the array of
addresses under `nodes` in the corresponding section of the
cluster config file.

> [!NOTE]
> **Note:** Do not modify the top-level control plane and load balancer node pool resources directly. Editing the control plane node pool directly doesn't work.

In a high availability (HA) configuration, an odd number of control plane
node pools (three or more) are required to establish a quorum to ensure
that if a control plane fails, others will take over. If you have an even number
of nodes temporarily while ading or removing nodes for maintenance or
replacement, your deployment maintains HA as long as you have enough quorum.

#### Worker nodes

You can add or remove worker nodes directly with the `bmctl` command. Worker
node pools must have at least one desired node. However, if you'd like to remove
the whole node pool, use the `kubectl` command. In the following example,
the command deletes a node pool named `np1`, where the variable for the cluster
namespace is `my-cluster`:

```bash
kubectl -n my-cluster delete nodepool np1
```

## Other mutable fields

Besides adding and removing nodes, you can also use the `bmctl update` command
to modify certain elements of your cluster configuration. Typically, to update
your cluster resource, you edit your local version of the cluster configuration
file and use `bmctl update` to apply your changes. The `bmctl update` command is
similar to the `kubectl apply` command.

> [!IMPORTANT]
> **Important:** Many aspects of your cluster configuration are immutable and can't be updated after you create your cluster. For a comprehensive list of mutable and immutable fields, see the [Cluster configuration field reference](https://docs.cloud.google.com/anthos/clusters/docs/bare-metal/1.9/reference/cluster-config-ref.md.txt#cluster_configuration_fields). The field reference is a sortable table. Click the column headings to change the sort order. Click a field name to view its description.

The following sections outline some common examples for updating an existing
cluster by either changing a field value or modifying a related custom resource.

### `loadBalancer.addressPools`

The [`addressPools`](https://docs.cloud.google.com/anthos/clusters/docs/bare-metal/1.9/reference/cluster-config-ref.md.txt#loadbalancer-addresspools)
section contains fields for specifying load-balancing pools for bundled load
balancers. You can add more load-balancing address pools at any time, but you
can't remove or modify any existing address pools.

```yaml
addressPools:
- name: pool1
  addresses:
  - 192.168.1.0-192.168.1.4
  - 192.168.1.240/28
- name: pool2
addresses:
- 192.168.1.224/28
```

### `bypassPreflightCheck`

The default value of the
[`bypassPreflightCheck`](https://docs.cloud.google.com/anthos/clusters/docs/bare-metal/1.9/reference/cluster-config-ref.md.txt#bypasspreflightcheck)
field is `false`. If you set this field to `true` in the cluster configuration
file, the internal preflight checks are ignored you apply resources to existing
clusters.

```yaml
apiVersion: baremetal.cluster.gke.io/v1
kind: Cluster
metadata:
  name: cluster1
  namespace: cluster-cluster1
  annotations:
    baremetal.cluster.gke.io/private-mode: "true"
spec:
  bypassPreflightCheck: true
```

### `loginUser`

You can set the [`loginUser`](https://docs.cloud.google.com/anthos/clusters/docs/bare-metal/1.9/reference/cluster-config-ref.md.txt#nodeaccess-loginuser)
field under the node access configuration. This field supports passwordless
`sudo` capability for machine login.

```yaml
apiVersion: baremetal.cluster.gke.io/v1
kind: Cluster
metadata:
  name: cluster1
  namespace: cluster-cluster1
  annotations:
    baremetal.cluster.gke.io/private-mode: "true"
spec:
  nodeAccess:
    loginUser: abm
```

### BGPLoadBalancer

`bmctl` allows you to apply and update BGP based load-balancers through the
Google packaged Bundled LB option.

```yaml
apiVersion: "networking.gke.io/v1alpha1"
kind: BGPLoadBalancer
metadata:
 name: default
 namespace: cluster-cluster1
spec:
 localASN: 65001
 peers:
 - peerASN: 65000
   peerIP: 10.0.1.1
```

### `NetworkAttachmentDefinition`

You can use the `bmctl update` command to modify `NetworkAttachmentDefinition`
custom resources that correspond to the network.

```yaml
apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
  name: gke-network-1
  namespace: cluster-my-cluster
spec:
  config: '{
  "type": "ipvlan",
  "master": "enp2342",
  "mode": "l2",
  "ipam": {
    "type": "whereabouts",
    "range": "172.120.0.0/24"
```

After you modify the config file, you can update the cluster by running the
`bmctl update` command. It will parse the cluster config file and apply custom
resources based on the parsed result.

For the self-managing clusters, such as admin and standalone clusters, run:

```bash
bmctl update cluster -c CLUSTER_NAME --kubeconfig=KUBECONFIG
```

For user clusters, run:

```bash
bmctl update cluster -c CLUSTER_NAME --admin-kubeconfig=ADMIN_KUBECONFIG
```

> [!NOTE]
> **Note:** `--kubeconfig` is the alias of `--admin-kubeconfig`. Use `--kubeconfig` for self-managing clusters and use `--admin-kubeconfig` for user clusters.