<button class="button button-with-icon cloud-select-dropdown__button button-primary" aria-haspopup="true">Version 1.11 </button>

- [Latest version](https://docs.cloud.google.com/anthos/clusters/docs/on-prem/latest/how-to/managing-node-pools)
- [Version 1.16](https://docs.cloud.google.com/anthos/clusters/docs/on-prem/1.16/how-to/managing-node-pools.md.txt)
- [Earlier versions (unsupported)](https://docs.cloud.google.com/anthos/clusters/docs/on-prem/latest/version-history#expandable-1)

<br />

When you create a user cluster, you have to configure at least one
[node pool](https://docs.cloud.google.com/kubernetes-engine/docs/concepts/node-pools.md.txt), which is a group of
nodes that all have the same configuration. After the cluster is created, you
can add new node pools, update node pool settings, and delete node pools.

How you create, update, and delete node pools depends on whether the cluster is
managed by the Anthos On-Prem API. A user cluster is managed by the
Anthos On-Prem API if one of the following is true:

- The cluster was created in the Google Cloud console, which automatically
  configures the Anthos On-Prem API to manage the cluster.

- The command `gkectl enroll cluster` was run on a user cluster, which
  [configures it to be managed by the Anthos On-Prem API](https://docs.cloud.google.com/anthos/clusters/docs/on-prem/1.11/how-to/enroll-user-cluster).

If the Anthos On-Prem API is managing a user cluster, you can use the
Google Cloud console to manage node pools. If the user cluster isn't managed
by the Anthos On-Prem API, use `gkectl` on the command line of your admin
workstation to manage node pools.

## Add a node pool

If you created the cluster in the Google Cloud console, you can use the
Google Cloud console to add a node pool; however, you need to use the
command line to configure the following node pool settings:

- [Windows for the OS image type](https://docs.cloud.google.com/anthos/clusters/docs/on-prem/1.11/how-to/user-cluster-configuration-file.md.txt#nodepool-osimage-field)
- [vSphere datastore](https://docs.cloud.google.com/anthos/clusters/docs/on-prem/1.11/how-to/user-cluster-configuration-file.md.txt#nodepool-vsphere-datastore-field)
- [vSphere tags](https://docs.cloud.google.com/anthos/clusters/docs/on-prem/1.11/how-to/user-cluster-configuration-file.md.txt#nodepool-vsphere-tags-field)
  If you want to attach tags to all VMs in a node pool, your vCenter user account
  must have these
  [vSphere tagging privileges](https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.security.doc/GUID-2F8C5B02-7EF6-428A-BDB6-B2CC3D307097.html):

  - vSphere Tagging.Assign or Unassign vSphere Tag
  - vSphere Tagging.Assign or Unassign vSphere Tag on Object (vSphere 7)
- [Automatic scaling for the node pool](https://docs.cloud.google.com/anthos/clusters/docs/on-prem/1.11/how-to/user-cluster-configuration-file.md.txt#nodepool-autoscaling-section)

Before you add another node pool,
[verify that enough IP addresses are available](https://docs.cloud.google.com/anthos/clusters/docs/on-prem/1.11/how-to/resizing-a-user-cluster.md.txt#verify_ips)
on the cluster.

### Console

1. In the Google Cloud console, go to the GKE Enterprise clusters page.

   [Go to the GKE Enterprise clusters page](https://console.cloud.google.com/anthos/clusters)
2. Select the Google Cloud project that the user cluster is in.

3. In the cluster list, click the name of the cluster, and then click
   **View details** in the **Details** panel.

4. Click **Add node pool**.

5. Configure the node pool:

   1. Enter the **Node pool name**.
   2. Enter the number of **vCPUs** for each node in the pool (minimum 4 per user cluster worker).
   3. Enter the **memory** size in mebibytes (MiB) for each node in the pool (minimum 8192 MiB per user cluster worker node and must be a multiple of 4).
   4. In the **Nodes** field, enter the number of nodes in the pool (minimum of 3).
   5. Select the **OS image type** : **Ubuntu Containerd** , **Ubuntu** ,
      or **COS**.

      > [!NOTE]
      > **Note:** The **Ubuntu** OS Image type for node pools will be deprecated in Google Distributed Cloud version 1.12 and unsupported in version 1.13. Consider changing your node pools now to use either **Ubuntu
      > Containerd** or **COS** for the OS Image type.

   6. Enter the **Boot disk size** in gibibytes (GiB) (default is 40 GiB).

6. In the **Node pool metadata (optional)** section, if you want to add
   Kubernetes
   [labels](https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/)
   and
   [taints](https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/),
   do the following:

   1. Click **+ Add Kubernetes Labels** . Enter the **Key** and **Value** for the label. Repeat as needed.
   2. Click **+ Add Taint** . Enter the **Key** , **Value** , and **Effect** for the taint. Repeat as needed.
7. Click **Create**.

8. The Google Cloud console displays **Cluster status: changes in
   progress** . Click **Show Details** to view the **Resource status
   condition** and **Status messages**.

### Command line

1. In your
   [user cluster configuration file](https://docs.cloud.google.com/anthos/clusters/docs/on-prem/1.11/how-to/user-cluster-configuration-file.md.txt),
   fill in the
   [`nodePools`](https://docs.cloud.google.com/anthos/clusters/docs/on-prem/1.11/how-to/user-cluster-configuration-file.md.txt#nodepools-section)
   section.

   You must specify the following fields:
   - `nodePools.[i].name`
   - `nodePools[i].cpus`
   - `nodePools.[i].memoryMB`
   - `nodePools.[i].replicas`

   The following fields are optional. If you don't include
   [`nodePools[i].bootDiskSizeGB`](https://docs.cloud.google.com/anthos/clusters/docs/on-prem/1.11/how-to/user-cluster-configuration-file.md.txt#nodepool-bootdisksizegb-field)
   or
   [`nodePools[i].osImageType`](https://docs.cloud.google.com/anthos/clusters/docs/on-prem/1.11/how-to/user-cluster-configuration-file.md.txt#nodepool-osimagetype-field),
   the default values are used.
   - `nodePools[i].labels`
   - `nodePools[i].taints`
   - `nodePools[i].bootDiskSizeGB`
   - `nodePools[i].osImageType`
   - `nodePools[i].vsphere.datastore`
   - `nodePools[i].vsphere.tags`

   > [!NOTE]
   > **Note:** The **Ubuntu** OS Image type for node pools will be deprecated in Google Distributed Cloud version 1.12 and unsupported in version 1.13. Consider changing your node pools now to use either **Ubuntu
   > Containerd** or **COS** for the OS Image type.

2. Run the following command:

   ```
   gkectl update cluster --kubeconfig ADMIN_CLUSTER_KUBECONFIG --config USER_CLUSTER_CONFIG
   ```

   Replace the following:
   - `[ADMIN_CLUSTER_KUBECONFIG]` with the path of the
     kubeconfig file for your admin cluster.

   - `[USER_CLUSTER_CONFIG]` with the path of your user cluster
     configuration file.

### Example configuration

In the following example configuration, there are four node pools, each with
different attributes:

- `pool-1`: only the minimum required attributes are specified
- `pool-2`: includes `vsphere.datastore` and `vsphere.tags`
- `pool-3`: includes `taints` and `labels`
- `pool-4`: includes `osImageType`and `bootDiskSizeGB`

```yaml
nodePools:
- name: pool-1
  cpus: 4
  memoryMB: 8192
  replicas: 5
- name: pool-2
  cpus: 8
  memoryMB: 16384
  replicas: 3
  vsphere:
    datastore: my_datastore
    tags:
    - category: "purpose"
      name: "testing"
- name: pool-3
  cpus: 4
  memoryMB: 8192
  replicas: 5
  taints:
    - key: "example-key"
      effect: NoSchedule
  labels:
    environment: production
    app: nginx
- name: pool-4
  cpus: 4
  memoryMB: 8192
  replicas: 5
  osImageType: cos
  bootDiskSizeGB: 40
```

## Update a node pool

You can use the command line to update all fields in the
in the
[`nodePools`](https://docs.cloud.google.com/anthos/clusters/docs/on-prem/1.11/how-to/user-cluster-configuration-file.md.txt#nodepools-section)
section of your user cluster configuration file. Currently, the only node pool
fields that you can update using the Google Cloud console are:

- Number of replicas
- Memory
- Number of vCPUs

When you increase the number of replicas, Google Distributed Cloud adds the
required number of nodes to the user cluster, and when you decrease the number
of replicas, nodes are removed. Changing the number of replicas for a node pool
doesn't interrupt workloads. Make sure that you have IP addresses available
if you increase the number of replicas.

If you update any other node pool field, this triggers a
[rolling update](https://kubernetes.io/docs/tutorials/kubernetes-basics/update/update-intro/)
on the cluster. In a rolling update, Google Distributed Cloud creates a new node
and then deletes an old node. This process is repeated until all the old nodes
have been replaced with new nodes. This process doesn't cause downtime, but
the cluster must have an extra IP address available to use during the update.

Suppose a node pool will have N nodes at the end of an update. Then you must
have at least N + 1 IP addresses available for nodes in that pool. This means
that if you are resizing a cluster by adding nodes to one or more pools, you
must have at least one more IP address than the total number of nodes that will
be in all of the cluster's node pools at the end of the resizing.
For more information, see
[Verify that enough IP addresses are available](https://docs.cloud.google.com/anthos/clusters/docs/on-prem/1.11/how-to/resizing-a-user-cluster.md.txt#verify_ips).

To update a node pool on a user cluster:

### Console

1. In the Google Cloud console, go to the GKE Enterprise clusters page.

   [Go to the GKE Enterprise clusters page](https://console.cloud.google.com/anthos/clusters)
2. Select the Google Cloud project that the user cluster is in.

3. In the cluster list, click the name of the cluster, and then click
   **View details** in the **Details** panel.

4. Click the **Nodes** tab.

5. Click the name of the node pool that you want to modify.

6. Click **Edit** next to the field
   that you want to modify, and click **Done**.

7. Click to go back to the
   previous page.

8. The Google Cloud console displays **Cluster status: changes in
   progress** . Click **Show Details** to view the **Resource status
   condition** and **Status messages**.

### Command line

1. Modify the values for the fields that you want to change in the
   [`nodePools`](https://docs.cloud.google.com/anthos/clusters/docs/on-prem/1.11/how-to/user-cluster-configuration-file.md.txt#nodepools-section)
   section of the user cluster configuration file.

2. Update the cluster:

   ```
   gkectl update cluster --kubeconfig ADMIN_CLUSTER_KUBECONFIG --config USER_CLUSTER_CONFIG
   ```

   Replace the following:
   - `[ADMIN_CLUSTER_KUBECONFIG]` with the path of the
     kubeconfig file for your admin cluster.

   - `[USER_CLUSTER_CONFIG]` with the path of your user cluster
     configuration file.

### Update the `osImageType` used by a node pool

To update a node pool to use a different `osImageType`, you have to use the
command line. To change the `osImageType` used by a node pool, update the
configuration file for the node pool, as shown in the following example, and run
`gkectl update cluster`.

```
nodePools:
- name: np-1
  cpus: 4
  memoryMB: 8192
  replicas: 3
  osImageType: ubuntu_containerd
```

> [!WARNING]
> **Warning:** As Docker Engine support will be removed in Kubernetes 1.24, `ubuntu` OS image type [will not be supported](https://docs.cloud.google.com/anthos/clusters/docs/on-prem/1.11/concepts/using-containerd.md.txt). You should plan to convert your node pools to use the `ubuntu_containerd` or `cos` OS image type as soon as possible.

## Verify your changes

To verify that your node pools have been created or updated as intended,
inspect the cluster nodes:

### Console

1. In the Google Cloud console, go to the GKE Enterprise clusters page.

   [Go to the GKE Enterprise clusters page](https://console.cloud.google.com/anthos/clusters)
2. Select the Google Cloud project that the user cluster is in.

3. In the cluster list, click the name of the cluster, and then click
   **View details** in the **Details** panel.

4. Click the **Nodes** tab.

5. Click the name of the node pool you want to view.

### Command line

Run the following command:

```
kubectl --kubeconfig USER_CLUSTER_KUBECONFIG get nodes -o wide
```

If you need to revert your changes, edit the cluster configuration file and
run `gkectl update cluster`.

## Delete a node pool

Although you can delete node pools, your user cluster must have at least one
node pool. Deleting a node pool causes immediate removal of the pool's nodes
regardless of whether those nodes are running workloads.

> [!CAUTION]
> **Caution:** If the cluster is managed by the Anthos On-Prem API, be sure to use the Google Cloud console to delete the node pool.

To delete a node pool from a user cluster:

### Console

1. Ensure that there are no workloads running on the affected nodes.

2. In the Google Cloud console, go to the GKE Enterprise clusters page.

   [Go to the GKE Enterprise clusters page](https://console.cloud.google.com/anthos/clusters)
3. Select the Google Cloud project that the user cluster is in.

4. In the cluster list, click the name of the cluster, and then click
   **View details** in the **Details** panel.

5. Click the **Nodes** tab.

6. Click the name of the node pool that you want to delete.

7. Click **Delete**.

8. Click to go back to the
   previous page.

9. The Google Cloud console displays **Cluster status: changes in
   progress** . Click **Show Details** to view the **Resource status
   condition** and **Status messages**.

### Command line

1. Ensure that there are no workloads running on the affected nodes.

2. Remove its definition from the
   [`nodePools`](https://docs.cloud.google.com/anthos/clusters/docs/on-prem/1.11/how-to/user-cluster-configuration-file.md.txt#nodepools-section)
   section of the user cluster configuration file.

3. Update the cluster:

   ```
   gkectl update cluster --kubeconfig ADMIN_CLUSTER_KUBECONFIG --config USER_CLUSTER_CONFIG
   ```

   Replace the following:
   - `[ADMIN_CLUSTER_KUBECONFIG]` with the path of the
     kubeconfig file for your admin cluster.

   - `[USER_CLUSTER_CONFIG]` with the path of your user cluster
     configuration file.

## Troubleshooting

- In general, the `gkectl update cluster` command provides specifics when it
  fails. If the command succeeded and you don't see the nodes, you can
  troubleshoot with the
  [Diagnosing cluster issues](https://docs.cloud.google.com/anthos/clusters/docs/on-prem/1.11/diagnose.md.txt#diagnose_user_cluster)
  guide.

- It is possible that there are insufficient cluster resources like a lack of
  available IP addresses during node pool creation or update. See the [Resizing a
  user cluster](https://docs.cloud.google.com/anthos/clusters/docs/on-prem/1.11/how-to/resizing-a-user-cluster) topic
  for details about verifying that IP addresses are available.

- You can also review the general
  [Troubleshooting](https://docs.cloud.google.com/anthos/clusters/docs/on-prem/1.11/troubleshooting) guide.

- Won't proceed past `Creating node MachineDeployment(s) in user cluster...`.

  It can take a while to create or update the node pools in your user cluster.
  However, if the wait time is extremely long and you suspect that something
  might have failed, you can run the following commands:
  1. Run `kubectl get nodes` to obtain the state of your nodes.
  2. For any nodes that are not ready, run `kubectl describe node NODE_NAME` to obtain details.