<button class="button button-with-icon cloud-select-dropdown__button button-primary" aria-haspopup="true">Version 1.10 (unsupported)</button>

- [Latest](https://docs.cloud.google.com/anthos/clusters/docs/bare-metal/latest/limits)
- [Version 1.16](https://docs.cloud.google.com/anthos/clusters/docs/bare-metal/1.16/limits.md.txt)
- [Earlier versions (unsupported)](https://docs.cloud.google.com/anthos/clusters/docs/bare-metal/1.10/getting-support.md.txt#version-support)

<br />

This page explains Google Distributed Cloud release 1.10 quotas and
limits for Google Cloud projects, clusters, and nodes.

## Limits

Note the following limits and recommendations for your clusters.

### Maximum number of pods per cluster

We recommend that you limit the number of pods per cluster to 15,000 or fewer.
For example, if your cluster has 200 nodes, you should restrict the number of
pods per node to 75 or fewer. Likewise, if you want to run 110 pods per node,
you should restrict the number of nodes in your cluster to 136 or fewer. The
following table provides examples of configurations that are and aren't
recommended.

| Pods per node | Nodes per cluster | Pods per Cluster | Result |
|---|---|---|---|
| 110 | 200 | 22,000 | No |
| 110 | 136 | 14,960 | Yes |
| 100 | 150 | 15,000 | Yes |
| 75 | 200 | 15,000 | Yes |

The maximum number of pods per cluster recommendation takes precedence over the
recommendations for pods per node and nodes per cluster in the following
sections.

### Maximum number of nodes per cluster

We test Google Distributed Cloud to run workloads with up to 500 nodes. However,
to ensure optimal performance and reliability, we recommend that you don't
exceed 200 nodes per cluster when running workloads in production.

| Cluster type | Minimum nodes | Recommended maximum nodes | Absolute maximum nodes |
|---|---|---|---|
| User, Standalone, or Hybrid | 1 | 200 | 500 |

For single-node clusters, you must remove the
`node-role.kubernetes.io/master:NoSchedule` taint to run workloads on the node.
For details, see
[Kubernetes taints and tolerations](https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/).

### Maximum number of pods per node

Google Distributed Cloud supports the configuration of maximum pods per node in
the `nodeConfig.PodDensity.MaxPodsPerNode` setting of the [cluster configuration
file](https://docs.cloud.google.com/anthos/clusters/docs/bare-metal/1.10/reference/cluster-config#cluster_configuration_file). The
following table shows the minimum and maximum values supported for
`MaxPodsPerNode`, which includes pods running add-on services:

| Cluster type | Minimum allowed value | Recommended maximum value | Maximum allowed value |
|---|---|---|---|
| All HA clusters and non-HA user clusters | 32 | 110 | 250 |
| All other non-HA clusters | 64 | 110 | 250 |

### Maximum number of endpoints

On RHEL and CentOS, there's a cluster-level limitation of 100,000 endpoints.
This number is the sum of all pods that are referenced by a Kubernetes service.
If two services reference the same set of pods, this situation counts as two
separate sets of endpoints. The underlying `nftable` implementation on RHEL and
CentOS causes this limitation; it's not an intrinsic limitation of
Google Distributed Cloud.

**Mitigation**

For RHEL and CentOS, there are no mitigations. For Ubuntu and Debian
systems, we recommend
[switching from the default `iptables` to legacy `iptables`](https://wiki.debian.org/nftables#Reverting_to_legacy_xtables)
on large-scale clusters.

### Dataplane V2 eBPF limit

The maximum number of entries in the BPF lbmap for Dataplane V2 is 65,536.
Increases in the following areas can cause the total number of entries to grow:

- Number of services
- Number of ports per service
- Number of backends per service

We recommend that you monitor the actual number of entries used by your cluster
to ensure that you don't exceed the limit. Use the following command to get the
current entries:

```bash
kubectl get po -n kube-system -l k8s-app=cilium | cut -d " " -f1 | grep anetd | head -n1 | \
    xargs -I % kubectl -n kube-system exec % -- cilium bpf lb list | wc -l
```

We also recommend that you use your own monitoring pipeline to collect metrics
from the `anetd` DaemonSet. Monitor for the following conditions to identify
when the number of entries are causing problems:

```json
cilium_bpf_map_ops_total{map_name="lb4_services_v2",operation="update",outcome="fail" } > 0
```

```json
cilium_bpf_map_ops_total{map_name="lb4_backends_v2",operation="update",outcome="fail" } > 0
```

### LoadBalancer and NodePort Services port limit

The port limit for LoadBalancer and NodePort Services is 2,768. The default
port range is 30000-32767. If you exceed the limit, you can't create new
LoadBalancer or NodePort Services and you can't add new node ports for existing
services.

Use the following command to check the number of ports currently allocated:

```bash
kubectl get svc -A | grep : | tr -s ' ' | cut -d ' '  -f6 | tr ',' '\n' | wc -l
```

### Bundled load balancer node connection limits

The number of connections allowed for each node used for bundled load balancing
(MetalLB) is 28,000. The default ephemeral port range for these connections is
32768-60999. If you exceed the connection limit, requests to the LoadBalancer
Service might fail.

If you need to expose a load balancer service that is capable of handling a
substantial number of connections (for Ingress, for example), we recommend that
you consider an alternate load balancing method to avoid this limitation with
MetalLB.

## Cluster quotas

You can register a maximum of 15 clusters by default. To register more clusters
in GKE Hub, you can submit a request to increase your quota in the Google Cloud
console:

[Go to Quotas](https://console.cloud.google.com/apis/api/gkehub.googleapis.com/quotas)

> [!NOTE]
> **Note:** Each user cluster must have a unique name across *all* admin clusters that are within a single Google Cloud project.

Didn't find what you were looking for? Click **Send feedback** and let us know
what's missing.