Google Distributed Cloud software only for VMware clusters need outbound access to Google Cloud services and
container registries, and they exchange traffic with vCenter Server, load
balancers, and each other. This document lists the addresses to allowlist in
your proxy server and the firewall rules to create for admin clusters, user
clusters, and other components so that your proxy and firewalls permit this
traffic.

## Allowlisting addresses for your proxy

If your organization requires outbound traffic to pass through a proxy server,
allowlist the following addresses in your proxy server. Note that
`www.googleapis.com` is needed, instead of `googleapis.com`:

- dl.google.com ^[**1**](https://docs.cloud.google.com/kubernetes-engine/distributed-cloud/vmware/docs/how-to/firewall-rules.md.txt#note1)^
- gcr.io
- us.gcr.io
- www.googleapis.com
- accounts.google.com
- anthos.googleapis.com
- anthosgke.googleapis.com
- cloudresourcemanager.googleapis.com
- compute.googleapis.com
- connectgateway.googleapis.com
- container.googleapis.com
- gkeconnect.googleapis.com ^[**2**](https://docs.cloud.google.com/kubernetes-engine/distributed-cloud/vmware/docs/how-to/firewall-rules.md.txt#note2)^
- gkehub.googleapis.com
- gkeonprem.googleapis.com
- gkeonprem.mtls.googleapis.com
- iam.googleapis.com
- iamcredentials.googleapis.com
- kubernetesmetadata.googleapis.com
- logging.googleapis.com
- monitoring.googleapis.com
- oauth2.googleapis.com
- opsconfigmonitoring.googleapis.com
- securetoken.googleapis.com
- servicecontrol.googleapis.com
- serviceusage.googleapis.com
- storage.googleapis.com
- sts.googleapis.com
- releases.hashicorp.com (Optional) ^[**3**](https://docs.cloud.google.com/kubernetes-engine/distributed-cloud/vmware/docs/how-to/firewall-rules.md.txt#note3)^

**Notes**:

^**1**^ `dl.google.com` is required by the Google Cloud
SDK installer.

^**2**^ If your cluster was registered to the fleet
using a Google Cloud region, you need to allowlist
`REGION-gkeconnect.googleapis.com` (for
example, `us-central1-gkeconnect.googleapis.com`). If you didn't specify a
region, the cluster uses the global Connect service instance, and you
allowlist `gkeconnect.googleapis.com`. If you need to find your cluster's
fleet membership location, run `gcloud container fleet memberships list`. For
more information, see
[`gkeConnect.location`](https://docs.cloud.google.com/kubernetes-engine/distributed-cloud/vmware/docs/how-to/user-cluster-configuration-file-latest.md.txt#gkeconnect-location-field).

^**3**^ If you don't use the Terraform client on your
admin workstation to run commands such as `terraform apply`, then you don't
need to allowlist `releases.hashicorp.com`. If you do use the Terraform client
on your admin workstation, you can optionally allowlist `releases.hashicorp.com`
so that you can check if the Terraform client version that you are using is the
latest by running the
[`terraform version`](https://developer.hashicorp.com/terraform/cli/commands/version)
command.

Also, if your vCenter Server has an external IP address, allowlist its address
in your proxy server.

> [!NOTE]
> **Note:** You cannot use [SSL MITM proxy](https://mitmproxy.org/) for your proxy, because this requires automatically deploying an extra CA certificate for the proxy, which is not supported.

## Firewall rules for admin clusters

The admin cluster IP addresses depend on whether Controlplane V2 is enabled
on the user cluster and the version in which the cluster was created.

- When Controlplane V2 is enabled, the control plane for a user cluster runs
  on the user cluster itself. When Controlplane V2 isn't enabled, the control
  plane for a user cluster runs on one or more nodes in the admin cluster,
  which is referred to as kubeception.

- In 1.28 and higher, new HA admin clusters don't have add-on nodes.

The IP addresses of admin cluster add-on nodes (if they exist) and kubeception
user cluster control plane nodes are listed in the admin cluster
[IP block file](https://docs.cloud.google.com/kubernetes-engine/distributed-cloud/vmware/docs/how-to/ip-block-file.md.txt). The admin cluster control
plane nodes are configured in the `network.controlPlaneIPBlock.ips`
section in the admin cluster configuration file.

Because the IP addresses in the admin cluster IP block file are not assigned
to specific nodes, you must make sure that all of the firewall rules listed in
the following table apply to all of the IP addresses available for the admin
cluster.

> [!NOTE]
> **Note:** In the following table, an admin cluster worker node refers to either an add-on node or a kubeception user cluster control plane node.

Set up your firewall rules to allow the following traffic.

| From | Source port | To | Port | Protocol | Description |
|---|---|---|---|---|---|
| Admin cluster control-plane node | 1024 - 65535 | vCenter Server API | 443 | TCP/https | Cluster resizing. |
| Admin cluster add-on nodes | 1024 - 65535 | vCenter Server API | 443 | TCP/https | User cluster lifecycle management. |
| Admin cluster add-on nodes | 32768 - 60999 | VIP of the admin cluster's Kubernetes API server VIPs of user clusters' Kubernetes API servers | 443 | TCP/https | User cluster create. User cluster update. User cluster upgrade. User cluster delete. |
| Admin cluster control-plane nodes | 32768 - 60999 | gcr.io cloudresourcemanager.googleapis.com compute.googleapis.com iam.googleapis.com oauth2.googleapis.com serviceusage.googleapis.com storage.googleapis.com Any \*.googleapis.com URL required for the services enabled for the admin or user clusters VIPs of user clusters' Kubernetes API servers VIP of the admin cluster's Kubernetes API server vCenter Server API Admin cluster F5 BIG-IP API User cluster F5 BIG-IP API Admin cluster NTP servers User cluster NTP servers Admin cluster DNS servers User cluster DNS servers | 443 | TCP/https | Preflight checks (validation). When you create, update, or upgrade user clusters. When you create, update or upgrade the admin cluster. |
| Admin cluster control plane nodes | 32768 - 60999 | User cluster On-premises local Docker registry | Depends on your registry | TCP/https | Preflight checks (validation). Required if a user cluster is configured to use a local private Docker registry instead of gcr.io. When you create or upgrade user clusters. When you create or upgrade the admin cluster. |
| Admin cluster control-plane nodes | 32768 - 60999 | Admin cluster nodes User cluster nodes Admin cluster Load Balancer VIPs User cluster Load Balancer VIPs | <br /> | icmp | Preflight checks (validation). When you create, update or upgrade user clusters. When you create, update or upgrade the admin cluster. |
| Admin cluster control-plane nodes | 32768 - 60999 | User cluster worker nodes | 22 | ssh | Preflight checks (validation). When you upgrade user clusters. When you upgrade the admin cluster. |
| User cluster control-plane node (kubeception only) | 1024 - 65535 | vCenter Server API | 443 | TCP/https | Cluster resizing. |
| User cluster control-plane node (kubeception only) | 1024 - 65535 | cloudresourcemanager.googleapis.com gkeconnect.googleapis.com or <var translate="no">REGION</var>-gkeconnect.googleapis.com gkehub.googleapis.com | 443 | TCP/https | Access is required for fleet registration. See [note 2](https://docs.cloud.google.com/kubernetes-engine/distributed-cloud/vmware/docs/how-to/firewall-rules.md.txt#note2) after the list of URLs to allowlist. |
| User cluster control-plane node (kubeception only) | 1024 - 65535 | F5 BIG-IP API | 443 | TCP/https | <br /> |
| User cluster control-plane node (kubeception only) | 1024 - 65535 | On-premises local Docker registry | Depends on your registry | TCP/https | Required if Google Distributed Cloud is configured to use a local private Docker registry instead of gcr.io. |
| User cluster control-plane node (kubeception only) | 1024 - 65535 | gcr.io oauth2.googleapis.com storage.googleapis.com Any \*.googleapis.com URL required for the services enabled for the admin cluster | 443 | TCP/https | Download images from public Docker registries. Not required if using a private Docker registry. |
| Cloud Logging Collector, which runs on an admin cluster add-on node | 1024 - 65535 | oauth2.googleapis.com logging.googleapis.com servicecontrol.googleapis.com storage.googleapis.com www.googleapis.com | 443 | TCP/https | <br /> |
| Cloud Metadata Collector, which runs on an admin cluster add-on node | 1024 - 65535 | opsconfigmonitoring.googleapis.com | 443 | TCP/https | <br /> |
| Cloud Monitoring Collector, which runs on an admin cluster add-on node | 1024 - 65535 | oauth2.googleapis.com monitoring.googleapis.com | 443 | TCP/https | <br /> |
| Admin cluster control-plane node | 1024 - 65535 | F5 BIG-IP API | 443 | TCP/https | <br /> |
| Admin cluster control-plane node | 1024 - 65535 | On-premises local Docker registry | Depends on your registry | TCP/https | Required if Google Distributed Cloud is configured to use a local private Docker registry instead of gcr.io. |
| Admin cluster control-plane node | 1024 - 65535 | gcr.io oauth2.googleapis.com storage.googleapis.com Any \*.googleapis.com URL required for the services enabled for the admin cluster | 443 | TCP/https | Download images from public Docker registries. Not required if using a private Docker registry. |
| Admin cluster worker nodes | 1024 - 65535 | Admin cluster worker nodes | All | 179 - bgp 443 - https 5473 - Calico/Typha 9443 - Envoy metrics 10250 - kubelet node port | All worker nodes must be layer-2 adjacent and without any firewall. |
| Admin cluster nodes | 1024 - 65535 | Admin cluster pod CIDR | all | any | External traffic gets SNAT'ed on the first node and sent to pod IP. |
| Admin cluster worker nodes | all | User cluster nodes | 22 | ssh | Required for kubeception. API server to kubelet communication over an SSH tunnel. This should be skipped for Controlplane V2. |
| Admin cluster nodes | 1024 - 65535 | IPs of Seesaw LB VMs of the admin cluster | 20255,20257 | TCP/http | LB config push and metrics monitoring. Only needed if you are using Bundled LB Seesaw. |
| Admin cluster nodes | 1024 - 65535 | Admin cluster nodes | 7946 | TCP/UDP | MetalLB health check. Only needed if you are using Bundled LB MetalLB. |
| Admin cluster nodes | All | User cluster control-plane VIP | 443 | https | Required for Controlplane V2. Allow nodes and Pods in the admin cluster to talk to the Kubernetes API server of the user cluster. |
| Admin cluster nodes | All | User cluster control-plane nodes | 443 | https | Required for Controlplane V2. Allow nodes and Pods in the admin cluster to talk to the Kubernetes API server of the user cluster by using the IP address of a user cluster control-plane node. |

## Firewall rules for user cluster nodes

The IP addresses of user cluster nodes are listed in the
[IP block file](https://docs.cloud.google.com/kubernetes-engine/distributed-cloud/vmware/docs/how-to/ip-block-file.md.txt).

As with the admin cluster nodes, you don't know which IP address will be used
for which node. Thus, all of the rules in the user cluster nodes apply to each
user cluster node.

| From | Source port | To | Port | Protocol | Description |
|---|---|---|---|---|---|
| User cluster control-plane node (Controlplane V2 only) | 1024 - 65535 | vCenter Server API | 443 | TCP/https | Cluster resizing. |
| User cluster control-plane node (Controlplane V2 only) | 1024 - 65535 | cloudresourcemanager.googleapis.com gkeconnect.googleapis.com or <var translate="no">REGION</var>-gkeconnect.googleapis.com gkehub.googleapis.com | 443 | TCP/https | Access is required for fleet registration. See [note 2](https://docs.cloud.google.com/kubernetes-engine/distributed-cloud/vmware/docs/how-to/firewall-rules.md.txt#note2) after the list of URLs to allowlist. |
| User cluster control-plane node (Controlplane V2 only) | 1024 - 65535 | On-premises local Docker registry | Depends on your registry | TCP/https | Required if Google Distributed Cloud is configured to use a local private Docker registry instead of gcr.io. |
| User cluster control-plane node (Controlplane V2 only) | 1024 - 65535 | gcr.io oauth2.googleapis.com storage.googleapis.com Any \*.googleapis.com URL required for the services enabled for the admin cluster | 443 | TCP/https | Download images from public Docker registries. Not required if using a private Docker registry. |
| User cluster control-plane node (Controlplane V2 only) | 1024 - 65535 | F5 BIG-IP API | 443 | TCP/https | <br /> |
| User cluster control-plane node (Controlplane V2 only) | 1024 - 65535 | User cluster nodes | 14443 | TCP | ANG Webhook Service. Required if `advancedNetworking` is enabled. `kube-apiserver` communicates with `ang-controller-manager` for admission checks. |
| User cluster worker nodes | all | gcr.io oauth2.googleapis.com storage.googleapis.com Any \*.googleapis.com URL required for the services enabled for this cluster | 443 | TCP/https | Download images from public Docker registries. Not required if using a private Docker registry. |
| User cluster worker nodes | all | F5 BIG-IP API | 443 | TCP/https | <br /> |
| User cluster worker nodes | all | VIP of the pushprox server, which runs in the Admin cluster. | 8443 | TCP/https | Prometheus traffic. |
| User cluster worker nodes | all | User cluster worker nodes | all | 22 - ssh 179 - bgp 443 - https 5473 - calico-typha 9443 - envoy metrics 10250 - kubelet node port | All worker nodes must be layer-2 adjacent and without any firewall. |
| User cluster worker nodes | all | User control plane VIP | 443 | TCP/https | <br /> |
| User cluster worker nodes | All | User control plane VIP | 8132 | GRPC | Required for kubeception. Konnectivity connection. This should be skipped for Controlplane V2. |
| Admin cluster nodes | All | User cluster vCenter Server | 443 | https | Allow the admin cluster to manage the lifecycle of the user cluster. Required if the admin and user clusters have different vCenter Servers. |
| Admin cluster nodes | 1024 - 65535 | User cluster nodes | 14443 | TCP | ANG Webhook Service. Required if `advancedNetworking` is enabled. `kube-apiserver` communicates with `ang-controller-manager` for admission checks. |
| User cluster nodes | 1024 - 65535 | User cluster pod CIDR | all | any | External traffic gets SNAT'ed on the first node and sent to pod IP. |
| Cloud Logging Collector, which runs on a random user cluster worker node | 1024 - 65535 | oauth2.googleapis.com logging.googleapis.com servicecontrol.googleapis.com www.googleapis.com | 443 | TCP/https | <br /> |
| Connect agent, which runs on a random user cluster worker node. | 1024 - 65535 | cloudresourcemanager.googleapis.com gkeconnect.googleapis.com or <var translate="no">REGION</var>-gkeconnect.googleapis.com gkehub.googleapis.com www.googleapis.com iam.googleapis.com iamcredentials.googleapis.com oauth2.googleapis.com securetoken.googleapis.com sts.googleapis.com accounts.google.com | 443 | TCP/https | [Connect](https://docs.cloud.google.com/kubernetes-engine/fleet-management/docs/connect-agent.md.txt) traffic. See [note 2](https://docs.cloud.google.com/kubernetes-engine/distributed-cloud/vmware/docs/how-to/firewall-rules.md.txt#note2) after the list of URLs to allowlist. |
| Cloud Metadata Collector, which runs on a random user cluster worker node | 1024 - 65535 | opsconfigmonitoring.googleapis.com kubernetesmetadata.googleapis.com | 443 | TCP/https | <br /> |
| Cloud Monitoring Collector, which runs on a random user cluster worker node | 1024 - 65535 | oauth2.googleapis.com monitoring.googleapis.com | 443 | TCP/https | <br /> |
| User cluster nodes | 1024 - 65535 | IPs of Seesaw LB VMs of the user cluster | 20255,20257 | TCP/http | LB config push and metrics monitoring. Only needed if you are using Bundled LB Seesaw. |
| User cluster nodes with enableLoadBalancer=true | 1024 - 65535 | User cluster nodes with enableLoadBalancer=true | 7946 | TCP/UDP | MetalLB health check. Only needed if you are using Bundled LB MetalLB. |
| User cluster network | all | User cluster control plane VIP | 443 | TCP/https | <br /> |

## Firewall rules for remaining components

These rules apply to all other components not listed in the tables for the admin cluster and user cluster nodes.

| From | Source port | To | Port | Protocol | Description |
|---|---|---|---|---|---|
| Admin cluster pod CIDR | 1024 - 65535 | Admin cluster pod CIDR | all | any | Inter-pod traffic does L2 forwarding directly using source and destination IP within Pod CIDR. |
| Admin cluster pod CIDR | 1024 - 65535 | Admin cluster nodes | all | any | Return traffic of external traffic. |
| User cluster pod CIDR | 1024 - 65535 | User cluster pod CIDR | all | any | Inter-pod traffic does L2 forwarding directly using source and destination IP within Pod CIDR. |
| User cluster pod CIDR | 1024 - 65535 | User cluster nodes | all | any | Return traffic of external traffic. |
| Clients and application end users | all | VIP of Istio ingress | 80, 443 | TCP | End user traffic to the ingress service of a user cluster. |
| Jump server to deploy the admin workstation | ephemeral port range | vCenter Server API ESXi VMkernel (mgt) IPs of hosts in target cluster | 443 | TCP/https | Check ephemeral port range from \`cat /proc/sys/net/ipv4/ip_local_port_range\`. |
| Admin workstation | 32768 - 60999 | gcr.io cloudresourcemanager.googleapis.com oauth2.googleapis.com storage.googleapis.com Any \*.googleapis.com URL required for the services enabled for this cluster | 443 | TCP/https | Download Docker images from public Docker registries. |
| Admin workstation | 32768 - 60999 | gcr.io cloudresourcemanager.googleapis.com compute.googleapis.com iam.googleapis.com oauth2.googleapis.com serviceusage.googleapis.com storage.googleapis.com Any \*.googleapis.com URL required for the services enabled for the admin or user clusters VIPs of user clusters' Kubernetes API servers VIP of the admin cluster's Kubernetes API server vCenter Server API F5 BIG-IP API | 443 | TCP/https | Preflight checks (validation). When you create, update, upgrade, or delete clusters using `gkectl`. |
| Admin workstation | 32768 - 60999 | vCenter Server API F5 BIG-IP API | 443 | TCP/https | Admin cluster create. User cluster create. |
| Admin workstation | 32768 - 60999 | ESXi VMkernel (mgt) IPs of hosts in target cluster | 443 | TCP/https | The admin workstation uploads the OVA to the datastore through the ESXi hosts. |
| Admin workstation | 32768 - 60999 | VIP of the admin cluster's Kubernetes API server VIPs of user clusters' Kubernetes API servers | 443 | TCP/https | Admin cluster create. Admin cluster update. User cluster create. User cluster update. User cluster delete. |
| Admin workstation | 32768 - 60999 | Admin cluster control-plane node and worker nodes | 443 | TCP/https | Admin cluster create. Control plane upgrades. |
| Admin workstation | 32768 - 60999 | All admin cluster nodes and all user cluster nodes | 443 | TCP/https | Network validation as part of the `gkectl check-config` command. |
| Admin workstation | 32768 - 60999 | VIP of the admin cluster's Istio ingress VIP of user clusters' Istio ingress | 443 | TCP/https | Network validation as part of the `gkectl check-config` command. |
| Admin workstation | 32768 - 60999 | oauth2.googleapis.com logging.googleapis.com monitoring.googleapis.com servicecontrol.googleapis.com storage.googleapis.com www.googleapis.com | 443 | TCP/https | Cloud logging and monitoring access. |
| Admin workstation | 32768 - 60999 | IPs of Seesaw LB VMs in both admin and user clusters Seesaw LB VIPs of both admin and user clusters | 20256,20258 | TCP/http/gRPC | Health check of LBs. Only needed if you are using Bundled LB Seesaw. |
| Admin workstation | 32768 - 60999 | Node IP of the cluster control plane | 22 | TCP | Required if you need SSH access from the admin workstation to the admin cluster control plane. |
| Admin workstation | 32768 - 60999 | releases.hashicorp.com | 443 | TCP/https | Optional. See [note 3](https://docs.cloud.google.com/kubernetes-engine/distributed-cloud/vmware/docs/how-to/firewall-rules.md.txt#note3) after the list of URLs to allowlist. |
| LB VM IPs | 32768 - 60999 | node IPs of the corresponding cluster | 10256: node health check 30000 - 32767: healthCheckNodePort | TCP/http | Node health check. healthCheckNodePort is for services with externalTrafficPolicy set to Local. Only needed if you are using Bundled LB Seesaw. |
| F5 Self-IP | 1024 - 65535 | All admin and all user cluster nodes | 30000 - 32767 | any | For the data plane traffic that F5 BIG-IP load balances via a virtual server VIP to the node ports on the Kubernetes cluster nodes. Typically the F5 self-ip is on the same network/subnet as the Kubernetes cluster nodes. |