This guide helps you diagnose why your workloads (Pods or VMs) cannot access
external networks using Cloud NAT. As a developer, you primarily
interact with the `CloudNatGateway` resource. The status of this resource is
your primary source of truth for debugging.

## Before you begin

To troubleshoot a Cloud NAT configuration, you must have the following:

- The necessary identity and access roles. Ask your IAM Project Admin to grant you one or both of the following roles:
  - **Cloud NAT Viewer (`cloud-nat-viewer`)**: This role offers read-only access to Cloud NAT resources. This role gets you started on diagnosing the problem.
  - **Cloud NAT Developer (`cloud-nat-developer`)**: This role provides the necessary permissions for application operators to Create, Read, Update, and Delete (CRUD) Cloud NAT objects within their assigned projects. This role lets you perform many of the fixes described on this page.
  - Additional roles may be needed for some specific diagnostic measures and fixes.

## Initial diagnostics

Before diving into error codes, ensure that the basic resources are present and
reachable.

**Command:**

    kubectl get cloudnatgateway GATEWAY_NAME -n PROJECT_NAMESPACE -o yaml

Replace the following:

- `GATEWAY_NAME`: The name of the `CloudNatGateway` resource.
- `PROJECT_NAMESPACE`: The namespace of your project.

**Check the Status Conditions:** A healthy gateway must have **all** the
following conditions set to `True`:

- `Ready`: Global health status.
- `SubnetsReady`: The IP pool configuration is valid.
- `PerimeterConfigurationReady`: The upstream network infrastructure is configured.
- `EgressRoutesReady`: The routing policies for your pods are active.

If any of these are `False`, check the `reason` and `message` fields in the
status output and refer to the tables in the following sections.

## Error code reference and remediation

The error codes returned by `kubectl get cloudnatgateway` fall into three
main categories.

### Subnet errors (`SubnetsReady` is False)

This condition indicates issues with the IP address pool assigned to the gateway.

| Error Code | What It Means | Remediation Steps |
|---|---|---|
| `CloudNATSelectorFieldOverlapsCode` | **Configuration Conflict.** The `workloadSelector` of this gateway matches the same workloads as another gateway in your project. Traffic cannot be deterministically routed. | 1. List all Cloud NAT gateways in your project: `kubectl get cloudnatgateway` 2. Compare the `workloadSelector` of this gateway with others. 3. Modify the labels so that no single Pod/VM is selected by more than one gateway. |
| `CloudNATSubnetRefsFieldInvalidCode` | **Invalid Subnet.** The subnet specified in `subnetRefs` is unusable. Common reasons: - Subnet does not exist. - Subnet is not in `Ready` state. - Subnet is not of type `Leaf`. | 1. Verify the subnet exists: `kubectl get subnet <SUBNET_NAME>` 2. Check the subnet status is `Ready`. 3. Ensure the subnet `type` is `Leaf` (Cloud NAT cannot use Root or Loopback subnets). |
| `CloudNATSubnetAlreadyInUseCode` | **Subnet Conflict.** The subnet you requested is already "owned" by another Cloud NAT gateway. A subnet can only be attached to one gateway at a time. | 1. Choose a different subnet for this gateway. 2. Or, remove the subnet from the other gateway first. |
| `UNETAPIServerErrorCode` | **System Error.** The controller cannot talk to the API server to validate subnets. | **Action:** This is likely a temporary platform issue. If it persists, contact your Platform Administrator. |

### Perimeter configuration errors (`PerimeterConfigurationReady` is False)

This condition reflects the status of the perimeter gateways.

| Error Code | What It Means | Remediation Steps |
|---|---|---|
| `NET-E0305` | **Configuration Conflict.** (Same as above). Overlapping selectors prevent the system from calculating the correct routing group. | 1. List all Cloud NAT gateways in your project: `kubectl get cloudnatgateway` 2. Compare the `workloadSelector` of this gateway with others. 3. Modify the labels so that no single Pod/VM is selected by more than one gateway. |
| `NET-E0301` | **Resource Exhaustion / Node Failure.** The system created the configuration, but couldn't assign your egress IPs to a healthy physical node. This usually means either the subnet is out of IPs or the gateway nodes are down. | 1. Check your \[Subnet\](/distributed-cloud/hosted/docs/1.16/gdcag/platform/pa-user/subnets-overview) usage. Is it full? 2. If the Subnet has free IPs and is `Ready`, this indicates a platform-side infrastructure failure (e.g., no healthy gateway nodes available). **Action**: Contact Platform Administrator. |
| `NET-E0001` | **System Error.** Controller communication failure. | **Action:** Contact Platform Administrator. |

### Egress route errors (`EgressRoutesReady` is False)

This condition reflects the status of the routing policies inside the cluster.

| Error Code | What It Means | Remediation Steps |
|---|---|---|
| `NET-E0305` | **Configuration Conflict.** (Same as above). | 1. List all Cloud NAT gateways in your project: `kubectl get cloudnatgateway` 2. Compare the `workloadSelector` of this gateway with others. 3. Modify the labels so that no single Pod/VM is selected by more than one gateway. |
| `NET-E0304` | **Programming Failure.** The system failed to program the routing rules (BPF) for your specific gateway IPs. | **Action:** This is an internal programming error or state inconsistency. 1. Try making a trivial update to the Gateway (e.g., edit a label) to trigger a reconciliation. 2. If it persists, contact the Platform Administrator. |

## Other common issues

If the Gateway status is `Ready: True` but traffic is still failing, check these common misconfigurations:

### Missing project-level permission

Your project must be explicitly authorized to egress traffic.

- **Check:** Does your Project resource have the label `networking.gdc.goog/enable-default-egress-allow-to-outside-the-org: "true"`?
- **Fix:** Ask your Project Admin to apply this label.

### Missing VM annotation (Virtual Machines Only)

VMs bypass the standard Pod egress path and need explicit instruction to use
Cloud NAT.

- **Check:** Does the `VirtualMachineExternalAccess` (VMEA) object for your VM have the annotation `egress.networking.gke.io/use-cloud-nat: "true"`?
- **Fix:** Add the annotation to the VMEA object.

### Standard cluster node egress

If you are running a Standard Cluster, the nodes themselves need permission to egress.

- **Check:** Does the `Cluster` object have the label `cluster.gdc.goog/enable-node-egress-to-outside-the-org: "true"`?
- **Fix:** Ask your Platform Admin to label the Cluster object.

### Default Egress NAT versus Cloud NAT collision

A common configuration error occurs when a workload is configured to use the
legacy Default Egress NAT mechanism while simultaneously being selected by a
Cloud NAT Gateway. This combination results in a collision where the data
plane receives conflicting routing instructions, leading to packet loss or
non-deterministic routing behavior.

#### Diagnose Pod collisions

For Pods, the Default Egress NAT is typically enabled by adding a specific
label. A Pod cannot have this label while also being targeted by a
Cloud NAT Gateway.

1. **Identify the Target Pod:** Get the labels of the Pod experiencing connectivity issues.

       kubectl get pod <POD_NAME> -n <NAMESPACE> --show-labels

2. **Check for Conflicting Labels:**

   - **Cloud NAT Selection:** Do the Pod's labels match the `workloadSelector` of any `CloudNatGateway` in the namespace?
   - **Default Egress Label:** Does the Pod have the label `egress.networking.gke.io/enabled: "true"`?

   **Condition:** If **BOTH** are true, you have a collision.
3. **Resolution:** Remove the legacy default egress label from the Pod (or its
   parent Deployment/StatefulSet) to allow Cloud NAT to take over
   exclusive control.

#### Diagnose VM collisions

For Virtual Machines, the mechanism is different. VMs with
`VirtualMachineExternalAccess` (VMEA) objects are often configured for default
access. To use Cloud NAT, they must explicitly opt-in by adding
an annotation to disable the default path and enable the
Cloud NAT path.

1. **Identify the VMEA:** Find the `VirtualMachineExternalAccess` object associated with the VM.

       kubectl get vmea -n <NAMESPACE>

2. **Check for Missing Annotation:**

   - **Cloud NAT Selection:** Do the VM's labels match a `CloudNatGateway`?
   - **Opt-in Annotation:** Check the VMEA for the annotation `egress.networking.gke.io/use-cloud-nat: "true"`.

   **Condition:** If the VM matches a gateway but **LACKS** this annotation,
   traffic will collide with the default egress system.
3. **Resolution:** Add the annotation to the VMEA object.

       kubectl annotate vmea <VMEA_NAME> -n <NAMESPACE> egress.networking.gke.io/use-cloud-nat="true"