This page explains how to configure clusters created in Google Kubernetes Engine (GKE)
Standard mode to perform [IP masquerade](https://docs.cloud.google.com/kubernetes-engine/docs/concepts/ip-masquerade-agent.md.txt)
with the `ip-masq-agent`. For more information about IP masquerading in
GKE [Autopilot mode](https://docs.cloud.google.com/kubernetes-engine/docs/concepts/autopilot-overview.md.txt),
see [Use Egress NAT Policy to configure IP masquerade in Autopilot clusters](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/egress-nat-policy-ip-masq-autopilot.md.txt).

## Before you begin


Before you start, make sure that you have performed the following tasks:

- Enable the Google Kubernetes Engine API.
[Enable Google Kubernetes Engine API](https://console.cloud.google.com/apis/enableflow?apiid=container.googleapis.com)
- To use the Google Cloud CLI for this task, [install](https://docs.cloud.google.com/sdk/docs/install) and then [initialize](https://docs.cloud.google.com/sdk/docs/initialize.md.txt) the gcloud CLI. If you previously installed the gcloud CLI, get the latest version by running the `gcloud components update` command. Earlier gcloud CLI versions might not support running the commands in this document.

  > [!NOTE]
  > **Note:** For existing gcloud CLI installations, make sure to set the `compute/region` [property](https://docs.cloud.google.com/sdk/docs/properties.md.txt#setting_properties). If you use primarily zonal clusters, set the `compute/zone` instead. By setting a default location, you can avoid errors in the gcloud CLI like the following: `One of [--zone, --region] must be supplied: Please specify location`. You might need to specify the location in certain commands if the location of your cluster differs from the default that you set.

<!-- -->

- Ensure that you have an existing Standard cluster. If you need one, [create a Standard cluster](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/creating-a-regional-cluster.md.txt).

## Checking `ip-masq-agent` status

This section shows you how to:

- Determine whether your cluster has an `ip-masq-agent` DaemonSet.
- Check the `ip-masq-agent` [ConfigMap](https://kubernetes.io/docs/concepts/configuration/configmap/) resource.

### Check the `ip-masq-agent` DaemonSet

To check if your cluster is running the `ip-masq-agent` DaemonSet, use the
Google Cloud CLI or the Google Cloud console.

### gcloud

1. Get the credentials for your cluster:

   ```bash
   gcloud container clusters get-credentials CLUSTER_NAME
   ```

   Replace `CLUSTER_NAME` with the name of your cluster.
2. Search for the `ip-masq-agent` in the `kube-system` namespace:

   ```bash
   kubectl get daemonsets/ip-masq-agent -n kube-system
   ```

   If the `ip-masq-agent` DaemonSet exists, then the output is
   similar to the following:

       NAME            DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
       ip-masq-agent   3         3         3       3            3           <none>          13d

   If the `ip-masq-agent` DaemonSet does not exist, then the output is
   similar to the following:

       Error from server (NotFound): daemonsets.apps "ip-masq-agent" not found

3. Check if the `ip-masq-agent` DaemonSet is running the latest version:

   ```bash
   kubectl get daemonsets/ip-masq-agent -n kube-system -o jsonpath='{.spec.template.spec.containers[].image}'
   ```

   This command must return the same container image as specified in
   [Deploying the ip-masq-agent DaemonSet](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/ip-masquerade-agent.md.txt#deploy-ip-masq-agent-deamonset).

### Console

1. Go to the **Workloads** page in the Google Cloud console.

   [Go to Workloads](https://console.cloud.google.com/kubernetes/workload)
2. For *Filter*, do the following:

   1. Click to clear the **Is system object: False** filter.
   2. Filter the following properties:
      - **Name** : `ip-masq-agent`.
      - **Cluster**: the name of your cluster.

   If `ip-masq-agent` DaemonSet exists, then you can see the DaemonSet record in the table.
   If `ip-masq-agent` DaemonSet does not exist, then no rows are displayed.

To create the `ip-masq-agent` ConfigMap and deploy the `ip-masq-agent` DaemonSet,
refer to [Configuring and deploying the `ip-masq-agent`](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/ip-masquerade-agent.md.txt#config-ip-masq-agent).

### Checking the `ip-masq-agent` ConfigMap

To check if your cluster is running the `ip-masq-agent` ConfigMap, use the
Google Cloud CLI or the Google Cloud console.

### gcloud

1. Get the credentials for your cluster:

   ```bash
   gcloud container clusters get-credentials CLUSTER_NAME
   ```

   Replace `CLUSTER_NAME` with the name of your cluster.
2. Describe the `ip-masq-agent` ConfigMap in the `kube-system` namespace:

   ```bash
   kubectl describe configmaps/ip-masq-agent -n kube-system
   ```

   If the `ip-masq-agent` ConfigMap exists, then the output is
   similar to the following:

       Name:         ip-masq-agent
       Namespace:    kube-system
       Labels:       <none>
       Annotations:  <none>

       Data
       ====
       config:
       ---
       nonMasqueradeCIDRs:
         - 198.15.5.92/24
         - 10.0.0.0/8
       masqLinkLocal: false
       resyncInterval: 60s

       BinaryData
       ====

       Events:  <none>

   If the `ip-masq-agent` ConfigMap does not exist, then the output is
   similar to the following:

       Error from server (NotFound): configmaps "ip-masq-agent" not found

### Console

1. Go to the **Configuration** page in the Google Cloud console.

   [Go to Configuration](https://console.cloud.google.com/kubernetes/config)
2. For *Filter*, do the following:

   1. Click to clear the **Is system object: False** filter.
   2. Filter the following properties:
      - **Name** : `ip-masq-agent`.
      - **Cluster**: the name of your cluster.

   If `ip-masq-agent` ConfigMap exists, then you can see the ConfigMap record in the table.
   If `ip-masq-agent` ConfigMap does not exist, then no rows are displayed.

If the cluster already has the `ip-masq-agent` ConfigMap, you can configure and
deploy it.

## Configuring and deploying the `ip-masq-agent`

This section shows you how to create or edit the `ip-masq-agent`
ConfigMap and how to deploy or delete the `ip-masq-agent` DaemonSet. To
determine which tasks you need to perform, you must first [determine whether your
cluster already has the `ip-masq-agent` ConfigMap and `ip-masq-agent`
DaemonSet](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/ip-masquerade-agent#check-ip-masq-agent).

### Creating the `ip-masq-agent` ConfigMap

The following steps show how to create the `ip-masq-agent` ConfigMap. If
your cluster already has the `ip-masq-agent` ConfigMap, [edit an existing
ip-masq-agent ConfigMap instead](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/ip-masquerade-agent#edit-ip-masq-agent-configmap).

1. Create a configuration file using the following template and save it locally.
   You can use any name for the local copy of this configuration file.

   ```bash
   nonMasqueradeCIDRs:
     - CIDR_1
     - CIDR_2
   masqLinkLocal: false
   resyncInterval: SYNC_INTERVALUNIT_OF_TIME
   ```

   Replace the following:
   - `CIDR_1` and
     `CIDR_2`: the IP address ranges in [CIDR
     format](https://en.wikipedia.org/wiki/Classless_Inter-Domain_Routing).
     When packets are sent to these destinations, your cluster does not
     masquerade IP address sources and preserves source Pod IP addresses.
     If you need more than two CIDRs, add more entries to the
     `nonMasqueradeCIDRs` list following the same format. At
     minimum, the `nonMasqueradeCIDRs` property should include the node and Pod IP
     address ranges of your cluster.

   - `SYNC_INTERVAL`: the amount of time after which
     each `ip-masq-agent` Pod checks the contents of the `ip-masq-agent`
     ConfigMap and writes any changes to its local `/etc/config/ip-masq-agent`
     file. Defaults to `60`.

   - `UNIT_OF_TIME`: the unit of time for the resyncInterval.
     Valid values include `s` (for seconds) or `ms` (for milliseconds). Defaults
     to `s`.

   Set `masqLinkLocal` to false (the default) unless you need to enable
   masquerading for packets sent to link local IPv4
   addresses. For more information, see
   [Masquerading to link-local destinations](https://docs.cloud.google.com/kubernetes-engine/docs/concepts/ip-masquerade-agent.md.txt#masqLinkLocal).

   > [!CAUTION]
   > **Caution:** A custom `nonMasqueradeCIDRs` must at least include the cluster's node and Pod IP address range(s). This is especially important if you enable [intranode visibility](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/intranode-visibility.md.txt). If the Pod IP address range(s) are not present in `nonMasqueradeCIDRs` and you have intranode visibility turned on, Pods won't be able to properly communicate with each other.

2. Create the ConfigMap resource:

   ```bash
   kubectl create configmap ip-masq-agent \
      --namespace=kube-system \
      --from-file=config=LOCAL_CONFIG_FILE_PATH
   ```

   Replace `LOCAL_CONFIG_FILE_PATH` with the path to the
   config file you created in the previous step.

   > [!NOTE]
   > **Note:** The ConfigMap content must be associated with a named key that the `ip-masq-agent` DaemonSet expects when reading the `ip-masq-agent` ConfigMap to parse it as a volume. In the following example, and for standard `ip-masq-agent` DaemonSets, the key is `config`. Specifying the `--from-file=config=LOCAL_CONFIG_FILE_PATH` flag helps ensure that the IP masquerade configuration information is associated with the `config` key.

3. Describe the `ip-masq-agent` ConfigMap in the `kube-system` namespace:

   ```bash
   kubectl describe configmaps/ip-masq-agent -n kube-system
   ```

   The output is similar to the following:

       Name:         ip-masq-agent
       Namespace:    kube-system
       Labels:       <none>
       Annotations:  <none>

       Data
       ====
       config:
       ---
       nonMasqueradeCIDRs:
         - 198.15.5.92/24
         - 10.0.0.0/8
       masqLinkLocal: false
       resyncInterval: 60s

       BinaryData
       ====
       Events:  <none>

   This output includes the `config` parameter with your configuration changes.
   You can now
   [deploy the `ip-masq-agent` DeamonSet](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/ip-masquerade-agent.md.txt#deploy-ip-masq-agent-deamonset).

### Editing an existing `ip-masq-agent` ConfigMap

You can modify the contents of an existing `ip-masq-agent` ConfigMap by
completing the following steps:

1. Open the ConfigMap in a text editor:

   ```bash
   kubectl edit configmap ip-masq-agent --namespace=kube-system
   ```
2. Edit the content of the ConfigMap file:

   ```yaml
   apiVersion: v1
   data:
     config: |
       nonMasqueradeCIDRs:
         - CIDR_1
         - CIDR_2
       masqLinkLocal: false
       resyncInterval: SYNC_INTERVALUNIT_OF_TIME
   kind: ConfigMap
   metadata:
     name: ip-masq-agent
     namespace: kube-system
   ```

   Replace the following:
   - `CIDR_1` and
     `CIDR_2`: the IP address ranges in [CIDR
     format](https://en.wikipedia.org/wiki/Classless_Inter-Domain_Routing).
     When packets are sent to these destinations, your cluster does not
     masquerade IP address sources and preserves source Pod IP addresses. If you
     need more than two CIDRs, add more entries to the `nonMasqueradeCIDRs`
     list following the same format. At minimum, the `nonMasqueradeCIDRs`
     property should include the node and Pod IP address ranges of your cluster.

   - `SYNC_INTERVAL`: the amount of time after which
     each `ip-masq-agent` Pod checks the contents of the `ip-masq-agent`
     ConfigMap and writes any changes to its local `/etc/config/ip-masq-agent`
     file. Defaults to `60`.

   - `UNIT_OF_TIME`: the unit of time for the resyncInterval.
     Valid values include `s` (for seconds) or `ms` (for milliseconds). Defaults
     to `s`.

   Set `masqLinkLocal` to false (the default) unless you need to enable
   masquerading for packets sent to link local IPv4
   addresses. For more information, see
   [Masquerading to link-local destinations](https://docs.cloud.google.com/kubernetes-engine/docs/concepts/ip-masquerade-agent.md.txt#masqLinkLocal).

   > [!CAUTION]
   > **Caution:** A custom `nonMasqueradeCIDRs` must at least include the cluster's node and Pod IP address range(s). This is especially important if you enable [intranode visibility](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/intranode-visibility.md.txt). If the Pod IP address range(s) are not present in `nonMasqueradeCIDRs` and you have intranode visibility turned on, Pods won't be able to properly communicate with each other.

3. Describe the `ip-masq-agent` ConfigMap in the `kube-system` namespace:

   ```bash
   kubectl describe configmaps/ip-masq-agent -n kube-system
   ```

   The output is similar to the following:

       Name:         ip-masq-agent
       Namespace:    kube-system
       Labels:       <none>
       Annotations:  <none>

       Data
       ====
       config:
       ---
       nonMasqueradeCIDRs:
         - 198.15.5.92/24
         - 10.0.0.0/8
       masqLinkLocal: false
       resyncInterval: 60s

       BinaryData
       ====

       Events:  <none>

   This output includes the `config` parameter which matches the configuration
   value from the file you created.

### Deploying the `ip-masq-agent` DaemonSet

After you create or edit your `ip-masq-agent` ConfigMap, deploy the
`ip-masq-agent` DaemonSet.

1. Save the following manifest as a YAML file:

   ```yaml
   apiVersion: apps/v1
   kind: DaemonSet
   metadata:
     name: ip-masq-agent
     namespace: kube-system
   spec:
     selector:
       matchLabels:
         k8s-app: ip-masq-agent
     template:
       metadata:
         labels:
           k8s-app: ip-masq-agent
       spec:
         hostNetwork: true
         containers:
         - name: ip-masq-agent
           image: gke.gcr.io/ip-masq-agent:v2.12.6-gke.15@sha256:c51e89bdc5073fdbaaff8bf42b32765f7f1eb0fec7d6189de5f14c48ac489448
           args:
           - --v=2
           - --logtostderr=false
           - --log_file=/dev/stdout
           - --log_file_max_size=0
           # The masq-chain must be IP-MASQ
           - --masq-chain=IP-MASQ
           # To non-masquerade reserved IP ranges by default,
           # uncomment the following line.
           # - --nomasq-all-reserved-ranges
           # Must be set to false when using Dataplane V2.
           - --random-fully=false
           securityContext:
             privileged: false
             capabilities:
               drop: ["ALL"]
               add: ["NET_ADMIN", "NET_RAW"]
             allowPrivilegeEscalation: false
             seccompProfile:
               type: RuntimeDefault
           volumeMounts:
           - name: config-volume
             mountPath: /etc/config
         volumes:
         - name: config-volume
           configMap:
             name: ip-masq-agent
             optional: true
             items:
             - key: config
               path: ip-masq-agent
         tolerations:
         - effect: NoSchedule
           operator: Exists
         - effect: NoExecute
           operator: Exists
         - key: "CriticalAddonsOnly"
           operator: "Exists"
   ```

   This manifest creates a volume named `config-volume` which is mounted as
   specified by the container's volumeMount.

   If you need to edit this manifest, consider the following conditions:
   - The volume name can be anything but must match the container's `volumeMount`
     name.

   - The ConfigMap name must match the name of the `configMap`
     referenced in the `config-volume` Volume in the Pod.

   - The name of the chain (`--masq-chain`) must be `IP-MASQ`. Otherwise,
     GKE does not override the default masquerading rules.

   - DaemonSet Pods read from the `ip-masq-agent` file. The `ip-masq-agent` file
     content is the value of the `config` key in the ConfigMap.

   - If you use non-masquerade reserved IP ranges by
     default, uncomment the `- --nomasq-all-reserved-ranges` line in the `arg`
     section.

2. Deploy the DaemonSet:

   ```bash
   kubectl apply -f LOCAL_FILE_PATH
   ```

   Replace `LOCAL_FILE_PATH` with the path to the file
   you created in the previous step.

You can manually update the `ip-masq-agent` DaemonSet you created. To learn more, see
[Updating a DaemonSet](https://kubernetes.io/docs/concepts/workloads/controllers/daemonset/#updating-a-daemonset).

## Deleting the `ip-masq-agent`

This section shows you how to delete the `ip-masq-agent` DaemonSet and the
`ip-masq-agent` ConfigMap. Deleting the `ip-masq-agent` doesn't revert the existing
IP masquerading settings on the nodes.

### Deleting the `ip-masq-agent` DaemonSet

If you manually created the `ip-masq-agent` DaemonSet, you can delete it by
running the following command:

```bash
kubectl delete daemonsets ip-masq-agent -n kube-system
```

> [!NOTE]
> **Note:** If the `ip-masq-agent` DaemonSet was deployed automatically, GKE will automatically recreate it.

### Deleting the `ip-masq-agent` ConfigMap

To completely delete the `ip-masq-agent` ConfigMap, run the following command:

```bash
kubectl delete configmap ip-masq-agent -n kube-system
```

## Troubleshooting

The following sections provide troubleshooting advice.

### General troubleshooting

The following steps help you diagnose issues with IP address masquerading:

- [Confirm the status of the `ip-masq-agent`](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/ip-masquerade-agent.md.txt#check-ip-masq-agent). If the ConfigMap is not defined, traffic to all default destinations is [not masqueraded](https://docs.cloud.google.com/kubernetes-engine/docs/concepts/ip-masquerade-agent.md.txt#how_ipmasq_works) and preserves the Pod IP address. Traffic to other destinations preserves the node IP address.
- [Confirm the version of the `ip-masq-agent` image specified in the DaemonSet](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/ip-masquerade-agent.md.txt#check-ip-masq-agent-daemonset). If the version of the `ip-masq-agent` DaemonSet is not the latest, [follow the deployment steps to update the DaemonSet](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/ip-masquerade-agent.md.txt#deploy-ip-masq-agent-deamonset).
- Check whether the IP-MASQ chain is correctly populated in the NAT IP tables by running the `sudo iptables -t nat -L IP-MASQ` command in the impacted node. If the `nonMasqueradeCIDRs` defined in the ConfigMap is not showing up on the NAT IP Tables, verify that there are no typos in the config file that was used to create the ConfigMap.
- Confirm that the destination allows both the node and Pod IP address ranges.
- If traffic is not accessible from the node or Pod, [run a connectivity test](https://docs.cloud.google.com/network-intelligence-center/docs/connectivity-tests/how-to/running-connectivity-tests.md.txt).

### Issue: Pod IP address changes to node IP address

While using an IP masquerade agent, you might notice the Pod's source IP
address unexpectedly use the node's IP address when your Pods communicate
with external destinations.

The issue is caused by a missing or incomplete custom Source Network Address
Translation (SNAT) list. When you use an IP masquerade agent, the default SNAT
of the cluster is used when the ConfigMap is missing or it doesn't contain a
`nonMasqueradeCIDRs` list. When a packet leaves a Pod, the default SNAT changes
the source IP address from the Pod's IP address to the node's internal IP
address. To learn more about SNAT, see
[IP masquerade agent](https://docs.cloud.google.com/kubernetes-engine/docs/concepts/ip-masquerade-agent.md.txt).

To resolve the issue, configure a custom SNAT list by defining a
`nonMasqueradeCIDRs` list in the `ip-masq-agent` ConfigMap:

1. Open the `ip-masq-agent` ConfigMap:

   ```bash
   kubectl edit configmap ip-masq-agent --namespace=kube-system
   ```
2. Review the `nonMasqueradeCIDRs` list within the ConfigMap. The
   `nonMasqueradeCIDRs` list should be present and complete. For example:

   ```yaml
   ...
   nonMasqueradeCIDRs:
     - 35.100.0.0/16
   ...
   ```
3. If the list is missing or incomplete, add or modify the `nonMasqueradeCIDRs`
   list to include the destination IP ranges for which you want to preserve
   source Pod IP addresses. This list should also include the addresses that
   you want to use Cloud NAT.

   If you don't want any external traffic to use SNAT, set the
   `nonMasqueradeCIDRs` field to `0.0.0.0/0`. For example:

   ```yaml
   ...
   nonMasqueradeCIDRs:
     - 0.0.0.0/0
   ...
   ```

   When your traffic doesn't use SNAT, all packets
   sent from Pods retain the Pod's IP address as their source IP address,
   regardless of destination.
4. Check the source IP address of outbound packets from your Pods. To check
   this address, take a packet capture. If that's not feasible, you can check
   the IP address of the node, which should be the source IP address of
   outbound packets subject to SNAT, with the following command:

   ```bash
   kubectl get nodes -o wide
   ```

## What's next

- Learn how to [use Egress NAT Policy to configure IP masquerade in Autopilot clusters](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/egress-nat-policy-ip-masq-autopilot.md.txt).
- Learn how to [create a VPC-native cluster](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/alias-ips.md.txt).
- Read the [GKE network overview](https://docs.cloud.google.com/kubernetes-engine/docs/concepts/network-overview).
- Learn about [configuring authorized networks](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/authorized-networks).