[Dual-stack networking](https://kubernetes.io/docs/concepts/services-networking/dual-stack/)
in Google Distributed Cloud lets your clusters accept traffic from external devices
that use either [IPv4](https://en.wikipedia.org/wiki/IPv4) or
[IPv6](https://en.wikipedia.org/wiki/IPv6). Pods and nodes get both IPv4 and
IPv6 addresses, and a Kubernetes Service can have an IPv4 address, an IPv6
address, or both.

Dual-stack clusters always use
[flat mode](https://docs.cloud.google.com/kubernetes-engine/distributed-cloud/bare-metal/docs/reference/flat-vs-island-network.md.txt)
for IPv6. For IPv4, dual-stack clusters use island mode by default, but you can
configure flat mode instead. To create a dual-stack cluster, your underlying
network must be dual-stack enabled. You can't start a dual-stack cluster on a
single-stack IPv4 or IPv6 network.

## Before you begin

If your cluster nodes are running [RedHat Enterprise Linux](https://docs.cloud.google.com/kubernetes-engine/distributed-cloud/bare-metal/docs/installing/os-reqs.md.txt),
and they have [SELinux](https://www.redhat.com/en/topics/linux/what-is-selinux)
enabled, then on each node:

- In `/etc/firewalld/firewalld.conf`, set `IPv6_rpfilter=no`.

- Run `systemctl restart firewalld`.

If your cluster nodes use Stateless Address Autoconfiguration (SLAAC) for
IPv6 network configuration, see
[Node IPv6 configuration for dual-stack networking](https://docs.cloud.google.com/kubernetes-engine/distributed-cloud/bare-metal/docs/how-to/dual-stack-networking.md.txt#node-ipv6-configuration)
to configure the nodes to accept Router Advertisements (RAs).

## Overview of creating a dual-stack cluster

You can enable dual-stack networking when you create a new cluster, but you
can't enable dual-stack networking for an existing cluster.

Follow the instructions in one of the
[cluster creation documents](https://docs.cloud.google.com/kubernetes-engine/distributed-cloud/bare-metal/docs/installing/creating-clusters/create-clusters-overview.md.txt).

In your configuration file, include manifests for the following:

- A Namespace resource
- A Cluster resource
- One or more NodePool resources
- One or more `ClusterCIDRConfig` resources

Fill in the Namespace manifest and the NodePool manifests as you would for a
single-stack cluster.

In the Cluster manifest, under `clusterNetwork.services.cidrBlocks`, specify both
an IPv4 CIDR range and an IPv6 CIDR range. This is the enabling criterion for a
dual-stack cluster. That is, if you provide Service CIDR ranges for both IPv4
and IPv6, your cluster will have a dual-stack network.

In the Cluster manifest, under `clusterNetwork.pods.cidrBlocks`, specify an IPv4
CIDR range, but don't specify an IPv6 CIDR range. IPv6 CIDR ranges for Pods
are specified in `ClusterCIDRConfig` manifests.

If you are using bundled load balancing, provide both IPv4 and IPv6 addresses
in the `loadBalancer.addressPools` section of the Cluster manifest.

The `ClusterCIDRConfig` resources are for specifying IPv4 and IPv6 CIDR ranges for
Pods. You can use a single `ClusterCIDRConfig` resource to specify CIDR ranges
that are cluster-wide. That is, the IPv4 Pod addresses for all nodes are taken
from a single CIDR range, and the IPv6 Pod addresses for all nodes are taken
from a single CIDR range. Or you can use several `ClusterCIDRConfig` resources to
specify CIDR ranges that apply to a particular node pool or a particular node.

> [!NOTE]
> **Note:** Dual-stack clusters cannot be reverted to single-stack. In addition, the `ClusterCIDRConfig` object is immutable; to modify CIDR configurations, create a new `ClusterCIDRConfig` object.

## Reachability for Pod IP addresses

A dual-stack cluster uses flat mode for IPv6 networking. The example given in
this document is for a cluster that uses static flat-mode networking for IPv6.
That is, the cluster is not configured to use Border Gateway Protocol (BGP).

For a cluster that uses static flat-mode networking, you must specify node and
Pod IP addresses that are all part of the same subnet. This configuration makes
it possible for clients outside the cluster, but in the same
[Layer 2 domain](https://en.wikipedia.org/wiki/Broadcast_domain)
as the cluster nodes, to send packets directly to Pod IP addresses.

For example, suppose your cluster nodes and some other machines are all in the
same Layer 2 domain. Here's one way you could specify address ranges:

| Purpose | Range | Number of addresses |
|---|---|---|
| Entire Layer 2 domain | `fd12::/108` | 2^20^ |
| Pods | `fd12::1:0/112` | 2^16^ |
| Nodes | `fd12::2:0/112` | 2^16^ |
| Other machines | `fd12::3:0/112` | 2^16^ |
| VIPs | `fd12::4:0/112` | 2^16^ |

In the preceding example, these are the key points to understand:

- All node, Pod, and machine addresses are in the large range: `fd12::/108`.

- The Pod IP addresses are in a subset of the large range.

- The node IP addresses are in a different subset of the large range.

- The IP addresses of other machines are in a different subset of the large
  range.

- All the subset ranges are distinct from each other.

In the preceding example, each machine in the Layer 2 domain, including the cluster
nodes, must have a forwarding rule for the large range. For example:

```
inet fd12::/108 scope global eth0
```

## Node IPv6 configuration for dual-stack networking

When you enable dual-stack networking, Google Distributed Cloud automatically configures
each node to enable IPv6 packet forwarding by setting the
`net.ipv6.conf.all.forwarding` sysctl parameter to `1`. This kernel setting is
required for the cluster networking layer to route IPv6 traffic for Pods and
Services between nodes.

### Router advertisement considerations

Enabling `net.ipv6.conf.all.forwarding` has a standard Linux kernel side effect:
network interfaces stop accepting IPv6 Router Advertisements (RAs) by default.
The Linux kernel assumes that a system with packet forwarding enabled functions
as a router, and routers don't process RAs from other routers by default.

This change can disrupt nodes that rely on Stateless Address Autoconfiguration
(SLAAC) to obtain IPv6 addresses, prefixes, or default routes from the network,
which can lead to a loss of node-level IPv6 connectivity.

### Required configuration for SLAAC-dependent nodes

If your nodes use SLAAC for their IPv6 network configuration, you must manually
configure the kernel to accept RAs when IPv6 packet forwarding is enabled.

To accept RAs on an interface when IPv6 forwarding is enabled, persistently set
the `net.ipv6.conf.INTERFACE_NAME.accept_ra` sysctl
parameter to `2` on each affected node:

1. Add the sysctl setting to a configuration file in `/etc/sysctl.d/`, such as
   `/etc/sysctl.d/60-ipv6-ra.conf`:

   ```
   net.ipv6.conf.INTERFACE_NAME.accept_ra=2
   ```

   Replace `INTERFACE_NAME` with the name of the network
   interface, such as `eth0`.
2. Apply the sysctl setting:

   ```
   sudo sysctl --system
   ```

Setting `net.ipv6.conf.INTERFACE_NAME.accept_ra` to `2`
ensures that the node can route IPv6 packets for the cluster while continuing to
receive its own network configuration through RAs.

## Example: Create a dual-stack cluster

When you create a dual-stack cluster, you have various options. For example, you
could have cluster-wide CIDR ranges or you could have CIDR ranges that apply to
particular node pools. You could combine an IPv6 flat network with an IPv4
island-mode network. Or both your IPv4 and IPv6 networks could be flat. You
could use bundled load balancing or manual load balancing.

This section gives one example of how to create a dual-stack cluster. The cluster
in this example has the following characteristics:

- An IPv4 network in island mode
- An IPv6 network in flat mode
- A cluster-wide IPv4 CIDR range for Pods
- A cluster-wide IPv6 CIDR range for Pods
- A cluster-wide IPv4 CIDR range for Services
- A cluster-wide IPv6 CIDR range for Services
- An IPv4 address pool to be used for Services of type `LoadBalancer`
- An IPv6 address pool to be used for Services of type `LoadBalancer`
- Bundled load balancing

For additional configuration examples, see
[Variations on using ClusterCIDRConfig](https://docs.cloud.google.com/kubernetes-engine/distributed-cloud/bare-metal/docs/how-to/dual-stack-networking.md.txt#variations-clustercidrconfig).

### Fill in a configuration file

Follow the instructions in one of the
[cluster creation documents](https://docs.cloud.google.com/kubernetes-engine/distributed-cloud/bare-metal/docs/installing/creating-clusters/create-clusters-overview.md.txt).

In your configuration file, in the `Cluster` manifest:

- For `clusterNetwork.pods.cidrBlocks`, provide a single IPv4 CIDR range.

- For `clusterNetwork.services.cidrBlocks`, provide two CIDR ranges: one for
  IPv4 and one for IPv6.

- For `loadBalancer.addressPools`, provide two address ranges: one for IPv4 and
  one for IPv6. When you create a
  [Service of type `LoadBalancer`](https://kubernetes.io/docs/concepts/services-networking/service/#loadbalancer),
  the external IP addresses for the Service are chosen from these ranges.

Here is an example that shows the relevant portions of a Cluster manifest:

> [!CAUTION]
> **Caution:** Don't use this example as-is. Replace the address ranges with values that are appropriate for your network.

```
apiVersion: baremetal.cluster.gke.io/v1
kind: Cluster
metadata:
  name: "dual-stack"
  namespace: "cluster-dual-stack"

spec:
  clusterNetwork:
    pods:
      cidrBlocks:
      - "192.168.0.0/16"
    services
      cidrBlocks:
       - "172.16.0.0/20"
       - "fd12::5:0/116"
...
  loadBalancer:
    mode: "bundled"
    ...
    addressPools:
    - name: "pool-1"
      addresses:
       - "10.2.0.212-10.2.0.221"
       - "fd12::4:101-fd12::4:110"
```

In the same configuration file, include a manifest for a `ClusterCIDRConfig`.

- Set `ipv4.cidr` to the same CIDR range that you provided in the `Cluster`
  manifest. This is a requirement if IPv4 is in island mode.

- Set `namespace` to the same value that you provided in the `Cluster` manifest.

- Set `ipv6.cidr` to an IPv6 CIDR range for Pods.

- For each CIDR range, provide a value for `perNodeMaskSize` to specify how many
  Pod addresses will be assigned to each node. The number of IPv4 addresses
  assigned to each node must be the same as the number of IPv6 addresses
  assigned to each node. You must set your values for `perNodeMaskSize`
  accordingly. For example, if you want 2^8^ addresses per node, set your
  `perNodeMaskSize` values as follows:

  - `ipv4.perNodeMaskSize: 24` # (32 - 8 = 24)
  - `ipv6.perNodeMaskSize: 120` # (128 - 8 = 120)

Here's an example of a `ClusterCIDRConfig` manifest:

> [!CAUTION]
> **Caution:** Don't use this example as-is. Replace the address ranges with values that are appropriate for your network.

```
apiVersion: baremetal.cluster.gke.io/v1alpha1
kind: ClusterCIDRConfig
metadata:
  name: "cluster-wide-ranges"
  namespace: "cluster-dual-stack"  # Must be the same as the Cluster namespace.
spec:
  ipv4:
    cidr: "192.168.0.0/16"  #  For island mode, must be the same as the Cluster CIDR.
    perNodeMaskSize: 24
  ipv6:
    cidr: "fd12::1:0/112"
    perNodeMaskSize: 120
```

In the preceding example:

- The IPv4 Pod CIDR range has 2^(32-16)^ = 2^16^ addresses.
  The per-node mask size is 24, so the number of addresses assigned to each
  node is 2^(32-24)^ = 2^8^.

- The IPv6 Pod CIDR range has 2^(128-112)^ = 2^16^
  addresses. The per-node mask size is 120, so the number of addresses
  assigned to each node is 2^(128-120)^ = 2^8^.

#### Example configuration file

Click to see an example of a completed configuration file.

**Caution**: Don't use this example as-is. Replace the address
ranges and key paths with values that are appropriate for your network.

```
gcrKeyPath: "bmctl-workspace/.sa-keys/gcr.json"
sshPrivateKeyPath: "/home/me/.ssh/id_rsa"
gkeConnectAgentServiceAccountKeyPath: "bmctl-workspace/.sa-keys/bm-connect.json"
gkeConnectRegisterServiceAccountKeyPath: "bmctl-workspace/.sa-keys/bm-register.json"
cloudOperationsServiceAccountKeyPath: "bmctl-workspace/.sa-keys/bm-cloud-ops.json"
---
apiVersion: v1
kind: Namespace
metadata:
  name: "cluster-dual-stack"
---
apiVersion: baremetal.cluster.gke.io/v1
kind: Cluster
metadata:
  name: "dual-stack"
  namespace: "cluster-dual-stack"
spec:
  type: "user"
  anthosBareMetalVersion: "1.14.0"
  gkeConnect:
    projectID: "my-project"
  controlPlane:
    nodePoolSpec:
      nodes:
      - address: "10.2.0.4"
  clusterNetwork:
    pods:
      cidrBlocks:
      - "192.168.0.0/16"
    services:
      cidrBlocks:
       - "172.16.0.0/20"
       - "fd12::5:0/116"
  loadBalancer:
    mode: "bundled"
    ports:
      controlPlaneLBPort: 443
    vips:
      controlPlaneVIP: "10.2.0.211
      ingressVIP: "10.2.0.212"
    addressPools:
    - name: "pool-1"
      addresses:
      - "10.2.0.212-10.2.0.221"
      - "fd12::4:101-fd12::4:110
  clusterOperations:
    projectID: "my-project"
    location: "us-central1"
  storage:
    lvpNodeMounts:
      path: "/mnt/localpv-disk"
      storageClassName: "node-disk"
    lvpShare:
      numPVUnderSharedPath: 5
      path: "/mnt/localpv-share"
      storageClassName: "local-shared"
  nodeConfig:
    podDensity:
      maxPodsPerNode: 110
---
apiVersion: baremetal.cluster.gke.io/v1
kind: NodePool
metadata:
  name: "pool-1"
  namespace: "cluster-dual-stack"
spec:
  clusterName: "dual-stack"
  nodes:
  - address: "10.2.0.5"
---
apiVersion: baremetal.cluster.gke.io/v1alpha1
kind: ClusterCIDRConfig
metadata:
  name: "cluster-wide-ranges"
  namespace: "cluster-dual-stack"
spec:
  ipv4:
    cidr: "192.168.0.0/16"
    perNodeMaskSize: 24
  ipv6:
    cidr: "fd12::1:0/112"
    perNodeMaskSize: 120
```

### Finish creating the cluster

Finish creating your cluster as described in your
[cluster creation document](https://docs.cloud.google.com/kubernetes-engine/distributed-cloud/bare-metal/docs/installing/creating-clusters/create-clusters-overview.md.txt).

## View cluster nodes and Pods

List the cluster nodes:

```
kubectl --kubeconfig CLUSTER_KUBECONFIG get nodes --output yaml
```

Replace `CLUSTER_KUBECONFIG` with the path of your cluster kubeconfig
file.

In the output, you can see the IPv4 and IPv6 addresses of each node. You can
also see the IPv4 and IPv6 address ranges for Pods on the node. For example:

```
- apiVersion: v1
  kind: Node
  ...
  spec:
    podCIDR: 192.168.1.0/24
    podCIDRs:
    - 192.168.1.0/24
    - fd12::1:100/120
    providerID: baremetal://10.2.0.5
  status:
    addresses:
    - address: 10.2.0.5
      type: InternalIP
    - address: fd12::2:5
      type: InternalIP
```

List the Pods in the cluster:

```
kubectl --kubeconfig CLUSTER_KUBECONFIG get pods --all-namespaces
```

Choose one Pod, and list the details. For example:

```
kubectl --kubeconfig CLUSTER_KUBECONFIG get pod gke-metrics-agent-b9qrv \
  --namespace kube-system \
  -- output yaml
```

In the output, you can see the IPv4 and IPv6 addresses of the Pod. For example:

```
apiVersion: v1
kind: Pod
metadata:
  ...
  name: gke-metrics-agent-b9qrv
  namespace: kube-system
...
status:
  ...
  podIPs:
  - ip: 192.168.1.146
  - ip: fd12::1:11a
```

## Variations on using ClusterCIDRConfig

The preceding example used a `ClusterCIDRConfig` object to specify cluster-wide
Pod CIDR ranges. That is, a single IPv4 CIDR range is used for all Pods in the
cluster. And a single IPv6 CIDR range is used for all Pods in the cluster.

In certain situations, you might not want to use a single CIDR range for all
Pods in a cluster. For example, you might want to specify a separate CIDR range
for each node pool, or you might want to specify a separate CIDR range for each
node. For more information on `ClusterCIDRConfig` and examples on
using it, see [Understand the ClusterCIDRConfig custom resource](https://docs.cloud.google.com/kubernetes-engine/distributed-cloud/bare-metal/docs/reference/cluster-cidr-config.md.txt).

For example, the following `ClusterCIDRConfig` specifies a CIDR range for a node
pool named `"workers"`.

> [!CAUTION]
> **Caution:** Don't use these examples as they appear here. Replace the address ranges with values that are appropriate for your network.

```
apiVersion: baremetal.cluster.gke.io/v1alpha1
kind: ClusterCIDRConfig
metadata:
  name: "worker-pool-ccc"
  namespace: "cluster-dual-stack"
spec:
  ipv4:
    cidr: "192.168.0.0/16"
    perNodeMaskSize: 24
  ipv6:
    cidr: "fd12::1:0/112"
    perNodeMaskSize: 120
  nodeSelector:
    matchLabels:
      baremetal.cluster.gke.io/node-pool: "workers"
```

The following `ClusterCIDRConfig` specifies a CIDR range for a single
node that has IP address `10.2.0.5`:

```
apiVersion: baremetal.cluster.gke.io/v1alpha1
kind: ClusterCIDRConfig
metadata:
  name: "range-node1"
  namespace: "cluster-dual-stack"
spec:
  ipv4:
    cidr: "192.168.1.0/24"
    perNodeMaskSize: 24
  ipv6:
    cidr: "fd12::1:0/120"
    perNodeMaskSize: 120
  nodeSelector:
    matchLabels:
      baremetal.cluster.gke.io/k8s-ip: "10.2.0.5"
```

## Create a dual-stack Service of type `ClusterIP`

Here is a manifest for a Deployment:

```
apiVersion: apps/v1
kind: Deployment
metadata:
  name: "my-deployment"
spec:
  selector:
    matchLabels:
      app: "try-dual-stack"
  replicas: 3
  template:
    metadata:
      labels:
        app: "try-dual-stack"
    spec:
      containers:
      - name: "hello"
        image: "us-docker.pkg.dev/google-samples/containers/gke/hello-app:2.0"
```

Save the manifest in a file named `my-deployment.yaml`, and create the Deployment:

```
kubectl --kubeconfig CLUSTER_KUBECONFIG apply -f my-deployment.yaml
```

Replace `CLUSTER_KUBECONFIG` with the path of your cluster kubeconfig
file.

Here is a manifest for a Service of type `ClusterIP`:

```
apiVersion: v1
kind: Service
metadata:
  name: "my-service"
spec:
  selector:
    app: "try-dual-stack"
  type: "ClusterIP"
  ipFamilyPolicy: "RequireDualStack"
  ipFamilies:
  - "IPv6"
  - "IPv4"
  ports:
  - port: 80
    targetPort: 8080
```

In the context of this exercise, these are the key points to understand about
the preceding Service manifest:

- The `ipFamilyPolicy` field is set to `RequireDualStack` This means both IPv6
  and IPv4 `ClusterIP` addresses are allocated for the Service.

- The `ipFamilies` field specifies the IPv6 family first and the IPv4 family
  second. This means that `spec.ClusterIP` for the Service will be an IPv6
  address chosen from `clusterNetwork.services.cidrBlocks` in the Cluster
  manifest.

Save the manifest in a file named `my-cip-service.yaml`, and create the Service:

```
kubectl --kubeconfig CLUSTER_KUBECONFIG apply -f my-cip-service.yaml
```

List details about the Service:

```
kubectl --kubeconfig CLUSTER_KUBECONFIG get service my-service --output yaml
```

In the output, you can see the cluster IP addresses for the Service. For
example:

```
apiVersion: v1
kind: Service
metadata:
  name: my-service
  …
spec:
  clusterIP: fd12::5:9af
  clusterIPs:
  - fd12::5:9af
  - 172.16.12.197
```

On a cluster node, call the Service:

```
curl IPV4_CLUSTER_IP
curl IPV6_CLUSTER_IP
```

The output displays a "Hello world" message:

```
Hello, world!
Version: 2.0.0
Hostname: my-deployment-xxx
```

## Create a dual-stack Service of type `LoadBalancer`

Here is a manifest for a Service of type `LoadBalancer`:

```
apiVersion: v1
kind: Service
metadata:
  name: "my-lb-service"
spec:
  selector:
    app: "try-dual-stack"
  type: "LoadBalancer"
  ipFamilyPolicy: "RequireDualStack"
  ipFamilies:
  - "IPv6"
  - "IPv4"
  ports:
  - port: 80
    targetPort: 8080
```

Save the manifest in a file named `my-lb-service.yaml`, and create the Service:

```
kubectl --kubeconfig CLUSTER_KUBECONFIG apply -f my-lb-service.yaml
```

Recall that in your Cluster manifest, you specified a range of IPv6 addresses
and a range of IPv4 addresses to be used for Services of type `LoadBalancer`:

```
  loadBalancer:
    mode: "bundled"
    ...
    addressPools:
    - name: "pool-1"
      addresses:
      - "10.2.0.112-10.2.0.221"
      - "fd12::4:101-fd12::4:110"
```

Your Service will be assigned an external IPv4 address chosen from the IPv4
range and an external IPv6 address chosen from the IPv6 range.

List details for the Service:

```
kubectl --kubeconfig CLUSTER_KUBECONFIG get service my-lb-service --output yaml
```

In the output, you can see the external addresses for the Service. For example:

```
apiVersion: v1
kind: Service
metadata:
  name: my-lb-service
...
status:
  loadBalancer:
    ingress:
    - ip: 10.2.0.213
    - ip: fd12::4:101
```

## Possible values for `ipFamilyPolicy`

When you create a dual-stack Service, you can set `ipFamilyPolicy` to one of
these values:

- `SingleStack`: The controller allocates a cluster IP address for the Service,
  chosen from the first range specified in the Cluster manifest under
  `clusterNetwork.services.cidrBlocks`.

- `PreferDualStack`: The controller allocates IPv4 and IPv6 cluster IP addresses
  for the Service, chosen from the ranges specified in the Cluster manifest
  under `clusterNetwork.services.cidrBlocks`. If the cluster is not a dual-stack
  cluster, the behavior is the same as with `SingleStack`.

- `RequireDualStack`: The controller allocates IPv4 and IPv6 cluster IP
  addresses for the Service, chosen from the ranges specified in the Cluster
  manifest under `clusterNetwork.services.cidrBlocks`. It sets the value of
  `spec.clusterIP` based on the first address family specified in the Service
  manifest under `ipFamilies`.

## More information

For more information about how to create dual-stack Services, see
[Dual-stack options on new Services](https://kubernetes.io/docs/concepts/services-networking/dual-stack/#dual-stack-options-on-new-services).