## Overview

The ClusterCIDRConfig is a custom CIDR allocator resource that enables you to
allocate more IP addresses ranges for Pods dynamically.

IP Address Management (IPAM) enables efficient use of IP subnets and avoids
having overlaps in address ranges, which prevents network conflicts and outages.
Kubernetes assigns Pod CIDRs per node, which are used as IP addresses for the Pods
running on that node.

Current Kubernetes NodeIPAM has following limitations:

- All Pod CIDRs are allocated from one cluster CIDR. You have to specify the
  entire IP address range accounting for the largest cluster at the time of
  cluster creation. This limitation can waste IP addresses.

- If you increase the cluster size, it is difficult to add more IP addresses.

- The cluster CIDR is one large range. It may be difficult to find a
  contiguous block of IP addresses that satisfy the needs of the cluster.

- Each node gets a fixed-size IP range within a cluster. If nodes are of
  different sizes and capacity, you can't allocate a bigger Pod range to a
  given node with larger capacity and a smaller range to nodes with lesser
  capacity. This wastes a lot of IP addresses. For a large cluster with many
  nodes, this waste gets compounded across all the nodes in the cluster.

With ClusterCIDRConfig functionality, you can avoid assigning a large CIDR block
to a cluster, map your cluster size to the scale of your Pods, and therefore
preserve IP addresses. You can save IP addresses by using ClusterCIDRConfigs
with different combinations of the CIDR and perNodeMaskSize. ClusterCIDRConfig
resource supports the following:

- Multiple discontiguous IP CIDR blocks for cluster CIDR at a more granular
  level

- Node affinity of CIDR blocks

- Different block sizes allocated to nodes

Google Distributed Cloud uses the ClusterCIDRConfig functionality in the following features:

- [IPv4/IPv6 dual-stack networking](https://docs.cloud.google.com/kubernetes-engine/distributed-cloud/bare-metal/docs/how-to/dual-stack-networking.md.txt)

- [Implement flat IPv4 mode network model](https://docs.cloud.google.com/kubernetes-engine/distributed-cloud/bare-metal/docs/how-to/flat-network.md.txt)

- [Implement flat-mode network model with BGP support](https://docs.cloud.google.com/kubernetes-engine/distributed-cloud/bare-metal/docs/how-to/flat-bgp-network.md.txt)

`Cluster.spec.clusterNetwork.pods.cidrBlocks` is an optional field and isn't
defined by default. You must define it if any of the features from the preceding
list doesn't have it defined. For example, it is required when the clusters are
created in IPv4 island mode and must be specified as it is used as a native
routing CIDR.

The following table lists the use of ClusterCIDRConfig's
`Cluster.spec.clusterNetwork.pods.cidrBlocks` field behavior for different
network modes.

| Network Mode | ClusterCIDRConfig Value |
|---|---|
| IPv4 Island (default) | (Required field) Specify `Cluster.spec.clusterNetwork.pods.cidrBlocks`. |
| IPv4 Flat (default) | `Cluster.spec.clusterNetwork.pods.cidrBlocks` are completely ignored, and can be skipped. Users have to explicitly define ClusterCIDRConfigs (per-node, per-nodepool and/or per-cluster). |
| Dual-stack (IPv4 Island, IPv4 Flat) | Specify the IPv4 CIDR. Do not specify IPv6 CIDR in `Cluster.spec.clusterNetwork.pods.cidrBlocks`. Specify ClusterCIDRConfigs with both IPv4 and IPv6 CIDRs. IPv4 CIDR configured in all the ClusterCIDRConfigs must be the same as the IPv4 CIDR from `Cluster.spec.clusterNetwork.pods.cidrBlocks` including the PerNodeMask value for IPv4. For more information on ClusterCIDRConfig and examples on using it, see [Examples: Dualstack (IPv4 island, IPv6 Flat)](https://docs.cloud.google.com/kubernetes-engine/distributed-cloud/bare-metal/docs/reference/cluster-cidr-config.md.txt#IslandFlat) |
| Dual-stack (Flat IPv4, Flat IPv6) | You can skip `Cluster.spec.clusterNetwork.pods.cidrBlocks` as these are completely ignored. You must explicitly define ClusterCIDRConfigs (per-node, per-nodepool, and/or per-cluster) with both IPv4 and IPv6 CIDRs. |

<br />

## Configuring the ClusterCIDRConfig custom CIDR allocator resource

### ClusterCIDRConfig

When you configure the ClusterCIDRConfig custom CIDR allocator resource,
consider the following points:

- Pod CIDR assignment from a particular ClusterCIDRConfig to a node is based
  on label selectors. This is similar to the nodeSelector mechanism used for
  scheduling Pods on a node.

- You must configure the ClusterCIDRConfig during the cluster creation process
  in the cluster configuration YAML file. Once you specify ClusterCIDRConfigs,
  you cannot modify the values later.

- You can specify multiple ClusterCIDRConfigs with overlapping CIDRs.

- If no matching ClusterCIDRConfig is found for a node, the node remains in a
  NotReady state, until a ClusterCIDRConfig with matching labels is created.

- If the best match ClusterCIDRConfig does not have more CIDRs available for
  allocation, the next best CIDR is chosen and the Pod CIDRs are allocated
  from the available CIDRs.

- In case of [dual-stack model](https://docs.cloud.google.com/kubernetes-engine/distributed-cloud/bare-metal/docs/how-to/dual-stack-networking.md.txt),
  if you want to assign dual-stack Pod CIDRs to the nodes, do the following:

  - Configure both IPv4 and IPv6 CIDRs in the ClusterCIDRConfig.

  - Ensure that all ClusterCIDRConfig have DualStack CIDRs, if multiple
    ClusterCIDRConfig are configured.

  - Ensure that both IPv4 and IPv6 CIDRs configured have an equal number of
    allocatable IP addresses per node.

  For example, `32 - spec.IPv4.PerNodeMaskSize == 128 -
  spec.IPv6.PerNodeMaskSize`

  `spec.IPv4.PerNodeMaskSize = 24`

  `spec.IPv6.PerNodeMaskSize = 120`

  Thus, 32 - 24 == 128 - 120, as the difference is 8.
- Multiple ClusterCIDRConfigs can match the labels from the nodeSelector to
  node labels.

### ClusterCIDRConfig assignment rules

To determine which ClusterCIDRConfig is used to assign Pod CIDRs to the current
node, use the following tie-breaking rules. Implement these rules in the given
order. Implement the next rule only if the tie is not broken by the preceding
rule.

1. Pick the ClusterCIDRConfig whose NodeSelector matches the most labels on the
   Node. For example, `{'node.kubernetes.io/instance-type':'medium', 'rack':
   'rack1'} (Match Count: 2)` is picked before
   `{'node.kubernetes.io/instance-type': 'medium'}. (Match Count: 1)`.

2. Pick the ClusterCIDRConfig with the fewest allocatable Pod CIDRs. For
   example, `{CIDR: "10.0.0.0/16", PerNodeMaskSize: "16"} (1 possible Pod
   CIDR)` is picked before `{CIDR: "192.168.0.0/20", PerNodeMaskSize: "22"} (4
   possible Pod CIDRs)`.

3. Pick the ClusterCIDRConfig whose PerNodeMaskSize has the fewest IP addresses.
   For example, 27 (2\^(32-27)= 32 IP addresses) picked before
   25 (2\^(32-25)=128 IP addresses).

4. Pick the ClusterCIDRConfig whose matching NodeSelector label has a lower
   alphanumeric value. For example, `{'kubernetes.io/hostname': 'node-1'}` is
   chosen over `{'node.kubernetes.io/instance-type':'medium'}`.

5. Pick the ClusterCIDRConfig whose CIDR IP has a lower value. Irrespective of
   whether the config is an IPv4 config or a DualStack config, only the IPv4
   CIDRs are compared. For example, `{CIDR: "10.0.0.0/16"} is picked over
   {CIDR: "192.168.0.0/16"}`.

## Configuration examples

This section lists configuration examples for Cluster and ClusterCIDRConfig for
all the networking modes.

> [!CAUTION]
> **Caution:** Do not use these examples as they appear here. Replace the address ranges with values that are appropriate for your network.

Examples: IPv4 Island Mode (Default)
Cluster Config (Default)

```
    apiVersion: baremetal.cluster.gke.io/v1
    kind: Cluster
    metadata:
      name: bm-cluster
      namespace: cluster-default
    spec:
    ...
      clusterNetwork:
        # Pods specify the IP ranges from which pod networks are allocated.
        pods:
          cidrBlocks:
          - 192.168.0.0/16
        services:
          cidrBlocks:
          - 10.96.0.0/12

      ... (other cluster config omitted)
  
```
Examples: IPv4 Flat Mode
Cluster Config

```
  apiVersion: baremetal.cluster.gke.io/v1
  kind: Cluster
  metadata:
    name: bm-cluster
    namespace: cluster-default
  spec:
  ...
    clusterNetwork:
      flatIPv4: true
      services:
        cidrBlocks:
        - 10.96.0.0/12

    ... (other cluster config omitted)
  
```
ClusterCIDRConfig
Example 1: Configure Per Node ClusterCIDRConfig

```
    ---
    apiVersion: baremetal.cluster.gke.io/v1alpha1
    kind: ClusterCIDRConfig
    metadata:
      name: range-node1
      namespace: cluster-default    # Cluster namespace
    spec:
      ipv4:
        cidr: "222.1.0.0/24"
        perNodeMaskSize: 24
      nodeSelector:
        matchLabels:
          kubernetes.io/hostname: node-1
    —
    apiVersion: baremetal.cluster.gke.io/v1alpha1
    kind: ClusterCIDRConfig
    metadata:
      name: range-node2
      namespace: cluster-default
    spec:
      ipv4:
        cidr: "222.1.1.0/24"
        perNodeMaskSize: 24
      nodeSelector:
        matchLabels:
          kubernetes.io/hostname: node-2
    —
    apiVersion: baremetal.cluster.gke.io/v1alpha1
    kind: ClusterCIDRConfig
    metadata:
      name: range-node3
      namespace: cluster-default
    spec:
      ipv4:
        cidr: "222.1.2.0/24"
        perNodeMaskSize: 24
      nodeSelector:
        matchLabels:
          kubernetes.io/hostname: node-3

  
```
Example 2: Example for specifying a NodePool wide PodCIDR

> [!NOTE]
> **Note:** The two flat IP modes have different constraints:
>
> - For Static Flat IP (without BGP): Can only be used if all the nodes are in the same Layer 2 Domain.
> - For Dynamic Flat IP (with BGP): Can be used even when the nodes span multiple subdomains.

```
    apiVersion: baremetal.cluster.gke.io/v1alpha1
    kind: ClusterCIDRConfig
    metadata:
      name: nodepool-wide-1
      namespace: cluster-default
    spec:
      ipv4:
        cidr: "222.2.0.0/18"
        perNodeMaskSize: 24
      nodeSelector:
        matchLabels:
          baremetal.cluster.gke.io/node-pool: workers
  
```
Example 3: Configure Cluster wide PodCIDR - No nodeSelector is configured

> [!NOTE]
> **Note:** The two flat IP modes have different constraints:
>
> - For Static Flat IP (without BGP): Can only be used if all the nodes are in the same Layer 2 Domain.
> - For Dynamic Flat IP (with BGP): Can be used even when the nodes span multiple subdomains.

```
    apiVersion: baremetal.cluster.gke.io/v1alpha1
    kind: ClusterCIDRConfig
    metadata:
      name: cluster-wide-1
      namespace: cluster-default
    spec:
      ipv4:
        cidr: "222.3.0.0/16"
        perNodeMaskSize: 24
  
```

<br />

<br />

Examples: Dualstack (IPv4 Island, IPv6 Flat)
Cluster Config

```
  apiVersion: baremetal.cluster.gke.io/v1
  kind: Cluster
  metadata:
    name: bm-cluster
    namespace: cluster-default
  spec:
  ...
    clusterNetwork:
      # Pods specify the IP ranges from which pod networks are allocated.
      pods:
        cidrBlocks:
        - 192.168.0.0/16
      services:
        cidrBlocks:
        - 10.96.0.0/12
        # Additional IPv6 cidrBlock, determines if the cluster is DualStack
        - 2620:0:1000:2630:5:2::/112

    ... (other cluster config omitted)
  
```

<br />

ClusterCIDRConfig
Example 1: Configure Per Node ClusterCIDRConfig

```
    ---
    apiVersion: baremetal.cluster.gke.io/v1alpha1
    kind: ClusterCIDRConfig
    metadata:
      name: range-node1
      namespace: cluster-default    # Cluster namespace
    spec:
      ipv4:
        cidr: "192.168.0.0/16"
        perNodeMaskSize: 24
      ipv6:
        cidr: "2222:1::/120"
        perNodeMaskSize: 120
      nodeSelector:
        matchLabels:
          kubernetes.io/hostname: node-1
    —
    apiVersion: baremetal.cluster.gke.io/v1alpha1
    kind: ClusterCIDRConfig
    metadata:
      name: range-node2
      namespace: cluster-default
    spec:
      ipv4:
        cidr: "192.168.0.0/16"
        perNodeMaskSize: 24
      ipv6:
        cidr: "2222:2::/120"
        perNodeMaskSize: 120
      nodeSelector:
        matchLabels:
          kubernetes.io/hostname: node-2
    —
    apiVersion: baremetal.cluster.gke.io/v1alpha1
    kind: ClusterCIDRConfig
    metadata:
      name: range-node3
      namespace: cluster-default
    spec:
      ipv4:
        cidr: "192.168.0.0/16"
        perNodeMaskSize: 24
      ipv6:
        cidr: "2222:3::/120"
        perNodeMaskSize: 120
      nodeSelector:
        matchLabels:
          kubernetes.io/hostname: node-3
    
```

<br />

<br />

Example 2: Example for specifying a NodePool wide PodCIDR

> [!NOTE]
> **Note:** The two flat IP modes have different constraints:
>
> - For Static Flat IP (without BGP): Can only be used if all the nodes are in the same Layer 2 Domain.
> - For Dynamic Flat IP (with BGP): Can be used even when the nodes span multiple subdomains.

```
    apiVersion: baremetal.cluster.gke.io/v1alpha1
    kind: ClusterCIDRConfig
    metadata:
      name: nodepool-wide-1
      namespace: cluster-default
    spec:
      ipv4:
        cidr: "192.168.0.0/16"
        perNodeMaskSize: 24
      ipv6:
        cidr: "2222:3::/112"
        perNodeMaskSize: 118
      nodeSelector:
        matchLabels:
          baremetal.cluster.gke.io/node-pool: workers
    
```

<br />

<br />

Example 3: Configure Cluster wide PodCIDR - No nodeSelector is configured

> [!NOTE]
> **Note:** The two flat IP modes have different constraints:
>
> - For Static Flat IP (without BGP): Can only be used if all the nodes are in the same Layer 2 Domain.
> - For Dynamic Flat IP (with BGP): Can be used even when the nodes span multiple subdomains.

```
    apiVersion: baremetal.cluster.gke.io/v1alpha1
    kind: ClusterCIDRConfig
    metadata:
      name: cluster-wide-1
      namespace: cluster-default
    spec:
      ipv4:
        cidr: "192.168.0.0/16"
        perNodeMaskSize: 24
      ipv6:
        cidr: "2222:3::/112"
        perNodeMaskSize: 120
    
```

<br />

Examples: Dualstack (IPv4 Flat, IPv6 Flat)
Cluster Config

```
    apiVersion: baremetal.cluster.gke.io/v1
    kind: Cluster
    metadata:
      name: bm-cluster
      namespace: cluster-default
    spec:
    ...
      clusterNetwork:
        flatIPv4: true
        services:
          cidrBlocks:
          - 10.96.0.0/12
          # Additional IPv6 cidrBlock, determines if the cluster is DualStack
          - 2620:0:1000:2630:5:2::/112

      ... (other cluster config omitted)
  
```

<br />

ClusterCIDRConfig

<br />

<br />

Example 1: Configure Per Node ClusterCIDRConfig

> [!NOTE]
> **Note:** The two flat IP modes have different constraints:
>
> - For Static Flat IP (without BGP): Can only be used if all the nodes are in the same Layer 2 Domain.
> - For Dynamic Flat IP (with BGP): Can be used even when the nodes span multiple subdomains.

```
    ---
    apiVersion: baremetal.cluster.gke.io/v1alpha1
    kind: ClusterCIDRConfig
    metadata:
      name: range-node1
      namespace: cluster-default    # Cluster namespace
    spec:
      ipv4:
        cidr: "222.1.0.0/24"
        perNodeMaskSize: 24
      ipv6:
        cidr: "2222:1::/120"
        perNodeMaskSize: 120
      nodeSelector:
        matchLabels:
          kubernetes.io/hostname: node-1
    —
    apiVersion: baremetal.cluster.gke.io/v1alpha1
    kind: ClusterCIDRConfig
    metadata:
      name: range-node2
      namespace: cluster-default
    spec:
      ipv4:
        cidr: "222.1.1.0/24"
        perNodeMaskSize: 24
      ipv6:
        cidr: "2222:2::/120"
        perNodeMaskSize: 120
      nodeSelector:
        matchLabels:
          kubernetes.io/hostname: node-2
    —
    apiVersion: baremetal.cluster.gke.io/v1alpha1
    kind: ClusterCIDRConfig
    metadata:
      name: range-node3
      namespace: cluster-default
    spec:
      ipv4:
        cidr: "222.1.2.0/24"
        perNodeMaskSize: 24
      ipv6:
        cidr: "2222:3::/120"
        perNodeMaskSize: 120
      nodeSelector:
        matchLabels:
          kubernetes.io/hostname: node-3
    
```

<br />

<br />

Example 2: Example for specifying a NodePool wide PodCIDR

> [!NOTE]
> **Note:** The two flat IP modes have different constraints:
>
> - For Static Flat IP (without BGP): Can only be used if all the nodes are in the same Layer 2 Domain.
> - For Dynamic Flat IP (with BGP): Can be used even when the nodes span multiple subdomains.

```
    apiVersion: baremetal.cluster.gke.io/v1alpha1
    kind: ClusterCIDRConfig
    metadata:
      name: nodepool-wide-1
      namespace: cluster-default
    spec:
      ipv4:
        cidr: "222.2.0.0/18"
        perNodeMaskSize: 24
      ipv6:
        cidr: "2222:3::/112"
        perNodeMaskSize: 118
      nodeSelector:
        matchLabels:
          baremetal.cluster.gke.io/node-pool: workers
    
```

<br />

<br />

Example 3: Configure Cluster wide PodCIDR - No nodeSelector is configured

```
    apiVersion: baremetal.cluster.gke.io/v1alpha1
    kind: ClusterCIDRConfig
    metadata:
      name: cluster-wide-1
      namespace: cluster-default
    spec:
      ipv4:
        cidr: "222.3.0.0/16"
        perNodeMaskSize: 24
      ipv6:
        cidr: "2222:3::/112"
        perNodeMaskSize: 120
    
```

<br />

<br />

<br />