Implement flat-mode network model with BGP support

This document describes how to implement a flat-mode network model with Border Gateway Protocol (BGP) support. When you implement a network model with BGP support, BGP dynamically ensures that pods in different Layer 2 domains can communicate with each other. Flat-mode networking with BGP is sometimes called dynamic flat IP.

For more information about flat-mode network models, see Flat vs island mode network models.

How to implement a flat-mode network that uses BGP

Flat-mode networking with BGP is enabled when you create a new cluster. You can't enable this feature for an existing cluster. Once this feature is enabled, you can make changes to some of the configuration settings.

To implement a cluster on a flat-mode network model with BGP support:

Edit the cluster configuration file:
- Set the spec.clusterNetwork.advancedNetworking field to true.
- If you want to enable flat-mode networking for IPv4, set the spec.clusterNetwork.flatIPv4 field to true.
  
  For an alternative, see Dual-stack cluster (IPv4 Island, IPv6 Dynamic Flat IP), which configures your cluster with flat-mode networking for IPv6 only.
```
apiVersion: baremetal.cluster.gke.io/v1
kind: Cluster
metadata:
  name: bm
  namespace: cluster-bm
spec:
  type: user
  ...
  clusterNetwork:
    advancedNetworking: true
    flatIPv4: true
  ...
```
When spec.clusterNetwork.flatIPv4is set to true, the field spec.clusterNetwork.pods.cidrBlocks is ignored and can be omitted. However, you need to add a ClusterCIDRConfigs manifest in the cluster configuration file (per-node, per-nodepool and/or per-cluster).
Append a NetworkGatewayGroup manifest to the cluster configuration file:

Specify the floating IPs to use for BGP peering. Ensure that the resource name is default and the namespace is the cluster namespace.
```
---
apiVersion: networking.gke.io/v1
kind: NetworkGatewayGroup
metadata:
  name: default
  namespace: cluster-bm
spec:
  floatingIPs:
  - 10.0.1.100
  - 10.0.2.100
```
The NetworkGatewayGroup custom resource manages a list of one or more floating IP addresses. The BGP peering sessions are initiated from floating IP addresses that you specify in the NetworkGatewayGroup custom resource.
Append a FlatIPMode manifest to the cluster configuration file:

The name of the FlatIPMode resource must be default and the namespace is the cluster namespace. The peerSelector value flatip-peer: "true" matches the labels in BGPPeer objects bgppeer1 and bgppeer2 (defined in the following step), so both peers are used for flat-mode networking.

The following FlatIPMode manifest is for IPv4 single-stack, flat-mode networking with BGP. For alternative configurations, see Configuration examples.
```
---
apiVersion: baremetal.cluster.gke.io/v1alpha1
kind: FlatIPMode
metadata:
  name: default
  namespace: cluster-bm
spec:
  enableBGPIPv4: true
  enableBGPIPv6: false
  peerSelector:
    flatip-peer: "true"
```

Append one or more BGPPeer manifests to the cluster configuration file:

You choose the names for the resources, but all BGPPeer resources must be in the cluster namespace.

---
apiVersion: networking.gke.io/v1
kind: BGPPeer
metadata:
  name: bgppeer1
  namespace: cluster-bm
  labels:
    flatip-peer: "true"
spec:
  localASN: 65001
  peerASN: 65000
  peerIP: 10.0.1.254
  sessions: 2
---
apiVersion: networking.gke.io/v1
kind: BGPPeer
metadata:
  name: bgppeer2
  namespace: cluster-bm
  labels:
    flatip-peer: "true"
spec:
  localASN: 65001
  peerASN: 65000
  peerIP: 10.0.2.254
  sessions: 2

Append a ClusterCIDRConfig manifest to the cluster configuration file:

The CusterCIDRConfig resource must also be in the cluster namespace.
```
apiVersion: baremetal.cluster.gke.io/v1alpha1
kind: ClusterCIDRConfig
metadata:
  name: cluster-wide-1
  namespace: cluster-bm
spec:
  ipv4:
    cidr: "192.168.0.0/16"
    perNodeMaskSize: 24
```
ClusterCIDRConfig is a custom resource that specifies Pod CIDR ranges to be allocated to nodes dynamically. The CNI uses the Pod CIDR ranges allocated on a Node to allocate IP addresses to the individual Pods running on the Node. The ClusterCIDRConfig is also used for dual-stack networking. For more information about the ClusterCIDRConfig custom resource, including usage examples, see Understand the ClusterCIDRConfig custom resource.
Create the cluster:
```
bmctl create cluster
```
For more information about creating clusters, see Cluster creation overview.

If your environment supports multi-protocol BGP (MP-BGP), IPv4 and IPv6 routes can be advertised over these IPv4 sessions. For examples of different configurations, including examples that use MP-BGP, see Configuration examples.

Modify your BGP-based flat-mode networking configuration

After you've created your cluster configured to use a flat-mode network model with BGP, some configuration settings can be updated. Use the admin cluster kubeconfig file when you make subsequent updates to the BGP-related resources (NetworkGatewayGroup, FlatIPMode, and BGPPeer). The admin cluster then reconciles the changes to the user cluster. If you edit these resources on the user cluster directly, the admin cluster overwrites your changes in subsequent reconciliations.

Example configurations

The following sections include cluster configuration examples for different variations of the flat-mode network model with BGP. The sample configuration files aren't complete. Most cluster settings that aren't relevant to flat-mode networking with BGP have been omitted.

Single-stack IPv4 cluster

The following cluster configuration file sample shows the settings for configuring a single-stack IPv4 cluster with flat-mode networking with BGP:

apiVersion: baremetal.cluster.gke.io/v1
kind: Cluster
metadata:
  name: bm
  namespace: cluster-bm
spec:
  ...
  clusterNetwork:
    advancedNetworking: true
    flatIPv4: true
    services:
      cidrBlocks:
      - 10.96.0.0/12
  ...
---
apiVersion: baremetal.cluster.gke.io/v1alpha1
kind: ClusterCIDRConfig          
metadata:
  name: cluster-wide-1
  namespace: cluster-bm          # Must match the cluster namespace
spec:
  ipv4:
    cidr: "222.2.0.0/16"
    perNodeMaskSize: 24
---
apiVersion: networking.gke.io/v1
kind: NetworkGatewayGroup
metadata:
  name: default
  namespace: cluster-bm           # Must match the cluster namespace
spec:
  floatingIPs:
  - 10.0.1.100
  - 10.0.3.100
---
apiVersion: baremetal.cluster.gke.io/v1alpha1
kind: FlatIPMode
metadata:
  name: default
  namespace: cluster-bm            # Must match the cluster namespace
spec:
  enableBGPIPv4: true
  enableBGPIPv6: false
  peerSelector:
    flatipmode-peer: "true"
---
apiVersion: networking.gke.io/v1
kind: BGPPeer
metadata:
  name: bgppeer1
  namespace: cluster-bm            # Must match the cluster namespace
  labels:
    flatipmode-peer: "true"
spec:
  localASN: 65001
  peerASN: 65002
  peerIP: 10.0.1.254
  sessions: 2
---
apiVersion: networking.gke.io/v1
kind: BGPPeer
metadata:
  name: bgppeer2
  namespace: cluster-bm            # Must match the cluster namespace
  labels:
    flatipmode-peer: "true"
spec:
  localASN: 65001
  peerASN: 65002
  peerIP: 10.0.3.254
  sessions: 2

Dual-stack cluster (IPv4 Island, IPv6 Dynamic Flat IP)

The following cluster configuration file sample shows the settings for configuring a dual-stack (IPv4/IPv6) cluster with flat-mode networking with BGP for just IPv6:

apiVersion: baremetal.cluster.gke.io/v1
kind: Cluster
metadata:
  name: bm
  namespace: cluster-bm
spec:
  ...
  clusterNetwork:
    advancedNetworking: true
    flatIPv4: false
    pods:
      cidrBlocks:
      - 192.168.0.0/16
    services:
      cidrBlocks:
      - 10.96.0.0/12
      # Additional IPv6 CIDR block determines if the cluster is dual-stack
      - 2620:0:1000:2630:5:2::/112
  ... 
---
apiVersion: baremetal.cluster.gke.io/v1alpha1
kind: ClusterCIDRConfig          
metadata:
  name: cluster-wide-1
  namespace: cluster-bm          # Must match the cluster namespace
spec:
  ipv4:
    cidr: "192.168.0.0/16"
    perNodeMaskSize: 24
  ipv6:
    cidr: "2222:3::/112"
    perNodeMaskSize: 120
---
apiVersion: networking.gke.io/v1
kind: NetworkGatewayGroup
metadata:
  name: default
  namespace: cluster-bm           # Must match the cluster namespace
spec:
  floatingIPs:
  - 10.0.1.100
  - 10.0.3.100
---
apiVersion: baremetal.cluster.gke.io/v1alpha1
kind: FlatIPMode
metadata:
  name: default
  namespace: cluster-bm            # Must match the cluster namespace
spec:
  enableBGPIPv4: false
  enableBGPIPv6: true
  peerSelector:
    flatipmode-peer: "true"
---
apiVersion: networking.gke.io/v1
kind: BGPPeer
metadata:
  name: bgppeer1
  namespace: cluster-bm            # Must match the cluster namespace
  labels:
    flatipmode-peer: "true"
spec:
  localASN: 65001
  peerASN: 65002
  peerIP: 10.0.1.254
  sessions: 2
---
apiVersion: networking.gke.io/v1
kind: BGPPeer
metadata:
  name: bgppeer2
  namespace: cluster-bm            # Must match the cluster namespace
  labels:
    flatipmode-peer: "true"
spec:
  localASN: 65001
  peerASN: 65002
  peerIP: 10.0.3.254
  sessions: 2

Dual-stack cluster (IPv4 Dynamic Flat IP, IPv6 Dynamic Flat IP)

The following cluster configuration file sample shows the settings for configuring a dual-stack cluster with flat-mode networking with BGP:

apiVersion: baremetal.cluster.gke.io/v1
kind: Cluster
metadata:
  name: bm
  namespace: cluster-bm
spec:
  ...
  clusterNetwork:
    advancedNetworking: true
    flatIPv4: true
    pods:
      cidrBlocks:
      - 192.168.0.0/16
    services:
      cidrBlocks:
      - 10.96.0.0/12
      # Additional IPv6 CIDR block determines if the cluster is dual-stack
      - 2620:0:1000:2630:5:2::/112
  ... 
---
apiVersion: baremetal.cluster.gke.io/v1alpha1
kind: ClusterCIDRConfig          
metadata:
  name: cluster-wide-1
  namespace: cluster-bm          # Must match the cluster namespace
spec:
  ipv4:
    cidr: "222.2.0.0/16"
    perNodeMaskSize: 24
  ipv6:
    cidr: "2222:3::/112"
    perNodeMaskSize: 120
---
apiVersion: networking.gke.io/v1
kind: NetworkGatewayGroup
metadata:
  name: default
  namespace: cluster-bm           # Must match the cluster namespace
spec:
  floatingIPs:
  - 10.0.1.100
  - 10.0.3.100
---
apiVersion: baremetal.cluster.gke.io/v1alpha1
kind: FlatIPMode
metadata:
  name: default
  namespace: cluster-bm            # Must match the cluster namespace
spec:
  enableBGPIPv4: true
  enableBGPIPv6: true
  peerSelector:
    flatipmode-peer: "true"
---
apiVersion: networking.gke.io/v1
kind: BGPPeer
metadata:
  name: bgppeer1
  namespace: cluster-bm            # Must match the cluster namespace
  labels:
    flatipmode-peer: "true"
spec:
  localASN: 65001
  peerASN: 65002
  peerIP: 10.0.1.254
  sessions: 2
---
apiVersion: networking.gke.io/v1
kind: BGPPeer
metadata:
  name: bgppeer2
  namespace: cluster-bm            # Must match the cluster namespace
  labels:
    flatipmode-peer: "true"
spec:
  localASN: 65001
  peerASN: 65002
  peerIP: 10.0.3.254
  sessions: 2

Troubleshooting

To help you troubleshoot issues related to flat-mode networking with BGP, this section includes instructions for checking your configuration:

Verify if a FlatIPModes object is created in the cluster namespace on the admin cluster:

kubectl get flatipmodes -A --kubeconfig ADMIN_KUBECONFIG

The response should look something like this:

NAMESPACE                 NAME      AGE
cluster-bm                default   2d17h

Verify if a flatipmodes.networking.gke.io object is created on the user cluster:

The flatipmodes.networking.gke.io object is cluster scoped.
```
kubectl get flatipmodes.networking.gke.io --kubeconfig USER_KUBECONFIG
```
The response should look something like this:
```
NAME      AGE
default   2d17h
```

Get the BGPSessions resources to view the current sessions:

kubectl get bgpsessions -A --kubeconfig USER_KUBECONFIG