This page describes the specialized Network Function Kubernetes operator that Google Distributed Cloud connected ships with. This operator implements a set of CustomResourceDefinitions (CRDs) that allow Distributed Cloud connected to execute high-performance workloads. Network Function operator and SR-IOV functionality is not available on Distributed Cloud connected servers. The Network Function operator lets you do the following: - Poll for existing network devices on a node. - Query the IP address and physical link state for each network device on a node. - Provision additional network interfaces on a node. - Configure low-level system features on the node's physical machine required to support high-performance workloads. - Use single-root input/output virtualization (SR-IOV) on PCI Express network interfaces to virtualize them into multiple virtual interfaces. You can then configure your Distributed Cloud connected workloads to use those virtual network interfaces. Distributed Cloud connected support for SR-IOV is based on the following open source projects: - [Kubernetes SR-IOV Network Operator](https://github.com/k8snetworkplumbingwg/sriov-network-operator) - [Multus-CNI plugin](https://github.com/k8snetworkplumbingwg/multus-cni) - [SRIOV-CNI plugin](https://github.com/k8snetworkplumbingwg/sriov-cni) ## Prerequisites The Network Function operator fetches network configuration from the Distributed Cloud Edge Network API. To allow this, you must grant the Network Function operator service account the Edge Network Viewer role (`roles/edgenetwork.viewer`) using the following command: ``` gcloud projects add-iam-policy-binding PROJECT_ID \ --role roles/edgenetwork.viewer \ --member "serviceAccount:PROJECT_ID.svc.id.goog[nf-operator/nf-angautomator-sa]" ``` Replace `PROJECT_ID` with the ID of the target Google Cloud project. ## Network Function operator resources The Distributed Cloud connected Network Function operator implements the following Kubernetes CRDs: - [`Network`](https://docs.cloud.google.com/distributed-cloud/connected/1.6.1/docs/network-function#Network). Defines a virtual network that Pods can use to communicate with internal and external resources. You must create the corresponding VLAN using the Distributed Cloud Edge Network API before specifying it in this resource. For instructions, see [Create a network](https://docs.cloud.google.com/distributed-cloud/connected/1.6.1/docs/networking#create_a_network). - [`NetworkInterfaceState`](https://docs.cloud.google.com/distributed-cloud/connected/1.6.1/docs/network-function#NetworkInterfaceState). Enables the discovery of network interface states and querying a network interface for link state and IP address. - [`NodeSystemConfigUpdate`](https://docs.cloud.google.com/distributed-cloud/connected/1.6.1/docs/network-function#NodeSystemConfigUpdate). Enables the configuration of low-level system features such as kernel options and `Kubelet` flags. - [`SriovNetworkNodePolicy`](https://docs.cloud.google.com/distributed-cloud/connected/1.6.1/docs/network-function#SriovNetworkNodePolicy). Selects a group of SR-IOV virtualized network interfaces and instantiates the group as a Kubernetes resource. You can use this resource in a `NetworkAttachmentDefinition` resource. - [`SriovNetworkNodeState`](https://docs.cloud.google.com/distributed-cloud/connected/1.6.1/docs/network-function#SriovNetworkNodeState). Lets you query the provisioning state of the `SriovNetworkNodePolicy` resource on a Distributed Cloud node. - [`NetworkAttachmentDefinition`](https://docs.cloud.google.com/distributed-cloud/connected/1.6.1/docs/network-function#NetworkAttachmentDefinition). Lets you attach Distributed Cloud Pods to one or more logical or physical networks on your Distributed Cloud connected node. You must create the corresponding VLAN using the Distributed Cloud Edge Network API before specifying it in this resource. For instructions, see [Create a network](https://docs.cloud.google.com/distributed-cloud/connected/1.6.1/docs/networking#create_a_network). The Network Function operator also lets you define secondary network interfaces that do not use SR-IOV virtual functions. ### `Network` resource > [!WARNING] > > **Preview** > > > This feature is > > subject to the "Pre-GA Offerings Terms" in the General Service Terms section of the > [Service Specific > Terms](https://docs.cloud.google.com/terms/service-terms#1). > > Pre-GA features are available "as is" and might have limited support. > > For more information, see the > [launch stage descriptions](https://cloud.google.com/products/#product-launch-stages). The `Network` resource defines a virtual network within the Distributed Cloud connected rack that Pods within your Distributed Cloud connected cluster can use to communicate with internal and external resources. The `Network` resource provides the following configurable parameters for the network interface exposed as writable fields: - `spec.type`: specifies the network transport layer for this network. The only valid value is `L2`. You must also specify a `nodeInterfaceMatcher.interfaceName` value. - `spec.nodeInterfaceMatcher.interfaceName`: the name of the physical network interface on the target Distributed Cloud connected node to use with this network. - `spec.gateway4`: the IP address of the network gateway for this network. - `spec.l2NetworkConfig.prefixLength4`: specifies the CIDR range for this network. The following example illustrates the structure of the resource: apiVersion: networking.gke.io/v1 kind: Network metadata: name: vlan200-network annotations: networking.gke.io/gdce-vlan-id: 200 networking.gke.io/gdce-vlan-mtu: 1500 spec: type: L2 nodeInterfaceMatcher: interfaceName: gdcenet0.200 gateway4: 10.53.0.1 ### `NetworkInterfaceState` resource The `NetworkInterfaceState` resource is a read-only resource that lets you discover physical network interfaces on the node and collect runtime statistics on the network traffic flowing through those interfaces. Distributed Cloud creates a `NetworkInterfaceState` resource for each node in a cluster. The default configuration of Distributed Cloud connected machines includes a bonded network interface on the Rack Select Network Daughter Card (rNDC) named `gdcenet0`. This interface bonds the `eno1np0` and `eno2np1` network interfaces. Each of those is connected to one Distributed Cloud ToR switch, respectively. The `NetworkInterfaceState` resource provides the following categories of network interface information exposed as read-only status fields. General information: - `status.interfaces.ifname`: the name of the target network interface. - `status.lastReportTime`: the time and date of the last status report for the target interface. IP address configuration information: - `status.interfaces.interfaceinfo.address`: the IP address assigned to the target interface. - `status.interfaces.interfaceinfo.dns`: the IP address of the DNS server assigned to the target interface. - `status.interfaces.interfaceinfo.gateway`: the IP address of the network gateway serving the target interface. - `status.interfaces.interfaceinfo.prefixlen`: the length of the IP prefix. Hardware information: - `status.interfaces.linkinfo.broadcast`: the broadcast MAC address of the target interface. - `status.interfaces.linkinfo.businfo`: the PCIe device path in `bus:slot.function` format. - `status.interfaces.linkinfo.flags`: the interface flags---for example, `BROADCAST`. - `status.interfaces.linkinfo.macAddress`: the Unicast MAC address of the target interface. - `status.interfaces.linkinfo.mtu`: the MTU value for the target interface. Reception statistics: - `status.interfaces.statistics.rx.bytes`: the total bytes received by the target interface. - `status.interfaces.statistics.rx.dropped`: the total packets dropped by the target interface. - `status.interfaces.statistics.rx.errors`: the total packet receive errors for the target interface. - `status.interfaces.statistics.rx.multicast`: the total multicast packets received by the target interface. - `status.interfaces.statistics.rx.overErrors`: the total packet receive over errors for the target interface. - `status.interfaces.statistics.rx.packets`: the total packets received by the target interface. Transmission statistics: - `status.interfaces.statistics.tx.bytes`: the total bytes transmitted by the target interface. - `status.interfaces.statistics.tx.carrierErrors`: the total carrier errors encountered by the target interface. - `status.interfaces.statistics.tx.collisions`: the total packet collisions encountered by the target interface. - `status.interfaces.statistics.tx.dropped`: the total packets dropped by the target interface. - `status.interfaces.statistics.tx.errors`: the total transmission errors for the target interface. - `status.interfaces.statistics.tx.packets`: the total packets transmitted by the target interface. The following example illustrates the structure of the resource: apiVersion: networking.gke.io/v1 kind: NetworkInterfaceState metadata: name: MyNode1 nodeName: MyNode1 status: interfaces: - ifname: eno1np0 linkinfo: businfo: 0000:1a:00.0 flags: up|broadcast|multicast macAddress: ba:16:03:9e:9c:87 mtu: 9000 statistics: rx: bytes: 1098522811 errors: 2 multicast: 190926 packets: 4988200 tx: bytes: 62157709961 packets: 169847139 - ifname: eno2np1 linkinfo: businfo: 0000:1a:00.1 flags: up|broadcast|multicast macAddress: ba:16:03:9e:9c:87 mtu: 9000 statistics: rx: bytes: 33061895405 multicast: 110203 packets: 110447356 tx: bytes: 2370516278 packets: 11324730 - ifname: enp95s0f0np0 interfaceinfo: - address: fe80::63f:72ff:fec4:2bf4 prefixlen: 64 linkinfo: businfo: 0000:5f:00.0 flags: up|broadcast|multicast macAddress: 04:3f:72:c4:2b:f4 mtu: 9000 statistics: rx: bytes: 37858381 multicast: 205645 packets: 205645 tx: bytes: 1207334 packets: 6542 - ifname: enp95s0f1np1 interfaceinfo: - address: fe80::63f:72ff:fec4:2bf5 prefixlen: 64 linkinfo: businfo: 0000:5f:00.1 flags: up|broadcast|multicast macAddress: 04:3f:72:c4:2b:f5 mtu: 9000 statistics: rx: bytes: 37852406 multicast: 205607 packets: 205607 tx: bytes: 1207872 packets: 6545 - ifname: enp134s0f0np0 interfaceinfo: - address: fe80::63f:72ff:fec4:2b6c prefixlen: 64 linkinfo: businfo: 0000:86:00.0 flags: up|broadcast|multicast macAddress: 04:3f:72:c4:2b:6c mtu: 9000 statistics: rx: bytes: 37988773 multicast: 205584 packets: 205584 tx: bytes: 1212385 packets: 6546 - ifname: enp134s0f1np1 interfaceinfo: - address: fe80::63f:72ff:fec4:2b6d prefixlen: 64 linkinfo: businfo: 0000:86:00.1 flags: up|broadcast|multicast macAddress: 04:3f:72:c4:2b:6d mtu: 9000 statistics: rx: bytes: 37980702 multicast: 205548 packets: 205548 tx: bytes: 1212297 packets: 6548 - ifname: gdcenet0 interfaceinfo: - address: 208.117.254.36 prefixlen: 28 - address: fe80::b816:3ff:fe9e:9c87 prefixlen: 64 linkinfo: flags: up|broadcast|multicast macAddress: ba:16:03:9e:9c:87 mtu: 9000 statistics: rx: bytes: 34160422968 errors: 2 multicast: 301129 packets: 115435591 tx: bytes: 64528301111 packets: 181171964 .. lastReportTime: "2022-03-30T07:35:44Z" ### `NodeSystemConfigUpdate` resource The `NodeSystemConfigUpdate` resource lets you make changes to the node's operating system configuration as well as modify `Kubelet` flags. Changes other than `sysctl` changes require a node reboot. When instantiating this resource, you must specify the target nodes in the `nodeSelector` field. You must include all key-value pairs for each target node in the `nodeSelector` field. When you specify more than one target node in this field, the target nodes are updated one node at a time. CAUTION: The `nodeName` field has been deprecated. Using it immediately reboots the target nodes, including local control plane nodes, which can halt critical workloads. The `NodeSystemConfigUpdate` resource provides the following configuration fields specific to Distributed Cloud connected: - `spec.containerRuntimeDNSConfig.ip`: specifies a list of IP addresses for private image registries. - `spec.containerRuntimeDNSConfig`: specifies a list of custom DNS entries used by the Container Runtime Environment on each Distributed Cloud connected node. Each entry consists of the following fields: - `ip`: specifies the target IPv4 address, - `domain`: specifies the corresponding domain, - `interface`: specifies the network egress interface through which the IP address specified in the `ip` field is reachable. You can specify an interface defined through the following resources: `CustomNetworkInterfaceConfig`, `Network` (by annotation), `NetworkAttachmentDefinition`, (by annotation). This is a preview-level feature. - `spec.kubeletConfig.cpuManagerPolicy`: specifies the Kubernetes CPUManager policy. Valid values are `None` and `Static`. - `spec.kubeletConfig.topologyManagerPolicy`: specifies the Kubernetes TopologyManager policy. Valid values are `None`, `BestEffort`, `Restricted`, and `SingleNumaMode`. - `spec.osConfig.hugePagesConfig`: specifies the huge page configuration per NUMA node. Valid values are `2MB` and `1GB`. The number of huge pages requested is evenly distributed across both NUMA nodes in the system. For example, if you allocate 16 huge pages at 1 GB each, then each node receives a pre-allocation of 8 GB. - `spec.osConfig.isolatedCpusPerSocket`: specifies the number of isolated CPUs per socket. Required if `cpuManagerPolicy` is set to `Static`. The maximum number of isolated CPUs must be fewer than 80% of the total CPUs in the node. - `spec.osConfig.cpuIsolationPolicy`: specifies the CPU isolation policy. The `Default` policy only isolates `systemd` tasks from CPUs reserved for workloads. The `Kernel` policy marks the CPUs as `isolcpus` and sets the `rcu_nocb`, `nohz_full`, and `rcu_nocb_poll` flags on each CPU. - `spec.sysctls.NodeLevel`: specifies the `sysctls` parameters that you can configure globally on a node by using the Network Function operator. The configurable parameters are as follows: - `fs.inotify.max_user_instances` - `fs.inotify.max_user_watches` - `kernel.sched_rt_runtime_us` - `kernel.core_pattern` - `net.ipv4.tcp_wmem` - `net.ipv4.tcp_rmem` - `net.ipv4.tcp_slow_start_after_idle` - `net.ipv4.udp_rmem_min` - `net.ipv4.udp_wmem_min` - `net.ipv4.tcp_rmem` - `net.ipv4.tcp_wmem` - `net.core.rmem_max` - `net.core.wmem_max` - `net.core.rmem_default` - `net.core.wmem_default` - `net.netfilter.nf_conntrack_tcp_timeout_unacknowledged` - `net.netfilter.nf_conntrack_tcp_timeout_max_retrans` - `net.sctp.auth_enable` - `net.sctp.sctp_mem` - `net.ipv4.udp_mem` - `net.ipv4.tcp_mem` - `net.ipv4.tcp_slow_start_after_idle` - `net.sctp.auth_enable` - `vm.max_map_count` You can also scope both safe and unsafe `sysctls` parameters to a specific Pod or namespace by using the [`tuning` Container Networking Interface (CNI) plug-in](https://www.cni.dev/plugins/current/meta/tuning/). The `NodeSystemConfigUpdate` resource provides the following read-only general status fields: - `status.lastReportTime`: the most recent time that status was reported for the target interface. - `status.conditions.lastTransitionTime`: the most recent time that the condition of the interface has changed. - `status.conditions.observedGeneration`: denotes the `.metadata.generation` value on which the initial condition was based. - `status.conditions.message`: an informative message describing the change of the interface's condition. - `status.conditions.reason`: a programmatic identifier denoting the reason for the last change of the interface's condition. - `status.conditions.status`: the status descriptor of the condition. Valid values are `True`, `False`, and `Unknown`. - `status.conditions.type`: the condition type in camelCase. The following example illustrates the structure of the resource: apiVersion: networking.gke.io/v1 kind: NodeSystemConfigUpdate metadata: name: node-pool-1-config namespace: default spec: nodeSelector: baremetal.cluster.gke.io/node-pool: node-pool-1 networking.gke.io/worker-network-sriov.capable: true sysctls: nodeLevel: "net.ipv4.udp_mem" : "12348035 16464042 24696060" kubeletConfig: topologyManagerPolicy: BestEffort cpuManagerPolicy: Static osConfig: hugePagesConfig: "TWO_MB": 0 "ONE_GB": 16 isolatedCpusPerSocket: "0": 10 "1": 10 ### `SriovNetworkNodePolicy` resource The `SriovNetworkNodePolicy` resource lets you allocate a group of SR-IOV virtual functions (VFs) on a Distributed Cloud connected physical machine and instantiate that group as a Kubernetes resource. You can then use this resource in a `NetworkAttachmentDefinition` resource. You can select each target VF by its PCIe vendor and device ID, its PCIe device addresses, or by its Linux enumerated device name. The SR-IOV Network Operator configures each physical network interface to provision the target VFs. This includes updating the network interface firmware, configuring the Linux kernel driver, and rebooting the Distributed Cloud connected machine, if necessary. To discover the network interfaces available on your node, you can look up the [`NetworkInterfaceState` resources](https://docs.cloud.google.com/distributed-cloud/connected/1.6.1/docs/network-function#NetworkInterfaceState) on that node in the `nf-operator` namespace. The following example illustrates the structure of the resource: apiVersion: sriovnetwork.k8s.cni.cncf.io/v1 kind: SriovNetworkNodePolicy metadata: name: mlnx6-p2-sriov-en2 namespace: sriov-network-operator spec: deviceType: netdevice isRdma: true mtu: 9000 nicSelector: pfNames: - enp134s0f1np1 nodeSelector: edgecontainer.googleapis.com/network-sriov.capable: "true" numVfs: 31 priority: 99 resourceName: mlnx6_p2_sriov_en2 The preceding example creates a maximum of 31 VFs from the second port on the network interface named `enp134s0f1np1` with an MTU value of `9000` (the maximum allowed value). Use the node selector label `edgecontainer.googleapis.com/network-sriov.capable`, which is present on all Distributed Cloud connected nodes capable of SR-IOV. For information about using this resource, see [`SriovNetworkNodeState`](https://github.com/k8snetworkplumbingwg/sriov-network-operator#sriovnetworknodepolicy). ### `SriovNetworkNodeState` resource The `SriovNetworkNodeState` read-only resource lets you query the provisioning state of the `SriovNetworkNodePolicy` resource on a Distributed Cloud connected node. It returns the complete configuration of the `SriovNetworkNodePolicy` resource on the node as well as a list of active VFs on the node. The `status.syncStatus` field indicates whether all `SriovNetworkNodePolicy` resources defined for the node have been properly applied. The following example illustrates the structure of the resource: apiVersion: sriovnetwork.k8s.cni.cncf.io/v1 kind: SriovNetworkNodeState metadata: name: MyNode1 namespace: sriov-network-operator spec: dpConfigVersion: "1969684" interfaces: - mtu: 9000 name: enp134s0f1np1 numVfs: 31 pciAddress: 0000:86:00.1 vfGroups: - deviceType: netdevice mtu: 9000 policyName: mlnx6-p2-sriov-en2 resourceName: mlnx6_p2_sriov_en2 vfRange: 0-30 status: Status: Interfaces: Device ID: 1015 Driver: mlx5_core Link Speed: 25000 Mb/s Link Type: ETH Mac: ba:16:03:9e:9c:87 Mtu: 9000 Name: eno1np0 Pci Address: 0000:1a:00.0 Vendor: 15b3 Device ID: 1015 Driver: mlx5_core Link Speed: 25000 Mb/s Link Type: ETH Mac: ba:16:03:9e:9c:87 Mtu: 9000 Name: eno2np1 Pci Address: 0000:1a:00.1 Vendor: 15b3 Vfs: - Vfs: - deviceID: 101e driver: mlx5_core mac: c2:80:29:b5:63:55 mtu: 9000 name: enp134s0f1v0 pciAddress: 0000:86:04.1 vendor: 15b3 vfID: 0 - deviceID: 101e driver: mlx5_core mac: 7e:36:0c:82:d4:20 mtu: 9000 name: enp134s0f1v1 pciAddress: 0000:86:04.2 vendor: 15b3 vfID: 1 .. syncStatus: Succeeded For information about using this resource, see [`SriovNetworkNodeState`](https://github.com/k8snetworkplumbingwg/sriov-network-operator#sriovnetworknodestate). ### `NetworkAttachmentDefinition` resource The `NetworkAttachmentDefinition` resource lets you attach Distributed Cloud Pods to one or more logical or physical networks on your Distributed Cloud connected node. It leverages the [Multus-CNI](https://github.com/k8snetworkplumbingwg/multus-cni) framework and the [SRIOV-CNI plugin](https://github.com/k8snetworkplumbingwg/sriov-cni). Use an annotation to reference the name of the appropriate `SriovNetworkNodePolicy` resource. When you create this annotation, do the following: - Use the key `k8s.v1.cni.cncf.io/resourceName`. - Use the prefix `gke.io/` in its value, followed by the name of the target `SriovNetworkNodePolicy` resource. The following example illustrates the structure of the resource: apiVersion: "k8s.cni.cncf.io/v1" kind: NetworkAttachmentDefinition metadata: name: sriov-net1 namespace: mynamespace annotations: k8s.v1.cni.cncf.io/resourceName: gke.io/mlnx6_p2_sriov_en2 spec: config: '{ "type": "sriov", "cniVersion": "0.3.1", "name": "sriov-network", "ipam": { "type": "host-local", "subnet": "10.56.217.0/24", "routes": [{ "dst": "0.0.0.0/0" }], "gateway": "10.56.217.1" } }' ## Configure a secondary interface on a Pod using SR-IOV VFs After you configure a `SriovNetworkNodePolicy` resource and a corresponding `NetworkAttachmentDefinition` resource, you can configure a secondary network interface on a Distributed Cloud Pod by using SR-IOV virtual functions. To do so, add an annotation to your Distributed Cloud Pod definition as follows: - Key: `k8s.v1.cni.cncf.io/networks` - Value: `nameSpace/ [!WARNING] > > **Preview** > > > This feature is > > subject to the "Pre-GA Offerings Terms" in the General Service Terms section of the > [Service Specific > Terms](https://docs.cloud.google.com/terms/service-terms#1). > > Pre-GA features are available "as is" and might have limited support. > > For more information, see the > [launch stage descriptions](https://cloud.google.com/products/#product-launch-stages). Distributed Cloud connected also supports creating a secondary network interface on a Pod by using the MacVLAN driver. Only the `gdcenet0` interface supports this configuration and only on Pods that run containerized workloads. To configure an interface to use the MacVLAN driver: 1. Configure a `NetworkAttachmentDefinition` resource as shown in the following example: apiVersion: "k8s.cni.cncf.io/v1" kind: NetworkAttachmentDefinition metadata: name: macvlan-b400-1 annotations: networking.gke.io/gdce-vlan-id: 400 spec: config: '{ "type": "macvlan", "master": "gdcenet0.400", "ipam": { "type": "static", "addresses": [ { "address": "192.168.100.20/27", "gateway": "192.168.100.1" } ] ... } }' 2. Add an annotation to your Distributed Cloud Pod definition as follows: apiVersion: v1 kind: Pod metadata: name: macvlan-testpod1 annotations: k8s.v1.cni.cncf.io/networks: macvlan-b400-1 ## Configure a secondary interface on a Pod using Distributed Cloud multi-networking > [!WARNING] > > **Preview** > > > This feature is > > subject to the "Pre-GA Offerings Terms" in the General Service Terms section of the > [Service Specific > Terms](https://docs.cloud.google.com/terms/service-terms#1). > > Pre-GA features are available "as is" and might have limited support. > > For more information, see the > [launch stage descriptions](https://cloud.google.com/products/#product-launch-stages). Distributed Cloud connected supports creating a secondary network interface on a Pod by using its multi-network feature. To do so, complete the following steps: 1. Configure a `Network` resource. For example: apiVersion: networking.gke.io/v1 kind: Network metadata: name: vlan200-network spec: type: L2 nodeInterfaceMatcher: interfaceName: vlan200-interface gateway4: 10.53.0.1 2. Add an annotation to your Distributed Cloud Pod definition as follows: apiVersion: v1 kind: Pod metadata: name: myPod annotations: networking.gke.io/interfaces: '[{"interfaceName":"eth1","network":"vlan200-network"}]' networking.gke.io/default-interface: eth1 ... ## What's next - [Installation requirements](https://docs.cloud.google.com/distributed-cloud/connected/1.6.1/docs/requirements)