This page describes how to run [NVIDIA Collective Communications Library (NCCL)](https://docs.cloud.google.com/ai-hypercomputer/docs/nccl/overview.md.txt) tests on *custom GKE clusters* that use
A3 Edge with
GPUDirect-TCPX
networking protocols. A custom GKE cluster is a cluster that you create by
using `gcloud` commands.

You can use the tests that are described on this page for the following
scenarios:

- If your GKE cluster uses Flex-start nodes, then use a basic [test on two nodes](https://docs.cloud.google.com/ai-hypercomputer/docs/nccl/test-gke-custom-a3-edge.md.txt#two-node-test).
- If your GKE cluster uses different types of nodes such as on-demand or reservation-bound nodes, then use an [NCCL test with Topology Aware Scheduling](https://docs.cloud.google.com/ai-hypercomputer/docs/nccl/test-gke-custom-a3-edge.md.txt#deploy-nccl-tas-workload).

## Before you begin

The tests on this page use [JobSet](https://jobset.sigs.k8s.io/docs/overview/)
and [Kueue with Topology Aware Scheduling (TAS)](https://kueue.sigs.k8s.io/docs/concepts/topology_aware_scheduling/). Before running any tests, you must set up
your cluster and do the following:

1. [Install JobSet](https://jobset.sigs.k8s.io/docs/installation/).

2. Install Kueue.

   ```bash
   kubectl apply --server-side -f https://github.com/kubernetes-sigs/kueue/releases/download/v0.16.5/manifests.yaml
   ```

## Set up your cluster with Jobset and Kueue

After you install JobSet and Kueue, take the following steps:

1. Save the following manifest as a `kueue-config.yaml` file:

   ```yaml
   apiVersion: kueue.x-k8s.io/v1beta2
   kind: Topology
   metadata:
     name: "gke-default"
   spec:
     levels:
     - nodeLabel: "cloud.google.com/gce-topology-block"
     - nodeLabel: "cloud.google.com/gce-topology-subblock"
     - nodeLabel: "cloud.google.com/gce-topology-host"
     - nodeLabel: "kubernetes.io/hostname"
   ---
   apiVersion: kueue.x-k8s.io/v1beta2
   kind: ResourceFlavor
   metadata:
     name: a3-edge-flavor
   spec:
     nodeLabels:
       cloud.google.com/gke-accelerator: nvidia-h100-80gb
     topologyName: "gke-default"
   ---
   apiVersion: kueue.x-k8s.io/v1beta2
   kind: ResourceFlavor
   metadata:
     name: a3-edge-dws-flavor
   spec:
     nodeLabels:
       cloud.google.com/gke-accelerator: nvidia-h100-80gb
     topologyName: "gke-default"
     tolerations:
     - key: "cloud.google.com/gke-queued"
       operator: "Exists"
       effect: NoSchedule
   ---
   apiVersion: kueue.x-k8s.io/v1beta2
   kind: AdmissionCheck
   metadata:
     name: dws-prov
   spec:
     controllerName: kueue.x-k8s.io/provisioning-request
     parameters:
       apiGroup: kueue.x-k8s.io
       kind: ProvisioningRequestConfig
       name: dws-config
   ---
   apiVersion: kueue.x-k8s.io/v1beta2
   kind: ProvisioningRequestConfig
   metadata:
     name: dws-config
   spec:
     provisioningClassName: queued-provisioning.gke.io
     podSetUpdates:
     - key: autoscaling.gke.io/provisioning-request
     valueFromProvisioningClassDetail: ResizeRequestName
     managedResources:
     - nvidia.com/gpu
   ---
   apiVersion: kueue.x-k8s.io/v1beta2
   kind: ClusterQueue
   metadata:
     name: cq-tas
   spec:
     namespaceSelector: {}
     clusterQueueingStrategy: BestEffortFIFO
     resourceGroups:
     - flavors:
       - name: a3-edge-flavor
         resources:
         - name: "cpu"
           nominalQuota: 1000
         - name: "memory"
           nominalQuota: 1000Ti
         - name: "nvidia.com/gpu"
           nominalQuota: 1000
       - name: a3-edge-dws-flavor
         resources:
         - name: "cpu"
           nominalQuota: 1000
         - name: "memory"
           nominalQuota: 1000Ti
         - name: "nvidia.com/gpu"
           nominalQuota: 1000
     admissionChecksStrategy:
       admissionChecks:
       - name: "dws-prov"
         onFlavors: [a3-edge-dws-flavor]
   ---
   apiVersion: kueue.x-k8s.io/v1beta2
   kind: LocalQueue
   metadata:
     namespace: default
     name: lq-tas
   spec:
     clusterQueue: cq-tas
   ```

   <br />

2. Apply the manifest:

   ```bash
   kubectl apply -f kueue-config.yaml
   ```

When running workloads with Topology Aware Scheduling (TAS) enabled, you can
specify how strictly topology constraints are enforced by using one of the
following annotations in your
workload manifest:

- `kueue.x-k8s.io/podset-required-topology`: If you use this annotation, Kueue
  blocks scheduling until the workload can be scheduled within the requested
  topology constraint. Use this annotation to ensure that pods are placed
  together for optimal performance.

- `kueue.x-k8s.io/podset-preferred-topology`: If you use this annotation, Kueue
  attempts to schedule pods within the requested topology constraint, but if
  that's not possible, it admits the workload without meeting topology constraints.

> [!NOTE]
> **Note:** Avoid using the required mode with **DWS Flex-start** . Because Flex-start provisions nodes dynamically, the resulting nodes might not satisfy strict topological requirements, which can result in unschedulable workloads. For these configurations, use `podset-preferred-topology` instead.

For either annotation, specify one of the following values as the topology
constraint:

- `cloud.google.com/gce-topology-block`: Schedules pods within the same network block.
- `cloud.google.com/gce-topology-subblock`: Schedules pods within the same rack.
- `cloud.google.com/gce-topology-host`: Schedules pods on the same physical host.

## Test on two Flex-start nodes

To run NCCL tests on a GKE cluster that uses
A3 Edge Flex-start VMs,
use the following procedure. This procedure uses a
[JobSet](https://jobset.sigs.k8s.io/) manifest to run an NCCL test on two nodes.

1. Save the following manifest as a `nccl-tas-jobset.yaml` file:

   ```yaml
   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: nccl-config
   data:
     allgather.sh: |
       #!/bin/bash
       for script in /configs/*; do
         name=$(basename $script)
         cp $script "/scripts/$name"
         chmod +x "/scripts/$name"
       done
       /scripts/init_ssh.sh ${@};
       pushd /scripts;
       /scripts/gen_hostfiles.sh ${@};
       popd;
       /scripts/run-allgather.sh 8 eth1,eth2,eth3,eth4 1M 512M ${#};
   ---
   apiVersion: jobset.x-k8s.io/v1alpha2
   kind: JobSet
   metadata:
     name: nccl-tas-test
     labels:
       kueue.x-k8s.io/queue-name: lq-tas
   spec:
     suspend: true
     network:
       enableDNSHostnames: true
     replicatedJobs:
     - name: worker
       replicas: 2
       template:
         spec:
           parallelism: 1
           completions: 1
           template:
             metadata:
               annotations:
                 kueue.x-k8s.io/podset-preferred-topology: "cloud.google.com/gce-topology-block"
                 networking.gke.io/default-interface: 'eth0'
                 networking.gke.io/interfaces: |
                           [
                             {"interfaceName":"eth0","network":"default"},
                             {"interfaceName":"eth1","network":"vpc0"},
                             {"interfaceName":"eth2","network":"vpc1"},
                             {"interfaceName":"eth3","network":"vpc2"},
                             {"interfaceName":"eth4","network":"vpc3"}
                           ]
             spec:
               terminationGracePeriodSeconds: 0
               nodeSelector:
                 cloud.google.com/gke-accelerator: nvidia-h100-80gb
               tolerations:
               - key: cloud.google.com/gke-queued
                 effect: NoSchedule
                 value: "true"
               - key: "nvidia.com/gpu"
                 operator: "Exists"
                 effect: "NoSchedule"
               setHostnameAsFQDN: true
               containers:
               - name: tcpx-daemon
                 image: us-docker.pkg.dev/gce-ai-infra/gpudirect-tcpx/tcpgpudmarxd-dev:v2.0.11
                 command:
                   - /tcpgpudmarxd/build/app/tcpgpudmarxd
                   - --gpu_nic_preset
                   - a3vm
                   - --gpu_shmem_type
                   - fd
                   - --uds_path
                   - /run/tcpx
                   - --setup_param
                   - "--verbose 128 2 0 "
                 securityContext:
                   privileged: true
                   capabilities:
                     add:
                       - NET_ADMIN
                 volumeMounts:
                   - name: libraries
                     mountPath: /usr/local/nvidia/lib64
                   - name: tcpx-socket
                     mountPath: /run/tcpx
                   - name: sys
                     mountPath: /hostsysfs
                   - name: proc-sys
                     mountPath: /hostprocsysfs
                 env:
                   - name: LD_LIBRARY_PATH
                     value: /usr/local/nvidia/lib64
               - name: nccl-test
                 image: us-docker.pkg.dev/gce-ai-infra/gpudirect-tcpx/nccl-plugin-gpudirecttcpx-dev:v3.1.8
                 command:
                   - bash
                   - -c
                   - |
                     /scripts/container_entry.sh daemon;
                     sleep infinity;
                 securityContext:
                   privileged: true
                 volumeMounts:
                   - name: tcpx-socket
                     mountPath: /tmp
                   - name: libraries
                     mountPath: /usr/local/nvidia/lib64
                   - name: nccl-config
                     mountPath: /configs
                   - name: shared-memory
                     mountPath: /dev/shm
                 resources:
                   limits:
                     cpu: "200"
                     memory: "1800Gi"
                     nvidia.com/gpu: 8
                   requests:
                     cpu: "200"
                     memory: "1800Gi"
                     nvidia.com/gpu: 8
               volumes:
               - name: libraries
                 hostPath:
                   path: /home/kubernetes/bin/nvidia/lib64
               - name: tcpx-socket
                 emptyDir: {}
               - name: sys
                 hostPath:
                   path: /sys
               - name: proc-sys
                 hostPath:
                   path: /proc/sys
               - name: shared-memory
                 emptyDir:
                   medium: Memory
                   sizeLimit: 250Gi
               - name: nccl-config
                 configMap:
                   name: nccl-config
                   defaultMode: 0777
   ```

   <br />

2. Apply the manifest to your cluster:

   ```bash
   kubectl apply -f nccl-tas-jobset.yaml
   ```
3. Check that the JobSet is admitted and running:

   ```bash
   kubectl get jobset nccl-tas-test
   ```

   Wait for the JobSet to be unsuspended and Pods to reach the `Running`
   status.
4. Trigger the NCCL test by executing the `allgather.sh` script
   from the first worker Pod:

   ```bash
   kubectl exec --stdin --tty --container=nccl-test nccl-tas-test-worker-0-0 -- /configs/allgather.sh nccl-tas-test-worker-0-0 nccl-tas-test-worker-1-0
   ```

   The output for a two-node test is similar to the following:

         #                                                              out-of-place                       in-place
         #       size         count      type   redop    root     time   algbw   busbw #wrong     time   algbw   busbw #wrong
         #        (B)    (elements)                               (us)  (GB/s)  (GB/s)            (us)  (GB/s)  (GB/s)
             1048576         16384     float    none      -1    696.8    1.50    1.41      0    729.0    1.44    1.35      0
             ...
             536870912       8388608     float    none      -1   7101.7   75.60   70.87      0   7060.9   76.03   71.28      0
         # Out of bounds values : 0 OK
         # Avg bus bandwidth    : 29.8293

   <br />

## Deploy an NCCL test workload with TAS

If you have more than two nodes, we recommend using the following test, which
uses (TAS). To run NCCL tests with TAS on a GKE cluster that uses
A3 Edge
Flex-start VMs, use the following procedure.

1. Save the following manifest as a `nccl-jobset-test.yaml` file. Replace
   `NUM_NODES` with the number of nodes in the node pool:

   ```yaml
   apiVersion: jobset.x-k8s.io/v1alpha2
   kind: JobSet
   metadata:
     name: nccl-ag
     labels:
       kueue.x-k8s.io/queue-name: lq-tas
   spec:
     ttlSecondsAfterFinished: 1200
     suspend: true
     network:
       enableDNSHostnames: true
     replicatedJobs:
       - name: worker
         template:
           spec:
             parallelism: NUM_NODES
             completions: NUM_NODES
             template:
               metadata:
                 annotations:
                   kueue.x-k8s.io/podset-preferred-topology: "cloud.google.com/gce-topology-subblock"
                   networking.gke.io/default-interface: 'eth0'
                   networking.gke.io/interfaces: |
                           [
                             {"interfaceName":"eth0","network":"default"},
                             {"interfaceName":"eth1","network":"vpc0"},
                             {"interfaceName":"eth2","network":"vpc1"},
                             {"interfaceName":"eth3","network":"vpc2"},
                             {"interfaceName":"eth4","network":"vpc3"}
                           ]
               spec:
                 activeDeadlineSeconds: 3600
                 restartPolicy: Never
                 nodeSelector:
                   cloud.google.com/gke-accelerator: nvidia-h100-80gb
                 tolerations:
                 - key: cloud.google.com/gke-queued
                   operator: "Exists"
                   effect: NoSchedule
                 - key: "nvidia.com/gpu"
                   operator: "Exists"
                   effect: "NoSchedule"
                 setHostnameAsFQDN: true
                 volumes:
                 - name: proc
                   hostPath:
                     path: /proc
                 - name: nvidia
                   hostPath:
                     path: /home/kubernetes/bin/nvidia
                 - name: libraries
                   hostPath:
                     path: /home/kubernetes/bin/nvidia/lib64
                 - name: tcpx-socket
                   emptyDir: {}
                 - name: shared-memory
                   emptyDir:
                     medium: "Memory"
                     sizeLimit: 250Gi
                 containers:
                 - name: tcpx-daemon
                   image: us-docker.pkg.dev/gce-ai-infra/gpudirect-tcpx/tcpgpudmarxd-dev:v2.0.11
                   command:
                     - /tcpgpudmarxd/build/app/tcpgpudmarxd
                     - --gpu_nic_preset
                     - a3vm
                     - --gpu_shmem_type
                     - fd
                     - --uds_path
                     - /run/tcpx
                     - --setup_param
                     - "--verbose 128 2 0 "
                   securityContext:
                     privileged: true
                   volumeMounts:
                     - name: tcpx-socket
                       mountPath: /run/tcpx
                     - name: libraries
                       mountPath: /usr/local/nvidia/lib64
                 - name: nccl-test
                   stdin: true
                   tty: true
                   image: us-docker.pkg.dev/gce-ai-infra/gpudirect-tcpx/nccl-plugin-gpudirecttcpx-dev:v3.1.8
                   securityContext:
                     privileged: true
                   env:
                   - name: MY_NODE_NAME
                     valueFrom:
                       fieldRef:
                         fieldPath: spec.nodeName
                   - name: OMPI_ALLOW_RUN_AS_ROOT
                     value: "1"
                   - name: OMPI_ALLOW_RUN_AS_ROOT_CONFIRM
                     value: "1"
                   - name: N_NODES
                     value: "NUM_NODES"
                   - name: LD_LIBRARY_PATH
                     value: /usr/local/nvidia/lib64
                   command:
                   - bash
                   - -c
                   - |
                     /scripts/container_entry.sh daemon &
                     export POSTFIX=$(hostname | cut -d . -f 2-)
                     export WORKERS_BASENAME=$(hostname | cut -d . -f 1 | rev | cut -d - -f 2- | rev )
                     export NODE_RANK=$JOB_COMPLETION_INDEX
                     for i in `seq 0 $(($N_NODES-1))`; do
                       OTHER=${WORKERS_BASENAME}-${i}.${POSTFIX}
                       until ssh -p 222 -o StrictHostKeyChecking=no $OTHER hostname; do
                         sleep 10
                       done
                       echo ${OTHER} port=222 slots=8 | tee -a /tmp/hostfile;
                     done
                     if [[ "${NODE_RANK}" -eq "0" ]]; then
                         /scripts/run-allgather.sh 8 eth1,eth2,eth3,eth4 1M 512M ${N_NODES}
                     else
                         while ping -c 1 ${WORKERS_BASENAME}-0.${POSTFIX}; do
                         sleep 5
                     done
                     fi
                     exit 0
                   volumeMounts:
                   - name: nvidia
                     mountPath: /usr/local/nvidia
                   - name: tcpx-socket
                     mountPath: /tmp
                   - name: libraries
                     mountPath: /usr/local/nvidia/lib64
                   - name: shared-memory
                     mountPath: /dev/shm
                   resources:
                     limits:
                       cpu: "200"
                       memory: "1800Gi"
                       nvidia.com/gpu: 8
                     requests:
                       cpu: "200"
                       memory: "1800Gi"
                       nvidia.com/gpu: 8
   ```

   <br />

2. Apply the manifest:

   ```bash
   kubectl apply -f nccl-jobset-test.yaml
   ```
3. Check that the workload is admitted and reaches the `Completed` state.

4. Fetch logs for the Pod matching `nccl-ag-worker-0-0-.*` to see the results:

   ```bash
   kubectl logs $(kubectl get pods -o go-template='{{range .items}}{{.metadata.name}}{{"\n"}}{{end}}' | grep nccl-ag-worker-0-0)
   ```

## What's next

- [Collect and Understand NCCL Logs for Troubleshooting](https://docs.cloud.google.com/ai-hypercomputer/docs/nccl/collect-and-understand.md.txt) to understand the test outputs and troubleshoot issues.
- Learn about [troubleshooting slow performance](https://docs.cloud.google.com/ai-hypercomputer/docs/troubleshooting/troubleshoot-slow-performance.md.txt).