在採用 A3 Edge 的 GKE 叢集上執行 NCCL

一般 GPU

本頁說明如何在自訂 GKE 叢集上執行 NVIDIA Collective Communications Library (NCCL) 測試，這些叢集使用 A3 Edge，並採用 GPUDirect-TCPX 網路通訊協定。自訂 GKE 叢集是指您使用 gcloud 指令建立的叢集。

您可以使用本頁面所述的測試，進行下列情境：

• 如果 GKE 叢集使用彈性啟動節點，請在兩個節點上執行基本測試。
• 如果 GKE 叢集使用不同類型的節點，例如隨選或預訂節點，請使用拓撲感知排程進行 NCCL 測試。

事前準備

本頁面的測試使用 JobSet 和 Kueue with Topology Aware Scheduling (TAS)。執行任何測試前，請務必設定叢集並完成下列步驟：

1. 安裝 JobSet。

2. 安裝 Kueue。

   kubectl apply --server-side -f https://github.com/kubernetes-sigs/kueue/releases/download/v0.16.5/manifests.yaml
   

使用 Jobset 和 Kueue 設定叢集

安裝 JobSet 和 Kueue 後，請按照下列步驟操作：

1. 將下列資訊清單儲存為 kueue-config.yaml 檔案：

   apiVersion: kueue.x-k8s.io/v1beta2
   kind: Topology
   metadata:
     name: "gke-default"
   spec:
     levels:
     - nodeLabel: "cloud.google.com/gce-topology-block"
     - nodeLabel: "cloud.google.com/gce-topology-subblock"
     - nodeLabel: "cloud.google.com/gce-topology-host"
     - nodeLabel: "kubernetes.io/hostname"
   ---
   apiVersion: kueue.x-k8s.io/v1beta2
   kind: ResourceFlavor
   metadata:
     name: a3-edge-flavor
   spec:
     nodeLabels:
       cloud.google.com/gke-accelerator: nvidia-h100-80gb
     topologyName: "gke-default"
   ---
   apiVersion: kueue.x-k8s.io/v1beta2
   kind: ResourceFlavor
   metadata:
     name: a3-edge-dws-flavor
   spec:
     nodeLabels:
       cloud.google.com/gke-accelerator: nvidia-h100-80gb
     topologyName: "gke-default"
     tolerations:
     - key: "cloud.google.com/gke-queued"
       operator: "Exists"
       effect: NoSchedule
   ---
   apiVersion: kueue.x-k8s.io/v1beta2
   kind: AdmissionCheck
   metadata:
     name: dws-prov
   spec:
     controllerName: kueue.x-k8s.io/provisioning-request
     parameters:
       apiGroup: kueue.x-k8s.io
       kind: ProvisioningRequestConfig
       name: dws-config
   ---
   apiVersion: kueue.x-k8s.io/v1beta2
   kind: ProvisioningRequestConfig
   metadata:
     name: dws-config
   spec:
     provisioningClassName: queued-provisioning.gke.io
     podSetUpdates:
     - key: autoscaling.gke.io/provisioning-request
     valueFromProvisioningClassDetail: ResizeRequestName
     managedResources:
     - nvidia.com/gpu
   ---
   apiVersion: kueue.x-k8s.io/v1beta2
   kind: ClusterQueue
   metadata:
     name: cq-tas
   spec:
     namespaceSelector: {}
     clusterQueueingStrategy: BestEffortFIFO
     resourceGroups:
     - flavors:
       - name: a3-edge-flavor
         resources:
         - name: "cpu"
           nominalQuota: 1000
         - name: "memory"
           nominalQuota: 1000Ti
         - name: "nvidia.com/gpu"
           nominalQuota: 1000
       - name: a3-edge-dws-flavor
         resources:
         - name: "cpu"
           nominalQuota: 1000
         - name: "memory"
           nominalQuota: 1000Ti
         - name: "nvidia.com/gpu"
           nominalQuota: 1000
     admissionChecksStrategy:
       admissionChecks:
       - name: "dws-prov"
         onFlavors: [a3-edge-dws-flavor]
   ---
   apiVersion: kueue.x-k8s.io/v1beta2
   kind: LocalQueue
   metadata:
     namespace: default
     name: lq-tas
   spec:
     clusterQueue: cq-tas
   

2. 套用資訊清單：

   kubectl apply -f kueue-config.yaml
   

啟用拓撲感知排程 (TAS) 執行工作負載時，您可以在工作負載資訊清單中使用下列其中一個註解，指定拓撲限制的強制執行嚴格程度：

• kueue.x-k8s.io/podset-required-topology：如果您使用這項註解，Kueue 會封鎖排程，直到工作負載可在要求的拓撲限制內排程為止。使用這項註解，確保廣告連播會放在一起，爭取最佳成效。

• kueue.x-k8s.io/podset-preferred-topology：如果您使用這個註解，Kueue 會嘗試在要求的拓撲限制內排定 Pod，但如果無法這麼做，就會允許工作負載，而不符合拓撲限制。

附註： 請避免搭配 DWS 彈性啟動使用必要模式。由於彈性啟動會動態佈建節點，因此產生的節點可能無法滿足嚴格的拓撲要求，導致工作負載無法排程。如要進行這些設定，請改用 podset-preferred-topology。

無論是哪種註解，請指定下列其中一個值做為拓撲限制：

• cloud.google.com/gce-topology-block：在相同網路區塊內排定 Pod。
• cloud.google.com/gce-topology-subblock：在同一個機架內排程 Pod。
• cloud.google.com/gce-topology-host：在同一部實體主機上排定 Pod。

在兩個彈性啟動節點上測試

如要在使用 A3 Edge Flex-start VM 的 GKE 叢集上執行 NCCL 測試，請按照下列程序操作。這個程序會使用 JobSet 資訊清單，在兩個節點上執行 NCCL 測試。

1. 將下列資訊清單儲存為 nccl-tas-jobset.yaml 檔案：

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: nccl-config
   data:
     allgather.sh: |
       #!/bin/bash
       for script in /configs/*; do
         name=$(basename $script)
         cp $script "/scripts/$name"
         chmod +x "/scripts/$name"
       done
       /scripts/init_ssh.sh ${@};
       pushd /scripts;
       /scripts/gen_hostfiles.sh ${@};
       popd;
       /scripts/run-allgather.sh 8 eth1,eth2,eth3,eth4 1M 512M ${#};
   ---
   apiVersion: jobset.x-k8s.io/v1alpha2
   kind: JobSet
   metadata:
     name: nccl-tas-test
     labels:
       kueue.x-k8s.io/queue-name: lq-tas
   spec:
     suspend: true
     network:
       enableDNSHostnames: true
     replicatedJobs:
     - name: worker
       replicas: 2
       template:
         spec:
           parallelism: 1
           completions: 1
           template:
             metadata:
               annotations:
                 kueue.x-k8s.io/podset-preferred-topology: "cloud.google.com/gce-topology-block"
                 networking.gke.io/default-interface: 'eth0'
                 networking.gke.io/interfaces: |
                           [
                             {"interfaceName":"eth0","network":"default"},
                             {"interfaceName":"eth1","network":"vpc0"},
                             {"interfaceName":"eth2","network":"vpc1"},
                             {"interfaceName":"eth3","network":"vpc2"},
                             {"interfaceName":"eth4","network":"vpc3"}
                           ]
             spec:
               terminationGracePeriodSeconds: 0
               nodeSelector:
                 cloud.google.com/gke-accelerator: nvidia-h100-80gb
               tolerations:
               - key: cloud.google.com/gke-queued
                 effect: NoSchedule
                 value: "true"
               - key: "nvidia.com/gpu"
                 operator: "Exists"
                 effect: "NoSchedule"
               setHostnameAsFQDN: true
               containers:
               - name: tcpx-daemon
                 image: us-docker.pkg.dev/gce-ai-infra/gpudirect-tcpx/tcpgpudmarxd-dev:v2.0.11
                 command:
                   - /tcpgpudmarxd/build/app/tcpgpudmarxd
                   - --gpu_nic_preset
                   - a3vm
                   - --gpu_shmem_type
                   - fd
                   - --uds_path
                   - /run/tcpx
                   - --setup_param
                   - "--verbose 128 2 0 "
                 securityContext:
                   privileged: true
                   capabilities:
                     add:
                       - NET_ADMIN
                 volumeMounts:
                   - name: libraries
                     mountPath: /usr/local/nvidia/lib64
                   - name: tcpx-socket
                     mountPath: /run/tcpx
                   - name: sys
                     mountPath: /hostsysfs
                   - name: proc-sys
                     mountPath: /hostprocsysfs
                 env:
                   - name: LD_LIBRARY_PATH
                     value: /usr/local/nvidia/lib64
               - name: nccl-test
                 image: us-docker.pkg.dev/gce-ai-infra/gpudirect-tcpx/nccl-plugin-gpudirecttcpx-dev:v3.1.8
                 command:
                   - bash
                   - -c
                   - |
                     /scripts/container_entry.sh daemon;
                     sleep infinity;
                 securityContext:
                   privileged: true
                 volumeMounts:
                   - name: tcpx-socket
                     mountPath: /tmp
                   - name: libraries
                     mountPath: /usr/local/nvidia/lib64
                   - name: nccl-config
                     mountPath: /configs
                   - name: shared-memory
                     mountPath: /dev/shm
                 resources:
                   limits:
                     cpu: "200"
                     memory: "1800Gi"
                     nvidia.com/gpu: 8
                   requests:
                     cpu: "200"
                     memory: "1800Gi"
                     nvidia.com/gpu: 8
               volumes:
               - name: libraries
                 hostPath:
                   path: /home/kubernetes/bin/nvidia/lib64
               - name: tcpx-socket
                 emptyDir: {}
               - name: sys
                 hostPath:
                   path: /sys
               - name: proc-sys
                 hostPath:
                   path: /proc/sys
               - name: shared-memory
                 emptyDir:
                   medium: Memory
                   sizeLimit: 250Gi
               - name: nccl-config
                 configMap:
                   name: nccl-config
                   defaultMode: 0777
   

2. 將資訊清單套用至叢集：

   kubectl apply -f nccl-tas-jobset.yaml
   

3. 確認 JobSet 已獲准並正在執行：

   kubectl get jobset nccl-tas-test
   

   等待 JobSet 取消暫停，以及 Pod 達到 Running 狀態。

4. 從第一個工作站 Pod 執行 allgather.sh 指令碼，觸發 NCCL 測試：

   kubectl exec --stdin --tty --container=nccl-test nccl-tas-test-worker-0-0 -- /configs/allgather.sh nccl-tas-test-worker-0-0 nccl-tas-test-worker-1-0
   

   雙節點測試的輸出內容如下：

     #                                                              out-of-place                       in-place
     #       size         count      type   redop    root     time   algbw   busbw #wrong     time   algbw   busbw #wrong
     #        (B)    (elements)                               (us)  (GB/s)  (GB/s)            (us)  (GB/s)  (GB/s)
         1048576         16384     float    none      -1    696.8    1.50    1.41      0    729.0    1.44    1.35      0
         ...
         536870912       8388608     float    none      -1   7101.7   75.60   70.87      0   7060.9   76.03   71.28      0
     # Out of bounds values : 0 OK
     # Avg bus bandwidth    : 29.8293
   

使用 TAS 部署 NCCL 測試工作負載

如果您有兩個以上的節點，建議使用下列測試 (TAS)。如要在使用 A3 Edge Flex-start VM 的 GKE 叢集上，透過 TAS 執行 NCCL 測試，請按照下列程序操作。

1. 將下列資訊清單儲存為 nccl-jobset-test.yaml 檔案。將 NUM_NODES 換成節點集區中的節點數量：

   apiVersion: jobset.x-k8s.io/v1alpha2
   kind: JobSet
   metadata:
     name: nccl-ag
     labels:
       kueue.x-k8s.io/queue-name: lq-tas
   spec:
     ttlSecondsAfterFinished: 1200
     suspend: true
     network:
       enableDNSHostnames: true
     replicatedJobs:
       - name: worker
         template:
           spec:
             parallelism: NUM_NODES
             completions: NUM_NODES
             template:
               metadata:
                 annotations:
                   kueue.x-k8s.io/podset-preferred-topology: "cloud.google.com/gce-topology-subblock"
                   networking.gke.io/default-interface: 'eth0'
                   networking.gke.io/interfaces: |
                           [
                             {"interfaceName":"eth0","network":"default"},
                             {"interfaceName":"eth1","network":"vpc0"},
                             {"interfaceName":"eth2","network":"vpc1"},
                             {"interfaceName":"eth3","network":"vpc2"},
                             {"interfaceName":"eth4","network":"vpc3"}
                           ]
               spec:
                 activeDeadlineSeconds: 3600
                 restartPolicy: Never
                 nodeSelector:
                   cloud.google.com/gke-accelerator: nvidia-h100-80gb
                 tolerations:
                 - key: cloud.google.com/gke-queued
                   operator: "Exists"
                   effect: NoSchedule
                 - key: "nvidia.com/gpu"
                   operator: "Exists"
                   effect: "NoSchedule"
                 setHostnameAsFQDN: true
                 volumes:
                 - name: proc
                   hostPath:
                     path: /proc
                 - name: nvidia
                   hostPath:
                     path: /home/kubernetes/bin/nvidia
                 - name: libraries
                   hostPath:
                     path: /home/kubernetes/bin/nvidia/lib64
                 - name: tcpx-socket
                   emptyDir: {}
                 - name: shared-memory
                   emptyDir:
                     medium: "Memory"
                     sizeLimit: 250Gi
                 containers:
                 - name: tcpx-daemon
                   image: us-docker.pkg.dev/gce-ai-infra/gpudirect-tcpx/tcpgpudmarxd-dev:v2.0.11
                   command:
                     - /tcpgpudmarxd/build/app/tcpgpudmarxd
                     - --gpu_nic_preset
                     - a3vm
                     - --gpu_shmem_type
                     - fd
                     - --uds_path
                     - /run/tcpx
                     - --setup_param
                     - "--verbose 128 2 0 "
                   securityContext:
                     privileged: true
                   volumeMounts:
                     - name: tcpx-socket
                       mountPath: /run/tcpx
                     - name: libraries
                       mountPath: /usr/local/nvidia/lib64
                 - name: nccl-test
                   stdin: true
                   tty: true
                   image: us-docker.pkg.dev/gce-ai-infra/gpudirect-tcpx/nccl-plugin-gpudirecttcpx-dev:v3.1.8
                   securityContext:
                     privileged: true
                   env:
                   - name: MY_NODE_NAME
                     valueFrom:
                       fieldRef:
                         fieldPath: spec.nodeName
                   - name: OMPI_ALLOW_RUN_AS_ROOT
                     value: "1"
                   - name: OMPI_ALLOW_RUN_AS_ROOT_CONFIRM
                     value: "1"
                   - name: N_NODES
                     value: "NUM_NODES"
                   - name: LD_LIBRARY_PATH
                     value: /usr/local/nvidia/lib64
                   command:
                   - bash
                   - -c
                   - |
                     /scripts/container_entry.sh daemon &
                     export POSTFIX=$(hostname | cut -d . -f 2-)
                     export WORKERS_BASENAME=$(hostname | cut -d . -f 1 | rev | cut -d - -f 2- | rev )
                     export NODE_RANK=$JOB_COMPLETION_INDEX
                     for i in `seq 0 $(($N_NODES-1))`; do
                       OTHER=${WORKERS_BASENAME}-${i}.${POSTFIX}
                       until ssh -p 222 -o StrictHostKeyChecking=no $OTHER hostname; do
                         sleep 10
                       done
                       echo ${OTHER} port=222 slots=8 | tee -a /tmp/hostfile;
                     done
                     if [[ "${NODE_RANK}" -eq "0" ]]; then
                         /scripts/run-allgather.sh 8 eth1,eth2,eth3,eth4 1M 512M ${N_NODES}
                     else
                         while ping -c 1 ${WORKERS_BASENAME}-0.${POSTFIX}; do
                         sleep 5
                     done
                     fi
                     exit 0
                   volumeMounts:
                   - name: nvidia
                     mountPath: /usr/local/nvidia
                   - name: tcpx-socket
                     mountPath: /tmp
                   - name: libraries
                     mountPath: /usr/local/nvidia/lib64
                   - name: shared-memory
                     mountPath: /dev/shm
                   resources:
                     limits:
                       cpu: "200"
                       memory: "1800Gi"
                       nvidia.com/gpu: 8
                     requests:
                       cpu: "200"
                       memory: "1800Gi"
                       nvidia.com/gpu: 8
   

2. 套用資訊清單：

   kubectl apply -f nccl-jobset-test.yaml
   

3. 確認工作負載已獲准，並達到 Completed 狀態。

4. 擷取與 nccl-ag-worker-0-0-.* 相符的 Pod 記錄，查看結果：

   kubectl logs $(kubectl get pods -o go-template='{{range .items}}{{.metadata.name}}{{"\n"}}{{end}}' | grep nccl-ag-worker-0-0)
   

後續步驟

• 收集並解讀 NCCL 記錄以排解問題，瞭解測試輸出內容並排解問題。
• 瞭解如何排解效能緩慢的問題。