Google menggunakan teknologi AI untuk menerjemahkan konten ke dalam bahasa pilihan Anda. Terjemahan AI mungkin mengandung kesalahan.

Menyesuaikan dan menskalakan reinforcement learning dengan verl di GKE

Autopilot Standard

Tutorial ini menunjukkan cara mengorkestrasi lingkungan pelatihan terdistribusi untuk reinforcement learning di Google Kubernetes Engine (GKE). Anda menggunakan Ray dan framework verl (Volcano Engine Reinforcement Learning) untuk menyiapkan lingkungan pelatihan terdistribusi guna menyempurnakan model Qwen2.5-32B-Instruct pada set data GSM8K.

Tutorial ini berfokus pada pipeline pelatihan Group Relative Policy Optimization (GRPO) di GKE dengan Ray dan verl. GRPO adalah algoritma pembelajaran penguatan yang dirancang untuk meningkatkan kemampuan penalaran model. Algoritma hemat memori ini menyederhanakan proses reinforcement learning (RL) dengan menghilangkan Critic, atau model nilai, dan menggunakan perhitungan berbasis grup relatif.

Tutorial ini adalah titik awal yang baik jika Anda perlu menyiapkan lingkungan pelatihan terdistribusi tempat data, bobot model, dan mesin pelatihan dipisahkan untuk efisiensi.

Tutorial ini mendukung arsitektur GPU berikut:

Node GPU berbasis Intel atau AMD: Siapkan dan lakukan penskalaan menggunakan GPU NVIDIA B200 atau H200, menggunakan Alokasi Resource Dinamis (DRA) GKE untuk jalur Autopilot.
Node A4X (GB200) berbasis Arm: Siapkan dan lakukan penskalaan menggunakan Superchip NVIDIA GB200 Grace Blackwell, menggunakan Alokasi Resource Dinamis (DRA) GKE dan Multi-Node NVLink (IMEX).

Latar belakang

Bagian berikut memberikan ringkasan singkat tentang konsep yang digunakan dalam tutorial ini.

Reinforcement learning (RL)

RL mengajari model melalui pengalaman, eksplorasi, dan masukan, bukan imitasi statis. Meskipun pra-pelatihan mengajarkan model apa yang harus dikatakan, Reinforcement Learning from Human Feedback (RLHF) mengajarkannya cara menjadi bermanfaat, aman, dan logis. RL berfungsi sebagai jembatan antara model dasar dan model yang di-fine-tune untuk kasus penggunaan khusus.

Untuk mengetahui informasi selengkapnya, lihat Apa itu reinforcement learning?

Pengoptimalan Kebijakan Relatif Grup (GRPO)

GRPO, sebuah algoritma yang dipopulerkan oleh DeepSeek, menawarkan alternatif yang hemat memori untuk Proximal Policy Optimization (PPO) dalam penyelarasan LLM dengan menghapus model Critic. Alih-alih jaringan Critic, GRPO menghasilkan sekelompok respons untuk perintah yang sama dan menggunakan reward rata-rata kelompok tersebut sebagai dasar.

Untuk mengetahui informasi selengkapnya, lihat GRPO.

Volcano Engine Reinforcement Learning (verl)

verl adalah framework berperforma tinggi yang dirancang untuk menangani pola memori dan komputasi yang kompleks dari RL berbasis LLM.

Untuk mengetahui informasi selengkapnya, lihat verl.

Tujuan

Tutorial ini menunjukkan cara menyiapkan reinforcement learning di GKE dengan verl, dengan menyelesaikan langkah-langkah berikut:

Siapkan cluster GKE dengan A4X (GB200 Superchips), A4 (GPU B200), atau A3 Ultra (GPU H200).
Konfigurasi KubeRay untuk mengelola cluster Ray terdistribusi.
Gunakan Cloud Storage FUSE untuk memasang bucket Cloud Storage di semua node.
Jalankan tugas pelatihan GRPO menggunakan verl untuk menyelaraskan model Qwen2.5-32B-Instruct dengan set data GSM8K.

Sebelum memulai

Login ke akun Google Cloud Anda. Jika Anda baru menggunakan Google Cloud, buat akun untuk mengevaluasi performa produk kami dalam skenario dunia nyata. Pelanggan baru juga mendapatkan kredit gratis senilai $300 untuk menjalankan, menguji, dan men-deploy workload.

Instal Google Cloud CLI.

Catatan: Jika Anda telah menginstal gcloud CLI sebelumnya, pastikan Anda memiliki versi terbaru dengan menjalankan gcloud components update.

Jika Anda menggunakan penyedia identitas (IdP) eksternal, Anda harus login ke gcloud CLI dengan identitas gabungan Anda terlebih dahulu.

Untuk melakukan inisialisasi gcloud CLI, jalankan perintah berikut:

gcloud init

Buat atau pilih Google Cloud project.

Peran yang diperlukan untuk memilih atau membuat project

Pilih project: Memilih project tidak memerlukan peran IAM tertentu—Anda dapat memilih project yang telah diberi peran.
Membuat project: Untuk membuat project, Anda memerlukan peran Project Creator (roles/resourcemanager.projectCreator), yang berisi izin resourcemanager.projects.create. Pelajari cara memberikan peran.

Buat Google Cloud project:
```
gcloud projects create PROJECT_ID
```
Ganti PROJECT_ID dengan nama untuk Google Cloud project yang Anda buat.
Pilih project Google Cloud yang Anda buat:
```
gcloud config set project PROJECT_ID
```
Ganti PROJECT_ID dengan nama project Google Cloud Anda.

Verifikasi bahwa penagihan diaktifkan untuk project Google Cloud Anda.

Aktifkan API yang diperlukan, jika ada yang belum diaktifkan:

Peran yang diperlukan untuk mengaktifkan API

Untuk mengaktifkan API, Anda memerlukan izin serviceusage.services.enable. Jika Anda membuat project, kemungkinan Anda sudah memiliki izin ini melalui peran Pemilik (roles/owner). Jika tidak, Anda bisa mendapatkan izin ini melalui peran Admin Penggunaan Layanan (roles/serviceusage.serviceUsageAdmin). Pelajari cara memberikan peran.

gcloud services enable container.googleapis.com storage.googleapis.com compute.googleapis.com

Instal Google Cloud CLI.

Catatan: Jika Anda telah menginstal gcloud CLI sebelumnya, pastikan Anda memiliki versi terbaru dengan menjalankan gcloud components update.

Jika Anda menggunakan penyedia identitas (IdP) eksternal, Anda harus login ke gcloud CLI dengan identitas gabungan Anda terlebih dahulu.

Untuk melakukan inisialisasi gcloud CLI, jalankan perintah berikut:

gcloud init

Buat atau pilih Google Cloud project.

Peran yang diperlukan untuk memilih atau membuat project

Pilih project: Memilih project tidak memerlukan peran IAM tertentu—Anda dapat memilih project yang telah diberi peran.
Membuat project: Untuk membuat project, Anda memerlukan peran Project Creator (roles/resourcemanager.projectCreator), yang berisi izin resourcemanager.projects.create. Pelajari cara memberikan peran.

Buat Google Cloud project:
```
gcloud projects create PROJECT_ID
```
Ganti PROJECT_ID dengan nama untuk Google Cloud project yang Anda buat.
Pilih project Google Cloud yang Anda buat:
```
gcloud config set project PROJECT_ID
```
Ganti PROJECT_ID dengan nama project Google Cloud Anda.

Verifikasi bahwa penagihan diaktifkan untuk project Google Cloud Anda.

Aktifkan API yang diperlukan, jika ada yang belum diaktifkan:

Peran yang diperlukan untuk mengaktifkan API

gcloud services enable container.googleapis.com storage.googleapis.com compute.googleapis.com

Memberikan peran ke akun pengguna Anda. Jalankan perintah berikut satu kali untuk setiap peran IAM berikut: roles/container.admin, roles/iam.serviceAccountAdmin, roles/storage.admin, roles/iam.serviceAccountUser
```
gcloud projects add-iam-policy-binding PROJECT_ID --member="user:USER_IDENTIFIER" --role=ROLE
```
Ganti kode berikut:
- PROJECT_ID: Project ID Anda.
- USER_IDENTIFIER: ID untuk akun pengguna Anda. Misalnya, myemail@example.com.
- ROLE: Peran IAM yang Anda berikan ke akun pengguna Anda.

Buat akun Hugging Face jika Anda belum memilikinya.
Pastikan Anda memiliki token Hugging Face.
Pastikan project Anda memiliki kuota yang cukup untuk A4X (GB200 Superchips), A4 (GPU B200), atau A3 Ultra (GPU H200). Untuk mempelajari lebih lanjut, lihat Merencanakan kuota GPU dan Kuota GPU.
Pastikan Anda memiliki reservasi kapasitas aktif untuk jenis mesin GPU Anda. Untuk mengetahui informasi selengkapnya, lihat Mencadangkan kapasitas melalui tim akun Anda.
Untuk penyiapan A4X (GB200), pastikan Anda telah menginstal Helm

Menyiapkan lingkungan Anda

Dalam tutorial ini, Anda akan menggunakan Cloud Shell.

Buka Google Cloud console.
Di bagian atas Google Cloud jendela konsol, klik tombol Activate Cloud Shell.

Tetapkan variabel lingkungan:

A4 dan A3 Ultra

Autopilot

export PROJECT_ID=$(gcloud config get project)
export PROJECT_NUMBER=$(gcloud projects describe ${PROJECT_ID} --format="value(projectNumber)")
export CONTROL_PLANE_REGION="YOUR_REGION"
export NODE_ZONE="YOUR_ZONE"
export CLUSTER_NAME="YOUR_CLUSTER_NAME"
export KSA_NAME="YOUR_KSA_NAME"
export GS_BUCKET="YOUR_GCS_BUCKET"
export NAMESPACE="default"
export GPU_TYPE="YOUR_GPU_TYPE"
export MACHINE_TYPE="YOUR_MACHINE_TYPE"
export RESERVATION="YOUR_RESERVATION_NAME"
export HF_TOKEN="YOUR_HF_TOKEN"

Standar

export PROJECT_ID=$(gcloud config get project)
export PROJECT_NUMBER=$(gcloud projects describe ${PROJECT_ID} --format="value(projectNumber)")
export CONTROL_PLANE_REGION="YOUR_REGION"
export NODE_ZONE="YOUR_ZONE"
export CLUSTER_NAME="YOUR_CLUSTER_NAME"
export KSA_NAME="YOUR_KSA_NAME"
export GS_BUCKET="YOUR_GCS_BUCKET"
export NAMESPACE="default"
export GPU_TYPE="YOUR_GPU_TYPE"
export MACHINE_TYPE="YOUR_MACHINE_TYPE"
export CPU_MACHINE_TYPE="YOUR_CPU_MACHINE_TYPE"
export RESERVATION="YOUR_RESERVATION_NAME"
export HF_TOKEN="YOUR_HF_TOKEN"

export GVNIC_NETWORK_PREFIX="YOUR_GVNIC_NAME"
export RDMA_NETWORK_PREFIX="YOUR_RDMA_NAME"

A4X

export PROJECT_ID=$(gcloud config get project)
export PROJECT_NUMBER=$(gcloud projects describe ${PROJECT_ID} --format="value(projectNumber)")
export CONTROL_PLANE_REGION=YOUR_REGION
export NODE_ZONE=YOUR_ZONE
export CLUSTER_NAME=YOUR_CLUSTER_NAME
export KSA_NAME=YOUR_KSA_NAME
export GS_BUCKET=YOUR_GCS_BUCKET-${PROJECT_ID}
export NAMESPACE=default
export GPU_TYPE=YOUR_GPU_TYPE
export MACHINE_TYPE=YOUR_MACINE_TYPE
export RESERVATION=YOUR_RESERVATION_NAME
export HF_TOKEN=YOUR_HF_TOKEN

# A4X (GB200 Superchips) only variables
export NUM_GPU_NODES=4
export VERL_IMAGE=verlai/verl:vllm023.aarch64.dev1
export VERL_REF=ddbcdb7

Ganti nilai berikut:

YOUR_REGION: region Compute Engine untuk bidang kontrol cluster GKE.
YOUR_ZONE: zona tempat node dipesan. Untuk mengetahui informasi selengkapnya, lihat Ketersediaan GPU.
YOUR_CLUSTER_NAME: nama cluster GKE Anda.
YOUR_KSA_NAME: nama akun layanan Kubernetes Anda.
YOUR_GCS_BUCKET: nama dasar untuk bucket Cloud Storage Anda. Anda tidak perlu menentukan awalan gs://.
YOUR_GPU_TYPE: akselerator yang Anda pesan dalam reservasi kapasitas Compute Engine. Nilainya harus berupa salah satu dari nilai berikut:
- nvidia-gb200: A4X (GB200 Superchips)
- nvidia-b200: A4 (GPU B200)
- nvidia-h200-141gb: A3 Ultra (GPU H200)
YOUR_MACHINE_TYPE: jenis mesin yang akan digunakan:
- Untuk A4X (GB200 Superchips), gunakan a4x-highgpu-4g.
- Untuk A4 (GPU B200), gunakan a4-highgpu-8g atau yang lebih baru.
- Untuk A3 Ultra (GPU H200), gunakan a3-ultragpu-8g atau yang lebih baru.
YOUR_CPU_MACHINE_TYPE: jenis mesin Compute Engine untuk node pool default CPU. Gunakan jenis mesin highmem dengan minimal 16 vCPU yang didukung di zona cluster Anda, seperti n4-highmem-16 atau n2d-highmem-16.
YOUR_RESERVATION_NAME: nama reservasi kapasitas Anda.
YOUR_HF_TOKEN: token Hugging Face Anda.
Gunakan variabel berikut saat menyiapkan lingkungan di cluster GKE Standard:
- YOUR_GVNIC_NAME (Khusus A4 atau A3 Ultra): awalan untuk nama jaringan gVNIC. Anda dapat menggunakan awalan apa pun yang Anda inginkan.
- YOUR_RDMA_NAME (khusus A4 atau A3 Ultra): awalan untuk jaringan remote direct memory access (RDMA). Anda dapat menggunakan awalan apa pun yang Anda inginkan.

Gandakan repositori contoh:

git clone https://github.com/GoogleCloudSamples/AIHypercomputerSamples.git

Buka direktori kerja untuk mode GKE pilihan Anda:
A4 dan A3 Ultra
Autopilot

cd AIHypercomputerSamples/gpu/tuning/verl_rl_autopilot

Standar

cd AIHypercomputerSamples/gpu/tuning/verl_rl_standard
A4X
Tidak ada perubahan direktori yang diperlukan. Anda dapat langsung melanjutkan ke bagian berikutnya.

Menyiapkan infrastruktur

Di bagian ini, Anda akan membuat jaringan VPC standar dan cluster GKE.

Buat jaringan dan subnet RDMA (GKE Standard - A4 dan A3 Ultra saja)

A4 dan A3 Ultra

Autopilot

Bagian ini hanya diperlukan untuk GPU Ultra A4 dan A3 GKE Standard.

Jika Anda menggunakan Autopilot, lewati bagian ini dan langsung lanjutkan ke Membuat cluster GKE. GKE secara otomatis menyediakan jaringan dan subnet VPC yang diperlukan, serta menggunakan DRANET yang dikelola GKE untuk mengalokasikan resource ini ke Pod Anda. Anda tidak perlu membuat infrastruktur jaringan secara manual.

Standar

Buat jaringan VPC untuk antarmuka gVNIC:

gcloud compute networks create ${GVNIC_NETWORK_PREFIX}-net \
  --subnet-mode=custom \
  --project=${PROJECT_ID}

gcloud compute networks subnets create ${GVNIC_NETWORK_PREFIX}-sub \
  --network=${GVNIC_NETWORK_PREFIX}-net \
  --region=${CONTROL_PLANE_REGION} \
  --range=192.168.0.0/24 \
  --project=${PROJECT_ID}

gcloud compute firewall-rules create ${GVNIC_NETWORK_PREFIX}-internal \
  --network=${GVNIC_NETWORK_PREFIX}-net \
  --action=ALLOW \
  --rules=tcp:0-65535,udp:0-65535,icmp \
  --source-ranges=192.168.0.0/16 \
  --project=${PROJECT_ID}

Buat jaringan VPC untuk RDMA:

gcloud compute networks create ${RDMA_NETWORK_PREFIX}-net \
  --network-profile=${NODE_ZONE}-vpc-roce \
  --subnet-mode=custom \
  --project=${PROJECT_ID}

Buat 8 subnet RDMA untuk 8 GPU:

for N in $(seq 0 7); do
  if ! gcloud compute networks subnets describe ${RDMA_NETWORK_PREFIX}-sub-$N --region=${CONTROL_PLANE_REGION} --project=${PROJECT_ID} >/dev/null 2>&1; then
    gcloud compute networks subnets create ${RDMA_NETWORK_PREFIX}-sub-$N \
      --network=${RDMA_NETWORK_PREFIX}-net \
      --region=${CONTROL_PLANE_REGION} \
      --range=192.168.$((N+1)).0/24 \
      --project=${PROJECT_ID} &
  else
    echo "Subnet ${RDMA_NETWORK_PREFIX}-sub-$N already exists."
  fi
done
wait

A4X

Bagian ini hanya diperlukan untuk GPU Ultra A4 dan A3 GKE Standard.

Jika Anda menggunakan GPU A4X (GB200), lewati bagian ini dan langsung lanjutkan ke Membuat cluster GKE. Untuk GPU A4X (GB200) atau Autopilot, GKE membuat jaringan secara otomatis saat node pool menggunakan profil jaringan akselerator auto. Blueprint Cluster Toolkit mengaktifkan profil ini dengan menggunakan flag enable_dranet:true.

Membuat cluster GKE

Buat cluster GKE yang sesuai dengan arsitektur GPU Anda:

A4 dan A3 Ultra

Pilih mode cluster GKE yang ingin Anda gunakan:

Autopilot

Buat cluster Autopilot:

gcloud container clusters create-auto ${CLUSTER_NAME} \
    --location=${CONTROL_PLANE_REGION} \
    --release-channel=rapid \
    --enable-ray-operator

Dapatkan kredensial untuk cluster Anda:

gcloud container clusters get-credentials ${CLUSTER_NAME} \
    --location=${CONTROL_PLANE_REGION}

Standar

Buat cluster Standard:

gcloud container clusters create ${CLUSTER_NAME} \
    --location=${CONTROL_PLANE_REGION} \
    --enable-dataplane-v2 \
    --workload-pool=${PROJECT_ID}.svc.id.goog \
    --enable-ip-alias \
    --enable-multi-networking \
    --addons=RayOperator,GcsFuseCsiDriver \
    --machine-type=${CPU_MACHINE_TYPE} \
    --num-nodes=1 \
    --min-nodes=1 \
    --max-nodes=5 \
    --enable-autoscaling \
    --project=${PROJECT_ID}

Dapatkan kredensial untuk cluster Anda:

gcloud container clusters get-credentials ${CLUSTER_NAME} \
    --location=${CONTROL_PLANE_REGION} \
    --project=${PROJECT_ID}

Buat node pool GPU. Node pool ini menggunakan reservasi Anda untuk memastikan ketersediaan. Anda mulai dengan dua node:

CMD=(
  gcloud container node-pools create gpu-pool
  --cluster="${CLUSTER_NAME}"
  --location="${CONTROL_PLANE_REGION}"
  --node-locations="${NODE_ZONE}"
  --machine-type="${MACHINE_TYPE}"
  --accelerator="type=${GPU_TYPE},count=8,gpu-driver-version=DEFAULT"
  --enable-autoscaling
  --num-nodes=2
  --total-max-nodes=10
  --additional-node-network="network=${GVNIC_NETWORK_PREFIX}-net,subnetwork=${GVNIC_NETWORK_PREFIX}-sub"
  --additional-node-network="network=${RDMA_NETWORK_PREFIX}-net,subnetwork=${RDMA_NETWORK_PREFIX}-sub-0"
  --additional-node-network="network=${RDMA_NETWORK_PREFIX}-net,subnetwork=${RDMA_NETWORK_PREFIX}-sub-1"
  --additional-node-network="network=${RDMA_NETWORK_PREFIX}-net,subnetwork=${RDMA_NETWORK_PREFIX}-sub-2"
  --additional-node-network="network=${RDMA_NETWORK_PREFIX}-net,subnetwork=${RDMA_NETWORK_PREFIX}-sub-3"
  --additional-node-network="network=${RDMA_NETWORK_PREFIX}-net,subnetwork=${RDMA_NETWORK_PREFIX}-sub-4"
  --additional-node-network="network=${RDMA_NETWORK_PREFIX}-net,subnetwork=${RDMA_NETWORK_PREFIX}-sub-5"
  --additional-node-network="network=${RDMA_NETWORK_PREFIX}-net,subnetwork=${RDMA_NETWORK_PREFIX}-sub-6"
  --additional-node-network="network=${RDMA_NETWORK_PREFIX}-net,subnetwork=${RDMA_NETWORK_PREFIX}-sub-7"
  --project="${PROJECT_ID}"
  --async
  --format="value(name)"
)

if [ -n "${RESERVATION:-}" ]; then
  CMD+=("--reservation-affinity=specific" "--reservation=${RESERVATION}")
else
  CMD+=("--reservation-affinity=none")
fi

if ! OPERATION_ID="$("${CMD[@]}")" || [[ -z "${OPERATION_ID}" ]]; then
  echo "Error: Failed to initiate nodepool creation."
  exit 1
fi

# Wait for nodepool creation to complete
if [[ "${NODEPOOL_TIMEOUT:-YOUR_NODEPOOL_TIMEOUT}" == "YOUR_NODEPOOL_TIMEOUT" ]]; then
  NODEPOOL_TIMEOUT="3600"
  echo "Debug: NODEPOOL_TIMEOUT is not set to a custom value, using default: ${NODEPOOL_TIMEOUT}"
fi
echo "Waiting for nodepool ${OPERATION_ID} to be created..."
if ! timeout "${NODEPOOL_TIMEOUT}" gcloud container operations wait "${OPERATION_ID}" --location="${CONTROL_PLANE_REGION}" --project="${PROJECT_ID}"; then
  echo "Error: Nodepool creation timed out or failed after ${NODEPOOL_TIMEOUT} seconds."

  # Describe the operation to get more details
  gcloud container operations describe "${OPERATION_ID}" \
    --location="${CONTROL_PLANE_REGION}" \
    --project="${PROJECT_ID}"
  exit 1
fi
echo "Nodepool creation completed successfully."

Instal penginstal NCCL RDMA yang digunakan untuk cluster Standard:

kubectl apply -f https://raw.githubusercontent.com/GoogleCloudPlatform/container-engine-accelerators/refs/heads/master/gpudirect-rdma/nccl-rdma-installer.yaml

A4X

Buat cluster GKE dan node pool menggunakan cetak biru Cluster Toolkit gke-a4x. Blueprint ini menyediakan cluster GKE, termasuk kumpulan node A4X yang terikat ke reservasi Anda, jaringan akselerator (satu gVNIC tambahan plus empat jalur RDMA), dan driver DRANET terkelola yang mengekspos NIC CX-7 sebagai perangkat DRA.

Gunakan petunjuk deployment blueprint untuk mengonfigurasi parameter Anda (seperti PROJECT_ID, CONTROL_PLANE_REGION, NODE_ZONE, reservasi, dan NUM_GPU_NODES), lalu deploy cluster. Atau, Anda dapat mengikuti panduan pembuatan cluster GKE A4X untuk membuat cluster secara manual.
1. Dapatkan kredensial untuk cluster Anda:
```
gcloud container clusters get-credentials ${CLUSTER_NAME} --location=${CONTROL_PLANE_REGION}
```
Verifikasi bahwa cluster mengekspos NIC RDMA melalui DRA:
```
kubectl get deviceclasses
```
Output harus menyertakan mrdma.google.com.

Pastikan node A4X ada:

kubectl get nodes -l cloud.google.com/gke-accelerator=nvidia-gb200

Instal plugin gIB NCCL (varian A4X):

kubectl apply -f https://raw.githubusercontent.com/GoogleCloudPlatform/container-engine-accelerators/master/gpudirect-rdma/nccl-rdma-installer-a4x.yaml

Instal driver NVIDIA DRA, yang menyediakan saluran ComputeDomain (IMEX) untuk NVLink multi-node:

helm repo add nvidia https://helm.ngc.nvidia.com/nvidia && helm repo update
kubectl create namespace nvidia-dra-driver-gpu
kubectl apply -f - <<EOF
apiVersion: v1
kind: ResourceQuota
metadata:
  name: nvidia-dra-driver-gpu-quota
  namespace: nvidia-dra-driver-gpu
spec:
  hard:
    pods: "$((2 * NUM_GPU_NODES + 1))"
  scopeSelector:
    matchExpressions:
    - operator: In
      scopeName: PriorityClass
      values:
      - system-node-critical
      - system-cluster-critical
EOF
helm upgrade --install nvidia-dra-driver-gpu nvidia/nvidia-dra-driver-gpu \
  --version=25.3.1 --namespace nvidia-dra-driver-gpu \
  --set nvidiaDriverRoot=/home/kubernetes/bin/nvidia \
  --set resources.gpus.enabled=false \
  --set kubeletPlugin.tolerations[0].key=nvidia.com/gpu \
  --set kubeletPlugin.tolerations[0].operator=Exists \
  --set kubeletPlugin.tolerations[1].key=kubernetes.io/arch \
  --set kubeletPlugin.tolerations[1].operator=Exists

Instal operator KubeRay, yang dicakup ke namespace beban kerja:

kubectl create namespace ${NAMESPACE}
helm repo add kuberay https://ray-project.github.io/kuberay-helm/ && helm repo update
helm upgrade --install kuberay-operator kuberay/kuberay-operator \
  --namespace ${NAMESPACE} \
  --set singleNamespaceInstall=true --set "watchNamespace={${NAMESPACE}}"

Mengonfigurasi pemetaan jaringan (khusus GKE Standard - A4 dan A3 Ultra)

A4 dan A3 Ultra

Autopilot

Langkah ini diperlukan untuk penyiapan GPU GKE Standard (khusus A4 dan A3 Ultra). Jika Anda menggunakan A4X (GB200), GKE akan mengelola antarmuka jaringan secara otomatis, jadi lewati bagian ini.

Standar

Periksa manifes network-mapping.yaml:

apiVersion: networking.gke.io/v1
kind: GKENetworkParamSet
metadata:
  name: gvnic-1
spec:
  vpc: ${GVNIC_NETWORK_PREFIX}-net
  vpcSubnet: ${GVNIC_NETWORK_PREFIX}-sub
  deviceMode: NetDevice
---
apiVersion: networking.gke.io/v1
kind: Network
metadata:
  name: gvnic-1
spec:
  type: "Device"
  parametersRef:
    group: networking.gke.io
    kind: GKENetworkParamSet
    name: gvnic-1
---
apiVersion: networking.gke.io/v1
kind: GKENetworkParamSet
metadata:
  name: rdma-0
spec:
  vpc: ${RDMA_NETWORK_PREFIX}-net
  vpcSubnet: ${RDMA_NETWORK_PREFIX}-sub-0
  deviceMode: RDMA
---
apiVersion: networking.gke.io/v1
kind: Network
metadata:
  name: rdma-0
spec:
  type: "Device"
  parametersRef:
    group: networking.gke.io
    kind: GKENetworkParamSet
    name: rdma-0
---
apiVersion: networking.gke.io/v1
kind: GKENetworkParamSet
metadata:
  name: rdma-1
spec:
  vpc: ${RDMA_NETWORK_PREFIX}-net
  vpcSubnet: ${RDMA_NETWORK_PREFIX}-sub-1
  deviceMode: RDMA
---
apiVersion: networking.gke.io/v1
kind: Network
metadata:
  name: rdma-1
spec:
  type: "Device"
  parametersRef:
    group: networking.gke.io
    kind: GKENetworkParamSet
    name: rdma-1
---
apiVersion: networking.gke.io/v1
kind: GKENetworkParamSet
metadata:
  name: rdma-2
spec:
  vpc: ${RDMA_NETWORK_PREFIX}-net
  vpcSubnet: ${RDMA_NETWORK_PREFIX}-sub-2
  deviceMode: RDMA
---
apiVersion: networking.gke.io/v1
kind: Network
metadata:
  name: rdma-2
spec:
  type: "Device"
  parametersRef:
    group: networking.gke.io
    kind: GKENetworkParamSet
    name: rdma-2
---
apiVersion: networking.gke.io/v1
kind: GKENetworkParamSet
metadata:
  name: rdma-3
spec:
  vpc: ${RDMA_NETWORK_PREFIX}-net
  vpcSubnet: ${RDMA_NETWORK_PREFIX}-sub-3
  deviceMode: RDMA
---
apiVersion: networking.gke.io/v1
kind: Network
metadata:
  name: rdma-3
spec:
  type: "Device"
  parametersRef:
    group: networking.gke.io
    kind: GKENetworkParamSet
    name: rdma-3
---
apiVersion: networking.gke.io/v1
kind: GKENetworkParamSet
metadata:
  name: rdma-4
spec:
  vpc: ${RDMA_NETWORK_PREFIX}-net
  vpcSubnet: ${RDMA_NETWORK_PREFIX}-sub-4
  deviceMode: RDMA
---
apiVersion: networking.gke.io/v1
kind: Network
metadata:
  name: rdma-4
spec:
  type: "Device"
  parametersRef:
    group: networking.gke.io
    kind: GKENetworkParamSet
    name: rdma-4
---
apiVersion: networking.gke.io/v1
kind: GKENetworkParamSet
metadata:
  name: rdma-5
spec:
  vpc: ${RDMA_NETWORK_PREFIX}-net
  vpcSubnet: ${RDMA_NETWORK_PREFIX}-sub-5
  deviceMode: RDMA
---
apiVersion: networking.gke.io/v1
kind: Network
metadata:
  name: rdma-5
spec:
  type: "Device"
  parametersRef:
    group: networking.gke.io
    kind: GKENetworkParamSet
    name: rdma-5
---
apiVersion: networking.gke.io/v1
kind: GKENetworkParamSet
metadata:
  name: rdma-6
spec:
  vpc: ${RDMA_NETWORK_PREFIX}-net
  vpcSubnet: ${RDMA_NETWORK_PREFIX}-sub-6
  deviceMode: RDMA
---
apiVersion: networking.gke.io/v1
kind: Network
metadata:
  name: rdma-6
spec:
  type: "Device"
  parametersRef:
    group: networking.gke.io
    kind: GKENetworkParamSet
    name: rdma-6
---
apiVersion: networking.gke.io/v1
kind: GKENetworkParamSet
metadata:
  name: rdma-7
spec:
  vpc: ${RDMA_NETWORK_PREFIX}-net
  vpcSubnet: ${RDMA_NETWORK_PREFIX}-sub-7
  deviceMode: RDMA
---
apiVersion: networking.gke.io/v1
kind: Network
metadata:
  name: rdma-7
spec:
  type: "Device"
  parametersRef:
    group: networking.gke.io
    kind: GKENetworkParamSet
    name: rdma-7

Terapkan manifes:

envsubst < network-mapping.yaml | kubectl apply -f -

A4X

Langkah ini diperlukan untuk penyiapan GPU GKE Standard (khusus A4 dan A3 Ultra). Jika Anda menggunakan A4X (GB200), GKE akan mengelola antarmuka jaringan secara otomatis, jadi lewati bagian ini.

Menyiapkan data dan penyimpanan

Mengonfigurasi resource Cloud Storage dan Kubernetes:

Membuat bucket Cloud Storage:

gcloud storage buckets create "gs://${GS_BUCKET}" \
  --location="${CONTROL_PLANE_REGION}" \
  --project="${PROJECT_ID}" \
  --enable-hierarchical-namespace \
  --uniform-bucket-level-access

Buat Akun Layanan Kubernetes (KSA) dan ikat ke bucket:

kubectl create serviceaccount ${KSA_NAME} -n ${NAMESPACE}
gcloud storage buckets add-iam-policy-binding "gs://${GS_BUCKET}" \
  --member="principal://iam.googleapis.com/projects/${PROJECT_NUMBER}/locations/global/workloadIdentityPools/${PROJECT_ID}.svc.id.goog/subject/ns/${NAMESPACE}/sa/${KSA_NAME}" \
  --role="roles/storage.objectUser"

Buat Secret untuk Hugging Face:

kubectl create secret generic hf-secret --from-literal=hf_token=${HF_TOKEN}

Periksa manifes gcsfuse-storage.yaml:

apiVersion: v1
kind: PersistentVolume
metadata:
  name: training-bucket-pv
spec:
  accessModes:
  -   ReadWriteMany
  capacity:
    storage: 768Gi
  persistentVolumeReclaimPolicy: Delete
  storageClassName: gcsfuse-sc
  mountOptions:
  -   implicit-dirs
  -   metadata-cache:negative-ttl-secs:0
  -   metadata-cache:ttl-secs:0
  -   metadata-cache:stat-cache-max-size-mb:-1
  -   metadata-cache:type-cache-max-size-mb:-1
  -   file-cache:max-size-mb:-1
  -   file-cache:cache-file-for-range-read:true
  -   file-cache:enable-parallel-downloads:true
  -   read_ahead_kb=1024
  -   write:enable-streaming-writes:true
  -   write:global-max-blocks:200000
  csi:
    driver: gcsfuse.csi.storage.gke.io
    volumeHandle: ${GS_BUCKET}
    volumeAttributes:
      skipCSIBucketAccessCheck: "true"
      gcsfuseMetadataPrefetchOnMount: "true"
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: training-bucket-pvc
spec:
  accessModes:
  -   ReadWriteMany
  resources:
    requests:
      storage: 768Gi
  storageClassName: gcsfuse-sc

Terapkan manifes:

envsubst < gcsfuse-storage.yaml | kubectl apply -f -

Menyiapkan DRANET

Konfigurasi DRANET Anda:

A4 dan A3 Ultra

Autopilot

Buat manifes ComputeClass:

echo "Generating computeclass-dranet.yaml..."
cat <<EOF > computeclass-dranet.yaml
apiVersion: cloud.google.com/v1
kind: ComputeClass
metadata:
  name: dranet-a4-computeclass-v3
spec:
  nodePoolAutoCreation:
    enabled: true
  nodePoolConfig:
    dra:
      networking:
        enabled: true
  priorities:
  - machineType: ${MACHINE_TYPE}
    gpu:
      count: 8
      type: ${GPU_TYPE}
    acceleratorNetworkProfile: auto
EOF

if [ -n "${RESERVATION:-}" ]; then
  echo "Adding reservation affinity for ${RESERVATION} to ComputeClass..."
  cat <<EOF >> computeclass-dranet.yaml
    reservations:
      affinity: Specific
      specific:
      - name: ${RESERVATION}
        project: ${PROJECT_ID}
EOF
fi

Terapkan manifes computeclass-dranet.yaml (dibuat di langkah sebelumnya) dan manifes resourceclaim-dranet.yaml (termasuk dalam repositori contoh):

echo "Applying ComputeClass..."
kubectl apply -f computeclass-dranet.yaml

echo "Applying ResourceClaimTemplate..."
kubectl apply -f resourceclaim-dranet.yaml

Standar

Tidak diperlukan penyiapan DRANET. Anda dapat langsung melanjutkan ke bagian berikutnya.

A4X

DRANET ditetapkan oleh Cluster Toolkit. Anda dapat langsung melanjutkan ke bagian berikutnya.

Menyiapkan model dan data

Isi bucket Cloud Storage Anda dengan bobot dan set data model. Anda dapat menjalankan perintah ini secara lokal atau di Pod GKE untuk mengisi bucket:

A4 dan A3 Ultra

Autopilot

Periksa manifes data-prep-job.yaml, yang mengonfigurasi tugas penyiapan data:

apiVersion: batch/v1
kind: Job
metadata:
  name: data-prep-job
  namespace: ${NAMESPACE}
spec:
  template:
    metadata:
      annotations:
        gke-gcsfuse/volumes: "true"
        gke-gcsfuse/cpu-limit: "4"
        gke-gcsfuse/memory-limit: "8Gi"
        gke-gcsfuse/ephemeral-storage-limit: "50Gi"
    spec:
      serviceAccountName: ${KSA_NAME}
      restartPolicy: OnFailure
      nodeSelector:
        cloud.google.com/compute-class: Performance
        cloud.google.com/machine-family: c3
      containers:
      - name: prep-data
        image: verlai/verl:vllm011.latest
        resources:
          requests:
            cpu: "8"
            memory: "16Gi"
            ephemeral-storage: "50Gi"
          limits:
            cpu: "8"
            memory: "16Gi"
            ephemeral-storage: "50Gi"
        env:
        - name: HF_TOKEN
          valueFrom:
            secretKeyRef:
              name: hf-secret
              key: hf_token
        - name: HF_HOME
          value: /data/.cache/huggingface
        - name: HF_HUB_DISABLE_XET
          value: "1"
        command: ["/bin/bash", "-c"]
        args:
        - |
          set -euo pipefail

          # Clone verl to GCS (for worker pods)
          if [ ! -d "/data/verl" ]; then
            echo "Cloning verl to GCS..."
            git clone --depth 1 --branch v0.6.1 https://github.com/volcengine/verl.git /data/verl

            echo "Installing verl on GCS to generate metadata..."
            pip3 install --no-deps -e /data/verl
          else
            echo "verl already exists in /data/verl"
          fi

          # Clone verl locally for fast installation
          echo "Cloning verl locally..."
          git clone --depth 1 --branch v0.6.1 https://github.com/volcengine/verl.git /tmp/verl

          # Install verl package from local clone
          echo "Installing verl package..."
          pip3 install --no-cache-dir --no-deps /tmp/verl
          rm -rf /tmp/verl

          # Preprocess GSM8K
          if [ ! -d "/data/gsm8k" ]; then
            echo "Preprocessing GSM8K..."
            python /data/verl/examples/data_preprocess/gsm8k.py --local_save_dir /data/gsm8k
          else
            echo "GSM8K data already exists in /data/gsm8k"
          fi

          # Download model sequentially to support GCS FUSE direct streaming writes
          if [ ! -d "/data/Qwen2.5-32B-Instruct" ]; then
            echo "Downloading Qwen2.5-32B-Instruct..."
            huggingface-cli download Qwen/Qwen2.5-32B-Instruct --local-dir /data/Qwen2.5-32B-Instruct --local-dir-use-symlinks False
          else
            echo "Model Qwen2.5-32B-Instruct already exists in /data/Qwen2.5-32B-Instruct"
          fi

          echo "Data preparation complete!"
        volumeMounts:
        - name: training-bucket-vol
          mountPath: /data
      volumes:
      - name: training-bucket-vol
        persistentVolumeClaim:
          claimName: training-bucket-pvc

Catatan: Di Autopilot, menentukan cloud.google.com/compute-class: Performance tanpa seri mesin akan menggunakan seri mesin C4 secara default. Jika Anda tidak memiliki kuota C4 di region Anda, Pod mungkin tetap dalam status Pending. Manifes ini menentukan cloud.google.com/machine-family: c3. Anda dapat mengubah nilai ini di data-prep-job.yaml ke seri mesin yang didukung (seperti c2 atau c4) berdasarkan ketersediaan regional dan kuota project Anda.

Luncurkan tugas:

envsubst < "data-prep-job.yaml" | kubectl apply -f -

Periksa status Pod:
```
watch kubectl get pods -n ${NAMESPACE}
```
Setelah Pod data-prep-job mencapai status Running, tekan Control+C untuk berhenti melihat.
Pantau tugas:
```
kubectl logs -n ${NAMESPACE} -l job-name=data-prep-job -f
```
Perintah kubectl logs menampilkan output minimal atau tidak ada output hingga Pod mencapai status Running.

Standar

Periksa manifes data-prep-job.yaml, yang mengonfigurasi Tugas penyiapan data:

apiVersion: batch/v1
kind: Job
metadata:
  name: data-prep-job
  namespace: ${NAMESPACE}
spec:
  template:
    metadata:
      annotations:
        gke-gcsfuse/volumes: "true"
        gke-gcsfuse/cpu-limit: "2"
        gke-gcsfuse/memory-limit: "4Gi"
        gke-gcsfuse/ephemeral-storage-limit: "20Gi"
    spec:
      serviceAccountName: ${KSA_NAME}
      restartPolicy: OnFailure
      nodeSelector:
        cloud.google.com/gke-nodepool: "default-pool"
      containers:
      - name: prep-data
        image: verlai/verl:vllm011.latest
        resources:
          requests:
            cpu: "4"
            memory: "8Gi"
            ephemeral-storage: "10Gi"
          limits:
            cpu: "4"
            memory: "8Gi"
            ephemeral-storage: "10Gi"
        env:
        - name: HF_TOKEN
          valueFrom:
            secretKeyRef:
              name: hf-secret
              key: hf_token
        - name: HF_HOME
          value: /data/.cache/huggingface
        - name: HF_HUB_DISABLE_XET
          value: "1"
        command: ["/bin/bash", "-c"]
        args:
        - |
          set -euo pipefail

          # Clone verl to GCS (for worker pods)
          if [ ! -d "/data/verl" ]; then
            echo "Cloning verl to GCS..."
            git clone --branch v0.6.1 https://github.com/volcengine/verl.git /data/verl
          else
            echo "verl already exists in /data/verl"
          fi

          # Clone verl locally for fast installation
          echo "Cloning verl locally..."
          git clone --branch v0.6.1 https://github.com/volcengine/verl.git /tmp/verl

          # Install verl package from local clone
          echo "Installing verl package..."
          pip3 install --no-cache-dir --no-deps /tmp/verl
          rm -rf /tmp/verl

          # Preprocess GSM8K
          if [ ! -d "/data/gsm8k" ]; then
            echo "Preprocessing GSM8K..."
            python /data/verl/examples/data_preprocess/gsm8k.py --local_save_dir /data/gsm8k
          else
            echo "GSM8K data already exists in /data/gsm8k"
          fi

          # Download model
          if [ ! -d "/data/Qwen2.5-32B-Instruct" ]; then
            echo "Downloading Qwen2.5-32B-Instruct..."
            huggingface-cli download Qwen/Qwen2.5-32B-Instruct --local-dir /data/Qwen2.5-32B-Instruct --local-dir-use-symlinks False
          else
            echo "Model Qwen2.5-32B-Instruct already exists in /data/Qwen2.5-32B-Instruct"
          fi

          echo "Data preparation complete!"
        volumeMounts:
        - name: training-bucket-vol
          mountPath: /data
      volumes:
      - name: training-bucket-vol
        persistentVolumeClaim:
          claimName: training-bucket-pvc

Luncurkan Tugas:

envsubst < data-prep-job.yaml | kubectl apply -f -

Memantau Tugas:

kubectl logs -n ${NAMESPACE} -l job-name=data-prep-job -f

A4X

Clone repositori verl, siapkan lingkungan virtual, dan proses set data GSM8K:

git clone https://github.com/volcengine/verl.git
git -C verl checkout ${VERL_REF}

VENV_DIR=.venv
python3 -m venv $VENV_DIR
source $VENV_DIR/bin/activate
pip install verl

python verl/examples/data_preprocess/gsm8k.py --local_save_dir ~/data/gsm8k

Download model Qwen2.5-32B-Instruct menggunakan Hugging Face CLI (download ini memerlukan ruang disk sekitar 66 GB):
```
hf download Qwen/Qwen2.5-32B-Instruct --local-dir Qwen2.5-32B-Instruct
```

Upload model, data, dan kode verifikasi ke bucket Cloud Storage Anda:

gcloud storage cp --recursive verl gs://${GS_BUCKET}/verl
gcloud storage cp --recursive Qwen2.5-32B-Instruct gs://${GS_BUCKET}/Qwen2.5-32B-Instruct
gcloud storage cp --recursive ~/data/gsm8k/* gs://${GS_BUCKET}/gsm8k/

Men-deploy resource kustom RayCluster

Deploy resource kustom RayCluster, yang terdiri dari satu Pod head sistem dan beberapa Pod worker yang didukung GPU.

A4 dan A3 Ultra

Pilih mode cluster GKE yang Anda gunakan untuk membuat cluster:

Autopilot

Periksa workload RayCluster:

apiVersion: ray.io/v1
kind: RayCluster
metadata:
  name: b200-ray-cluster-dranet
spec:
  rayVersion: '2.47.0'
  headGroupSpec:
    rayStartParams:
      dashboard-host: '0.0.0.0'
    template:
      metadata:
        annotations:
          gke-gcsfuse/volumes: "true"
      spec:
        serviceAccountName: ${KSA_NAME}
        nodeSelector:
          cloud.google.com/gke-spot: "true"
          cloud.google.com/machine-family: "c2"
          cloud.google.com/compute-class: Performance
        containers:
        - name: ray-head
          image: verlai/verl:vllm011.latest 
          ports:
            - containerPort: 6379
              name: gcs-server
            - containerPort: 8265
              name: dashboard
            - containerPort: 10001
              name: client
          resources:
            limits:
              cpu: "12"
              memory: "32G"
              ephemeral-storage: "9Gi"
            requests:
              cpu: "12"
              memory: "32G"
              ephemeral-storage: "9Gi"
          volumeMounts:
            - mountPath: /tmp/ray
              name: ray-logs
            - name: training-bucket-vol
              mountPath: /data
        volumes:
          - name: ray-logs
            emptyDir: {}
          - name: training-bucket-vol
            persistentVolumeClaim:
              claimName: training-bucket-pvc
  workerGroupSpecs:
  - replicas: 2
    minReplicas: 2
    maxReplicas: 2
    groupName: gpu-group
    rayStartParams:
      num-cpus: "220"
    template:
      metadata:
        annotations:
          gke-gcsfuse/volumes: "true"
      spec:
        resourceClaims:
          - name: rdma-claim
            resourceClaimTemplateName: all-mrdma
        initContainers:
        - name: verl-setup
          image: verlai/verl:vllm011.latest
          command: ["/bin/bash", "-c"]
          args:
            - |
              echo "Installing GIB plugins..."
              curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | gpg --dearmor -o /etc/apt/trusted.gpg.d/cloud.google.gpg
              echo 'deb https://packages.cloud.google.com/apt gpudirect-gib-apt main' | tee /etc/apt/sources.list.d/nccl-gib.list
              apt-get update && apt-get install -y nccl-gib-plugins

              echo "Local install skipped (handled by data prep)"
          volumeMounts:
          - name: training-bucket-vol
            mountPath: /data
          - name: gib
            mountPath: /usr/local/gib
        serviceAccountName: ${KSA_NAME}
        nodeSelector:
          cloud.google.com/compute-class: dranet-a4-computeclass-v3
        tolerations:
          - key: "nvidia.com/gpu"
            operator: "Exists"
            effect: "NoSchedule"
        containers:
        - name: ray-worker
          image: verlai/verl:vllm011.latest
          env:
           - name: LD_LIBRARY_PATH
             value: /usr/local/nvidia/lib64:/usr/local/gib/lib64
          resources:
            limits:
              cpu: "180"
              memory: "2000Gi"
              nvidia.com/gpu: "8"
              ephemeral-storage: "1000Gi"
            requests:
              cpu: "180"
              memory: "2000Gi"
              nvidia.com/gpu: "8"
              ephemeral-storage: "1000Gi"
            claims:
            - name: rdma-claim
          volumeMounts:
          - name: gib
            mountPath: /usr/local/gib
            readOnly: true
          - name: shared-memory
            mountPath: /dev/shm
          - name: ray-tmp-storage
            mountPath: /tmp
          - name: training-bucket-vol
            mountPath: /data
        volumes:
        - name: gib
          emptyDir: {}
        - name: shared-memory
          emptyDir:
            medium: "Memory"
            sizeLimit: 250Gi 
        - name: ray-tmp-storage
          emptyDir: {}
        - name: training-bucket-vol
          persistentVolumeClaim:
            claimName: training-bucket-pvc

Terapkan RayCluster:

envsubst < "ray-cluster-auto-dranet.yaml" | kubectl apply -n "${NAMESPACE}" -f -

Standar

Periksa workload RayCluster:

apiVersion: ray.io/v1
kind: RayCluster
metadata:
  name: b200-ray-cluster
  annotations:
spec:
  rayVersion: '2.47.0'
  headGroupSpec:
    rayStartParams:
      dashboard-host: '0.0.0.0'
    template:
      metadata:
        annotations:
          gke-gcsfuse/volumes: "true"
      spec:
        serviceAccountName: ${KSA_NAME}
        nodeSelector:
          cloud.google.com/gke-nodepool: "default-pool"
        containers:
        - name: ray-head
          image: verlai/verl:vllm011.latest 
          ports:
            - containerPort: 6379
              name: gcs-server
            - containerPort: 8265
              name: dashboard
            - containerPort: 10001
              name: client
          resources:
            limits:
              cpu: "12"
              memory: "32G"
              ephemeral-storage: "9Gi"
            requests:
              cpu: "12"
              memory: "32G"
              ephemeral-storage: "9Gi"
          volumeMounts:
            - mountPath: /tmp/ray
              name: ray-logs
            - name: training-bucket-vol
              mountPath: /data
        volumes:
          - name: ray-logs
            emptyDir: {}
          - name: training-bucket-vol
            persistentVolumeClaim:
              claimName: training-bucket-pvc
  workerGroupSpecs:
  - replicas: 2
    minReplicas: 2
    maxReplicas: 2
    groupName: gpu-group
    rayStartParams:
      num-cpus: "220"
    template:
      metadata:
        annotations:
          gke-gcsfuse/volumes: "true"
          networking.gke.io/default-interface: 'eth0'
          networking.gke.io/interfaces: |
            [
              {"interfaceName":"eth0","network":"default"},
              {"interfaceName":"eth1","network":"gvnic-1"},
              {"interfaceName":"eth2","network":"rdma-0"},
              {"interfaceName":"eth3","network":"rdma-1"},
              {"interfaceName":"eth4","network":"rdma-2"},
              {"interfaceName":"eth5","network":"rdma-3"},
              {"interfaceName":"eth6","network":"rdma-4"},
              {"interfaceName":"eth7","network":"rdma-5"},
              {"interfaceName":"eth8","network":"rdma-6"},
              {"interfaceName":"eth9","network":"rdma-7"}
            ]
      spec:
        initContainers:
        - name: verl-setup
          image: verlai/verl:vllm011.latest
          command: ["/bin/bash", "-c"]
          args:
            - |
              echo "Performing local editable install..."
              cd /data/verl && pip3 install --no-deps -e .
          volumeMounts:
          - name: training-bucket-vol
            mountPath: /data
        serviceAccountName: ${KSA_NAME}
        nodeSelector:
          cloud.google.com/gke-accelerator: ${GPU_TYPE}
        tolerations:
          - key: "nvidia.com/gpu"
            operator: "Exists"
            effect: "NoSchedule"
        containers:
        - name: ray-worker
          image: verlai/verl:vllm011.latest
          env:
           - name: LD_LIBRARY_PATH
             value: /usr/local/nvidia/lib64
          resources:
            limits:
              cpu: "220"
              memory: "2800Gi"
              nvidia.com/gpu: "8"
              ephemeral-storage: "1000Gi"
            requests:
              cpu: "220"
              memory: "2800Gi"
              nvidia.com/gpu: "8"
              ephemeral-storage: "1000Gi"
          volumeMounts:
          - name: nvidia
            mountPath: /usr/local/nvidia
          - name: gib
            mountPath: /usr/local/gib
          - name: shared-memory
            mountPath: /dev/shm
          - name: ray-tmp-storage
            mountPath: /tmp
          - name: training-bucket-vol
            mountPath: /data
        volumes:
        - name: gib
          hostPath:
            path: /home/kubernetes/bin/gib
        - name: nvidia
          hostPath:
            path: /home/kubernetes/bin/nvidia
        - name: lib64
          hostPath:
            path: /lib64
        - name: shared-memory
          emptyDir:
            medium: "Memory"
            sizeLimit: 250Gi 
        - name: sys
          hostPath:
            path: /sys
        - name: proc-sys
          hostPath:
            path: /proc/sys
        - name: ray-tmp-storage
          emptyDir: {}
        - name: training-bucket-vol
          persistentVolumeClaim:
            claimName: training-bucket-pvc

Terapkan RayCluster:

envsubst < "ray-cluster-standard.yaml" | kubectl apply -f -

A4X

Buat RDMA ResourceClaimTemplate dan NVIDIA ComputeDomain. Setiap Pod pekerja GPU mengklaim empat NIC RDMA (semua jalur node-nya) dan satu channel IMEX. Simpan manifes berikut ke compute-domain-a4x.yaml:

apiVersion: resource.k8s.io/v1
kind: ResourceClaimTemplate
metadata:
  name: verl-rdma-nic
  namespace: ${NAMESPACE}
spec:
  spec:
    devices:
      requests:
      - name: nic
        exactly:
          deviceClassName: mrdma.google.com
          allocationMode: ExactCount
          count: 1
---
apiVersion: resource.nvidia.com/v1beta1
kind: ComputeDomain
metadata:
  name: verl-compute-domain
  namespace: ${NAMESPACE}
spec:
  numNodes: ${NUM_GPU_NODES}
  channel:
    resourceClaimTemplate:
      name: verl-compute-domain-channel

Terapkan manifes:

kubectl apply -f compute-domain-a4x.yaml

Deploy RayCluster. Pod head Ray berjalan di node A4X tanpa meminta GPU (karena image hanya arm64). Simpan konfigurasi berikut ke ray-cluster-a4x.yaml:

apiVersion: ray.io/v1
kind: RayCluster
metadata:
  name: gb200-ray-cluster
  namespace: ${NAMESPACE}
spec:
  rayVersion: '2.49.0'
  headGroupSpec:
    rayStartParams:
      dashboard-host: '0.0.0.0'
      num-cpus: "0"
    template:
      metadata:
        annotations:
          gke-gcsfuse/volumes: "true"
      spec:
        serviceAccountName: ${KSA_NAME}
        nodeSelector:
          cloud.google.com/gke-accelerator: nvidia-gb200
        tolerations:
        - key: nvidia.com/gpu
          operator: Exists
          effect: NoSchedule
        - key: kubernetes.io/arch
          operator: Exists
          effect: NoSchedule
        containers:
        - name: ray-head
          image: ${VERL_IMAGE}
          lifecycle:
            postStart:
              exec:
                command:
                - /bin/bash
                - -c
                - pip3 install --quiet TransferQueue==0.1.8
          ports:
          - containerPort: 6379
            name: gcs-server
          - containerPort: 8265
            name: dashboard
          - containerPort: 10001
            name: client
          resources:
            limits:
              cpu: "12"
              memory: 32Gi
              ephemeral-storage: 20Gi
            requests:
              cpu: "12"
              memory: 32Gi
              ephemeral-storage: 20Gi
          volumeMounts:
          - mountPath: /tmp/ray
            name: ray-logs
          - name: training-bucket-vol
            mountPath: /data
        volumes:
        - name: ray-logs
          emptyDir: {}
        - name: training-bucket-vol
          persistentVolumeClaim:
            claimName: training-bucket-pvc
  workerGroupSpecs:
  - replicas: ${NUM_GPU_NODES}
    minReplicas: ${NUM_GPU_NODES}
    maxReplicas: ${NUM_GPU_NODES}
    groupName: gpu-group
    rayStartParams:
      num-cpus: "120"
    template:
      metadata:
        annotations:
          gke-gcsfuse/volumes: "true"
      spec:
        serviceAccountName: ${KSA_NAME}
        nodeSelector:
          cloud.google.com/gke-accelerator: nvidia-gb200
        affinity:
          podAntiAffinity:
            requiredDuringSchedulingIgnoredDuringExecution:
            - labelSelector:
                matchLabels:
                  ray.io/group: gpu-group
              topologyKey: kubernetes.io/hostname
        tolerations:
        - key: nvidia.com/gpu
          operator: Exists
          effect: NoSchedule
        - key: kubernetes.io/arch
          operator: Exists
          effect: NoSchedule
        containers:
        - name: ray-worker
          image: ${VERL_IMAGE}
          lifecycle:
            postStart:
              exec:
                command:
                - /bin/bash
                - -c
                - pip3 install --quiet TransferQueue==0.1.8
          env:
          - name: LD_LIBRARY_PATH
            value: /usr/local/nvidia/lib64
          resources:
            limits:
              cpu: "120"
              memory: 600Gi
              nvidia.com/gpu: "4"
              ephemeral-storage: 500Gi
            requests:
              cpu: "120"
              memory: 600Gi
              nvidia.com/gpu: "4"
              ephemeral-storage: 500Gi
            claims:
            - name: rdma-nic-0
            - name: rdma-nic-1
            - name: rdma-nic-2
            - name: rdma-nic-3
            - name: compute-domain-channel
          volumeMounts:
          - name: nvidia
            mountPath: /usr/local/nvidia
          - name: gib
            mountPath: /usr/local/gib
          - name: shared-memory
            mountPath: /dev/shm
          - name: ray-tmp-storage
            mountPath: /tmp
          - name: training-bucket-vol
            mountPath: /data
        resourceClaims:
        - name: rdma-nic-0
          resourceClaimTemplateName: verl-rdma-nic
        - name: rdma-nic-1
          resourceClaimTemplateName: verl-rdma-nic
        - name: rdma-nic-2
          resourceClaimTemplateName: verl-rdma-nic
        - name: rdma-nic-3
          resourceClaimTemplateName: verl-rdma-nic
        - name: compute-domain-channel
          resourceClaimTemplateName: verl-compute-domain-channel
        volumes:
        - name: gib
          hostPath:
            path: /home/kubernetes/bin/gib
        - name: nvidia
          hostPath:
            path: /home/kubernetes/bin/nvidia
        - name: shared-memory
          emptyDir:
            medium: Memory
            sizeLimit: 200Gi
        - name: ray-tmp-storage
          emptyDir: {}
        - name: training-bucket-vol
          persistentVolumeClaim:
            claimName: training-bucket-pvc

Terapkan manifes RayCluster:

envsubst < ray-cluster-a4x.yaml | kubectl apply -f -

Tunggu hingga satu head Pod dan empat worker Pod berada dalam status Running:
```
kubectl get pods -w
```

Luncurkan Tugas GRPO

Konfigurasi dan kirim tugas pelatihan reinforcement learning:

A4 dan A3 Ultra

Pilih mode cluster GKE yang Anda gunakan untuk membuat cluster:

Autopilot

Siapkan Klien Ray:

if [ ! -d "env" ]; then
  python3 -m venv env
else
  echo "Found virtual environment env, not recreating"
fi

source env/bin/activate
pip3 install --upgrade pip
pip3 install ray[default]

Memulihkan Layanan Head Ray:

SVC_NAME="$(kubectl get svc -l "ray.io/node-type=head" -o jsonpath='{..metadata.name}')"
echo "Ray head service name: ${SVC_NAME}"

Siapkan penerusan port ke node dasbor Ray. Proses berjalan di latar belakang, sehingga Anda dapat terus menjalankan perintah di sesi terminal yang sama.

echo "Starting auto-restarting port-forwarding to ${SVC_NAME} on port 8265..."
run_port_forward() {
    while true; do
        kubectl port-forward svc/"${SVC_NAME}" 8265:8265 -n "${NAMESPACE}" >/dev/null 2>&1 || true
        sleep 1
    done
}
run_port_forward &
PF_LOOP_PID=$!

Untuk menghentikan loop penerusan port, gunakan perintah kill ${PF_LOOP_PID}.

Periksa manifes runtime-env.yaml:

py_modules: ["."]
working_dir: "."
excludes:
  - ".git"
  - "env"
  - "venv"
  - ".venv"
  - "__pycache__"
  - "*.pyc"
  - ".cache"
  - "*.log"
  - "*.tmp"
  - "build"
  - "dist"
env_vars:
  PYTHONPATH: "/data/verl"
  LD_LIBRARY_PATH: "/usr/local/nvidia/lib64:/usr/local/gib/lib64"
  NCCL_DEBUG: "INFO"
  NCCL_ENV_PLUGIN: "gcp"
  NCCL_CONF_FILE: "/usr/local/gib/configs/nccl.a4.conf"
  NCCL_IB_GID_INDEX: "3"
  NCCL_NVLS_ENABLE: "0"
  NCCL_TUNER_CONFIG_PATH: "/usr/local/gib/configs"
  NUM_WORKERS: "2"
  CPUS_PER_WORKER: "192"
  GPUS_PER_WORKER: "8"
  HF_HOME: "/data/huggingface_cache"
  GLOO_SOCKET_IFNAME: "eth0"

Jika Anda menggunakan GPU H200, ubah nilai variabel lingkungan NCCL_CONF_FILE menjadi /usr/local/gib/configs/nccl.a3u.conf.

File ini digunakan oleh klien Ray. Anda tidak perlu menerapkan manifes ini ke cluster.

Kirimkan tugas menggunakan perintah ray job submit:

SUBMIT_OUT=$(ray job submit \
  --address "http://localhost:8265" \
  --no-wait \
  --runtime-env runtime-env.yaml \
  -- \
  bash -c "
      cd /data/verl && PYTHONUNBUFFERED=1 python3 -m verl.trainer.main_ppo \
      data.train_files=/data/gsm8k/train.parquet \
      data.val_files=/data/gsm8k/test.parquet \
      data.train_batch_size=256 \
      data.max_prompt_length=512 \
      data.max_response_length=512 \
      actor_rollout_ref.model.path=/data/Qwen2.5-32B-Instruct \
      actor_rollout_ref.actor.optim.lr=1e-5 \
      actor_rollout_ref.actor.ppo_mini_batch_size=256 \
      actor_rollout_ref.actor.ppo_micro_batch_size_per_gpu=16 \
      actor_rollout_ref.rollout.name=vllm \
      actor_rollout_ref.rollout.log_prob_micro_batch_size_per_gpu=8 \
      actor_rollout_ref.rollout.tensor_model_parallel_size=8 \
      actor_rollout_ref.rollout.gpu_memory_utilization=0.6 \
      actor_rollout_ref.ref.log_prob_micro_batch_size_per_gpu=4 \
      actor_rollout_ref.actor.strategy=fsdp2 \
      algorithm.kl_ctrl.kl_coef=0.001 \
      trainer.logger=console \
      trainer.val_before_train=False \
      trainer.n_gpus_per_node=8 \
      trainer.nnodes=2 \
      trainer.save_freq=10 \
      trainer.test_freq=10 \
      trainer.total_training_steps=10 \
      trainer.default_local_dir=/data/verl/checkpoints \
      algorithm.adv_estimator=grpo \
      actor_rollout_ref.rollout.n=8 \
      actor_rollout_ref.actor.use_torch_compile=False \
      actor_rollout_ref.actor.fsdp_config.use_torch_compile=False \
      actor_rollout_ref.ref.use_torch_compile=False \
      actor_rollout_ref.ref.fsdp_config.use_torch_compile=False \
      trainer.total_epochs=1")
echo "${SUBMIT_OUT}"
JOB_ID=$(echo "${SUBMIT_OUT}" | grep "submitted successfully" | awk -F"'" '{print $2}' || true)

Pantau log tugas pelatihan dan cari critic/score/mean metrik yang meningkat, yang menunjukkan bahwa model sedang belajar:
- Dasbor Ray:
  1. Buka http://localhost:8265 di browser Anda. Jika Anda menggunakan Cloud Shell, klik Web Preview, pilih Change port, lalu masukkan 8265.
  2. Buka tab Tugas.
  3. Klik ID pekerjaan Anda.
  4. Filter log menurut kata kunci critic untuk memeriksa critic/score/mean.
- Terminal: Karena tugas dikirimkan secara asinkron dengan flag --no-wait, streaming log tugas di terminal Anda dengan menjalankan perintah berikut:
```
ray job logs "${JOB_ID}" --address "http://localhost:8265" --follow
```
Setelah pelatihan selesai, verifikasi bahwa checkpoint model telah disimpan ke bucket Cloud Storage Anda dengan mencantumkan konten direktori:
```
gcloud storage ls "gs://${GS_BUCKET}/verl/checkpoints/"
```
Atau, lihat titik pemeriksaan di konsol Google Cloud dengan membuka Cloud Storage > Buckets, memilih YOUR_GCS_BUCKET, lalu membuka folder verl/checkpoints.

Standar

Siapkan Klien Ray:

if [ ! -d "env" ]; then
  echo "Creating local virtual environment for Ray client..."
  python3 -m venv env
else
  echo "Found virtual environment env, not recreating"
fi
source env/bin/activate
pip3 install --upgrade pip
pip3 install ray[default]

Memulihkan Layanan Head Ray:

SVC_NAME="$(kubectl get svc -l "ray.io/node-type=head" -o jsonpath='{..metadata.name}')"
echo "Ray head service name: ${SVC_NAME}"

Siapkan penerusan port ke node dasbor Ray. Proses berjalan di latar belakang, sehingga Anda dapat terus menjalankan perintah di sesi terminal yang sama.
```
echo "Starting port-forwarding to ${SVC_NAME} on port 8265..."
kubectl port-forward svc/"${SVC_NAME}" 8265:8265 -n "${NAMESPACE}" &
PF_PID=$!
```
Untuk menghentikan loop penerusan port, gunakan perintah kill ${PF_PID}.

Periksa manifes runtime-env.yaml:

working_dir: "."
excludes:
  - ".git"
  - "env"
  - ".cache"
  - ".zip"
  - ".log"
env_vars:
  PYTHONPATH: "/data/verl"
  LD_LIBRARY_PATH: "/usr/local/nvidia/lib64"
  NCCL_DEBUG: "INFO"
  NUM_WORKERS: "2"
  CPUS_PER_WORKER: "192"
  GPUS_PER_WORKER: "8"
  NCCL_NET_PLUGIN: "/usr/local/gib/lib64/libnccl-net_internal.so"
  NCCL_CROSS_NIC: "0"
  NCCL_NET_GDR_LEVEL: "PIX"
  NCCL_P2P_NET_CHUNKSIZE: "131072"
  NCCL_NVLS_CHUNKSIZE: "524288"
  NCCL_IB_ADAPTIVE_ROUTING: "1"
  NCCL_IB_QPS_PER_CONNECTION: "4"
  NCCL_IB_TC: "52"
  NCCL_IB_FIFO_TC: "84"
  NCCL_TUNER_CONFIG_PATH: "/usr/local/gib/configs/tuner_config_a4.txtpb"
  HF_HOME: "/data/huggingface_cache"
  GLOO_SOCKET_IFNAME: "eth0"

Jika Anda menggunakan GPU H200, ubah nilai variabel lingkungan NCCL_TUNER_CONFIG_PATH menjadi /usr/local/gib/configs/tuner_config_a3u.txtpb.

File ini digunakan oleh klien Ray. Anda tidak perlu menerapkan manifes ini ke cluster.

Kirim tugas:

ray job submit \
--address "http://localhost:8265" \
--runtime-env runtime-env-local.yaml \
-- \
bash -c "
    cd /data/verl && PYTHONUNBUFFERED=1 python3 -m verl.trainer.main_ppo \
    data.train_files=/data/gsm8k/train.parquet \
    data.val_files=/data/gsm8k/test.parquet \
    data.train_batch_size=256 \
    data.max_prompt_length=512 \
    data.max_response_length=512 \
    actor_rollout_ref.model.path=/data/Qwen2.5-32B-Instruct \
    actor_rollout_ref.actor.optim.lr=1e-5 \
    actor_rollout_ref.actor.ppo_mini_batch_size=256 \
    actor_rollout_ref.actor.ppo_micro_batch_size_per_gpu=8 \
    actor_rollout_ref.rollout.name=vllm \
    actor_rollout_ref.rollout.log_prob_micro_batch_size_per_gpu=2 \
    actor_rollout_ref.rollout.tensor_model_parallel_size=8 \
    actor_rollout_ref.rollout.gpu_memory_utilization=0.5 \
    actor_rollout_ref.ref.log_prob_micro_batch_size_per_gpu=2 \
    actor_rollout_ref.actor.strategy=fsdp2 \
    algorithm.kl_ctrl.kl_coef=0.001 \
    trainer.logger=console \
    trainer.val_before_train=False \
    trainer.n_gpus_per_node=8 \
    trainer.nnodes=2 \
    trainer.save_freq=10 \
    trainer.test_freq=10 \
    trainer.default_local_dir=/data/verl/checkpoints \
    algorithm.adv_estimator=grpo \
    actor_rollout_ref.rollout.n=8 \
    trainer.total_epochs=2"

Pantau log tugas pelatihan dan cari critic/score/mean metrik yang meningkat, yang menunjukkan bahwa model sedang belajar:
- Dasbor Ray:
  1. Buka http://localhost:8265 di browser Anda. Jika Anda menggunakan Cloud Shell, klik Web Preview, pilih Change port, lalu masukkan 8265.
  2. Buka tab Tugas.
  3. Klik ID pekerjaan Anda.
  4. Filter log menurut kata kunci critic untuk memeriksa metrik critic/score/mean.
- Output konsol: Karena tugas dikirimkan secara asinkron dengan tanda --no-wait, streaming log tugas di terminal dengan menjalankan perintah berikut:
```
ray job logs "${JOB_ID}" --address "http://localhost:8265" --follow
```
Setelah pelatihan selesai, verifikasi bahwa checkpoint model telah disimpan ke bucket Cloud Storage Anda dengan mencantumkan konten direktori:
```
gcloud storage ls "gs://${GS_BUCKET}/verl/checkpoints/"
```
Atau, lihat titik pemeriksaan di konsol Google Cloud dengan membuka Cloud Storage > Buckets, memilih YOUR_GCS_BUCKET, lalu membuka folder verl/checkpoints.

A4X

Dapatkan nama Pod head Ray:

export HEAD_POD=$(kubectl get pod -n ${NAMESPACE} -l ray.io/node-type=head -o jsonpath='{.items[0].metadata.name}')

Konfigurasi file lingkungan runtime Ray langsung di Pod head:

kubectl exec ${HEAD_POD} -c ray-head -- bash -c 'mkdir -p /tmp/submit && cat > /tmp/submit/runtime-env.yaml <<EOF
working_dir: "."
env_vars:
  PYTHONPATH: "/data/verl"
  LD_LIBRARY_PATH: "/usr/local/nvidia/lib64:/usr/local/gib/lib64"
  NCCL_DEBUG: "INFO"
  NCCL_ENV_PLUGIN: "gcp"
  HF_HOME: "/data/huggingface_cache"
  GLOO_SOCKET_IFNAME: "eth0"
EOF'

Kirimkan tugas pelatihan GRPO dengan menjalankannya di Pod head Ray:

kubectl exec ${HEAD_POD} -c ray-head -- bash -c 'cd /tmp/submit && \
ray job submit --runtime-env runtime-env.yaml --no-wait -- \
  python3 -m verl.trainer.main_ppo \
    algorithm.adv_estimator=grpo \
    data.train_files=/data/gsm8k/train.parquet \
    data.val_files=/data/gsm8k/test.parquet \
    data.train_batch_size=256 \
    data.max_prompt_length=512 \
    data.max_response_length=512 \
    actor_rollout_ref.model.path=/data/Qwen2.5-32B-Instruct \
    actor_rollout_ref.actor.optim.lr=1e-5 \
    actor_rollout_ref.actor.ppo_mini_batch_size=64 \
    actor_rollout_ref.actor.ppo_micro_batch_size_per_gpu=8 \
    actor_rollout_ref.actor.use_kl_loss=True \
    actor_rollout_ref.actor.strategy=fsdp2 \
    actor_rollout_ref.rollout.name=vllm \
    actor_rollout_ref.rollout.tensor_model_parallel_size=4 \
    actor_rollout_ref.rollout.gpu_memory_utilization=0.6 \
    actor_rollout_ref.rollout.n=8 \
    actor_rollout_ref.rollout.log_prob_micro_batch_size_per_gpu=16 \
    actor_rollout_ref.ref.log_prob_micro_batch_size_per_gpu=16 \
    algorithm.kl_ctrl.kl_coef=0.001 \
    trainer.logger=console \
    trainer.n_gpus_per_node=4 \
    trainer.nnodes=4 \
    trainer.save_freq=10 \
    trainer.test_freq=10 \
    trainer.total_epochs=2 \
    trainer.default_local_dir=/data/verl/checkpoints'

Pantau log tugas (menggunakan ID unik yang ditampilkan oleh ray job submit):
```
kubectl exec ${HEAD_POD} -c ray-head -- ray job logs <var>JOB_ID</var> --follow
```
Ganti JOB_ID. Konfirmasi bahwa NVLink lintas node aktif dengan mencari baris NCCL di log yang berisi via P2P/MNNVL.

Pembersihan

Untuk menghindari timbulnya biaya, hapus resource:

A4 dan A3 Ultra

Autopilot

Jika loop penerusan port masih berjalan di sesi latar belakang Anda, hentikan:
```
kill ${PF_LOOP_PID}
```

Hapus cluster Ray:

envsubst < "ray-cluster-auto-dranet.yaml" | kubectl delete -f - --ignore-not-found=true || true

Hapus Cloud Storage FUSE:

envsubst < "gcsfuse-storage.yaml" | kubectl delete -f - --ignore-not-found=true || true

Hapus resource DRANET:

kubectl delete -f "resourceclaim-dranet.yaml" --ignore-not-found=true 2>/dev/null || true
kubectl delete -f "computeclass-dranet.yaml" --ignore-not-found=true 2>/dev/null || true

Hapus bucket Cloud Storage:

gcloud storage rm -r "gs://${GS_BUCKET}" --project="${PROJECT_ID:-$(gcloud config get project)}" --quiet || true

Hapus cluster GKE:

gcloud container clusters delete "${CLUSTER_NAME}" --location="${CONTROL_PLANE_REGION}" --quiet || true

Standar

Jika loop penerusan port masih berjalan di sesi latar belakang Anda, hentikan:
```
kill ${PF_PID}
```

Hapus cluster Ray:

envsubst < "ray-cluster-standard.yaml" | kubectl delete --timeout=30s -f - --ignore-not-found=true || true

Hapus Cloud Storage FUSE:

envsubst < "gcsfuse-storage.yaml" | kubectl delete --timeout=30s -f - --ignore-not-found=true || true

Hapus bucket Cloud Storage:

gcloud storage rm -r "gs://${GS_BUCKET}" --project="${PROJECT_ID}" || true

Hapus cluster GKE:

gcloud container clusters delete ${CLUSTER_NAME} --location=${CONTROL_PLANE_REGION} --project=${PROJECT_ID} --quiet || true

Hapus jaringan dan subnet VPC:

# Function to safely delete all firewall rules for a given network
delete_network_firewalls() {
  local NET_NAME=$1
  echo "Deleting all firewall rules associated with network ${NET_NAME}..."
  for rule in $(gcloud compute firewall-rules list --filter="network:${NET_NAME}" --format="value(name)" --project=${PROJECT_ID} 2>/dev/null); do
    echo "Deleting firewall rule ${rule}..."
    gcloud compute firewall-rules delete ${rule} --project=${PROJECT_ID} --quiet || true
  done
}

# Delete Firewalls for both networks
delete_network_firewalls "${RDMA_NETWORK_PREFIX}-net"
delete_network_firewalls "${GVNIC_NETWORK_PREFIX}-net"

# Delete RDMA subnets
echo "Deleting RDMA subnets..."
for N in $(seq 0 7); do
  if gcloud compute networks subnets describe ${RDMA_NETWORK_PREFIX}-sub-$N --region=${CONTROL_PLANE_REGION} --project=${PROJECT_ID} >/dev/null 2>&1; then
    echo "Deleting subnet ${RDMA_NETWORK_PREFIX}-sub-$N..."
    gcloud compute networks subnets delete ${RDMA_NETWORK_PREFIX}-sub-$N --region=${CONTROL_PLANE_REGION} --project=${PROJECT_ID} --quiet || true
  fi
done

# Delete GVNIC subnet
if gcloud compute networks subnets describe ${GVNIC_NETWORK_PREFIX}-sub --region=${CONTROL_PLANE_REGION} --project=${PROJECT_ID} >/dev/null 2>&1; then
  echo "Deleting GVNIC subnet ${GVNIC_NETWORK_PREFIX}-sub..."
  gcloud compute networks subnets delete ${GVNIC_NETWORK_PREFIX}-sub --region=${CONTROL_PLANE_REGION} --project=${PROJECT_ID} --quiet || true
fi

# Function to delete VPC Network with retry and firewall re-purge
delete_vpc_network_with_retry() {
  local NET_NAME=$1
  if gcloud compute networks describe ${NET_NAME} --project=${PROJECT_ID} >/dev/null 2>&1; then
    echo "Deleting network ${NET_NAME}..."
    for attempt in {1..6}; do
      delete_network_firewalls "${NET_NAME}"
      if gcloud compute networks delete ${NET_NAME} --project=${PROJECT_ID} --quiet; then
        echo "Successfully deleted network ${NET_NAME}"
        break
      fi
      echo "Failed to delete network ${NET_NAME} on attempt ${attempt}. Retrying in 10s..."
      sleep 10
    done
  fi
}

# Delete Networks safely with retry
delete_vpc_network_with_retry "${RDMA_NETWORK_PREFIX}-net"
delete_vpc_network_with_retry "${GVNIC_NETWORK_PREFIX}-net"

A4X

kubectl delete raycluster gb200-ray-cluster
kubectl delete computedomain verl-compute-domain
gcloud storage rm -r gs://${GS_BUCKET}
gcloud container clusters delete ${CLUSTER_NAME} --location=${CONTROL_PLANE_REGION}

Menyesuaikan dan menskalakan reinforcement learning dengan verl di GKE Tetap teratur dengan koleksi Simpan dan kategorikan konten berdasarkan preferensi Anda.

Latar belakang

Reinforcement learning (RL)

Pengoptimalan Kebijakan Relatif Grup (GRPO)

Volcano Engine Reinforcement Learning (verl)

Tujuan

Sebelum memulai

Menyiapkan lingkungan Anda

A4 dan A3 Ultra

Autopilot

Standar

A4X

A4 dan A3 Ultra

Autopilot

Standar

A4X

Menyiapkan infrastruktur

Buat jaringan dan subnet RDMA (GKE Standard - A4 dan A3 Ultra saja)

A4 dan A3 Ultra

Autopilot

Standar

A4X

Membuat cluster GKE

A4 dan A3 Ultra

Autopilot

Standar

A4X

Mengonfigurasi pemetaan jaringan (khusus GKE Standard - A4 dan A3 Ultra)

A4 dan A3 Ultra

Autopilot

Standar

A4X

Menyiapkan data dan penyimpanan

Menyiapkan DRANET

A4 dan A3 Ultra

Autopilot

Standar

A4X

Menyiapkan model dan data

A4 dan A3 Ultra

Autopilot

Standar

A4X

Men-deploy resource kustom RayCluster

A4 dan A3 Ultra

Autopilot

Standar

A4X

Luncurkan Tugas GRPO

A4 dan A3 Ultra

Autopilot

Standar

A4X

Pembersihan

A4 dan A3 Ultra

Autopilot

Standar

A4X

Langkah berikutnya

Menyesuaikan dan menskalakan reinforcement learning dengan verl di GKE