Google menggunakan teknologi AI untuk menerjemahkan konten ke dalam bahasa pilihan Anda. Terjemahan AI mungkin mengandung kesalahan.

Panduan pengguna model Mistral dengan vLLM di GDC dengan air gap

Dokumen ini memberikan petunjuk langkah demi langkah untuk men-deploy model open weight dari Mistral AI di lingkungan Google Distributed Cloud (GDC) dengan air gap. Panduan ini membahas penggunaan vLLM untuk inferensi dengan throughput tinggi, menggunakan kemampuan platform GDC, termasuk resource Kubernetes, Harbor, dan GPU. Pilih model dan opsi GPU di bagian Model; setiap langkah lainnya sama untuk semua model.

Model yang didukung

Model	Repositori Hugging Face	Parameter	Lisensi	Opsi GPU
Mistral 7B Instruct	`mistralai/Mistral-7B-Instruct-v0.2`	7,2 miliar (padat)	Apache 2.0	1 x GPU apa pun
Mistral NeMo 12B Instruct	`mistralai/Mistral-Nemo-Instruct-2407`	12B (padat)	Apache 2.0	1 x GPU apa pun
Mistral Small 3.2 24B Instruct	`mistralai/Mistral-Small-3.2-24B-Instruct-2506`	24B (padat, dengan encoder visi)	Apache 2.0	1 x NVIDIA A100 80 GB 1 x NVIDIA H100 94 GB 1 x NVIDIA H200 141 GB atau B300 288 GB
Ministral 8B Instruct	`mistralai/Ministral-8B-Instruct-2410`	8B (padat)	Mistral AI Research License (khusus untuk penggunaan riset dan non-komersial)	1 x GPU apa pun
Mixtral 8x7B Instruct	`mistralai/Mixtral-8x7B-Instruct-v0.1`	Total 46,7 miliar / 12,9 miliar aktif (mixture of experts, 8x7B)	Apache 2.0	2 x NVIDIA A100 80 GB 2 x NVIDIA H100 94 GB 1 x NVIDIA A100 80 GB atau H100 94 GB dengan `--quantization=fp8` 1 x NVIDIA H200 141 GB atau B300 288 GB
Magistral Small 1.2 24B	`mistralai/Magistral-Small-2509`	24B (dense, penalaran, dengan encoder visi)	Apache 2.0	1 x NVIDIA A100 80 GB 1 x NVIDIA H100 94 GB 1 x NVIDIA H200 141 GB atau B300 288 GB

Arsitektur

Solusi ini melibatkan deployment backend inferensi dalam container (vLLM) sebagai Deployment dalam cluster standar. Bobot model disimpan di Persistent Volume, diisi dari file yang didownload secara lokal dan diupload menggunakan pod loader. Service Kubernetes berjenis LoadBalancer mengekspos API backend, dan objek NetworkPolicy / ProjectNetworkPolicy mengamankan akses ke API tersebut.

Sebelum memulai

Selesaikan implementasi referensi Model Open Weight dengan vLLM di workstation, dan pastikan Anda memiliki akses ke direktori ${HOME}/gdcag-solutions.

Identity and Access Management

Pastikan akun, peran, dan izin IAM yang diperlukan telah dikonfigurasi dengan benar.

Peran GDC User di organisasi:

Opsional: Standard Cluster Admin (Melihat cluster dari konsol)

Peran GDC User di project:

Standard Cluster Admin (standard-cluster-admin)

Peran GDC User di cluster standar:

StandardClusterRoleBinding ke StandardClusterRole cluster-admin (dibuat oleh Project IAM Admin, lihat bagian Identity and Access Management pada implementasi referensi)

Workstation

Jalankan semua perintah dalam panduan ini dari workstation dengan alat dan konektivitas yang diperlukan ke lingkungan dan internet, kecuali jika ada langkah yang menyatakan sebaliknya.

Konfigurasi lingkungan berlapis:

${HOME}/gdcag-solutions/env.sh memuat setelan platform dan registry.
${HOME}/gdcag-solutions/open-weight-models/vllm/env.sh menambahkan setelan vLLM.
Panduan pengguna ini menambahkan env.d/model.sh (model) dan env.d/accelerator.sh (opsi GPU dan setelan runtime), yang Anda tulis di bagian Model; dan env.d/model-common.sh (nilai yang berasal dari keduanya).

Buat struktur direktori panduan pengguna:

mkdir -p ${HOME}/gdcag-solutions/open-weight-models/vllm/mistral/env.d

Buat file konfigurasi lingkungan model umum. Setiap jalur dan nama resource berasal dari GDCS_HF_MODEL, sehingga identik untuk semua model:

cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/mistral/env.d/model-common.sh && echo "Successfully created." || echo "Failed to create!"
# Model (derived from env.d/model.sh)
export GDCS_HF_DOWNLOAD_PATH="${GDCS_USER_GUIDE_HOME}/hf-models"
export GDCS_HF_MODEL_LOWERCASE="${GDCS_HF_MODEL,,}"
export GDCS_HF_MODEL_DOWNLOAD_DIR="${GDCS_HF_DOWNLOAD_PATH}/${GDCS_HF_MODEL_LOWERCASE}"
export GDCS_HF_PV_MOUNT_PATH="/mnt/hf-model"

GDCS_HF_MODEL_KUBERNETES="${GDCS_HF_MODEL_LOWERCASE//\//-}"
export GDCS_HF_MODEL_KUBERNETES="${GDCS_HF_MODEL_KUBERNETES//./-}"

# Workload
export GDCS_WORKLOAD_NAMESPACE="vllm-mistral"
export GDCS_WORKLOAD_NAME="${GDCS_HF_MODEL_KUBERNETES}"
export GDCS_MODEL_LOADER_POD_NAME="loader-${GDCS_HF_MODEL_KUBERNETES}"
EOF

Buat file loader lingkungan panduan pengguna:

cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/mistral/env.sh && echo "Successfully created." || echo "Failed to create!"
source "${HOME}/gdcag-solutions/open-weight-models/vllm/env.sh"

export GDCS_USER_GUIDE_HOME="${HOME}/gdcag-solutions/open-weight-models/vllm/mistral"
echo "GDCS_USER_GUIDE_HOME=${GDCS_USER_GUIDE_HOME}"

# Sourced in dependency order: the model selection, the accelerator and runtime settings, then the values derived from them
source "${GDCS_USER_GUIDE_HOME}/env.d/model.sh"
source "${GDCS_USER_GUIDE_HOME}/env.d/accelerator.sh"
source "${GDCS_USER_GUIDE_HOME}/env.d/model-common.sh"
echo "GDCS_HF_MODEL=${GDCS_HF_MODEL} (${GDCS_WORKLOAD_ACCELERATOR_COUNT} x ${GDCS_WORKLOAD_ACCELERATOR_TYPE:-unset})"
EOF

Model

Pilih model dan opsi GPU yang cocok dengan node GPU cluster standar Anda. GDC dengan air gap menawarkan GPU berikut (lihat Jenis mesin node cluster); jenis resource akselerator adalah nama yang digunakan node untuk mengekspos GPU ke Kubernetes.

GPU	Memori	Jenis mesin	Jenis resource akselerator
NVIDIA A100	40 GB	`a2-highgpu-1g-gdc`	`nvidia.com/gpu-pod-NVIDIA_A100_PCIE_40GB`
NVIDIA A100	80 GB	`a2-ultragpu-1g-gdc`, `a2-ultragpu-2g-gdc`	`nvidia.com/gpu-pod-NVIDIA_A100_80GB_PCIE`
NVIDIA H100	94 GB	`a3-highgpu-1g-gdc` (Pratinjau), `a3-highgpu-2g-gdc` (Pratinjau), `a3-highgpu-4g-gdc`	`nvidia.com/gpu-pod-NVIDIA_H100_NVL`
NVIDIA H200	141 GB	`a3-ultragpu-8g-gdc`	`nvidia.com/gpu-pod-NVIDIA_H200`
NVIDIA B300	288 GB	`a4-ultragpu-8g-gdc`	`nvidia.com/gpu-pod-NVIDIA_B300_SXM6_AC`

Setiap tab mencantumkan fakta model, perintah yang menulis file lingkungan model env.d/model.sh, dan tab bertingkat yang menulis file lingkungan akselerator env.d/accelerator.sh untuk opsi GPU yang Anda pilih dengan jenis resource yang telah diisi otomatis dan setelan vLLM. Untuk beralih model atau opsi GPU nanti, tulis file lingkungan yang berbeda, ambil sumber lingkungan lagi, dan ulangi bagian Implementasi; workload beberapa model dapat berjalan berdampingan di namespace yang sama karena setiap nama resource berasal dari nama model.

Buat file lingkungan model dan akselerator untuk opsi model dan GPU yang ingin Anda deploy.

Mistral 7B Instruct

Atribut	Nilai
Repositori Hugging Face	`mistralai/Mistral-7B-Instruct-v0.2`
Parameter	7,2 miliar (padat)
Lisensi	Apache 2.0
Bobot yang digunakan oleh vLLM	Format Hugging Face (`model-*.safetensors`), sekitar 14,5 GB
Panjang konteks	32 ribu token; panduan ini menyajikan `--max-model-len 32768`

Buat file lingkungan model:

cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/mistral/env.d/model.sh && echo "Successfully created." || echo "Failed to create!"
# Model: Mistral 7B Instruct
export GDCS_HF_MODEL="mistralai/Mistral-7B-Instruct-v0.2"
export GDCS_HF_MODEL_REVISION="main"  # Hugging Face branch, tag, or commit to download
export GDCS_HF_DOWNLOAD_EXCLUDE="consolidated* *.bin"
EOF

Buat file lingkungan akselerator untuk opsi GPU Anda:

NVIDIA A100 40 GB

cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/mistral/env.d/accelerator.sh && echo "Successfully created." || echo "Failed to create!"
# Accelerator: NVIDIA A100 40 GB
export GDCS_WORKLOAD_ACCELERATOR_TYPE="nvidia.com/gpu-pod-NVIDIA_A100_PCIE_40GB"
export GDCS_WORKLOAD_ACCELERATOR_COUNT="1"

# vLLM
export GDCS_VLLM_MAX_MODEL_LEN="32768"
export GDCS_VLLM_EXTRA_ARGS=""
export GDCS_VLLM_STARTUP_FAILURE_THRESHOLD="60"
export GDCS_VLLM_TEST_MAX_TOKENS="128"
EOF

NVIDIA A100 80 GB

cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/mistral/env.d/accelerator.sh && echo "Successfully created." || echo "Failed to create!"
# Accelerator: NVIDIA A100 80 GB
export GDCS_WORKLOAD_ACCELERATOR_TYPE="nvidia.com/gpu-pod-NVIDIA_A100_80GB_PCIE"
export GDCS_WORKLOAD_ACCELERATOR_COUNT="1"

# vLLM
export GDCS_VLLM_MAX_MODEL_LEN="32768"
export GDCS_VLLM_EXTRA_ARGS=""
export GDCS_VLLM_STARTUP_FAILURE_THRESHOLD="60"
export GDCS_VLLM_TEST_MAX_TOKENS="128"
EOF

NVIDIA H100 94 GB

cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/mistral/env.d/accelerator.sh && echo "Successfully created." || echo "Failed to create!"
# Accelerator: NVIDIA H100 94 GB
export GDCS_WORKLOAD_ACCELERATOR_TYPE="nvidia.com/gpu-pod-NVIDIA_H100_NVL"
export GDCS_WORKLOAD_ACCELERATOR_COUNT="1"

# vLLM
export GDCS_VLLM_MAX_MODEL_LEN="32768"
export GDCS_VLLM_EXTRA_ARGS=""
export GDCS_VLLM_STARTUP_FAILURE_THRESHOLD="60"
export GDCS_VLLM_TEST_MAX_TOKENS="128"
EOF

NVIDIA H200 141 GB

cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/mistral/env.d/accelerator.sh && echo "Successfully created." || echo "Failed to create!"
# Accelerator: NVIDIA H200 141 GB
export GDCS_WORKLOAD_ACCELERATOR_TYPE="nvidia.com/gpu-pod-NVIDIA_H200"
export GDCS_WORKLOAD_ACCELERATOR_COUNT="1"

# vLLM
export GDCS_VLLM_MAX_MODEL_LEN="32768"
export GDCS_VLLM_EXTRA_ARGS=""
export GDCS_VLLM_STARTUP_FAILURE_THRESHOLD="60"
export GDCS_VLLM_TEST_MAX_TOKENS="128"
EOF

NVIDIA B300 288 GB

cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/mistral/env.d/accelerator.sh && echo "Successfully created." || echo "Failed to create!"
# Accelerator: NVIDIA B300 288 GB
export GDCS_WORKLOAD_ACCELERATOR_TYPE="nvidia.com/gpu-pod-NVIDIA_B300_SXM6_AC"
export GDCS_WORKLOAD_ACCELERATOR_COUNT="1"

# vLLM
export GDCS_VLLM_MAX_MODEL_LEN="32768"
export GDCS_VLLM_EXTRA_ARGS=""
export GDCS_VLLM_STARTUP_FAILURE_THRESHOLD="60"
export GDCS_VLLM_TEST_MAX_TOKENS="128"
EOF

Mistral NeMo 12B Instruct

Atribut	Nilai
Repositori Hugging Face	`mistralai/Mistral-Nemo-Instruct-2407`
Parameter	12B (padat)
Lisensi	Apache 2.0
Bobot yang digunakan oleh vLLM	Format Mistral (`consolidated.safetensors`, `params.json`, `tekken.json`), sekitar 24,5 GB
Panjang konteks	128 ribu token; panduan ini menyajikan `--max-model-len 32768`

Buat file lingkungan model:

cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/mistral/env.d/model.sh && echo "Successfully created." || echo "Failed to create!"
# Model: Mistral NeMo 12B Instruct
export GDCS_HF_MODEL="mistralai/Mistral-Nemo-Instruct-2407"
export GDCS_HF_MODEL_REVISION="main"  # Hugging Face branch, tag, or commit to download
export GDCS_HF_DOWNLOAD_EXCLUDE="model-*.safetensors *.bin"
EOF

Buat file lingkungan akselerator untuk opsi GPU Anda:

NVIDIA A100 40 GB

cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/mistral/env.d/accelerator.sh && echo "Successfully created." || echo "Failed to create!"
# Accelerator: NVIDIA A100 40 GB
export GDCS_WORKLOAD_ACCELERATOR_TYPE="nvidia.com/gpu-pod-NVIDIA_A100_PCIE_40GB"
export GDCS_WORKLOAD_ACCELERATOR_COUNT="1"

# vLLM
export GDCS_VLLM_MAX_MODEL_LEN="32768"
export GDCS_VLLM_EXTRA_ARGS=""
export GDCS_VLLM_STARTUP_FAILURE_THRESHOLD="180"
export GDCS_VLLM_TEST_MAX_TOKENS="128"
EOF

NVIDIA A100 80 GB

cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/mistral/env.d/accelerator.sh && echo "Successfully created." || echo "Failed to create!"
# Accelerator: NVIDIA A100 80 GB
export GDCS_WORKLOAD_ACCELERATOR_TYPE="nvidia.com/gpu-pod-NVIDIA_A100_80GB_PCIE"
export GDCS_WORKLOAD_ACCELERATOR_COUNT="1"

# vLLM
export GDCS_VLLM_MAX_MODEL_LEN="32768"
export GDCS_VLLM_EXTRA_ARGS=""
export GDCS_VLLM_STARTUP_FAILURE_THRESHOLD="180"
export GDCS_VLLM_TEST_MAX_TOKENS="128"
EOF

NVIDIA H100 94 GB

cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/mistral/env.d/accelerator.sh && echo "Successfully created." || echo "Failed to create!"
# Accelerator: NVIDIA H100 94 GB
export GDCS_WORKLOAD_ACCELERATOR_TYPE="nvidia.com/gpu-pod-NVIDIA_H100_NVL"
export GDCS_WORKLOAD_ACCELERATOR_COUNT="1"

# vLLM
export GDCS_VLLM_MAX_MODEL_LEN="32768"
export GDCS_VLLM_EXTRA_ARGS=""
export GDCS_VLLM_STARTUP_FAILURE_THRESHOLD="180"
export GDCS_VLLM_TEST_MAX_TOKENS="128"
EOF

NVIDIA H200 141 GB

cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/mistral/env.d/accelerator.sh && echo "Successfully created." || echo "Failed to create!"
# Accelerator: NVIDIA H200 141 GB
export GDCS_WORKLOAD_ACCELERATOR_TYPE="nvidia.com/gpu-pod-NVIDIA_H200"
export GDCS_WORKLOAD_ACCELERATOR_COUNT="1"

# vLLM
export GDCS_VLLM_MAX_MODEL_LEN="32768"
export GDCS_VLLM_EXTRA_ARGS=""
export GDCS_VLLM_STARTUP_FAILURE_THRESHOLD="180"
export GDCS_VLLM_TEST_MAX_TOKENS="128"
EOF

NVIDIA B300 288 GB

cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/mistral/env.d/accelerator.sh && echo "Successfully created." || echo "Failed to create!"
# Accelerator: NVIDIA B300 288 GB
export GDCS_WORKLOAD_ACCELERATOR_TYPE="nvidia.com/gpu-pod-NVIDIA_B300_SXM6_AC"
export GDCS_WORKLOAD_ACCELERATOR_COUNT="1"

# vLLM
export GDCS_VLLM_MAX_MODEL_LEN="32768"
export GDCS_VLLM_EXTRA_ARGS=""
export GDCS_VLLM_STARTUP_FAILURE_THRESHOLD="180"
export GDCS_VLLM_TEST_MAX_TOKENS="128"
EOF

Mistral Small 3.2 24B Instruct

Atribut	Nilai
Repositori Hugging Face	`mistralai/Mistral-Small-3.2-24B-Instruct-2506`
Parameter	24B (padat, dengan encoder visi)
Lisensi	Apache 2.0
Bobot yang digunakan oleh vLLM	Format Mistral (`consolidated.safetensors`, `params.json`, `tekken.json`), sekitar 48 GB
Panjang konteks	128 ribu token; panduan ini menyajikan `--max-model-len 32768`

Buat file lingkungan model:

cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/mistral/env.d/model.sh && echo "Successfully created." || echo "Failed to create!"
# Model: Mistral Small 3.2 24B Instruct
export GDCS_HF_MODEL="mistralai/Mistral-Small-3.2-24B-Instruct-2506"
export GDCS_HF_MODEL_REVISION="main"  # Hugging Face branch, tag, or commit to download
export GDCS_HF_DOWNLOAD_EXCLUDE="model-*.safetensors *.bin"
EOF

Buat file lingkungan akselerator untuk opsi GPU Anda:

NVIDIA A100 80 GB

cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/mistral/env.d/accelerator.sh && echo "Successfully created." || echo "Failed to create!"
# Accelerator: NVIDIA A100 80 GB
export GDCS_WORKLOAD_ACCELERATOR_TYPE="nvidia.com/gpu-pod-NVIDIA_A100_80GB_PCIE"
export GDCS_WORKLOAD_ACCELERATOR_COUNT="1"

# vLLM
export GDCS_VLLM_MAX_MODEL_LEN="32768"
export GDCS_VLLM_EXTRA_ARGS=""
export GDCS_VLLM_STARTUP_FAILURE_THRESHOLD="300"
export GDCS_VLLM_TEST_MAX_TOKENS="128"
EOF

NVIDIA H100 94 GB

cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/mistral/env.d/accelerator.sh && echo "Successfully created." || echo "Failed to create!"
# Accelerator: NVIDIA H100 94 GB
export GDCS_WORKLOAD_ACCELERATOR_TYPE="nvidia.com/gpu-pod-NVIDIA_H100_NVL"
export GDCS_WORKLOAD_ACCELERATOR_COUNT="1"

# vLLM
export GDCS_VLLM_MAX_MODEL_LEN="32768"
export GDCS_VLLM_EXTRA_ARGS=""
export GDCS_VLLM_STARTUP_FAILURE_THRESHOLD="300"
export GDCS_VLLM_TEST_MAX_TOKENS="128"
EOF

NVIDIA H200 141 GB

cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/mistral/env.d/accelerator.sh && echo "Successfully created." || echo "Failed to create!"
# Accelerator: NVIDIA H200 141 GB
export GDCS_WORKLOAD_ACCELERATOR_TYPE="nvidia.com/gpu-pod-NVIDIA_H200"
export GDCS_WORKLOAD_ACCELERATOR_COUNT="1"

# vLLM
export GDCS_VLLM_MAX_MODEL_LEN="32768"
export GDCS_VLLM_EXTRA_ARGS=""
export GDCS_VLLM_STARTUP_FAILURE_THRESHOLD="300"
export GDCS_VLLM_TEST_MAX_TOKENS="128"
EOF

NVIDIA B300 288 GB

cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/mistral/env.d/accelerator.sh && echo "Successfully created." || echo "Failed to create!"
# Accelerator: NVIDIA B300 288 GB
export GDCS_WORKLOAD_ACCELERATOR_TYPE="nvidia.com/gpu-pod-NVIDIA_B300_SXM6_AC"
export GDCS_WORKLOAD_ACCELERATOR_COUNT="1"

# vLLM
export GDCS_VLLM_MAX_MODEL_LEN="32768"
export GDCS_VLLM_EXTRA_ARGS=""
export GDCS_VLLM_STARTUP_FAILURE_THRESHOLD="300"
export GDCS_VLLM_TEST_MAX_TOKENS="128"
EOF

Ministral 8B Instruct

Atribut	Nilai
Repositori Hugging Face	`mistralai/Ministral-8B-Instruct-2410`
Parameter	8B (padat)
Lisensi	Mistral AI Research License (khusus untuk penggunaan riset dan non-komersial)
Bobot yang digunakan oleh vLLM	Format Mistral (`consolidated.safetensors`, `params.json`, `tekken.json`), sekitar 16 GB
Panjang konteks	128 ribu token; panduan ini menyajikan `--max-model-len 32768`

Buat file lingkungan model:

cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/mistral/env.d/model.sh && echo "Successfully created." || echo "Failed to create!"
# Model: Ministral 8B Instruct
export GDCS_HF_MODEL="mistralai/Ministral-8B-Instruct-2410"
export GDCS_HF_MODEL_REVISION="main"  # Hugging Face branch, tag, or commit to download
export GDCS_HF_DOWNLOAD_EXCLUDE="model-*.safetensors *.bin"
EOF

Buat file lingkungan akselerator untuk opsi GPU Anda:

NVIDIA A100 40 GB

cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/mistral/env.d/accelerator.sh && echo "Successfully created." || echo "Failed to create!"
# Accelerator: NVIDIA A100 40 GB
export GDCS_WORKLOAD_ACCELERATOR_TYPE="nvidia.com/gpu-pod-NVIDIA_A100_PCIE_40GB"
export GDCS_WORKLOAD_ACCELERATOR_COUNT="1"

# vLLM
export GDCS_VLLM_MAX_MODEL_LEN="32768"
export GDCS_VLLM_EXTRA_ARGS=""
export GDCS_VLLM_STARTUP_FAILURE_THRESHOLD="120"
export GDCS_VLLM_TEST_MAX_TOKENS="128"
EOF

NVIDIA A100 80 GB

cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/mistral/env.d/accelerator.sh && echo "Successfully created." || echo "Failed to create!"
# Accelerator: NVIDIA A100 80 GB
export GDCS_WORKLOAD_ACCELERATOR_TYPE="nvidia.com/gpu-pod-NVIDIA_A100_80GB_PCIE"
export GDCS_WORKLOAD_ACCELERATOR_COUNT="1"

# vLLM
export GDCS_VLLM_MAX_MODEL_LEN="32768"
export GDCS_VLLM_EXTRA_ARGS=""
export GDCS_VLLM_STARTUP_FAILURE_THRESHOLD="120"
export GDCS_VLLM_TEST_MAX_TOKENS="128"
EOF

NVIDIA H100 94 GB

cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/mistral/env.d/accelerator.sh && echo "Successfully created." || echo "Failed to create!"
# Accelerator: NVIDIA H100 94 GB
export GDCS_WORKLOAD_ACCELERATOR_TYPE="nvidia.com/gpu-pod-NVIDIA_H100_NVL"
export GDCS_WORKLOAD_ACCELERATOR_COUNT="1"

# vLLM
export GDCS_VLLM_MAX_MODEL_LEN="32768"
export GDCS_VLLM_EXTRA_ARGS=""
export GDCS_VLLM_STARTUP_FAILURE_THRESHOLD="120"
export GDCS_VLLM_TEST_MAX_TOKENS="128"
EOF

NVIDIA H200 141 GB

cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/mistral/env.d/accelerator.sh && echo "Successfully created." || echo "Failed to create!"
# Accelerator: NVIDIA H200 141 GB
export GDCS_WORKLOAD_ACCELERATOR_TYPE="nvidia.com/gpu-pod-NVIDIA_H200"
export GDCS_WORKLOAD_ACCELERATOR_COUNT="1"

# vLLM
export GDCS_VLLM_MAX_MODEL_LEN="32768"
export GDCS_VLLM_EXTRA_ARGS=""
export GDCS_VLLM_STARTUP_FAILURE_THRESHOLD="120"
export GDCS_VLLM_TEST_MAX_TOKENS="128"
EOF

NVIDIA B300 288 GB

cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/mistral/env.d/accelerator.sh && echo "Successfully created." || echo "Failed to create!"
# Accelerator: NVIDIA B300 288 GB
export GDCS_WORKLOAD_ACCELERATOR_TYPE="nvidia.com/gpu-pod-NVIDIA_B300_SXM6_AC"
export GDCS_WORKLOAD_ACCELERATOR_COUNT="1"

# vLLM
export GDCS_VLLM_MAX_MODEL_LEN="32768"
export GDCS_VLLM_EXTRA_ARGS=""
export GDCS_VLLM_STARTUP_FAILURE_THRESHOLD="120"
export GDCS_VLLM_TEST_MAX_TOKENS="128"
EOF

Mixtral 8x7B Instruct

Atribut	Nilai
Repositori Hugging Face	`mistralai/Mixtral-8x7B-Instruct-v0.1`
Parameter	Total 46,7 miliar / 12,9 miliar aktif (mixture of experts, 8x7B)
Lisensi	Apache 2.0
Bobot yang digunakan oleh vLLM	Format Hugging Face (`model-.safetensors`), sekitar 93 GB (repositori juga berisi file PyTorch `consolidated..pt`, dikecualikan)
Panjang konteks	32 ribu token; panduan ini menyajikan `--max-model-len 32768`

Buat file lingkungan model:

cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/mistral/env.d/model.sh && echo "Successfully created." || echo "Failed to create!"
# Model: Mixtral 8x7B Instruct
export GDCS_HF_MODEL="mistralai/Mixtral-8x7B-Instruct-v0.1"
export GDCS_HF_MODEL_REVISION="main"  # Hugging Face branch, tag, or commit to download
export GDCS_HF_DOWNLOAD_EXCLUDE="consolidated* *.bin"
EOF

Buat file lingkungan akselerator untuk opsi GPU Anda:

NVIDIA A100 80 GB (2 GPU)

a2-ultragpu-2g-gdc: Memerlukan node dengan dua GPU A100 80 GB; vLLM berjalan dengan --tensor-parallel-size 2.

cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/mistral/env.d/accelerator.sh && echo "Successfully created." || echo "Failed to create!"
# Accelerator: NVIDIA A100 80 GB (2 GPUs)
export GDCS_WORKLOAD_ACCELERATOR_TYPE="nvidia.com/gpu-pod-NVIDIA_A100_80GB_PCIE"
export GDCS_WORKLOAD_ACCELERATOR_COUNT="2"

# vLLM
export GDCS_VLLM_MAX_MODEL_LEN="32768"
export GDCS_VLLM_EXTRA_ARGS=""
export GDCS_VLLM_STARTUP_FAILURE_THRESHOLD="480"
export GDCS_VLLM_TEST_MAX_TOKENS="128"
EOF

NVIDIA H100 94 GB (2 GPU)

Memerlukan node dengan setidaknya dua GPU H100 (a3-highgpu-2g-gdc atau a3-highgpu-4g-gdc); vLLM berjalan dengan --tensor-parallel-size 2.

cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/mistral/env.d/accelerator.sh && echo "Successfully created." || echo "Failed to create!"
# Accelerator: NVIDIA H100 94 GB (2 GPUs)
export GDCS_WORKLOAD_ACCELERATOR_TYPE="nvidia.com/gpu-pod-NVIDIA_H100_NVL"
export GDCS_WORKLOAD_ACCELERATOR_COUNT="2"

# vLLM
export GDCS_VLLM_MAX_MODEL_LEN="32768"
export GDCS_VLLM_EXTRA_ARGS=""
export GDCS_VLLM_STARTUP_FAILURE_THRESHOLD="480"
export GDCS_VLLM_TEST_MAX_TOKENS="128"
EOF

NVIDIA A100 80 GB (FP8)

--quantization=fp8 mengkuantisasi bobot ke FP8 saat memuat (sekitar 47 GB, bukan 93 GB), dengan kualitas output yang sedikit lebih rendah.

cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/mistral/env.d/accelerator.sh && echo "Successfully created." || echo "Failed to create!"
# Accelerator: NVIDIA A100 80 GB (FP8)
export GDCS_WORKLOAD_ACCELERATOR_TYPE="nvidia.com/gpu-pod-NVIDIA_A100_80GB_PCIE"
export GDCS_WORKLOAD_ACCELERATOR_COUNT="1"

# vLLM
export GDCS_VLLM_MAX_MODEL_LEN="32768"
export GDCS_VLLM_EXTRA_ARGS="--quantization=fp8"
export GDCS_VLLM_STARTUP_FAILURE_THRESHOLD="480"
export GDCS_VLLM_TEST_MAX_TOKENS="128"
EOF

NVIDIA H100 94 GB (FP8)

--quantization=fp8 mengkuantisasi bobot ke FP8 saat memuat (sekitar 47 GB, bukan 93 GB), dengan kualitas output yang sedikit lebih rendah.

cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/mistral/env.d/accelerator.sh && echo "Successfully created." || echo "Failed to create!"
# Accelerator: NVIDIA H100 94 GB (FP8)
export GDCS_WORKLOAD_ACCELERATOR_TYPE="nvidia.com/gpu-pod-NVIDIA_H100_NVL"
export GDCS_WORKLOAD_ACCELERATOR_COUNT="1"

# vLLM
export GDCS_VLLM_MAX_MODEL_LEN="32768"
export GDCS_VLLM_EXTRA_ARGS="--quantization=fp8"
export GDCS_VLLM_STARTUP_FAILURE_THRESHOLD="480"
export GDCS_VLLM_TEST_MAX_TOKENS="128"
EOF

NVIDIA H200 141 GB

cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/mistral/env.d/accelerator.sh && echo "Successfully created." || echo "Failed to create!"
# Accelerator: NVIDIA H200 141 GB
export GDCS_WORKLOAD_ACCELERATOR_TYPE="nvidia.com/gpu-pod-NVIDIA_H200"
export GDCS_WORKLOAD_ACCELERATOR_COUNT="1"

# vLLM
export GDCS_VLLM_MAX_MODEL_LEN="32768"
export GDCS_VLLM_EXTRA_ARGS=""
export GDCS_VLLM_STARTUP_FAILURE_THRESHOLD="480"
export GDCS_VLLM_TEST_MAX_TOKENS="128"
EOF

NVIDIA B300 288 GB

cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/mistral/env.d/accelerator.sh && echo "Successfully created." || echo "Failed to create!"
# Accelerator: NVIDIA B300 288 GB
export GDCS_WORKLOAD_ACCELERATOR_TYPE="nvidia.com/gpu-pod-NVIDIA_B300_SXM6_AC"
export GDCS_WORKLOAD_ACCELERATOR_COUNT="1"

# vLLM
export GDCS_VLLM_MAX_MODEL_LEN="32768"
export GDCS_VLLM_EXTRA_ARGS=""
export GDCS_VLLM_STARTUP_FAILURE_THRESHOLD="480"
export GDCS_VLLM_TEST_MAX_TOKENS="128"
EOF

Magistral Small 1.2 24B

Atribut	Nilai
Repositori Hugging Face	`mistralai/Magistral-Small-2509`
Parameter	24B (dense, penalaran, dengan encoder visi)
Lisensi	Apache 2.0
Bobot yang digunakan oleh vLLM	Format Mistral (`consolidated.safetensors`, `params.json`, `tekken.json`), sekitar 48 GB
Panjang konteks	Token 128k (40k direkomendasikan oleh Mistral); panduan ini menyajikan `--max-model-len 40960`

Buat file lingkungan model:

cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/mistral/env.d/model.sh && echo "Successfully created." || echo "Failed to create!"
# Model: Magistral Small 1.2 24B
export GDCS_HF_MODEL="mistralai/Magistral-Small-2509"
export GDCS_HF_MODEL_REVISION="main"  # Hugging Face branch, tag, or commit to download
export GDCS_HF_DOWNLOAD_EXCLUDE="model-*.safetensors *.bin"
EOF

Buat file lingkungan akselerator untuk opsi GPU Anda:

NVIDIA A100 80 GB

cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/mistral/env.d/accelerator.sh && echo "Successfully created." || echo "Failed to create!"
# Accelerator: NVIDIA A100 80 GB
export GDCS_WORKLOAD_ACCELERATOR_TYPE="nvidia.com/gpu-pod-NVIDIA_A100_80GB_PCIE"
export GDCS_WORKLOAD_ACCELERATOR_COUNT="1"

# vLLM
export GDCS_VLLM_MAX_MODEL_LEN="40960"
export GDCS_VLLM_EXTRA_ARGS="--reasoning-parser=mistral"
export GDCS_VLLM_STARTUP_FAILURE_THRESHOLD="300"
export GDCS_VLLM_TEST_MAX_TOKENS="1024"
EOF

NVIDIA H100 94 GB

cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/mistral/env.d/accelerator.sh && echo "Successfully created." || echo "Failed to create!"
# Accelerator: NVIDIA H100 94 GB
export GDCS_WORKLOAD_ACCELERATOR_TYPE="nvidia.com/gpu-pod-NVIDIA_H100_NVL"
export GDCS_WORKLOAD_ACCELERATOR_COUNT="1"

# vLLM
export GDCS_VLLM_MAX_MODEL_LEN="40960"
export GDCS_VLLM_EXTRA_ARGS="--reasoning-parser=mistral"
export GDCS_VLLM_STARTUP_FAILURE_THRESHOLD="300"
export GDCS_VLLM_TEST_MAX_TOKENS="1024"
EOF

NVIDIA H200 141 GB

cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/mistral/env.d/accelerator.sh && echo "Successfully created." || echo "Failed to create!"
# Accelerator: NVIDIA H200 141 GB
export GDCS_WORKLOAD_ACCELERATOR_TYPE="nvidia.com/gpu-pod-NVIDIA_H200"
export GDCS_WORKLOAD_ACCELERATOR_COUNT="1"

# vLLM
export GDCS_VLLM_MAX_MODEL_LEN="40960"
export GDCS_VLLM_EXTRA_ARGS="--reasoning-parser=mistral"
export GDCS_VLLM_STARTUP_FAILURE_THRESHOLD="300"
export GDCS_VLLM_TEST_MAX_TOKENS="1024"
EOF

NVIDIA B300 288 GB

cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/mistral/env.d/accelerator.sh && echo "Successfully created." || echo "Failed to create!"
# Accelerator: NVIDIA B300 288 GB
export GDCS_WORKLOAD_ACCELERATOR_TYPE="nvidia.com/gpu-pod-NVIDIA_B300_SXM6_AC"
export GDCS_WORKLOAD_ACCELERATOR_COUNT="1"

# vLLM
export GDCS_VLLM_MAX_MODEL_LEN="40960"
export GDCS_VLLM_EXTRA_ARGS="--reasoning-parser=mistral"
export GDCS_VLLM_STARTUP_FAILURE_THRESHOLD="300"
export GDCS_VLLM_TEST_MAX_TOKENS="1024"
EOF

Ambil file lingkungan:

source ${HOME}/gdcag-solutions/open-weight-models/vllm/mistral/env.sh

Outputnya mirip dengan hal berikut ini:

GDCS_ROOT_HOME=HOME_DIRECTORY_PATH/gdcag-solutions
GDCS_IMPLEMENTATION_HOME=HOME_DIRECTORY_PATH/gdcag-solutions/open-weight-models/vllm
GDCS_USER_GUIDE_HOME=HOME_DIRECTORY_PATH/gdcag-solutions/open-weight-models/vllm/mistral
GDCS_HF_MODEL=HUGGING_FACE_REPOSITORY (ACCELERATOR_COUNT x ACCELERATOR_RESOURCE_TYPE)

Cluster

Konfigurasikan gdcloud:

gdcloud config set core/account "default-user"
gdcloud config set core/organization_console_url "${GDC_ZONAL_CONSOLE_URL}"
gdcloud config set core/project "${GDC_PROJECT}"
gdcloud config set core/zone "${GDC_ZONE}"

Lakukan autentikasi ke lingkungan GDC:
```
gdcloud auth login
```

Ambil kredensial cluster:

gdcloud clusters get-credentials "${GDC_STANDARD_CLUSTER_NAME}" \
--project="${GDC_PROJECT}" \
--standard \
--zone="${GDC_ZONE}"

Verifikasi konektivitas ke cluster:

kubectl get nodes -L node.cluster.private.gdc.goog/machine-class

Pastikan jenis resource akselerator dari file lingkungan akselerator tersedia di cluster. Perintah ini mencantumkan jenis resource akselerator yang diekspos node (nama bergantung pada model GPU yang diinstal; kosong pada cluster tanpa node GPU) dan nilai yang dikonfigurasi:
```
kubectl get nodes --output=json | jq -r '[.items[].status.allocatable | keys[] | select(startswith("nvidia.com/"))] | unique[]'
echo "GDCS_WORKLOAD_ACCELERATOR_TYPE=${GDCS_WORKLOAD_ACCELERATOR_TYPE}"
```
Catatan: Jika nilai yang dikonfigurasi tidak ada dalam daftar, tetapkan GDCS_WORKLOAD_ACCELERATOR_TYPE di env.d/accelerator.sh ke salah satu jenis resource yang tercantum dan sumber file lingkungan lagi.

Penerapan

Namespace

Buat namespace:

kubectl create namespace "${GDCS_WORKLOAD_NAMESPACE}" \
--dry-run=client \
--output=yaml | kubectl apply -f -

Tambahkan imagePullSecret:

kubectl create secret docker-registry "${GDCS_HARBOR_K8S_PULL_SECRET}" \
--dry-run=client \
--from-file=.dockerconfigjson=${GDCS_HARBOR_K8S_DOCKER_CONFIG}/config.json \
--namespace="${GDCS_WORKLOAD_NAMESPACE}" \
--output=yaml | kubectl apply -f -

Bobot model

Download bobot model (hanya file yang diperlukan vLLM untuk model yang dipilih, GDCS_HF_DOWNLOAD_EXCLUDE akan melewati sisanya) dan hapus direktori .cache:

EXCLUDE_ARGS=()
if [[ -n "${GDCS_HF_DOWNLOAD_EXCLUDE:-}" ]]; then
  read -r -a EXCLUDE_PATTERNS <<< "${GDCS_HF_DOWNLOAD_EXCLUDE}"
  EXCLUDE_ARGS=(--exclude "${EXCLUDE_PATTERNS[@]}")
fi
hf download "${GDCS_HF_MODEL}" --revision "${GDCS_HF_MODEL_REVISION}" --local-dir "${GDCS_HF_MODEL_DOWNLOAD_DIR}" "${EXCLUDE_ARGS[@]}" && \
rm -rf "${GDCS_HF_MODEL_DOWNLOAD_DIR}/.cache"

Hitung ukuran PersistentVolumeClaim yang diperlukan dari bobot model yang didownload dengan overhead 25% (dibulatkan ke atas ke GiB berikutnya):

MODEL_KIB=$(du -sk "${GDCS_HF_MODEL_DOWNLOAD_DIR}" | awk '{print $1}')
export GDCS_MODEL_PVC_SIZE="$(( (MODEL_KIB * 125 / 100 + 1048575) / 1048576 ))Gi"
echo "GDCS_MODEL_PVC_SIZE=${GDCS_MODEL_PVC_SIZE}"

Buat manifes untuk PersistentVolumeClaim ReadWriteOnce untuk bobot model:

cat <<EOF > "${GDCS_USER_GUIDE_HOME}/${GDCS_WORKLOAD_NAME}-rwo-pvc.yaml"
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: ${GDCS_WORKLOAD_NAME}-rwo
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: ${GDCS_MODEL_PVC_SIZE}
  storageClassName: standard-rwo
  volumeMode: Filesystem
EOF

Terapkan manifes untuk PersistentVolumeClaim untuk bobot model:

kubectl apply \
--filename="${GDCS_USER_GUIDE_HOME}/${GDCS_WORKLOAD_NAME}-rwo-pvc.yaml" \
--namespace="${GDCS_WORKLOAD_NAMESPACE}"

Tunggu hingga PersistentVolumeClaim mengikat dan mencetak nilai yang diperlukan Operator Infrastruktur untuk menyesuaikan kebijakan quality of service (QoS) I/O:

kubectl wait \
--for=jsonpath='{.status.phase}'=Bound \
"pvc/${GDCS_WORKLOAD_NAME}-rwo" \
--namespace="${GDCS_WORKLOAD_NAMESPACE}" \
--timeout=120s
BACKING_PVC=$(kubectl get pvc "${GDCS_WORKLOAD_NAME}-rwo" \
--namespace="${GDCS_WORKLOAD_NAMESPACE}" \
--output=jsonpath='{.spec.volumeName}')
echo "BACKING_PVC=${BACKING_PVC} ORG=${GDC_ORG}"

Jika performance-rwo tidak tersedia di cluster Anda, minta Operator Infrastruktur Anda untuk menerapkan kebijakan QoS berperforma tinggi (system-performance-<ORG>-admin, 30 IOPS per GiB dengan batas minimum 6.148 IOPS) ke volume yang mendukung PersistentVolumeClaim. Berikan nilai BACKING_PVC dan ORG yang dicetak pada langkah sebelumnya. Operator Infrastruktur menerapkan kebijakan seperti yang dijelaskan dalam FILE-R0097; kebijakan ini berlaku pada volume aktif, sehingga Anda dapat melanjutkan ke langkah berikutnya.

Catatan: Jika block storage berperforma tinggi diaktifkan di cluster, Anda dapat menetapkan storageClassName: performance-rwo dalam manifes PersistentVolumeClaim, bukan meminta kebijakan QoS. Jika kubectl get storageclass performance-rwo tidak menampilkan apa pun, Operator Infrastruktur belum mengaktifkan block storage berperforma tinggi.

Buat manifes untuk Pod loader model:

cat <<EOF > "${GDCS_USER_GUIDE_HOME}/${GDCS_MODEL_LOADER_POD_NAME}-pod.yaml"
apiVersion: v1
kind: Pod
metadata:
  name: ${GDCS_MODEL_LOADER_POD_NAME}
spec:
  containers:
    - name: uploader
      image: ${GDCS_HARBOR_PROJECT_URI}/library/busybox:${GDCS_BUSYBOX_IMAGE_TAG}
      command: ["sleep", "infinity"]
      volumeMounts:
        - name: model-data
          mountPath: ${GDCS_HF_PV_MOUNT_PATH}
  imagePullSecrets:
    - name: ${GDCS_HARBOR_K8S_PULL_SECRET}
  volumes:
    - name: model-data
      persistentVolumeClaim:
        claimName: ${GDCS_WORKLOAD_NAME}-rwo
EOF

Terapkan manifes untuk Pod pemuat model:

kubectl apply \
--filename="${GDCS_USER_GUIDE_HOME}/${GDCS_MODEL_LOADER_POD_NAME}-pod.yaml" \
--namespace="${GDCS_WORKLOAD_NAMESPACE}"

Tunggu hingga Pod pemuat model berstatus Running:

watch --color --interval 5 --no-title \
"kubectl get pod/${GDCS_MODEL_LOADER_POD_NAME} \
--namespace=${GDCS_WORKLOAD_NAMESPACE} | GREP_COLORS='mt=01;92' grep -E --color=always -e '^' -e 'Running'"

Upload bobot model ke PersistentVolume (skrip akan melewati file yang sudah ada dengan ukuran yang sama, sehingga dapat dijalankan ulang):

export LOCAL_MODEL_DIR="${GDCS_HF_MODEL_DOWNLOAD_DIR}"
export REMOTE_MODEL_DIR="${GDCS_HF_PV_MOUNT_PATH}/${GDCS_HF_MODEL_LOWERCASE}"
export REMOTE_MOUNT_DIR="${GDCS_HF_PV_MOUNT_PATH}"

${GDCS_IMPLEMENTATION_HOME}/model_weight_upload.sh

Hapus Pod pemuat model:

kubectl delete pod/${GDCS_MODEL_LOADER_POD_NAME} \
--namespace="${GDCS_WORKLOAD_NAMESPACE}"

Server inferensi

Buat manifes untuk Deployment vLLM. ConfigMap membawa setelan vLLM khusus model dari lingkungan, sehingga manifesnya sama untuk setiap opsi model dan GPU:

cat <<EOF > "${GDCS_USER_GUIDE_HOME}/${GDCS_WORKLOAD_NAME}-deployment.yaml"
apiVersion: v1
kind: ConfigMap
metadata:
  name: ${GDCS_WORKLOAD_NAME}
data:
  MAX_MODEL_LEN: "${GDCS_VLLM_MAX_MODEL_LEN}"
  MODEL: "${GDCS_HF_PV_MOUNT_PATH}/${GDCS_HF_MODEL_LOWERCASE}"
  PORT: "8000"
  TENSOR_PARALLEL_SIZE: "${GDCS_WORKLOAD_ACCELERATOR_COUNT}"
  VLLM_EXTRA_ARGS: "${GDCS_VLLM_EXTRA_ARGS}"
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: ${GDCS_WORKLOAD_NAME}
  labels:
    app: ${GDCS_WORKLOAD_NAME}
spec:
  replicas: 1
  selector:
    matchLabels:
      app: ${GDCS_WORKLOAD_NAME}
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: ${GDCS_WORKLOAD_NAME}
    spec:
      containers:
        - command:
            - /bin/sh
            - -c
          args:
            - exec vllm serve "\$MODEL" --port="\$PORT" --tensor-parallel-size="\$TENSOR_PARALLEL_SIZE" --max-model-len="\$MAX_MODEL_LEN" \$VLLM_EXTRA_ARGS
          envFrom:
            - configMapRef:
                name: ${GDCS_WORKLOAD_NAME}
          image: ${GDCS_HARBOR_PROJECT_URI}/vllm/vllm-openai:${GDCS_VLLM_IMAGE_TAG}-fips
          livenessProbe:
            failureThreshold: 3
            httpGet:
              path: /health
              port: 8000
            periodSeconds: 20
            timeoutSeconds: 5
          name: vllm
          ports:
            - containerPort: 8000
          readinessProbe:
            failureThreshold: 3
            httpGet:
              path: /health
              port: 8000
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 5
          resources:
            limits:
              ${GDCS_WORKLOAD_ACCELERATOR_TYPE}: "${GDCS_WORKLOAD_ACCELERATOR_COUNT}"
            requests:
              ${GDCS_WORKLOAD_ACCELERATOR_TYPE}: "${GDCS_WORKLOAD_ACCELERATOR_COUNT}"
          startupProbe:
            httpGet:
              path: /health
              port: 8000
            failureThreshold: ${GDCS_VLLM_STARTUP_FAILURE_THRESHOLD}
            periodSeconds: 10
            timeoutSeconds: 5
          volumeMounts:
            - name: model-weights
              mountPath: ${GDCS_HF_PV_MOUNT_PATH}
            - name: shm
              mountPath: /dev/shm
      securityContext:
        seLinuxOptions:
          type: spc_t
      imagePullSecrets:
        - name: ${GDCS_HARBOR_K8S_PULL_SECRET}
      volumes:
        - name: model-weights
          persistentVolumeClaim:
            claimName: ${GDCS_WORKLOAD_NAME}-rwo
        - name: shm
          emptyDir:
            medium: Memory
            sizeLimit: "8Gi"
---
apiVersion: v1
kind: Service
metadata:
  name: ${GDCS_WORKLOAD_NAME}
spec:
  ports:
    - name: vllm-http
      port: 80
      protocol: TCP
      targetPort: 8000
  selector:
    app: ${GDCS_WORKLOAD_NAME}
  type: LoadBalancer
EOF

Terapkan manifes untuk Deployment vLLM:

kubectl apply \
--filename="${GDCS_USER_GUIDE_HOME}/${GDCS_WORKLOAD_NAME}-deployment.yaml" \
--namespace="${GDCS_WORKLOAD_NAMESPACE}"

Buat manifes untuk NetworkPolicy vLLM:

cat << EOF > "${GDCS_USER_GUIDE_HOME}/vllm-networkpolicy.yaml" && echo "Successfully created." || echo "Failed to create!"
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-vllm-ingress
  namespace: ${GDCS_WORKLOAD_NAMESPACE}
spec:
  ingress:
  - from:
    - ipBlock:
        cidr: 0.0.0.0/0
    - namespaceSelector: {}
    ports:
    - port: 8000
      protocol: TCP
  podSelector: {}
  policyTypes:
  - Ingress
EOF

Terapkan manifes untuk NetworkPolicy vLLM:

kubectl apply \
--filename="${GDCS_USER_GUIDE_HOME}/vllm-networkpolicy.yaml"

Tunggu hingga Pod berstatus Running:

watch --color --interval 5 --no-title \
"kubectl get pods --selector=app=${GDCS_WORKLOAD_NAME} \
--namespace=${GDCS_WORKLOAD_NAMESPACE} | GREP_COLORS='mt=01;92' grep -E --color=always -e '^' -e 'Running'"

Tunggu hingga Deployment Siap:

watch --color --interval 5 --no-title \
"kubectl get deployment/${GDCS_WORKLOAD_NAME} \
--namespace=${GDCS_WORKLOAD_NAMESPACE} | GREP_COLORS='mt=01;92' grep -E --color=always -e '^' -e '1/1     1            1'"

Opsional: Alirkan log container untuk memantau progres startup:

kubectl logs deployment/${GDCS_WORKLOAD_NAME} \
--follow \
--namespace="${GDCS_WORKLOAD_NAMESPACE}"

Validasi

Ambil alamat IP LoadBalancer untuk vLLM:

export VLLM_LB_IP=$(kubectl get service/${GDCS_WORKLOAD_NAME} --namespace="${GDCS_WORKLOAD_NAMESPACE}" --output=jsonpath='{.status.loadBalancer.ingress[*].ip}')
echo "VLLM_LB_IP=${VLLM_LB_IP}"

Kirim permintaan pengujian ke vLLM. Nilai max_tokens adalah GDCS_VLLM_TEST_MAX_TOKENS dari env.d/accelerator.sh; model penalaran memerlukan anggaran yang lebih besar karena fase penalaran menggunakan token output:

curl http://${VLLM_LB_IP}/v1/models

curl http://${VLLM_LB_IP}/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
  "model": "'${GDCS_HF_PV_MOUNT_PATH}/${GDCS_HF_MODEL_LOWERCASE}'",
  "messages": [
    {"role": "user", "content": "Explain Google Distributed Cloud air-gapped briefly."}
  ],
  "max_tokens": '${GDCS_VLLM_TEST_MAX_TOKENS}'
}'

Jika alamat IP LoadBalancer tidak dapat dijangkau dari workstation, misalnya karena belum ada ProjectNetworkPolicy yang menerimanya, kirim permintaan melalui penerusan port:

kubectl port-forward service/${GDCS_WORKLOAD_NAME} \
--namespace="${GDCS_WORKLOAD_NAMESPACE}" 8000:80 &
PF_PID=$!

sleep 1

curl http://127.0.0.1:8000/v1/models

kill ${PF_PID}

Panduan pengguna model Mistral dengan vLLM di GDC dengan air gap Tetap teratur dengan koleksi Simpan dan kategorikan konten berdasarkan preferensi Anda.

Model yang didukung

Arsitektur

Sebelum memulai

Identity and Access Management

Workstation

Model

Mistral 7B Instruct

NVIDIA A100 40 GB

NVIDIA A100 80 GB

NVIDIA H100 94 GB

NVIDIA H200 141 GB

NVIDIA B300 288 GB

Mistral NeMo 12B Instruct

NVIDIA A100 40 GB

NVIDIA A100 80 GB

NVIDIA H100 94 GB

NVIDIA H200 141 GB

NVIDIA B300 288 GB

Mistral Small 3.2 24B Instruct

NVIDIA A100 80 GB

NVIDIA H100 94 GB

NVIDIA H200 141 GB

NVIDIA B300 288 GB

Ministral 8B Instruct

NVIDIA A100 40 GB

NVIDIA A100 80 GB

NVIDIA H100 94 GB

NVIDIA H200 141 GB

NVIDIA B300 288 GB

Mixtral 8x7B Instruct

NVIDIA A100 80 GB (2 GPU)

NVIDIA H100 94 GB (2 GPU)

NVIDIA A100 80 GB (FP8)

NVIDIA H100 94 GB (FP8)

NVIDIA H200 141 GB

NVIDIA B300 288 GB

Magistral Small 1.2 24B

NVIDIA A100 80 GB

NVIDIA H100 94 GB

NVIDIA H200 141 GB

NVIDIA B300 288 GB

Cluster

Penerapan

Namespace

Bobot model

Server inferensi

Validasi

Panduan pengguna model Mistral dengan vLLM di GDC dengan air gap