本文件提供逐步說明,協助您在 Google Distributed Cloud (GDC) 實體隔離環境中,部署 Google DeepMind 的開放權重模型,特別是 Gemma 4 系列。課程內容涵蓋使用 vLLM 進行高處理量服務,以及運用 GDC 平台的功能,包括 Kubernetes、Harbor 和 GPU 資源。在「Model」(模型) 區段中選取模型和 GPU 選項;所有模型的其他步驟都相同。
支援的模型
| 模型 | Hugging Face 存放區 | 參數 | 授權 | GPU 選項 |
|---|---|---|---|---|
| Gemma 4 E2B Instruct | google/gemma-4-E2B-it |
5.1B (2B effective) | Gemma 使用條款 | 1 個任何 GPU |
| Gemma 4 E4B Instruct | google/gemma-4-E4B-it |
80 億 (有效 40 億) | Gemma 使用條款 | 1 個任何 GPU |
| Gemma 4 12B Instruct | google/gemma-4-12B-it |
12B (密集) | Gemma 使用條款 | 1 個任何 GPU |
| Gemma 4 26B-A4B Instruct | google/gemma-4-26B-A4B-it |
258 億個參數/40 億個有效參數 (混合專家模型) | Gemma 使用條款 | 1 個 NVIDIA A100 80 GB 1 個 NVIDIA H100 94 GB 1 個 NVIDIA H200 141 GB 或 B300 288 GB |
| Gemma 4 31B Instruct | google/gemma-4-31B-it |
31.3B (密集) | Gemma 使用條款 | 1 個 NVIDIA A100 80 GB 1 個 NVIDIA H100 94 GB 1 個 NVIDIA H200 141 GB 或 B300 288 GB |
架構
這項解決方案會將容器化服務後端 (vLLM) 部署為標準叢集中的 Deployment。模型權重會儲存在永久磁碟區,並從本機下載的檔案填入,然後使用載入器 Pod 上傳。LoadBalancer 類型的 Kubernetes Service 會公開後端 API,而 NetworkPolicy/ProjectNetworkPolicy 物件則可保護對該 API 的存取。
事前準備
在工作站上完成使用 vLLM 參考實作開放權重模型,並確認您有權存取 ${HOME}/gdcag-solutions 目錄。
Identity and Access Management
請確認已正確設定必要的 IAM 帳戶、角色和權限。
組織的 GDC 使用者角色:
- 選用:標準叢集管理員 (從控制台查看叢集)
專案的 GDC 使用者角色:
- 標準叢集管理員 (
standard-cluster-admin)
標準叢集的 GDC 使用者角色:
StandardClusterRoleBinding至StandardClusterRolecluster-admin(由 IAM 專案管理員建立,請參閱參考實作的 Identity and Access Management 部分)
工作站
除非步驟另有說明,否則請在工作站執行本指南中的所有指令,並確保工作站具備必要工具,且可連線至環境和網際網路。
環境設定分為多層:
${HOME}/gdcag-solutions/env.sh會載入平台和登錄設定。${HOME}/gdcag-solutions/open-weight-models/vllm/env.sh會新增 vLLM 設定。本使用手冊會新增
env.d/model.sh(模型) 和env.d/accelerator.sh(GPU 選項和執行階段設定),您可以在「模型」部分編寫這些內容;以及env.d/model-common.sh(從中衍生的值)。
建立使用手冊目錄架構:
mkdir -p ${HOME}/gdcag-solutions/open-weight-models/vllm/gemma/env.d建立通用模型環境設定檔。它會從
GDCS_HF_MODEL衍生每個路徑和資源名稱,因此所有模型都相同:cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/gemma/env.d/model-common.sh && echo "Successfully created." || echo "Failed to create!" # Model (derived from env.d/model.sh) export GDCS_HF_DOWNLOAD_PATH="${GDCS_USER_GUIDE_HOME}/hf-models" export GDCS_HF_MODEL_LOWERCASE="${GDCS_HF_MODEL,,}" export GDCS_HF_MODEL_DOWNLOAD_DIR="${GDCS_HF_DOWNLOAD_PATH}/${GDCS_HF_MODEL_LOWERCASE}" export GDCS_HF_PV_MOUNT_PATH="/mnt/hf-model" GDCS_HF_MODEL_KUBERNETES="${GDCS_HF_MODEL_LOWERCASE//\//-}" export GDCS_HF_MODEL_KUBERNETES="${GDCS_HF_MODEL_KUBERNETES//./-}" # Workload export GDCS_WORKLOAD_NAMESPACE="vllm-gemma" export GDCS_WORKLOAD_NAME="${GDCS_HF_MODEL_KUBERNETES}" export GDCS_MODEL_LOADER_POD_NAME="loader-${GDCS_HF_MODEL_KUBERNETES}" EOF建立使用手冊環境載入器檔案:
cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/gemma/env.sh && echo "Successfully created." || echo "Failed to create!" source "${HOME}/gdcag-solutions/open-weight-models/vllm/env.sh" export GDCS_USER_GUIDE_HOME="${HOME}/gdcag-solutions/open-weight-models/vllm/gemma" echo "GDCS_USER_GUIDE_HOME=${GDCS_USER_GUIDE_HOME}" # Sourced in dependency order: the model selection, the accelerator and runtime settings, then the values derived from them source "${GDCS_USER_GUIDE_HOME}/env.d/model.sh" source "${GDCS_USER_GUIDE_HOME}/env.d/accelerator.sh" source "${GDCS_USER_GUIDE_HOME}/env.d/model-common.sh" echo "GDCS_HF_MODEL=${GDCS_HF_MODEL} (${GDCS_WORKLOAD_ACCELERATOR_COUNT} x ${GDCS_WORKLOAD_ACCELERATOR_TYPE:-unset})" EOF
模型
選取與標準叢集 GPU 節點相符的模型和 GPU 選項。GDC 實體隔離方案提供下列 GPU (請參閱「叢集節點機型」);加速器資源類型是節點向 Kubernetes 公開 GPU 的名稱。
| GPU | 記憶體 | 機型 | 加速器資源類型 |
|---|---|---|---|
| NVIDIA A100 | 40 GB | a2-highgpu-1g-gdc |
nvidia.com/gpu-pod-NVIDIA_A100_PCIE_40GB |
| NVIDIA A100 | 80 GB | a2-ultragpu-1g-gdc、a2-ultragpu-2g-gdc |
nvidia.com/gpu-pod-NVIDIA_A100_80GB_PCIE |
| NVIDIA H100 | 94 GB | a3-highgpu-1g-gdc (預先發布版)、a3-highgpu-2g-gdc (預先發布版)、a3-highgpu-4g-gdc |
nvidia.com/gpu-pod-NVIDIA_H100_NVL |
| NVIDIA H200 | 141 GB | a3-ultragpu-8g-gdc |
nvidia.com/gpu-pod-NVIDIA_H200 |
| NVIDIA B300 | 288 GB | a4-ultragpu-8g-gdc |
nvidia.com/gpu-pod-NVIDIA_B300_SXM6_AC |
每個分頁都會列出模型資訊、寫入模型環境檔案 env.d/model.sh 的指令,以及寫入加速器環境檔案 env.d/accelerator.sh 的巢狀分頁,其中包含所選 GPU 選項、預填的資源類型和 vLLM 設定。如要日後切換模型或 GPU 選項,請編寫不同的環境檔案、再次設定環境,然後重複「實作」一節的步驟。多個模型的運作負載可以在同一個命名空間中並行執行,因為每個資源名稱都是衍生自模型名稱。
為要部署的模型和 GPU 選項,建立模型和加速器環境檔案。
Gemma 4 E2B Instruct
屬性 值 Hugging Face 存放區 google/gemma-4-E2B-it參數 5.1B (2B effective) 授權 Gemma 使用條款 vLLM 使用的權重 Hugging Face 格式 ( model-*.safetensors),約 10.2 GB脈絡長度 128k 個詞元;本指南適用於 --max-model-len 32768建立模型環境檔案:
cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/gemma/env.d/model.sh && echo "Successfully created." || echo "Failed to create!" # Model: Gemma 4 E2B Instruct export GDCS_HF_MODEL="google/gemma-4-E2B-it" export GDCS_HF_MODEL_REVISION="main" # Hugging Face branch, tag, or commit to download export GDCS_HF_DOWNLOAD_EXCLUDE="" EOF為 GPU 選項建立加速器環境檔案:
NVIDIA A100 40 GB
cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/gemma/env.d/accelerator.sh && echo "Successfully created." || echo "Failed to create!" # Accelerator: NVIDIA A100 40 GB export GDCS_WORKLOAD_ACCELERATOR_TYPE="nvidia.com/gpu-pod-NVIDIA_A100_PCIE_40GB" export GDCS_WORKLOAD_ACCELERATOR_COUNT="1" # vLLM export GDCS_VLLM_MAX_MODEL_LEN="32768" export GDCS_VLLM_EXTRA_ARGS="" export GDCS_VLLM_STARTUP_FAILURE_THRESHOLD="60" export GDCS_VLLM_TEST_MAX_TOKENS="128" EOFNVIDIA A100 80 GB
cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/gemma/env.d/accelerator.sh && echo "Successfully created." || echo "Failed to create!" # Accelerator: NVIDIA A100 80 GB export GDCS_WORKLOAD_ACCELERATOR_TYPE="nvidia.com/gpu-pod-NVIDIA_A100_80GB_PCIE" export GDCS_WORKLOAD_ACCELERATOR_COUNT="1" # vLLM export GDCS_VLLM_MAX_MODEL_LEN="32768" export GDCS_VLLM_EXTRA_ARGS="" export GDCS_VLLM_STARTUP_FAILURE_THRESHOLD="60" export GDCS_VLLM_TEST_MAX_TOKENS="128" EOFNVIDIA H100 94 GB
cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/gemma/env.d/accelerator.sh && echo "Successfully created." || echo "Failed to create!" # Accelerator: NVIDIA H100 94 GB export GDCS_WORKLOAD_ACCELERATOR_TYPE="nvidia.com/gpu-pod-NVIDIA_H100_NVL" export GDCS_WORKLOAD_ACCELERATOR_COUNT="1" # vLLM export GDCS_VLLM_MAX_MODEL_LEN="32768" export GDCS_VLLM_EXTRA_ARGS="" export GDCS_VLLM_STARTUP_FAILURE_THRESHOLD="60" export GDCS_VLLM_TEST_MAX_TOKENS="128" EOFNVIDIA H200 141 GB
cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/gemma/env.d/accelerator.sh && echo "Successfully created." || echo "Failed to create!" # Accelerator: NVIDIA H200 141 GB export GDCS_WORKLOAD_ACCELERATOR_TYPE="nvidia.com/gpu-pod-NVIDIA_H200" export GDCS_WORKLOAD_ACCELERATOR_COUNT="1" # vLLM export GDCS_VLLM_MAX_MODEL_LEN="32768" export GDCS_VLLM_EXTRA_ARGS="" export GDCS_VLLM_STARTUP_FAILURE_THRESHOLD="60" export GDCS_VLLM_TEST_MAX_TOKENS="128" EOFNVIDIA B300 288 GB
cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/gemma/env.d/accelerator.sh && echo "Successfully created." || echo "Failed to create!" # Accelerator: NVIDIA B300 288 GB export GDCS_WORKLOAD_ACCELERATOR_TYPE="nvidia.com/gpu-pod-NVIDIA_B300_SXM6_AC" export GDCS_WORKLOAD_ACCELERATOR_COUNT="1" # vLLM export GDCS_VLLM_MAX_MODEL_LEN="32768" export GDCS_VLLM_EXTRA_ARGS="" export GDCS_VLLM_STARTUP_FAILURE_THRESHOLD="60" export GDCS_VLLM_TEST_MAX_TOKENS="128" EOFGemma 4 E4B Instruct
屬性 值 Hugging Face 存放區 google/gemma-4-E4B-it參數 80 億 (有效 40 億) 授權 Gemma 使用條款 vLLM 使用的權重 Hugging Face 格式 ( model-*.safetensors),約 16.0 GB脈絡長度 128k 個詞元;本指南適用於 --max-model-len 32768建立模型環境檔案:
cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/gemma/env.d/model.sh && echo "Successfully created." || echo "Failed to create!" # Model: Gemma 4 E4B Instruct export GDCS_HF_MODEL="google/gemma-4-E4B-it" export GDCS_HF_MODEL_REVISION="main" # Hugging Face branch, tag, or commit to download export GDCS_HF_DOWNLOAD_EXCLUDE="" EOF為 GPU 選項建立加速器環境檔案:
NVIDIA A100 40 GB
cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/gemma/env.d/accelerator.sh && echo "Successfully created." || echo "Failed to create!" # Accelerator: NVIDIA A100 40 GB export GDCS_WORKLOAD_ACCELERATOR_TYPE="nvidia.com/gpu-pod-NVIDIA_A100_PCIE_40GB" export GDCS_WORKLOAD_ACCELERATOR_COUNT="1" # vLLM export GDCS_VLLM_MAX_MODEL_LEN="32768" export GDCS_VLLM_EXTRA_ARGS="" export GDCS_VLLM_STARTUP_FAILURE_THRESHOLD="120" export GDCS_VLLM_TEST_MAX_TOKENS="128" EOFNVIDIA A100 80 GB
cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/gemma/env.d/accelerator.sh && echo "Successfully created." || echo "Failed to create!" # Accelerator: NVIDIA A100 80 GB export GDCS_WORKLOAD_ACCELERATOR_TYPE="nvidia.com/gpu-pod-NVIDIA_A100_80GB_PCIE" export GDCS_WORKLOAD_ACCELERATOR_COUNT="1" # vLLM export GDCS_VLLM_MAX_MODEL_LEN="32768" export GDCS_VLLM_EXTRA_ARGS="" export GDCS_VLLM_STARTUP_FAILURE_THRESHOLD="120" export GDCS_VLLM_TEST_MAX_TOKENS="128" EOFNVIDIA H100 94 GB
cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/gemma/env.d/accelerator.sh && echo "Successfully created." || echo "Failed to create!" # Accelerator: NVIDIA H100 94 GB export GDCS_WORKLOAD_ACCELERATOR_TYPE="nvidia.com/gpu-pod-NVIDIA_H100_NVL" export GDCS_WORKLOAD_ACCELERATOR_COUNT="1" # vLLM export GDCS_VLLM_MAX_MODEL_LEN="32768" export GDCS_VLLM_EXTRA_ARGS="" export GDCS_VLLM_STARTUP_FAILURE_THRESHOLD="120" export GDCS_VLLM_TEST_MAX_TOKENS="128" EOFNVIDIA H200 141 GB
cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/gemma/env.d/accelerator.sh && echo "Successfully created." || echo "Failed to create!" # Accelerator: NVIDIA H200 141 GB export GDCS_WORKLOAD_ACCELERATOR_TYPE="nvidia.com/gpu-pod-NVIDIA_H200" export GDCS_WORKLOAD_ACCELERATOR_COUNT="1" # vLLM export GDCS_VLLM_MAX_MODEL_LEN="32768" export GDCS_VLLM_EXTRA_ARGS="" export GDCS_VLLM_STARTUP_FAILURE_THRESHOLD="120" export GDCS_VLLM_TEST_MAX_TOKENS="128" EOFNVIDIA B300 288 GB
cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/gemma/env.d/accelerator.sh && echo "Successfully created." || echo "Failed to create!" # Accelerator: NVIDIA B300 288 GB export GDCS_WORKLOAD_ACCELERATOR_TYPE="nvidia.com/gpu-pod-NVIDIA_B300_SXM6_AC" export GDCS_WORKLOAD_ACCELERATOR_COUNT="1" # vLLM export GDCS_VLLM_MAX_MODEL_LEN="32768" export GDCS_VLLM_EXTRA_ARGS="" export GDCS_VLLM_STARTUP_FAILURE_THRESHOLD="120" export GDCS_VLLM_TEST_MAX_TOKENS="128" EOFGemma 4 12B Instruct
屬性 值 Hugging Face 存放區 google/gemma-4-12B-it參數 12B (密集) 授權 Gemma 使用條款 vLLM 使用的權重 Hugging Face 格式 ( model-*.safetensors),約 23.9 GB脈絡長度 25.6 萬個詞元;本指南使用 --max-model-len 32768建立模型環境檔案:
cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/gemma/env.d/model.sh && echo "Successfully created." || echo "Failed to create!" # Model: Gemma 4 12B Instruct export GDCS_HF_MODEL="google/gemma-4-12B-it" export GDCS_HF_MODEL_REVISION="main" # Hugging Face branch, tag, or commit to download export GDCS_HF_DOWNLOAD_EXCLUDE="" EOF為 GPU 選項建立加速器環境檔案:
NVIDIA A100 40 GB
cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/gemma/env.d/accelerator.sh && echo "Successfully created." || echo "Failed to create!" # Accelerator: NVIDIA A100 40 GB export GDCS_WORKLOAD_ACCELERATOR_TYPE="nvidia.com/gpu-pod-NVIDIA_A100_PCIE_40GB" export GDCS_WORKLOAD_ACCELERATOR_COUNT="1" # vLLM export GDCS_VLLM_MAX_MODEL_LEN="32768" export GDCS_VLLM_EXTRA_ARGS="" export GDCS_VLLM_STARTUP_FAILURE_THRESHOLD="180" export GDCS_VLLM_TEST_MAX_TOKENS="128" EOFNVIDIA A100 80 GB
cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/gemma/env.d/accelerator.sh && echo "Successfully created." || echo "Failed to create!" # Accelerator: NVIDIA A100 80 GB export GDCS_WORKLOAD_ACCELERATOR_TYPE="nvidia.com/gpu-pod-NVIDIA_A100_80GB_PCIE" export GDCS_WORKLOAD_ACCELERATOR_COUNT="1" # vLLM export GDCS_VLLM_MAX_MODEL_LEN="32768" export GDCS_VLLM_EXTRA_ARGS="" export GDCS_VLLM_STARTUP_FAILURE_THRESHOLD="180" export GDCS_VLLM_TEST_MAX_TOKENS="128" EOFNVIDIA H100 94 GB
cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/gemma/env.d/accelerator.sh && echo "Successfully created." || echo "Failed to create!" # Accelerator: NVIDIA H100 94 GB export GDCS_WORKLOAD_ACCELERATOR_TYPE="nvidia.com/gpu-pod-NVIDIA_H100_NVL" export GDCS_WORKLOAD_ACCELERATOR_COUNT="1" # vLLM export GDCS_VLLM_MAX_MODEL_LEN="32768" export GDCS_VLLM_EXTRA_ARGS="" export GDCS_VLLM_STARTUP_FAILURE_THRESHOLD="180" export GDCS_VLLM_TEST_MAX_TOKENS="128" EOFNVIDIA H200 141 GB
cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/gemma/env.d/accelerator.sh && echo "Successfully created." || echo "Failed to create!" # Accelerator: NVIDIA H200 141 GB export GDCS_WORKLOAD_ACCELERATOR_TYPE="nvidia.com/gpu-pod-NVIDIA_H200" export GDCS_WORKLOAD_ACCELERATOR_COUNT="1" # vLLM export GDCS_VLLM_MAX_MODEL_LEN="32768" export GDCS_VLLM_EXTRA_ARGS="" export GDCS_VLLM_STARTUP_FAILURE_THRESHOLD="180" export GDCS_VLLM_TEST_MAX_TOKENS="128" EOFNVIDIA B300 288 GB
cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/gemma/env.d/accelerator.sh && echo "Successfully created." || echo "Failed to create!" # Accelerator: NVIDIA B300 288 GB export GDCS_WORKLOAD_ACCELERATOR_TYPE="nvidia.com/gpu-pod-NVIDIA_B300_SXM6_AC" export GDCS_WORKLOAD_ACCELERATOR_COUNT="1" # vLLM export GDCS_VLLM_MAX_MODEL_LEN="32768" export GDCS_VLLM_EXTRA_ARGS="" export GDCS_VLLM_STARTUP_FAILURE_THRESHOLD="180" export GDCS_VLLM_TEST_MAX_TOKENS="128" EOFGemma 4 26B-A4B Instruct
屬性 值 Hugging Face 存放區 google/gemma-4-26B-A4B-it參數 258 億個參數/40 億個有效參數 (混合專家模型) 授權 Gemma 使用條款 vLLM 使用的權重 Hugging Face 格式 ( model-*.safetensors),約 51.6 GB脈絡長度 25.6 萬個詞元;本指南使用 --max-model-len 32768建立模型環境檔案:
cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/gemma/env.d/model.sh && echo "Successfully created." || echo "Failed to create!" # Model: Gemma 4 26B-A4B Instruct export GDCS_HF_MODEL="google/gemma-4-26B-A4B-it" export GDCS_HF_MODEL_REVISION="main" # Hugging Face branch, tag, or commit to download export GDCS_HF_DOWNLOAD_EXCLUDE="" EOF為 GPU 選項建立加速器環境檔案:
NVIDIA A100 80 GB
cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/gemma/env.d/accelerator.sh && echo "Successfully created." || echo "Failed to create!" # Accelerator: NVIDIA A100 80 GB export GDCS_WORKLOAD_ACCELERATOR_TYPE="nvidia.com/gpu-pod-NVIDIA_A100_80GB_PCIE" export GDCS_WORKLOAD_ACCELERATOR_COUNT="1" # vLLM export GDCS_VLLM_MAX_MODEL_LEN="32768" export GDCS_VLLM_EXTRA_ARGS="" export GDCS_VLLM_STARTUP_FAILURE_THRESHOLD="300" export GDCS_VLLM_TEST_MAX_TOKENS="128" EOFNVIDIA H100 94 GB
cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/gemma/env.d/accelerator.sh && echo "Successfully created." || echo "Failed to create!" # Accelerator: NVIDIA H100 94 GB export GDCS_WORKLOAD_ACCELERATOR_TYPE="nvidia.com/gpu-pod-NVIDIA_H100_NVL" export GDCS_WORKLOAD_ACCELERATOR_COUNT="1" # vLLM export GDCS_VLLM_MAX_MODEL_LEN="32768" export GDCS_VLLM_EXTRA_ARGS="" export GDCS_VLLM_STARTUP_FAILURE_THRESHOLD="300" export GDCS_VLLM_TEST_MAX_TOKENS="128" EOFNVIDIA H200 141 GB
cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/gemma/env.d/accelerator.sh && echo "Successfully created." || echo "Failed to create!" # Accelerator: NVIDIA H200 141 GB export GDCS_WORKLOAD_ACCELERATOR_TYPE="nvidia.com/gpu-pod-NVIDIA_H200" export GDCS_WORKLOAD_ACCELERATOR_COUNT="1" # vLLM export GDCS_VLLM_MAX_MODEL_LEN="32768" export GDCS_VLLM_EXTRA_ARGS="" export GDCS_VLLM_STARTUP_FAILURE_THRESHOLD="300" export GDCS_VLLM_TEST_MAX_TOKENS="128" EOFNVIDIA B300 288 GB
cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/gemma/env.d/accelerator.sh && echo "Successfully created." || echo "Failed to create!" # Accelerator: NVIDIA B300 288 GB export GDCS_WORKLOAD_ACCELERATOR_TYPE="nvidia.com/gpu-pod-NVIDIA_B300_SXM6_AC" export GDCS_WORKLOAD_ACCELERATOR_COUNT="1" # vLLM export GDCS_VLLM_MAX_MODEL_LEN="32768" export GDCS_VLLM_EXTRA_ARGS="" export GDCS_VLLM_STARTUP_FAILURE_THRESHOLD="300" export GDCS_VLLM_TEST_MAX_TOKENS="128" EOFGemma 4 31B Instruct
屬性 值 Hugging Face 存放區 google/gemma-4-31B-it參數 31.3B (密集) 授權 Gemma 使用條款 vLLM 使用的權重 Hugging Face 格式 ( model-*.safetensors),約 62.5 GB脈絡長度 25.6 萬個詞元;本指南使用 --max-model-len 32768建立模型環境檔案:
cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/gemma/env.d/model.sh && echo "Successfully created." || echo "Failed to create!" # Model: Gemma 4 31B Instruct export GDCS_HF_MODEL="google/gemma-4-31B-it" export GDCS_HF_MODEL_REVISION="main" # Hugging Face branch, tag, or commit to download export GDCS_HF_DOWNLOAD_EXCLUDE="" EOF為 GPU 選項建立加速器環境檔案:
NVIDIA A100 80 GB
在 80 GB GPU 上提供服務時,KV 快取可用的記憶體很少,因此這個選項會設定
--max-model-len 16384。cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/gemma/env.d/accelerator.sh && echo "Successfully created." || echo "Failed to create!" # Accelerator: NVIDIA A100 80 GB export GDCS_WORKLOAD_ACCELERATOR_TYPE="nvidia.com/gpu-pod-NVIDIA_A100_80GB_PCIE" export GDCS_WORKLOAD_ACCELERATOR_COUNT="1" # vLLM export GDCS_VLLM_MAX_MODEL_LEN="16384" export GDCS_VLLM_EXTRA_ARGS="" export GDCS_VLLM_STARTUP_FAILURE_THRESHOLD="360" export GDCS_VLLM_TEST_MAX_TOKENS="128" EOFNVIDIA H100 94 GB
cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/gemma/env.d/accelerator.sh && echo "Successfully created." || echo "Failed to create!" # Accelerator: NVIDIA H100 94 GB export GDCS_WORKLOAD_ACCELERATOR_TYPE="nvidia.com/gpu-pod-NVIDIA_H100_NVL" export GDCS_WORKLOAD_ACCELERATOR_COUNT="1" # vLLM export GDCS_VLLM_MAX_MODEL_LEN="32768" export GDCS_VLLM_EXTRA_ARGS="" export GDCS_VLLM_STARTUP_FAILURE_THRESHOLD="360" export GDCS_VLLM_TEST_MAX_TOKENS="128" EOFNVIDIA H200 141 GB
cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/gemma/env.d/accelerator.sh && echo "Successfully created." || echo "Failed to create!" # Accelerator: NVIDIA H200 141 GB export GDCS_WORKLOAD_ACCELERATOR_TYPE="nvidia.com/gpu-pod-NVIDIA_H200" export GDCS_WORKLOAD_ACCELERATOR_COUNT="1" # vLLM export GDCS_VLLM_MAX_MODEL_LEN="32768" export GDCS_VLLM_EXTRA_ARGS="" export GDCS_VLLM_STARTUP_FAILURE_THRESHOLD="360" export GDCS_VLLM_TEST_MAX_TOKENS="128" EOFNVIDIA B300 288 GB
cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/gemma/env.d/accelerator.sh && echo "Successfully created." || echo "Failed to create!" # Accelerator: NVIDIA B300 288 GB export GDCS_WORKLOAD_ACCELERATOR_TYPE="nvidia.com/gpu-pod-NVIDIA_B300_SXM6_AC" export GDCS_WORKLOAD_ACCELERATOR_COUNT="1" # vLLM export GDCS_VLLM_MAX_MODEL_LEN="32768" export GDCS_VLLM_EXTRA_ARGS="" export GDCS_VLLM_STARTUP_FAILURE_THRESHOLD="360" export GDCS_VLLM_TEST_MAX_TOKENS="128" EOF提供環境檔案來源:
source ${HOME}/gdcag-solutions/open-weight-models/vllm/gemma/env.sh輸出結果會與下列內容相似:
GDCS_ROOT_HOME=HOME_DIRECTORY_PATH/gdcag-solutions GDCS_IMPLEMENTATION_HOME=HOME_DIRECTORY_PATH/gdcag-solutions/open-weight-models/vllm GDCS_USER_GUIDE_HOME=HOME_DIRECTORY_PATH/gdcag-solutions/open-weight-models/vllm/gemma GDCS_HF_MODEL=HUGGING_FACE_REPOSITORY (ACCELERATOR_COUNT x ACCELERATOR_RESOURCE_TYPE)
叢集
設定
gdcloud:gdcloud config set core/account "default-user" gdcloud config set core/organization_console_url "${GDC_ZONAL_CONSOLE_URL}" gdcloud config set core/project "${GDC_PROJECT}" gdcloud config set core/zone "${GDC_ZONE}"向 GDC 環境進行驗證:
gdcloud auth login擷取叢集憑證:
gdcloud clusters get-credentials "${GDC_STANDARD_CLUSTER_NAME}" \ --project="${GDC_PROJECT}" \ --standard \ --zone="${GDC_ZONE}"驗證與叢集的連線:
kubectl get nodes -L node.cluster.private.gdc.goog/machine-class確認叢集中有加速器環境檔案的加速器資源類型。這個指令會列出節點公開的加速器資源類型 (名稱取決於安裝的 GPU 模型;在沒有 GPU 節點的叢集上,這個名稱會是空白) 和設定的值:
kubectl get nodes --output=json | jq -r '[.items[].status.allocatable | keys[] | select(startswith("nvidia.com/"))] | unique[]' echo "GDCS_WORKLOAD_ACCELERATOR_TYPE=${GDCS_WORKLOAD_ACCELERATOR_TYPE}"
實作
命名空間
建立命名空間:
kubectl create namespace "${GDCS_WORKLOAD_NAMESPACE}" \ --dry-run=client \ --output=yaml | kubectl apply -f -新增
imagePullSecret:kubectl create secret docker-registry "${GDCS_HARBOR_K8S_PULL_SECRET}" \ --dry-run=client \ --from-file=.dockerconfigjson=${GDCS_HARBOR_K8S_DOCKER_CONFIG}/config.json \ --namespace="${GDCS_WORKLOAD_NAMESPACE}" \ --output=yaml | kubectl apply -f -
模型權重
下載模型權重 (僅限 vLLM 所需的所選模型檔案,
GDCS_HF_DOWNLOAD_EXCLUDE會略過其餘檔案),並移除.cache目錄:EXCLUDE_ARGS=() if [[ -n "${GDCS_HF_DOWNLOAD_EXCLUDE:-}" ]]; then read -r -a EXCLUDE_PATTERNS <<< "${GDCS_HF_DOWNLOAD_EXCLUDE}" EXCLUDE_ARGS=(--exclude "${EXCLUDE_PATTERNS[@]}") fi hf download "${GDCS_HF_MODEL}" --revision "${GDCS_HF_MODEL_REVISION}" --local-dir "${GDCS_HF_MODEL_DOWNLOAD_DIR}" "${EXCLUDE_ARGS[@]}" && \ rm -rf "${GDCS_HF_MODEL_DOWNLOAD_DIR}/.cache"根據下載的模型權重,計算所需的
PersistentVolumeClaim大小,並加上 25% 的額外空間 (無條件進位至下一個整數 GiB):MODEL_KIB=$(du -sk "${GDCS_HF_MODEL_DOWNLOAD_DIR}" | awk '{print $1}') export GDCS_MODEL_PVC_SIZE="$(( (MODEL_KIB * 125 / 100 + 1048575) / 1048576 ))Gi" echo "GDCS_MODEL_PVC_SIZE=${GDCS_MODEL_PVC_SIZE}"為模型權重建立 ReadWriteOnce PersistentVolumeClaim 的資訊清單:
cat <<EOF > "${GDCS_USER_GUIDE_HOME}/${GDCS_WORKLOAD_NAME}-rwo-pvc.yaml" apiVersion: v1 kind: PersistentVolumeClaim metadata: name: ${GDCS_WORKLOAD_NAME}-rwo spec: accessModes: - ReadWriteOnce resources: requests: storage: ${GDCS_MODEL_PVC_SIZE} storageClassName: standard-rwo volumeMode: Filesystem EOF為模型權重套用 PersistentVolumeClaim 的資訊清單:
kubectl apply \ --filename="${GDCS_USER_GUIDE_HOME}/${GDCS_WORKLOAD_NAME}-rwo-pvc.yaml" \ --namespace="${GDCS_WORKLOAD_NAMESPACE}"等待
PersistentVolumeClaim繫結並輸出基礎架構運算子所需的值,以調整 I/O 服務品質 (QoS) 政策:kubectl wait \ --for=jsonpath='{.status.phase}'=Bound \ "pvc/${GDCS_WORKLOAD_NAME}-rwo" \ --namespace="${GDCS_WORKLOAD_NAMESPACE}" \ --timeout=120s BACKING_PVC=$(kubectl get pvc "${GDCS_WORKLOAD_NAME}-rwo" \ --namespace="${GDCS_WORKLOAD_NAMESPACE}" \ --output=jsonpath='{.spec.volumeName}') echo "BACKING_PVC=${BACKING_PVC} ORG=${GDC_ORG}"如果叢集不支援
performance-rwo,請要求基礎架構操作員將高效能 QoS 政策 (system-performance-<ORG>-admin,每 GiB 30 IOPS,最低 6,148 IOPS) 套用至支援PersistentVolumeClaim的磁碟區。提供上一個步驟輸出的BACKING_PVC和ORG值。基礎架構操作員會按照基礎架構操作員說明文件中的 FILE-R0097 執行手冊套用政策,並在實際磁碟區上生效,因此您可以繼續執行後續步驟。為模型載入器 Pod 建立資訊清單:
cat <<EOF > "${GDCS_USER_GUIDE_HOME}/${GDCS_MODEL_LOADER_POD_NAME}-pod.yaml" apiVersion: v1 kind: Pod metadata: name: ${GDCS_MODEL_LOADER_POD_NAME} spec: containers: - name: uploader image: ${GDCS_HARBOR_PROJECT_URI}/library/busybox:${GDCS_BUSYBOX_IMAGE_TAG} command: ["sleep", "infinity"] volumeMounts: - name: model-data mountPath: ${GDCS_HF_PV_MOUNT_PATH} imagePullSecrets: - name: ${GDCS_HARBOR_K8S_PULL_SECRET} volumes: - name: model-data persistentVolumeClaim: claimName: ${GDCS_WORKLOAD_NAME}-rwo EOF套用模型載入器 Pod 的資訊清單:
kubectl apply \ --filename="${GDCS_USER_GUIDE_HOME}/${GDCS_MODEL_LOADER_POD_NAME}-pod.yaml" \ --namespace="${GDCS_WORKLOAD_NAMESPACE}"等待模型載入器 Pod 進入「執行中」狀態:
watch --color --interval 5 --no-title \ "kubectl get pod/${GDCS_MODEL_LOADER_POD_NAME} \ --namespace=${GDCS_WORKLOAD_NAMESPACE} | GREP_COLORS='mt=01;92' grep -E --color=always -e '^' -e 'Running'"將模型權重上傳至
PersistentVolume(指令碼會略過大小相同的現有檔案,因此可以重新執行):export LOCAL_MODEL_DIR="${GDCS_HF_MODEL_DOWNLOAD_DIR}" export REMOTE_MODEL_DIR="${GDCS_HF_PV_MOUNT_PATH}/${GDCS_HF_MODEL_LOWERCASE}" export REMOTE_MOUNT_DIR="${GDCS_HF_PV_MOUNT_PATH}" ${GDCS_IMPLEMENTATION_HOME}/model_weight_upload.sh刪除模型載入器 Pod:
kubectl delete pod/${GDCS_MODEL_LOADER_POD_NAME} \ --namespace="${GDCS_WORKLOAD_NAMESPACE}"
推論伺服器
建立 vLLM Deployment 的資訊清單。ConfigMap 會從環境中擷取模型專屬的 vLLM 設定,因此每個模型和 GPU 選項的資訊清單都相同:
cat <<EOF > "${GDCS_USER_GUIDE_HOME}/${GDCS_WORKLOAD_NAME}-deployment.yaml" apiVersion: v1 kind: ConfigMap metadata: name: ${GDCS_WORKLOAD_NAME} data: MAX_MODEL_LEN: "${GDCS_VLLM_MAX_MODEL_LEN}" MODEL: "${GDCS_HF_PV_MOUNT_PATH}/${GDCS_HF_MODEL_LOWERCASE}" PORT: "8000" TENSOR_PARALLEL_SIZE: "${GDCS_WORKLOAD_ACCELERATOR_COUNT}" VLLM_EXTRA_ARGS: "${GDCS_VLLM_EXTRA_ARGS}" --- apiVersion: apps/v1 kind: Deployment metadata: name: ${GDCS_WORKLOAD_NAME} labels: app: ${GDCS_WORKLOAD_NAME} spec: replicas: 1 selector: matchLabels: app: ${GDCS_WORKLOAD_NAME} strategy: type: Recreate template: metadata: labels: app: ${GDCS_WORKLOAD_NAME} spec: containers: - command: - /bin/sh - -c args: - exec vllm serve "\$MODEL" --port="\$PORT" --tensor-parallel-size="\$TENSOR_PARALLEL_SIZE" --max-model-len="\$MAX_MODEL_LEN" \$VLLM_EXTRA_ARGS envFrom: - configMapRef: name: ${GDCS_WORKLOAD_NAME} image: ${GDCS_HARBOR_PROJECT_URI}/vllm/vllm-openai:${GDCS_VLLM_IMAGE_TAG}-fips livenessProbe: failureThreshold: 3 httpGet: path: /health port: 8000 periodSeconds: 20 timeoutSeconds: 5 name: vllm ports: - containerPort: 8000 readinessProbe: failureThreshold: 3 httpGet: path: /health port: 8000 periodSeconds: 10 successThreshold: 1 timeoutSeconds: 5 resources: limits: ${GDCS_WORKLOAD_ACCELERATOR_TYPE}: "${GDCS_WORKLOAD_ACCELERATOR_COUNT}" requests: ${GDCS_WORKLOAD_ACCELERATOR_TYPE}: "${GDCS_WORKLOAD_ACCELERATOR_COUNT}" startupProbe: httpGet: path: /health port: 8000 failureThreshold: ${GDCS_VLLM_STARTUP_FAILURE_THRESHOLD} periodSeconds: 10 timeoutSeconds: 5 volumeMounts: - name: model-weights mountPath: ${GDCS_HF_PV_MOUNT_PATH} - name: shm mountPath: /dev/shm securityContext: seLinuxOptions: type: spc_t imagePullSecrets: - name: ${GDCS_HARBOR_K8S_PULL_SECRET} volumes: - name: model-weights persistentVolumeClaim: claimName: ${GDCS_WORKLOAD_NAME}-rwo - name: shm emptyDir: medium: Memory sizeLimit: "8Gi" --- apiVersion: v1 kind: Service metadata: name: ${GDCS_WORKLOAD_NAME} spec: ports: - name: vllm-http port: 80 protocol: TCP targetPort: 8000 selector: app: ${GDCS_WORKLOAD_NAME} type: LoadBalancer EOF套用 vLLM Deployment 的資訊清單:
kubectl apply \ --filename="${GDCS_USER_GUIDE_HOME}/${GDCS_WORKLOAD_NAME}-deployment.yaml" \ --namespace="${GDCS_WORKLOAD_NAMESPACE}"建立 vLLM
NetworkPolicy的資訊清單:cat << EOF > "${GDCS_USER_GUIDE_HOME}/vllm-networkpolicy.yaml" && echo "Successfully created." || echo "Failed to create!" apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: allow-vllm-ingress namespace: ${GDCS_WORKLOAD_NAMESPACE} spec: ingress: - from: - ipBlock: cidr: 0.0.0.0/0 - namespaceSelector: {} ports: - port: 8000 protocol: TCP podSelector: {} policyTypes: - Ingress EOF套用 vLLM
NetworkPolicy的資訊清單:kubectl apply \ --filename="${GDCS_USER_GUIDE_HOME}/vllm-networkpolicy.yaml"
等待 Pod 進入「執行中」狀態:
watch --color --interval 5 --no-title \ "kubectl get pods --selector=app=${GDCS_WORKLOAD_NAME} \ --namespace=${GDCS_WORKLOAD_NAMESPACE} | GREP_COLORS='mt=01;92' grep -E --color=always -e '^' -e 'Running'"等待 Deployment 準備就緒:
watch --color --interval 5 --no-title \ "kubectl get deployment/${GDCS_WORKLOAD_NAME} \ --namespace=${GDCS_WORKLOAD_NAMESPACE} | GREP_COLORS='mt=01;92' grep -E --color=always -e '^' -e '1/1 1 1'"選用:串流容器記錄,監控啟動進度:
kubectl logs deployment/${GDCS_WORKLOAD_NAME} \ --follow \ --namespace="${GDCS_WORKLOAD_NAMESPACE}"
驗證
擷取 vLLM 的 LoadBalancer IP 位址:
export VLLM_LB_IP=$(kubectl get service/${GDCS_WORKLOAD_NAME} --namespace="${GDCS_WORKLOAD_NAMESPACE}" --output=jsonpath='{.status.loadBalancer.ingress[*].ip}') echo "VLLM_LB_IP=${VLLM_LB_IP}"傳送測試要求至 vLLM。
max_tokens值是env.d/accelerator.sh的GDCS_VLLM_TEST_MAX_TOKENS:curl http://${VLLM_LB_IP}/v1/models curl http://${VLLM_LB_IP}/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "'${GDCS_HF_PV_MOUNT_PATH}/${GDCS_HF_MODEL_LOWERCASE}'", "messages": [ {"role": "user", "content": "Explain Google Distributed Cloud air-gapped briefly."} ], "max_tokens": '${GDCS_VLLM_TEST_MAX_TOKENS}' }'如果無法從工作站連上 LoadBalancer IP 位址 (例如沒有
ProjectNetworkPolicy允許),請透過通訊埠轉送傳送要求:kubectl port-forward service/${GDCS_WORKLOAD_NAME} \ --namespace="${GDCS_WORKLOAD_NAMESPACE}" 8000:80 & PF_PID=$! sleep 1 curl http://127.0.0.1:8000/v1/models kill ${PF_PID}