在 GDC 實體隔離方案使用 Gemma 模型搭配 vLLM 的使用手冊

本文件提供逐步說明,協助您在 Google Distributed Cloud (GDC) 實體隔離環境中,部署 Google DeepMind 的開放權重模型,特別是 Gemma 4 系列。課程內容涵蓋使用 vLLM 進行高處理量服務,以及運用 GDC 平台的功能,包括 Kubernetes、Harbor 和 GPU 資源。在「Model」(模型) 區段中選取模型和 GPU 選項;所有模型的其他步驟都相同。

支援的模型

模型 Hugging Face 存放區 參數 授權 GPU 選項
Gemma 4 E2B Instruct google/gemma-4-E2B-it 5.1B (2B effective) Gemma 使用條款 1 個任何 GPU
Gemma 4 E4B Instruct google/gemma-4-E4B-it 80 億 (有效 40 億) Gemma 使用條款 1 個任何 GPU
Gemma 4 12B Instruct google/gemma-4-12B-it 12B (密集) Gemma 使用條款 1 個任何 GPU
Gemma 4 26B-A4B Instruct google/gemma-4-26B-A4B-it 258 億個參數/40 億個有效參數 (混合專家模型) Gemma 使用條款 1 個 NVIDIA A100 80 GB
1 個 NVIDIA H100 94 GB
1 個 NVIDIA H200 141 GB 或 B300 288 GB
Gemma 4 31B Instruct google/gemma-4-31B-it 31.3B (密集) Gemma 使用條款 1 個 NVIDIA A100 80 GB
1 個 NVIDIA H100 94 GB
1 個 NVIDIA H200 141 GB 或 B300 288 GB

架構

這項解決方案會將容器化服務後端 (vLLM) 部署為標準叢集中的 Deployment。模型權重會儲存在永久磁碟區,並從本機下載的檔案填入,然後使用載入器 Pod 上傳。LoadBalancer 類型的 Kubernetes Service 會公開後端 API,而 NetworkPolicy/ProjectNetworkPolicy 物件則可保護對該 API 的存取。

事前準備

在工作站上完成使用 vLLM 參考實作開放權重模型,並確認您有權存取 ${HOME}/gdcag-solutions 目錄。

Identity and Access Management

請確認已正確設定必要的 IAM 帳戶、角色和權限。

組織的 GDC 使用者角色:

  • 選用:標準叢集管理員 (從控制台查看叢集)

專案的 GDC 使用者角色:

  • 標準叢集管理員 (standard-cluster-admin)

標準叢集的 GDC 使用者角色:

  • StandardClusterRoleBinding 至 StandardClusterRole cluster-admin (由 IAM 專案管理員建立,請參閱參考實作的 Identity and Access Management 部分)

工作站

除非步驟另有說明,否則請在工作站執行本指南中的所有指令,並確保工作站具備必要工具,且可連線至環境和網際網路。

環境設定分為多層:

  • ${HOME}/gdcag-solutions/env.sh 會載入平台和登錄設定。
  • ${HOME}/gdcag-solutions/open-weight-models/vllm/env.sh 會新增 vLLM 設定。

  • 本使用手冊會新增 env.d/model.sh (模型) 和 env.d/accelerator.sh (GPU 選項和執行階段設定),您可以在「模型」部分編寫這些內容;以及 env.d/model-common.sh (從中衍生的值)。

  1. 建立使用手冊目錄架構:

    mkdir -p ${HOME}/gdcag-solutions/open-weight-models/vllm/gemma/env.d
    
  2. 建立通用模型環境設定檔。它會從 GDCS_HF_MODEL 衍生每個路徑和資源名稱,因此所有模型都相同:

    cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/gemma/env.d/model-common.sh && echo "Successfully created." || echo "Failed to create!"
    # Model (derived from env.d/model.sh)
    export GDCS_HF_DOWNLOAD_PATH="${GDCS_USER_GUIDE_HOME}/hf-models"
    export GDCS_HF_MODEL_LOWERCASE="${GDCS_HF_MODEL,,}"
    export GDCS_HF_MODEL_DOWNLOAD_DIR="${GDCS_HF_DOWNLOAD_PATH}/${GDCS_HF_MODEL_LOWERCASE}"
    export GDCS_HF_PV_MOUNT_PATH="/mnt/hf-model"
    
    GDCS_HF_MODEL_KUBERNETES="${GDCS_HF_MODEL_LOWERCASE//\//-}"
    export GDCS_HF_MODEL_KUBERNETES="${GDCS_HF_MODEL_KUBERNETES//./-}"
    
    # Workload
    export GDCS_WORKLOAD_NAMESPACE="vllm-gemma"
    export GDCS_WORKLOAD_NAME="${GDCS_HF_MODEL_KUBERNETES}"
    export GDCS_MODEL_LOADER_POD_NAME="loader-${GDCS_HF_MODEL_KUBERNETES}"
    EOF
    
  3. 建立使用手冊環境載入器檔案:

    cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/gemma/env.sh && echo "Successfully created." || echo "Failed to create!"
    source "${HOME}/gdcag-solutions/open-weight-models/vllm/env.sh"
    
    export GDCS_USER_GUIDE_HOME="${HOME}/gdcag-solutions/open-weight-models/vllm/gemma"
    echo "GDCS_USER_GUIDE_HOME=${GDCS_USER_GUIDE_HOME}"
    
    # Sourced in dependency order: the model selection, the accelerator and runtime settings, then the values derived from them
    source "${GDCS_USER_GUIDE_HOME}/env.d/model.sh"
    source "${GDCS_USER_GUIDE_HOME}/env.d/accelerator.sh"
    source "${GDCS_USER_GUIDE_HOME}/env.d/model-common.sh"
    echo "GDCS_HF_MODEL=${GDCS_HF_MODEL} (${GDCS_WORKLOAD_ACCELERATOR_COUNT} x ${GDCS_WORKLOAD_ACCELERATOR_TYPE:-unset})"
    EOF
    

模型

選取與標準叢集 GPU 節點相符的模型和 GPU 選項。GDC 實體隔離方案提供下列 GPU (請參閱「叢集節點機型」);加速器資源類型是節點向 Kubernetes 公開 GPU 的名稱。

GPU 記憶體 機型 加速器資源類型
NVIDIA A100 40 GB a2-highgpu-1g-gdc nvidia.com/gpu-pod-NVIDIA_A100_PCIE_40GB
NVIDIA A100 80 GB a2-ultragpu-1g-gdc、a2-ultragpu-2g-gdc nvidia.com/gpu-pod-NVIDIA_A100_80GB_PCIE
NVIDIA H100 94 GB a3-highgpu-1g-gdc (預先發布版)、a3-highgpu-2g-gdc (預先發布版)、a3-highgpu-4g-gdc nvidia.com/gpu-pod-NVIDIA_H100_NVL
NVIDIA H200 141 GB a3-ultragpu-8g-gdc nvidia.com/gpu-pod-NVIDIA_H200
NVIDIA B300 288 GB a4-ultragpu-8g-gdc nvidia.com/gpu-pod-NVIDIA_B300_SXM6_AC

每個分頁都會列出模型資訊、寫入模型環境檔案 env.d/model.sh 的指令,以及寫入加速器環境檔案 env.d/accelerator.sh 的巢狀分頁,其中包含所選 GPU 選項、預填的資源類型和 vLLM 設定。如要日後切換模型或 GPU 選項,請編寫不同的環境檔案、再次設定環境,然後重複「實作」一節的步驟。多個模型的運作負載可以在同一個命名空間中並行執行,因為每個資源名稱都是衍生自模型名稱。

  1. 為要部署的模型和 GPU 選項,建立模型和加速器環境檔案。

    Gemma 4 E2B Instruct

    屬性 值
    Hugging Face 存放區 google/gemma-4-E2B-it
    參數 5.1B (2B effective)
    授權 Gemma 使用條款
    vLLM 使用的權重 Hugging Face 格式 (model-*.safetensors),約 10.2 GB
    脈絡長度 128k 個詞元;本指南適用於 --max-model-len 32768

    建立模型環境檔案:

    cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/gemma/env.d/model.sh && echo "Successfully created." || echo "Failed to create!"
    # Model: Gemma 4 E2B Instruct
    export GDCS_HF_MODEL="google/gemma-4-E2B-it"
    export GDCS_HF_MODEL_REVISION="main"  # Hugging Face branch, tag, or commit to download
    export GDCS_HF_DOWNLOAD_EXCLUDE=""
    EOF
    

    為 GPU 選項建立加速器環境檔案:

    NVIDIA A100 40 GB

    cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/gemma/env.d/accelerator.sh && echo "Successfully created." || echo "Failed to create!"
    # Accelerator: NVIDIA A100 40 GB
    export GDCS_WORKLOAD_ACCELERATOR_TYPE="nvidia.com/gpu-pod-NVIDIA_A100_PCIE_40GB"
    export GDCS_WORKLOAD_ACCELERATOR_COUNT="1"
    
    # vLLM
    export GDCS_VLLM_MAX_MODEL_LEN="32768"
    export GDCS_VLLM_EXTRA_ARGS=""
    export GDCS_VLLM_STARTUP_FAILURE_THRESHOLD="60"
    export GDCS_VLLM_TEST_MAX_TOKENS="128"
    EOF
    

    NVIDIA A100 80 GB

    cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/gemma/env.d/accelerator.sh && echo "Successfully created." || echo "Failed to create!"
    # Accelerator: NVIDIA A100 80 GB
    export GDCS_WORKLOAD_ACCELERATOR_TYPE="nvidia.com/gpu-pod-NVIDIA_A100_80GB_PCIE"
    export GDCS_WORKLOAD_ACCELERATOR_COUNT="1"
    
    # vLLM
    export GDCS_VLLM_MAX_MODEL_LEN="32768"
    export GDCS_VLLM_EXTRA_ARGS=""
    export GDCS_VLLM_STARTUP_FAILURE_THRESHOLD="60"
    export GDCS_VLLM_TEST_MAX_TOKENS="128"
    EOF
    

    NVIDIA H100 94 GB

    cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/gemma/env.d/accelerator.sh && echo "Successfully created." || echo "Failed to create!"
    # Accelerator: NVIDIA H100 94 GB
    export GDCS_WORKLOAD_ACCELERATOR_TYPE="nvidia.com/gpu-pod-NVIDIA_H100_NVL"
    export GDCS_WORKLOAD_ACCELERATOR_COUNT="1"
    
    # vLLM
    export GDCS_VLLM_MAX_MODEL_LEN="32768"
    export GDCS_VLLM_EXTRA_ARGS=""
    export GDCS_VLLM_STARTUP_FAILURE_THRESHOLD="60"
    export GDCS_VLLM_TEST_MAX_TOKENS="128"
    EOF
    

    NVIDIA H200 141 GB

    cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/gemma/env.d/accelerator.sh && echo "Successfully created." || echo "Failed to create!"
    # Accelerator: NVIDIA H200 141 GB
    export GDCS_WORKLOAD_ACCELERATOR_TYPE="nvidia.com/gpu-pod-NVIDIA_H200"
    export GDCS_WORKLOAD_ACCELERATOR_COUNT="1"
    
    # vLLM
    export GDCS_VLLM_MAX_MODEL_LEN="32768"
    export GDCS_VLLM_EXTRA_ARGS=""
    export GDCS_VLLM_STARTUP_FAILURE_THRESHOLD="60"
    export GDCS_VLLM_TEST_MAX_TOKENS="128"
    EOF
    

    NVIDIA B300 288 GB

    cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/gemma/env.d/accelerator.sh && echo "Successfully created." || echo "Failed to create!"
    # Accelerator: NVIDIA B300 288 GB
    export GDCS_WORKLOAD_ACCELERATOR_TYPE="nvidia.com/gpu-pod-NVIDIA_B300_SXM6_AC"
    export GDCS_WORKLOAD_ACCELERATOR_COUNT="1"
    
    # vLLM
    export GDCS_VLLM_MAX_MODEL_LEN="32768"
    export GDCS_VLLM_EXTRA_ARGS=""
    export GDCS_VLLM_STARTUP_FAILURE_THRESHOLD="60"
    export GDCS_VLLM_TEST_MAX_TOKENS="128"
    EOF
    

    Gemma 4 E4B Instruct

    屬性 值
    Hugging Face 存放區 google/gemma-4-E4B-it
    參數 80 億 (有效 40 億)
    授權 Gemma 使用條款
    vLLM 使用的權重 Hugging Face 格式 (model-*.safetensors),約 16.0 GB
    脈絡長度 128k 個詞元;本指南適用於 --max-model-len 32768

    建立模型環境檔案:

    cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/gemma/env.d/model.sh && echo "Successfully created." || echo "Failed to create!"
    # Model: Gemma 4 E4B Instruct
    export GDCS_HF_MODEL="google/gemma-4-E4B-it"
    export GDCS_HF_MODEL_REVISION="main"  # Hugging Face branch, tag, or commit to download
    export GDCS_HF_DOWNLOAD_EXCLUDE=""
    EOF
    

    為 GPU 選項建立加速器環境檔案:

    NVIDIA A100 40 GB

    cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/gemma/env.d/accelerator.sh && echo "Successfully created." || echo "Failed to create!"
    # Accelerator: NVIDIA A100 40 GB
    export GDCS_WORKLOAD_ACCELERATOR_TYPE="nvidia.com/gpu-pod-NVIDIA_A100_PCIE_40GB"
    export GDCS_WORKLOAD_ACCELERATOR_COUNT="1"
    
    # vLLM
    export GDCS_VLLM_MAX_MODEL_LEN="32768"
    export GDCS_VLLM_EXTRA_ARGS=""
    export GDCS_VLLM_STARTUP_FAILURE_THRESHOLD="120"
    export GDCS_VLLM_TEST_MAX_TOKENS="128"
    EOF
    

    NVIDIA A100 80 GB

    cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/gemma/env.d/accelerator.sh && echo "Successfully created." || echo "Failed to create!"
    # Accelerator: NVIDIA A100 80 GB
    export GDCS_WORKLOAD_ACCELERATOR_TYPE="nvidia.com/gpu-pod-NVIDIA_A100_80GB_PCIE"
    export GDCS_WORKLOAD_ACCELERATOR_COUNT="1"
    
    # vLLM
    export GDCS_VLLM_MAX_MODEL_LEN="32768"
    export GDCS_VLLM_EXTRA_ARGS=""
    export GDCS_VLLM_STARTUP_FAILURE_THRESHOLD="120"
    export GDCS_VLLM_TEST_MAX_TOKENS="128"
    EOF
    

    NVIDIA H100 94 GB

    cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/gemma/env.d/accelerator.sh && echo "Successfully created." || echo "Failed to create!"
    # Accelerator: NVIDIA H100 94 GB
    export GDCS_WORKLOAD_ACCELERATOR_TYPE="nvidia.com/gpu-pod-NVIDIA_H100_NVL"
    export GDCS_WORKLOAD_ACCELERATOR_COUNT="1"
    
    # vLLM
    export GDCS_VLLM_MAX_MODEL_LEN="32768"
    export GDCS_VLLM_EXTRA_ARGS=""
    export GDCS_VLLM_STARTUP_FAILURE_THRESHOLD="120"
    export GDCS_VLLM_TEST_MAX_TOKENS="128"
    EOF
    

    NVIDIA H200 141 GB

    cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/gemma/env.d/accelerator.sh && echo "Successfully created." || echo "Failed to create!"
    # Accelerator: NVIDIA H200 141 GB
    export GDCS_WORKLOAD_ACCELERATOR_TYPE="nvidia.com/gpu-pod-NVIDIA_H200"
    export GDCS_WORKLOAD_ACCELERATOR_COUNT="1"
    
    # vLLM
    export GDCS_VLLM_MAX_MODEL_LEN="32768"
    export GDCS_VLLM_EXTRA_ARGS=""
    export GDCS_VLLM_STARTUP_FAILURE_THRESHOLD="120"
    export GDCS_VLLM_TEST_MAX_TOKENS="128"
    EOF
    

    NVIDIA B300 288 GB

    cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/gemma/env.d/accelerator.sh && echo "Successfully created." || echo "Failed to create!"
    # Accelerator: NVIDIA B300 288 GB
    export GDCS_WORKLOAD_ACCELERATOR_TYPE="nvidia.com/gpu-pod-NVIDIA_B300_SXM6_AC"
    export GDCS_WORKLOAD_ACCELERATOR_COUNT="1"
    
    # vLLM
    export GDCS_VLLM_MAX_MODEL_LEN="32768"
    export GDCS_VLLM_EXTRA_ARGS=""
    export GDCS_VLLM_STARTUP_FAILURE_THRESHOLD="120"
    export GDCS_VLLM_TEST_MAX_TOKENS="128"
    EOF
    

    Gemma 4 12B Instruct

    屬性 值
    Hugging Face 存放區 google/gemma-4-12B-it
    參數 12B (密集)
    授權 Gemma 使用條款
    vLLM 使用的權重 Hugging Face 格式 (model-*.safetensors),約 23.9 GB
    脈絡長度 25.6 萬個詞元;本指南使用 --max-model-len 32768

    建立模型環境檔案:

    cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/gemma/env.d/model.sh && echo "Successfully created." || echo "Failed to create!"
    # Model: Gemma 4 12B Instruct
    export GDCS_HF_MODEL="google/gemma-4-12B-it"
    export GDCS_HF_MODEL_REVISION="main"  # Hugging Face branch, tag, or commit to download
    export GDCS_HF_DOWNLOAD_EXCLUDE=""
    EOF
    

    為 GPU 選項建立加速器環境檔案:

    NVIDIA A100 40 GB

    cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/gemma/env.d/accelerator.sh && echo "Successfully created." || echo "Failed to create!"
    # Accelerator: NVIDIA A100 40 GB
    export GDCS_WORKLOAD_ACCELERATOR_TYPE="nvidia.com/gpu-pod-NVIDIA_A100_PCIE_40GB"
    export GDCS_WORKLOAD_ACCELERATOR_COUNT="1"
    
    # vLLM
    export GDCS_VLLM_MAX_MODEL_LEN="32768"
    export GDCS_VLLM_EXTRA_ARGS=""
    export GDCS_VLLM_STARTUP_FAILURE_THRESHOLD="180"
    export GDCS_VLLM_TEST_MAX_TOKENS="128"
    EOF
    

    NVIDIA A100 80 GB

    cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/gemma/env.d/accelerator.sh && echo "Successfully created." || echo "Failed to create!"
    # Accelerator: NVIDIA A100 80 GB
    export GDCS_WORKLOAD_ACCELERATOR_TYPE="nvidia.com/gpu-pod-NVIDIA_A100_80GB_PCIE"
    export GDCS_WORKLOAD_ACCELERATOR_COUNT="1"
    
    # vLLM
    export GDCS_VLLM_MAX_MODEL_LEN="32768"
    export GDCS_VLLM_EXTRA_ARGS=""
    export GDCS_VLLM_STARTUP_FAILURE_THRESHOLD="180"
    export GDCS_VLLM_TEST_MAX_TOKENS="128"
    EOF
    

    NVIDIA H100 94 GB

    cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/gemma/env.d/accelerator.sh && echo "Successfully created." || echo "Failed to create!"
    # Accelerator: NVIDIA H100 94 GB
    export GDCS_WORKLOAD_ACCELERATOR_TYPE="nvidia.com/gpu-pod-NVIDIA_H100_NVL"
    export GDCS_WORKLOAD_ACCELERATOR_COUNT="1"
    
    # vLLM
    export GDCS_VLLM_MAX_MODEL_LEN="32768"
    export GDCS_VLLM_EXTRA_ARGS=""
    export GDCS_VLLM_STARTUP_FAILURE_THRESHOLD="180"
    export GDCS_VLLM_TEST_MAX_TOKENS="128"
    EOF
    

    NVIDIA H200 141 GB

    cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/gemma/env.d/accelerator.sh && echo "Successfully created." || echo "Failed to create!"
    # Accelerator: NVIDIA H200 141 GB
    export GDCS_WORKLOAD_ACCELERATOR_TYPE="nvidia.com/gpu-pod-NVIDIA_H200"
    export GDCS_WORKLOAD_ACCELERATOR_COUNT="1"
    
    # vLLM
    export GDCS_VLLM_MAX_MODEL_LEN="32768"
    export GDCS_VLLM_EXTRA_ARGS=""
    export GDCS_VLLM_STARTUP_FAILURE_THRESHOLD="180"
    export GDCS_VLLM_TEST_MAX_TOKENS="128"
    EOF
    

    NVIDIA B300 288 GB

    cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/gemma/env.d/accelerator.sh && echo "Successfully created." || echo "Failed to create!"
    # Accelerator: NVIDIA B300 288 GB
    export GDCS_WORKLOAD_ACCELERATOR_TYPE="nvidia.com/gpu-pod-NVIDIA_B300_SXM6_AC"
    export GDCS_WORKLOAD_ACCELERATOR_COUNT="1"
    
    # vLLM
    export GDCS_VLLM_MAX_MODEL_LEN="32768"
    export GDCS_VLLM_EXTRA_ARGS=""
    export GDCS_VLLM_STARTUP_FAILURE_THRESHOLD="180"
    export GDCS_VLLM_TEST_MAX_TOKENS="128"
    EOF
    

    Gemma 4 26B-A4B Instruct

    屬性 值
    Hugging Face 存放區 google/gemma-4-26B-A4B-it
    參數 258 億個參數/40 億個有效參數 (混合專家模型)
    授權 Gemma 使用條款
    vLLM 使用的權重 Hugging Face 格式 (model-*.safetensors),約 51.6 GB
    脈絡長度 25.6 萬個詞元;本指南使用 --max-model-len 32768

    建立模型環境檔案:

    cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/gemma/env.d/model.sh && echo "Successfully created." || echo "Failed to create!"
    # Model: Gemma 4 26B-A4B Instruct
    export GDCS_HF_MODEL="google/gemma-4-26B-A4B-it"
    export GDCS_HF_MODEL_REVISION="main"  # Hugging Face branch, tag, or commit to download
    export GDCS_HF_DOWNLOAD_EXCLUDE=""
    EOF
    

    為 GPU 選項建立加速器環境檔案:

    NVIDIA A100 80 GB

    cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/gemma/env.d/accelerator.sh && echo "Successfully created." || echo "Failed to create!"
    # Accelerator: NVIDIA A100 80 GB
    export GDCS_WORKLOAD_ACCELERATOR_TYPE="nvidia.com/gpu-pod-NVIDIA_A100_80GB_PCIE"
    export GDCS_WORKLOAD_ACCELERATOR_COUNT="1"
    
    # vLLM
    export GDCS_VLLM_MAX_MODEL_LEN="32768"
    export GDCS_VLLM_EXTRA_ARGS=""
    export GDCS_VLLM_STARTUP_FAILURE_THRESHOLD="300"
    export GDCS_VLLM_TEST_MAX_TOKENS="128"
    EOF
    

    NVIDIA H100 94 GB

    cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/gemma/env.d/accelerator.sh && echo "Successfully created." || echo "Failed to create!"
    # Accelerator: NVIDIA H100 94 GB
    export GDCS_WORKLOAD_ACCELERATOR_TYPE="nvidia.com/gpu-pod-NVIDIA_H100_NVL"
    export GDCS_WORKLOAD_ACCELERATOR_COUNT="1"
    
    # vLLM
    export GDCS_VLLM_MAX_MODEL_LEN="32768"
    export GDCS_VLLM_EXTRA_ARGS=""
    export GDCS_VLLM_STARTUP_FAILURE_THRESHOLD="300"
    export GDCS_VLLM_TEST_MAX_TOKENS="128"
    EOF
    

    NVIDIA H200 141 GB

    cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/gemma/env.d/accelerator.sh && echo "Successfully created." || echo "Failed to create!"
    # Accelerator: NVIDIA H200 141 GB
    export GDCS_WORKLOAD_ACCELERATOR_TYPE="nvidia.com/gpu-pod-NVIDIA_H200"
    export GDCS_WORKLOAD_ACCELERATOR_COUNT="1"
    
    # vLLM
    export GDCS_VLLM_MAX_MODEL_LEN="32768"
    export GDCS_VLLM_EXTRA_ARGS=""
    export GDCS_VLLM_STARTUP_FAILURE_THRESHOLD="300"
    export GDCS_VLLM_TEST_MAX_TOKENS="128"
    EOF
    

    NVIDIA B300 288 GB

    cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/gemma/env.d/accelerator.sh && echo "Successfully created." || echo "Failed to create!"
    # Accelerator: NVIDIA B300 288 GB
    export GDCS_WORKLOAD_ACCELERATOR_TYPE="nvidia.com/gpu-pod-NVIDIA_B300_SXM6_AC"
    export GDCS_WORKLOAD_ACCELERATOR_COUNT="1"
    
    # vLLM
    export GDCS_VLLM_MAX_MODEL_LEN="32768"
    export GDCS_VLLM_EXTRA_ARGS=""
    export GDCS_VLLM_STARTUP_FAILURE_THRESHOLD="300"
    export GDCS_VLLM_TEST_MAX_TOKENS="128"
    EOF
    

    Gemma 4 31B Instruct

    屬性 值
    Hugging Face 存放區 google/gemma-4-31B-it
    參數 31.3B (密集)
    授權 Gemma 使用條款
    vLLM 使用的權重 Hugging Face 格式 (model-*.safetensors),約 62.5 GB
    脈絡長度 25.6 萬個詞元;本指南使用 --max-model-len 32768

    建立模型環境檔案:

    cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/gemma/env.d/model.sh && echo "Successfully created." || echo "Failed to create!"
    # Model: Gemma 4 31B Instruct
    export GDCS_HF_MODEL="google/gemma-4-31B-it"
    export GDCS_HF_MODEL_REVISION="main"  # Hugging Face branch, tag, or commit to download
    export GDCS_HF_DOWNLOAD_EXCLUDE=""
    EOF
    

    為 GPU 選項建立加速器環境檔案:

    NVIDIA A100 80 GB

    在 80 GB GPU 上提供服務時,KV 快取可用的記憶體很少,因此這個選項會設定 --max-model-len 16384。

    cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/gemma/env.d/accelerator.sh && echo "Successfully created." || echo "Failed to create!"
    # Accelerator: NVIDIA A100 80 GB
    export GDCS_WORKLOAD_ACCELERATOR_TYPE="nvidia.com/gpu-pod-NVIDIA_A100_80GB_PCIE"
    export GDCS_WORKLOAD_ACCELERATOR_COUNT="1"
    
    # vLLM
    export GDCS_VLLM_MAX_MODEL_LEN="16384"
    export GDCS_VLLM_EXTRA_ARGS=""
    export GDCS_VLLM_STARTUP_FAILURE_THRESHOLD="360"
    export GDCS_VLLM_TEST_MAX_TOKENS="128"
    EOF
    

    NVIDIA H100 94 GB

    cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/gemma/env.d/accelerator.sh && echo "Successfully created." || echo "Failed to create!"
    # Accelerator: NVIDIA H100 94 GB
    export GDCS_WORKLOAD_ACCELERATOR_TYPE="nvidia.com/gpu-pod-NVIDIA_H100_NVL"
    export GDCS_WORKLOAD_ACCELERATOR_COUNT="1"
    
    # vLLM
    export GDCS_VLLM_MAX_MODEL_LEN="32768"
    export GDCS_VLLM_EXTRA_ARGS=""
    export GDCS_VLLM_STARTUP_FAILURE_THRESHOLD="360"
    export GDCS_VLLM_TEST_MAX_TOKENS="128"
    EOF
    

    NVIDIA H200 141 GB

    cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/gemma/env.d/accelerator.sh && echo "Successfully created." || echo "Failed to create!"
    # Accelerator: NVIDIA H200 141 GB
    export GDCS_WORKLOAD_ACCELERATOR_TYPE="nvidia.com/gpu-pod-NVIDIA_H200"
    export GDCS_WORKLOAD_ACCELERATOR_COUNT="1"
    
    # vLLM
    export GDCS_VLLM_MAX_MODEL_LEN="32768"
    export GDCS_VLLM_EXTRA_ARGS=""
    export GDCS_VLLM_STARTUP_FAILURE_THRESHOLD="360"
    export GDCS_VLLM_TEST_MAX_TOKENS="128"
    EOF
    

    NVIDIA B300 288 GB

    cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/gemma/env.d/accelerator.sh && echo "Successfully created." || echo "Failed to create!"
    # Accelerator: NVIDIA B300 288 GB
    export GDCS_WORKLOAD_ACCELERATOR_TYPE="nvidia.com/gpu-pod-NVIDIA_B300_SXM6_AC"
    export GDCS_WORKLOAD_ACCELERATOR_COUNT="1"
    
    # vLLM
    export GDCS_VLLM_MAX_MODEL_LEN="32768"
    export GDCS_VLLM_EXTRA_ARGS=""
    export GDCS_VLLM_STARTUP_FAILURE_THRESHOLD="360"
    export GDCS_VLLM_TEST_MAX_TOKENS="128"
    EOF
    
  2. 提供環境檔案來源:

    source ${HOME}/gdcag-solutions/open-weight-models/vllm/gemma/env.sh
    

    輸出結果會與下列內容相似:

    GDCS_ROOT_HOME=HOME_DIRECTORY_PATH/gdcag-solutions
    GDCS_IMPLEMENTATION_HOME=HOME_DIRECTORY_PATH/gdcag-solutions/open-weight-models/vllm
    GDCS_USER_GUIDE_HOME=HOME_DIRECTORY_PATH/gdcag-solutions/open-weight-models/vllm/gemma
    GDCS_HF_MODEL=HUGGING_FACE_REPOSITORY (ACCELERATOR_COUNT x ACCELERATOR_RESOURCE_TYPE)
    

叢集

  1. 設定 gdcloud:

    gdcloud config set core/account "default-user"
    gdcloud config set core/organization_console_url "${GDC_ZONAL_CONSOLE_URL}"
    gdcloud config set core/project "${GDC_PROJECT}"
    gdcloud config set core/zone "${GDC_ZONE}"
    
  2. 向 GDC 環境進行驗證:

    gdcloud auth login
    
  3. 擷取叢集憑證:

    gdcloud clusters get-credentials "${GDC_STANDARD_CLUSTER_NAME}" \
    --project="${GDC_PROJECT}" \
    --standard \
    --zone="${GDC_ZONE}"
    
  4. 驗證與叢集的連線:

    kubectl get nodes -L node.cluster.private.gdc.goog/machine-class
    
  5. 確認叢集中有加速器環境檔案的加速器資源類型。這個指令會列出節點公開的加速器資源類型 (名稱取決於安裝的 GPU 模型;在沒有 GPU 節點的叢集上,這個名稱會是空白) 和設定的值:

    kubectl get nodes --output=json | jq -r '[.items[].status.allocatable | keys[] | select(startswith("nvidia.com/"))] | unique[]'
    echo "GDCS_WORKLOAD_ACCELERATOR_TYPE=${GDCS_WORKLOAD_ACCELERATOR_TYPE}"
    

實作

命名空間

  1. 建立命名空間:

    kubectl create namespace "${GDCS_WORKLOAD_NAMESPACE}" \
    --dry-run=client \
    --output=yaml | kubectl apply -f -
    
  2. 新增 imagePullSecret:

    kubectl create secret docker-registry "${GDCS_HARBOR_K8S_PULL_SECRET}" \
    --dry-run=client \
    --from-file=.dockerconfigjson=${GDCS_HARBOR_K8S_DOCKER_CONFIG}/config.json \
    --namespace="${GDCS_WORKLOAD_NAMESPACE}" \
    --output=yaml | kubectl apply -f -
    

模型權重

  1. 下載模型權重 (僅限 vLLM 所需的所選模型檔案,GDCS_HF_DOWNLOAD_EXCLUDE 會略過其餘檔案),並移除 .cache 目錄:

    EXCLUDE_ARGS=()
    if [[ -n "${GDCS_HF_DOWNLOAD_EXCLUDE:-}" ]]; then
      read -r -a EXCLUDE_PATTERNS <<< "${GDCS_HF_DOWNLOAD_EXCLUDE}"
      EXCLUDE_ARGS=(--exclude "${EXCLUDE_PATTERNS[@]}")
    fi
    hf download "${GDCS_HF_MODEL}" --revision "${GDCS_HF_MODEL_REVISION}" --local-dir "${GDCS_HF_MODEL_DOWNLOAD_DIR}" "${EXCLUDE_ARGS[@]}" && \
    rm -rf "${GDCS_HF_MODEL_DOWNLOAD_DIR}/.cache"
    
  2. 根據下載的模型權重,計算所需的 PersistentVolumeClaim 大小,並加上 25% 的額外空間 (無條件進位至下一個整數 GiB):

    MODEL_KIB=$(du -sk "${GDCS_HF_MODEL_DOWNLOAD_DIR}" | awk '{print $1}')
    export GDCS_MODEL_PVC_SIZE="$(( (MODEL_KIB * 125 / 100 + 1048575) / 1048576 ))Gi"
    echo "GDCS_MODEL_PVC_SIZE=${GDCS_MODEL_PVC_SIZE}"
    
  3. 為模型權重建立 ReadWriteOnce PersistentVolumeClaim 的資訊清單:

    cat <<EOF > "${GDCS_USER_GUIDE_HOME}/${GDCS_WORKLOAD_NAME}-rwo-pvc.yaml"
    apiVersion: v1
    kind: PersistentVolumeClaim
    metadata:
      name: ${GDCS_WORKLOAD_NAME}-rwo
    spec:
      accessModes:
        - ReadWriteOnce
      resources:
        requests:
          storage: ${GDCS_MODEL_PVC_SIZE}
      storageClassName: standard-rwo
      volumeMode: Filesystem
    EOF
    
  4. 為模型權重套用 PersistentVolumeClaim 的資訊清單:

    kubectl apply \
    --filename="${GDCS_USER_GUIDE_HOME}/${GDCS_WORKLOAD_NAME}-rwo-pvc.yaml" \
    --namespace="${GDCS_WORKLOAD_NAMESPACE}"
    
  5. 等待 PersistentVolumeClaim 繫結並輸出基礎架構運算子所需的值,以調整 I/O 服務品質 (QoS) 政策:

    kubectl wait \
    --for=jsonpath='{.status.phase}'=Bound \
    "pvc/${GDCS_WORKLOAD_NAME}-rwo" \
    --namespace="${GDCS_WORKLOAD_NAMESPACE}" \
    --timeout=120s
    BACKING_PVC=$(kubectl get pvc "${GDCS_WORKLOAD_NAME}-rwo" \
    --namespace="${GDCS_WORKLOAD_NAMESPACE}" \
    --output=jsonpath='{.spec.volumeName}')
    echo "BACKING_PVC=${BACKING_PVC} ORG=${GDC_ORG}"
    
  6. 如果叢集不支援 performance-rwo,請要求基礎架構操作員將高效能 QoS 政策 (system-performance-<ORG>-admin,每 GiB 30 IOPS,最低 6,148 IOPS) 套用至支援 PersistentVolumeClaim 的磁碟區。提供上一個步驟輸出的 BACKING_PVC 和 ORG 值。基礎架構操作員會按照基礎架構操作員說明文件中的 FILE-R0097 執行手冊套用政策,並在實際磁碟區上生效,因此您可以繼續執行後續步驟。

  7. 為模型載入器 Pod 建立資訊清單:

    cat <<EOF > "${GDCS_USER_GUIDE_HOME}/${GDCS_MODEL_LOADER_POD_NAME}-pod.yaml"
    apiVersion: v1
    kind: Pod
    metadata:
      name: ${GDCS_MODEL_LOADER_POD_NAME}
    spec:
      containers:
        - name: uploader
          image: ${GDCS_HARBOR_PROJECT_URI}/library/busybox:${GDCS_BUSYBOX_IMAGE_TAG}
          command: ["sleep", "infinity"]
          volumeMounts:
            - name: model-data
              mountPath: ${GDCS_HF_PV_MOUNT_PATH}
      imagePullSecrets:
        - name: ${GDCS_HARBOR_K8S_PULL_SECRET}
      volumes:
        - name: model-data
          persistentVolumeClaim:
            claimName: ${GDCS_WORKLOAD_NAME}-rwo
    EOF
    
  8. 套用模型載入器 Pod 的資訊清單:

    kubectl apply \
    --filename="${GDCS_USER_GUIDE_HOME}/${GDCS_MODEL_LOADER_POD_NAME}-pod.yaml" \
    --namespace="${GDCS_WORKLOAD_NAMESPACE}"
    
  9. 等待模型載入器 Pod 進入「執行中」狀態:

    watch --color --interval 5 --no-title \
    "kubectl get pod/${GDCS_MODEL_LOADER_POD_NAME} \
    --namespace=${GDCS_WORKLOAD_NAMESPACE} | GREP_COLORS='mt=01;92' grep -E --color=always -e '^' -e 'Running'"
    
  10. 將模型權重上傳至 PersistentVolume (指令碼會略過大小相同的現有檔案,因此可以重新執行):

    export LOCAL_MODEL_DIR="${GDCS_HF_MODEL_DOWNLOAD_DIR}"
    export REMOTE_MODEL_DIR="${GDCS_HF_PV_MOUNT_PATH}/${GDCS_HF_MODEL_LOWERCASE}"
    export REMOTE_MOUNT_DIR="${GDCS_HF_PV_MOUNT_PATH}"
    
    ${GDCS_IMPLEMENTATION_HOME}/model_weight_upload.sh
    
  11. 刪除模型載入器 Pod:

    kubectl delete pod/${GDCS_MODEL_LOADER_POD_NAME} \
    --namespace="${GDCS_WORKLOAD_NAMESPACE}"
    

推論伺服器

  1. 建立 vLLM Deployment 的資訊清單。ConfigMap 會從環境中擷取模型專屬的 vLLM 設定,因此每個模型和 GPU 選項的資訊清單都相同:

    cat <<EOF > "${GDCS_USER_GUIDE_HOME}/${GDCS_WORKLOAD_NAME}-deployment.yaml"
    apiVersion: v1
    kind: ConfigMap
    metadata:
      name: ${GDCS_WORKLOAD_NAME}
    data:
      MAX_MODEL_LEN: "${GDCS_VLLM_MAX_MODEL_LEN}"
      MODEL: "${GDCS_HF_PV_MOUNT_PATH}/${GDCS_HF_MODEL_LOWERCASE}"
      PORT: "8000"
      TENSOR_PARALLEL_SIZE: "${GDCS_WORKLOAD_ACCELERATOR_COUNT}"
      VLLM_EXTRA_ARGS: "${GDCS_VLLM_EXTRA_ARGS}"
    ---
    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: ${GDCS_WORKLOAD_NAME}
      labels:
        app: ${GDCS_WORKLOAD_NAME}
    spec:
      replicas: 1
      selector:
        matchLabels:
          app: ${GDCS_WORKLOAD_NAME}
      strategy:
        type: Recreate
      template:
        metadata:
          labels:
            app: ${GDCS_WORKLOAD_NAME}
        spec:
          containers:
            - command:
                - /bin/sh
                - -c
              args:
                - exec vllm serve "\$MODEL" --port="\$PORT" --tensor-parallel-size="\$TENSOR_PARALLEL_SIZE" --max-model-len="\$MAX_MODEL_LEN" \$VLLM_EXTRA_ARGS
              envFrom:
                - configMapRef:
                    name: ${GDCS_WORKLOAD_NAME}
              image: ${GDCS_HARBOR_PROJECT_URI}/vllm/vllm-openai:${GDCS_VLLM_IMAGE_TAG}-fips
              livenessProbe:
                failureThreshold: 3
                httpGet:
                  path: /health
                  port: 8000
                periodSeconds: 20
                timeoutSeconds: 5
              name: vllm
              ports:
                - containerPort: 8000
              readinessProbe:
                failureThreshold: 3
                httpGet:
                  path: /health
                  port: 8000
                periodSeconds: 10
                successThreshold: 1
                timeoutSeconds: 5
              resources:
                limits:
                  ${GDCS_WORKLOAD_ACCELERATOR_TYPE}: "${GDCS_WORKLOAD_ACCELERATOR_COUNT}"
                requests:
                  ${GDCS_WORKLOAD_ACCELERATOR_TYPE}: "${GDCS_WORKLOAD_ACCELERATOR_COUNT}"
              startupProbe:
                httpGet:
                  path: /health
                  port: 8000
                failureThreshold: ${GDCS_VLLM_STARTUP_FAILURE_THRESHOLD}
                periodSeconds: 10
                timeoutSeconds: 5
              volumeMounts:
                - name: model-weights
                  mountPath: ${GDCS_HF_PV_MOUNT_PATH}
                - name: shm
                  mountPath: /dev/shm
          securityContext:
            seLinuxOptions:
              type: spc_t
          imagePullSecrets:
            - name: ${GDCS_HARBOR_K8S_PULL_SECRET}
          volumes:
            - name: model-weights
              persistentVolumeClaim:
                claimName: ${GDCS_WORKLOAD_NAME}-rwo
            - name: shm
              emptyDir:
                medium: Memory
                sizeLimit: "8Gi"
    ---
    apiVersion: v1
    kind: Service
    metadata:
      name: ${GDCS_WORKLOAD_NAME}
    spec:
      ports:
        - name: vllm-http
          port: 80
          protocol: TCP
          targetPort: 8000
      selector:
        app: ${GDCS_WORKLOAD_NAME}
      type: LoadBalancer
    EOF
    
  2. 套用 vLLM Deployment 的資訊清單:

    kubectl apply \
    --filename="${GDCS_USER_GUIDE_HOME}/${GDCS_WORKLOAD_NAME}-deployment.yaml" \
    --namespace="${GDCS_WORKLOAD_NAMESPACE}"
    
  3. 建立 vLLM NetworkPolicy 的資訊清單:

    cat << EOF > "${GDCS_USER_GUIDE_HOME}/vllm-networkpolicy.yaml" && echo "Successfully created." || echo "Failed to create!"
    apiVersion: networking.k8s.io/v1
    kind: NetworkPolicy
    metadata:
      name: allow-vllm-ingress
      namespace: ${GDCS_WORKLOAD_NAMESPACE}
    spec:
      ingress:
      - from:
        - ipBlock:
            cidr: 0.0.0.0/0
        - namespaceSelector: {}
        ports:
        - port: 8000
          protocol: TCP
      podSelector: {}
      policyTypes:
      - Ingress
    EOF
    
  4. 套用 vLLM NetworkPolicy 的資訊清單:

    kubectl apply \
    --filename="${GDCS_USER_GUIDE_HOME}/vllm-networkpolicy.yaml"
    
  1. 等待 Pod 進入「執行中」狀態:

    watch --color --interval 5 --no-title \
    "kubectl get pods --selector=app=${GDCS_WORKLOAD_NAME} \
    --namespace=${GDCS_WORKLOAD_NAMESPACE} | GREP_COLORS='mt=01;92' grep -E --color=always -e '^' -e 'Running'"
    
  2. 等待 Deployment 準備就緒:

    watch --color --interval 5 --no-title \
    "kubectl get deployment/${GDCS_WORKLOAD_NAME} \
    --namespace=${GDCS_WORKLOAD_NAMESPACE} | GREP_COLORS='mt=01;92' grep -E --color=always -e '^' -e '1/1     1            1'"
    
  3. 選用:串流容器記錄,監控啟動進度:

    kubectl logs deployment/${GDCS_WORKLOAD_NAME} \
    --follow \
    --namespace="${GDCS_WORKLOAD_NAMESPACE}"
    

驗證

  1. 擷取 vLLM 的 LoadBalancer IP 位址:

    export VLLM_LB_IP=$(kubectl get service/${GDCS_WORKLOAD_NAME} --namespace="${GDCS_WORKLOAD_NAMESPACE}" --output=jsonpath='{.status.loadBalancer.ingress[*].ip}')
    echo "VLLM_LB_IP=${VLLM_LB_IP}"
    
  2. 傳送測試要求至 vLLM。max_tokens 值是 env.d/accelerator.sh 的 GDCS_VLLM_TEST_MAX_TOKENS:

    curl http://${VLLM_LB_IP}/v1/models
    
    curl http://${VLLM_LB_IP}/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
      "model": "'${GDCS_HF_PV_MOUNT_PATH}/${GDCS_HF_MODEL_LOWERCASE}'",
      "messages": [
        {"role": "user", "content": "Explain Google Distributed Cloud air-gapped briefly."}
      ],
      "max_tokens": '${GDCS_VLLM_TEST_MAX_TOKENS}'
    }'
    
  3. 如果無法從工作站連上 LoadBalancer IP 位址 (例如沒有 ProjectNetworkPolicy 允許),請透過通訊埠轉送傳送要求:

    kubectl port-forward service/${GDCS_WORKLOAD_NAME} \
    --namespace="${GDCS_WORKLOAD_NAMESPACE}" 8000:80 &
    PF_PID=$!
    
    sleep 1
    
    curl http://127.0.0.1:8000/v1/models
    
    kill ${PF_PID}