本文提供逐步操作說明，協助您在 Google Distributed Cloud (GDC) 實體隔離環境中部署開放權重模型。課程內容涵蓋如何使用 vLLM，透過 GDC 平台的功能 (包括 Kubernetes、Harbor 和加速器 (GPU) 資源) 提供高處理量。
架構
解決方案涉及在標準叢集中，將 vLLM 部署為容器化推論服務後端 (Deployment)。模型權重會儲存在永久磁碟區，這些磁碟區會根據下載的模型目錄動態調整大小，並加上 25% 的額外負擔，設定 system-performance-<org>-admin 儲存空間 QoS 政策 (每 GiB 30 IOPS，最低 6,148 IOPS，或啟用高效能區塊儲存空間時為 performance-rwo)，並使用模型載入器工作流程填入。LoadBalancer 類型的 Kubernetes Service 會公開後端 API，而 NetworkPolicy/ProjectNetworkPolicy 物件則會保護其存取權。
事前準備
繼續部署之前，請確認環境符合所有必要條件，且已正確設定必要的指令列公用程式。在工作站設定這些工具，對於管理 Container Registry、與叢集互動，以及自動化部署程序至關重要。
- 建立標準叢集，並提供足夠的資源 (CPU、記憶體、加速器)。加速器最低需求會因模型而異。
- Harbor 執行個體可用且可存取。
- 已套用必要的 IAM。
- 工作站，且具備連至環境和網際網路的必要連線能力
- 如要使用受管制模型，請提供 Hugging Face 權杖並接受條款。
環境設定
Identity and Access Management
請確認已正確設定必要的 IAM 帳戶、角色和權限。
組織的 GDC 使用者角色：
- 選用：標準叢集管理員 (透過控制台查看叢集)
專案的 GDC 使用者角色 (由專案 IAM 管理員在專案命名空間中授予 RoleBinding)：
- Harbor 執行個體檢視者 (
harbor-instance-viewer) - Harbor 專案建立者 (
harbor-project-creator，僅限 Harbor 專案尚不存在時) - 標準叢集管理員 (
standard-cluster-admin，gdcloud clusters get-credentials必備)
標準叢集的 GDC 使用者角色：上述專案角色不會授予叢集內的任何權限。專案 IAM 管理員必須額外將使用者繫結至 StandardClusterRole cluster-admin，並在管理 API 伺服器的專案命名空間中加入 StandardClusterRoleBinding；繫結會在幾秒內傳播至專案的標準叢集 (status.clusters[].conditions 顯示 Propagated=True)。
cat <<EOF | kubectl apply --kubeconfig MANAGEMENT_API_SERVER -f -
apiVersion: iam.gdc.goog/v1
kind: StandardClusterRoleBinding
metadata:
name: BINDING_NAME
namespace: PROJECT
spec:
roleRef:
apiGroup: iam.gdc.goog
kind: StandardClusterRole
name: cluster-admin
subjects:
- apiGroup: rbac.authorization.k8s.io
kind: User
name: USER
EOF
更改下列內容：
MANAGEMENT_API_SERVER：管理 API 伺服器 kubeconfig 檔案的路徑。BINDING_NAME：繫結名稱，例如alex-cluster-admin(小寫英文字母、數字和連字號)。PROJECT：專案名稱。USER：要繫結的使用者名稱，必須與識別資訊提供者回報的名稱完全相同 (例如電子郵件地址)。
基礎架構操作員協助 (僅限標準叢集不支援 performance-rwo 時)：使用者手冊會使用預設的 standard-rwo StorageClass 建立模型 PersistentVolumeClaim，將適當大小的模型磁碟區限制為每 GiB 3 個 IOPS。請要求基礎架構操作員將 system-performance-<ORG>-admin QoS 政策套用至支援模型 PersistentVolumeClaim 的磁碟區，使其獲得 6,148 IOPS 的下限。使用手冊會列印基礎架構操作員所需的值。
Harbor 機器人帳戶對 crane 的權限：
- 列出存放區
- 提取存放區
- 推送存放區
- 讀取構件
- 列出構件
- 建立標記
- 清單標記
Harbor 機器人帳戶的 Kubernetes 映像檔提取權限：
- 列出存放區
- 提取存放區
- 讀取構件
- 列出構件
- 清單標記
工作站
除非步驟另有說明，否則請在工作站執行本指南中的所有指令，並確保工作站具備必要工具，且可連線至環境和網際網路。
需求條件
工作站必須安裝下列工具：
crane：在登錄檔之間管理及複製容器映像檔 (說明文件)。docker：建構並推送修補後的 vLLM 容器映像檔。gdcloud：用於管理 GDC 資源的指令列介面 (CLI) (說明文件)。hf：用於存取 Hugging Face 的指令列介面 (CLI)。kubectl：用於與 Kubernetes 叢集通訊及管理叢集的指令列介面 (CLI)。jq：輕量且靈活的指令列 JSON 處理器。yq：可攜式指令列 YAML 處理器。
工作站設定
工作站設定需要下列環境資訊：
GDC_STANDARD_CLUSTER_NAME：GDC 標準叢集的名稱。GDC_DOMAIN_SUFFIX：GDC 環境的網域字尾 (例如gdc.example.com)。GDC_ORG：：GDC 組織的名稱。GDC_PROJECT：GDC 專案名稱。GDC_ZONE：GDC 部署區域的名稱。GDC_HARBOR_INSTANCE_NAME：專案中的 Harbor 執行個體名稱。GDCS_HARBOR_PROJECT_NAME：要用於映像檔的 Harbor 專案名稱 (預設：solutions)。GDCS_HARBOR_CRANE_ROBOT_NAME：Harborcrane機器人帳戶的名稱。GDCS_HARBOR_CRANE_ROBOT_TOKEN：Harborcrane機器人帳戶的驗證權杖。GDCS_HARBOR_K8S_ROBOT_NAME：Harbor Kubernetes 映像檔提取機器人帳戶的名稱。GDCS_HARBOR_K8S_ROBOT_TOKEN：Harbor Kubernetes 映像檔提取機器人帳戶的驗證權杖。
收集完所有必要變數的值後，請繼續產生環境變數檔案。建立後，您隨時可以手動編輯檔案。
建立根解決方案目錄和 Secret 資料夾：
mkdir -p ${HOME}/gdcag-solutions/env.d mkdir -p ${HOME}/gdcag-solutions/secrets touch ${HOME}/gdcag-solutions/secrets/harbor_crane_robot_token touch ${HOME}/gdcag-solutions/secrets/harbor_k8s_robot_token chmod u=rwx,go= ${HOME}/gdcag-solutions/secrets chmod -R u=rw,go= ${HOME}/gdcag-solutions/secrets/*建立平台環境設定檔：
cat << 'EOF' > ${HOME}/gdcag-solutions/env.d/platform.sh && echo "Successfully created." || echo "Failed to create!" # Infrastructure (Platform Native) export GDC_STANDARD_CLUSTER_NAME="STANDARD_CLUSTER_NAME" export GDC_DOMAIN_SUFFIX="DOMAIN_SUFFIX" export GDC_ORG="ORG" export GDC_PROJECT="PROJECT" export GDC_ZONE="ZONE" export GDC_HARBOR_INSTANCE_NAME="HARBOR_INSTANCE_NAME" # Derived platform values export GDC_ZONAL_HOSTNAME="${GDC_ORG}.${GDC_ZONE}.${GDC_DOMAIN_SUFFIX}" export GDC_ZONAL_CONSOLE_URL="https://console.${GDC_ZONAL_HOSTNAME}" export GDC_HARBOR_HOST="${GDC_HARBOR_INSTANCE_NAME}-${GDC_PROJECT}.${GDC_ORG}.${GDC_ZONE}.${GDC_DOMAIN_SUFFIX}" EOF更改下列內容：
STANDARD_CLUSTER_NAME：GDC 標準叢集名稱。DOMAIN_SUFFIX：GDC 網域字尾。ORG：GDC 組織。PROJECT：GDC 專案。ZONE：GDC 可用區。HARBOR_INSTANCE_NAME：GDC Harbor 執行個體名稱。
建立登錄檔環境設定檔：
cat << 'EOF' > ${HOME}/gdcag-solutions/env.d/registry.sh && echo "Successfully created." || echo "Failed to create!" # GDC Solutions Registry & Secrets export GDCS_HARBOR_PROJECT_NAME="solutions" export GDCS_HARBOR_CRANE_ROBOT_NAME='HARBOR_CRANE_ROBOT_NAME' export GDCS_HARBOR_CRANE_ROBOT_TOKEN="$(cat ${GDCS_ROOT_HOME}/secrets/harbor_crane_robot_token)" export GDCS_HARBOR_K8S_ROBOT_NAME='HARBOR_K8S_ROBOT_NAME' export GDCS_HARBOR_K8S_ROBOT_TOKEN="$(cat ${GDCS_ROOT_HOME}/secrets/harbor_k8s_robot_token)" export GDCS_HARBOR_K8S_PULL_SECRET="gdcs-image-pull-secret" # Derived registry values export GDCS_HARBOR_PROJECT_URI="${GDC_HARBOR_HOST}/${GDCS_HARBOR_PROJECT_NAME}" export GDCS_HARBOR_CHART_OCI_URI="oci://${GDCS_HARBOR_PROJECT_URI}" EOF更改下列內容：
HARBOR_CRANE_ROBOT_NAME：crane的 Harbor 機器人帳戶全名，如 Harbor 所示，包括robot$前置字元。請保留單引號，以免殼層展開$。HARBOR_K8S_ROBOT_NAME：Kubernetes 映像檔提取作業的 Harbor 機器人帳戶全名，如 Harbor 所示，包括robot$前置字串。請保留單引號，以免殼層展開$。
將權杖新增至密鑰檔案：
set +o history echo "CRANE_ROBOT_TOKEN" > ${HOME}/gdcag-solutions/secrets/harbor_crane_robot_token echo "KUBERNETES_ROBOT_TOKEN" > ${HOME}/gdcag-solutions/secrets/harbor_k8s_robot_token set -o history更改下列內容：
CRANE_ROBOT_TOKEN：crane機器人帳戶的密鑰。KUBERNETES_ROBOT_TOKEN：Kubernetes 映像檔提取機器人帳戶的密鑰。
建立根環境載入器檔案：
cat << 'EOF' > ${HOME}/gdcag-solutions/env.sh && echo "Successfully created." || echo "Failed to create!" export GDCS_ROOT_HOME="${HOME}/gdcag-solutions" echo "GDCS_ROOT_HOME=${GDCS_ROOT_HOME}" # Sourced in dependency order source "${GDCS_ROOT_HOME}/env.d/platform.sh" source "${GDCS_ROOT_HOME}/env.d/registry.sh" EOF
設定解決方案變數
建立解決方案實作目錄和密鑰資料夾：
mkdir -p ${HOME}/gdcag-solutions/open-weight-models/vllm/env.d mkdir -p ${HOME}/gdcag-solutions/open-weight-models/vllm/secrets touch ${HOME}/gdcag-solutions/open-weight-models/vllm/secrets/huggingface_token chmod u=rwx,go= ${HOME}/gdcag-solutions/open-weight-models/vllm/secrets chmod -R u=rw,go= ${HOME}/gdcag-solutions/open-weight-models/vllm/secrets/*將 Hugging Face 權杖新增至 Secret 檔案：
set +o history echo "HUGGINGFACE_TOKEN" > ${HOME}/gdcag-solutions/open-weight-models/vllm/secrets/huggingface_token set -o history更改下列內容：
HUGGINGFACE_TOKEN：您的 Hugging Face 存取權杖。
建立加速器設定檔：
cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/env.d/accelerator.sh && echo "Successfully created." || echo "Failed to create!" # Workload Accelerator (NVIDIA GPU) export GDCS_WORKLOAD_ACCELERATOR_TYPE="WORKLOAD_ACCELERATOR_TYPE" export GDCS_WORKLOAD_ACCELERATOR_COUNT="1" EOF更改下列內容：
WORKLOAD_ACCELERATOR_TYPE：GPU 節點公開的 GPU 資源名稱，例如nvidia.com/gpu-pod-NVIDIA_A100_80GB_PCIE。使用手冊會列出 GDC 實體隔離方案提供的每個 GPU 資源名稱。擷取叢集憑證後，您可以使用kubectl get nodes --output=json | jq -r '[.items[].status.allocatable | keys[] | select(startswith("nvidia.com/"))] | unique[]'列出叢集中的名稱。
建立解決方案環境設定檔：
cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/env.d/vllm.sh && echo "Successfully created." || echo "Failed to create!" # Hugging Face export HF_TOKEN="$(cat ${GDCS_IMPLEMENTATION_HOME}/secrets/huggingface_token 2>/dev/null || echo '')" # Images export GDCS_BUSYBOX_IMAGE_TAG="1.38.0" export GDCS_VLLM_IMAGE_TAG="v0.26.0" # Docker configuration directories for crane and Kubernetes export GDCS_HARBOR_CRANE_DOCKER_CONFIG="${GDCS_IMPLEMENTATION_HOME}/docker/crane" export GDCS_HARBOR_K8S_DOCKER_CONFIG="${GDCS_IMPLEMENTATION_HOME}/docker/k8s" EOF建立實作環境載入器檔案：
cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/env.sh && echo "Successfully created." || echo "Failed to create!" source "${HOME}/gdcag-solutions/env.sh" export GDCS_IMPLEMENTATION_HOME="${HOME}/gdcag-solutions/open-weight-models/vllm" echo "GDCS_IMPLEMENTATION_HOME=${GDCS_IMPLEMENTATION_HOME}" # Sourced in dependency order source "${GDCS_IMPLEMENTATION_HOME}/env.d/accelerator.sh" source "${GDCS_IMPLEMENTATION_HOME}/env.d/vllm.sh" EOF使用偏好的編輯器編輯及查看環境檔案：
${EDITOR:-vi} ${HOME}/gdcag-solutions/env.d/platform.sh ${EDITOR:-vi} ${HOME}/gdcag-solutions/env.d/registry.sh ${EDITOR:-vi} ${HOME}/gdcag-solutions/open-weight-models/vllm/env.d/accelerator.sh ${EDITOR:-vi} ${HOME}/gdcag-solutions/open-weight-models/vllm/env.d/vllm.sh提供環境檔案來源：
source ${HOME}/gdcag-solutions/open-weight-models/vllm/env.sh輸出結果會與下列內容相似：
GDCS_ROOT_HOME=HOME_DIRECTORY_PATH/gdcag-solutions GDCS_IMPLEMENTATION_HOME=HOME_DIRECTORY_PATH/gdcag-solutions/open-weight-models/vllm
GDC
本指南假設工作站已設為信任 GDC 環境和 Harbor 執行個體的 TLS 憑證。
設定
gdcloud：gdcloud config set core/account "default-user" gdcloud config set core/organization_console_url "${GDC_ZONAL_CONSOLE_URL}" gdcloud config set core/project "${GDC_PROJECT}" gdcloud config set core/zone "${GDC_ZONE}"向 GDC 環境進行驗證：
gdcloud auth login
叢集
擷取叢集憑證：
gdcloud clusters get-credentials "${GDC_STANDARD_CLUSTER_NAME}" \ --project="${GDC_PROJECT}" \ --standard \ --zone="${GDC_ZONE}"驗證與叢集的連線：
kubectl get nodes -L node.cluster.private.gdc.goog/machine-class
構件遷移準備
- 請驗證 Artifact Registry 的連線和設定。
為 crane 建立 Docker 設定檔。機器人帳戶用於推送大型映像檔層，避免在使用代管 Harbor 服務 (MHS) 憑證輔助程式 (docker-credential-mhs) 搭配使用者帳戶時，發生驗證權杖逾時：
export DOCKER_CONFIG="${GDCS_HARBOR_CRANE_DOCKER_CONFIG}" crane auth login "${GDC_HARBOR_HOST}" \ --password="${GDCS_HARBOR_CRANE_ROBOT_TOKEN}" \ --username="${GDCS_HARBOR_CRANE_ROBOT_NAME}"建立 Kubernetes 的 Docker 設定檔：
export DOCKER_CONFIG="${GDCS_HARBOR_K8S_DOCKER_CONFIG}" crane auth login "${GDC_HARBOR_HOST}" \ --password="${GDCS_HARBOR_K8S_ROBOT_TOKEN}" \ --username="${GDCS_HARBOR_K8S_ROBOT_NAME}"建立
seed_registry.sh指令碼：cat << 'EOF' > ${GDCS_IMPLEMENTATION_HOME}/seed_registry.sh && echo "Successfully created." || echo "Failed to create!" #!/bin/bash # seed_registry.sh: Modular artifact migration for GDC Solutions # Requires the env.sh file to be sourced first. SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" set -o nounset source "${SCRIPT_DIR}/env.sh" # Set the Docker config export DOCKER_CONFIG="${GDCS_HARBOR_CRANE_DOCKER_CONFIG}" eval "${SERIALIZED_IMAGES}" # Ensure GDCS_REGISTRY_IMAGES is set if [[ ${#GDCS_REGISTRY_IMAGES[@]} -eq 0 ]]; then echo "GDCS_REGISTRY_IMAGES must be set, exiting..." exit 1 fi # Migrate the images for source_image in "${GDCS_REGISTRY_IMAGES[@]}"; do # Strip the registry host only when the first path segment is a host (contains a dot or a port) first_segment="${source_image%%/*}" if [[ "${first_segment}" == *.* || "${first_segment}" == *:* ]]; then image_path="${source_image#*/}" else image_path="${source_image}" fi destination_image="${GDCS_HARBOR_PROJECT_URI}/${image_path}" # Ensure the folder structure is created crane append \ --new_layer=<(tar czf - -T /dev/null) \ --new_tag="${destination_image%:*}:create" \ --oci-empty-base 2> /dev/null || true # Copy the linux/amd64 platform only to avoid transferring multi-arch layers over air-gapped links crane copy --platform linux/amd64 "${source_image}" "${destination_image}" || { echo "Failed to copy ${source_image} to ${destination_image}"; exit 1; } done echo "Migration complete: Images are available at ${GDCS_HARBOR_PROJECT_URI}" EOF chmod u+x "${GDCS_IMPLEMENTATION_HOME}/seed_registry.sh"定義解決方案所需的容器映像檔清單：
declare -a GDCS_REGISTRY_IMAGES=( "docker.io/library/busybox:${GDCS_BUSYBOX_IMAGE_TAG}" "docker.io/vllm/vllm-openai:${GDCS_VLLM_IMAGE_TAG}" ) export SERIALIZED_IMAGES=$(declare -p GDCS_REGISTRY_IMAGES)將必要的容器映像檔植入 Artifact Registry：
${GDCS_IMPLEMENTATION_HOME}/seed_registry.sh
建立 Dockerfile 來修補 vLLM 容器，直到上游變更實作為止：
cat <<EOF > ${GDCS_IMPLEMENTATION_HOME}/docker/vllm-${GDCS_VLLM_IMAGE_TAG}-patch.dockerfile && echo "Successfully created." || echo "Failed to create!" FROM docker.io/vllm/vllm-openai:${GDCS_VLLM_IMAGE_TAG} RUN pip install --upgrade "opencv-python-headless<=4.12.0.88" EOF建構並推送修補後的映像檔：
context_dir="${GDCS_IMPLEMENTATION_HOME}/docker/empty-context" image="${GDCS_HARBOR_PROJECT_URI}/vllm/vllm-openai:${GDCS_VLLM_IMAGE_TAG}-fips" mkdir -p "${context_dir}" rm -rf "${context_dir}"/* export DOCKER_CONFIG="${GDCS_HARBOR_CRANE_DOCKER_CONFIG}" docker build \ --file="${GDCS_IMPLEMENTATION_HOME}/docker/vllm-${GDCS_VLLM_IMAGE_TAG}-patch.dockerfile" \ --tag="${image}" \ "${context_dir}" docker push "${image}"建立
model_weight_upload.sh指令碼。這項指令會透過模型載入器Pod(GNUtar串流，包括大於 8 GiB 的檔案)，將本機模型目錄複製到PersistentVolume，並略過已存在且大小相同的檔案：cat << 'EOF' > ${GDCS_IMPLEMENTATION_HOME}/model_weight_upload.sh && echo "Successfully created." || echo "Failed to create!" #!/bin/bash # model_weight_upload.sh: copy LOCAL_MODEL_DIR into REMOTE_MODEL_DIR on the PersistentVolume mounted in the loader Pod. # Requires: GDCS_WORKLOAD_NAMESPACE, GDCS_MODEL_LOADER_POD_NAME, LOCAL_MODEL_DIR, REMOTE_MODEL_DIR # Optional: REMOTE_MOUNT_DIR (PersistentVolume mount path, defaults to REMOTE_MODEL_DIR) - its lost+found directory is removed. SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" set -o errexit set -o nounset set -o pipefail # Ensure LOCAL_MODEL_DIR is set if [[ -z "${LOCAL_MODEL_DIR:-}" ]]; then echo "LOCAL_MODEL_DIR must be set, exiting..." exit 1 fi # Ensure REMOTE_MODEL_DIR is set if [[ -z "${REMOTE_MODEL_DIR:-}" ]]; then echo "REMOTE_MODEL_DIR must be set, exiting..." exit 1 fi REMOTE_MOUNT_DIR="${REMOTE_MOUNT_DIR:-${REMOTE_MODEL_DIR}}" KEEPALIVE_INTERVAL_SEC=300 start_session_keepalive() { echo "[KeepAlive] Started background heartbeat (every ${KEEPALIVE_INTERVAL_SEC}s)." while true; do sleep "${KEEPALIVE_INTERVAL_SEC}" # Lightweight API request to prevent session inactivity logout if kubectl get namespace "${GDCS_WORKLOAD_NAMESPACE}" >/dev/null 2>&1; then echo "[KeepAlive] Session heartbeat sent at $(date)" else echo "[KeepAlive] Heartbeat failed or context expired at $(date)" fi done } start_session_keepalive & MY_KEEPALIVE_PID=$! cleanup() { echo "[Cleanup] Stopping keep-alive process (PID: ${MY_KEEPALIVE_PID})..." kill "${MY_KEEPALIVE_PID}" 2>/dev/null || true wait "${MY_KEEPALIVE_PID}" 2>/dev/null || true echo "[Cleanup] Session keep-alive terminated." } trap cleanup EXIT INT TERM # Remove the lost+found directory kubectl exec "pod/${GDCS_MODEL_LOADER_POD_NAME}" \ --namespace="${GDCS_WORKLOAD_NAMESPACE}" \ -- rm -rf "${REMOTE_MOUNT_DIR}/lost+found" # Create the model directory kubectl exec "pod/${GDCS_MODEL_LOADER_POD_NAME}" \ --namespace="${GDCS_WORKLOAD_NAMESPACE}" \ -- mkdir -p "${REMOTE_MODEL_DIR}" # Loop through all files in the local directory FAILED_FILES=() MAX_UPLOAD_ATTEMPTS="${MAX_UPLOAD_ATTEMPTS:-3}" remote_size_of() { # Prints the remote file size (0 if the file does not exist) kubectl exec "pod/${GDCS_MODEL_LOADER_POD_NAME}" --namespace="${GDCS_WORKLOAD_NAMESPACE}" -- stat -c%s "$1" 2>/dev/null || echo 0 } # Loop through all files in the local directory (process substitution keeps FAILED_FILES in this shell) while read -r local_file; do # Get relative path rel_path="${local_file#"${LOCAL_MODEL_DIR}"/}" remote_file="${REMOTE_MODEL_DIR}/${rel_path}" # Get local file size local_size=$(stat -L -c%s "${local_file}") # Get remote file size (returns 0 if file does not exist) remote_size=$(remote_size_of "${remote_file}") if [ "${local_size}" -eq "${remote_size}" ]; then echo "[Loader] Skipping: ${rel_path} (Size matches)" continue fi if [ "${remote_size}" -ne "0" ]; then echo "[Loader] Mismatch: ${rel_path} (${remote_size} remote vs ${local_size} local)" fi # Ensure remote parent subdirectories exist if your model has folders remote_parent=$(dirname "${remote_file}") kubectl exec "pod/${GDCS_MODEL_LOADER_POD_NAME}" \ --namespace="${GDCS_WORKLOAD_NAMESPACE}" \ -- mkdir -p "${remote_parent}" attempt=1 while [ "${attempt}" -le "${MAX_UPLOAD_ATTEMPTS}" ]; do echo "[Loader] Uploading: ${rel_path} (${local_size} bytes, attempt ${attempt}/${MAX_UPLOAD_ATTEMPTS})..." # Copy the individual file using GNU tar streaming. # Note: 'kubectl cp' uses Go's archive/tar, which defaults to PAX format for files > 8 GiB. # BusyBox's tar in the loader pod does not parse PAX size headers and fails with 'tar: invalid tar magic'. # Streaming via GNU tar (--format=gnu) uses base-256 binary encoding for files > 8 GiB, which BusyBox tar supports. tar --format=gnu -C "$(dirname "${local_file}")" -cf - "$(basename "${local_file}")" | \ kubectl exec -i "pod/${GDCS_MODEL_LOADER_POD_NAME}" \ --namespace="${GDCS_WORKLOAD_NAMESPACE}" \ -- tar -xf - -C "${remote_parent}" || true # Verify: an interrupted exec stream leaves a truncated file behind without an error remote_size=$(remote_size_of "${remote_file}") if [ "${local_size}" -eq "${remote_size}" ]; then echo "[Loader] Verified: ${rel_path} (${remote_size} bytes)" break fi echo "[Loader] Incomplete upload: ${rel_path} (${remote_size} remote vs ${local_size} local), retrying..." attempt=$((attempt + 1)) done if [ "${local_size}" -ne "${remote_size}" ]; then FAILED_FILES+=("${rel_path}") fi done < <(find -L "${LOCAL_MODEL_DIR}" -type f) if [ "${#FAILED_FILES[@]}" -ne 0 ]; then echo "[Loader] FAILED: ${#FAILED_FILES[@]} file(s) could not be uploaded completely after ${MAX_UPLOAD_ATTEMPTS} attempts:" printf '[Loader] %s\n' "${FAILED_FILES[@]}" exit 1 fi echo "[Loader] Completed" # Fix file permissions kubectl exec "pod/${GDCS_MODEL_LOADER_POD_NAME}" \ --namespace="${GDCS_WORKLOAD_NAMESPACE}" \ -- chown 0:0 -R "${REMOTE_MODEL_DIR}" # Flush the volume before the loader Pod is deleted (a detached volume with unwritten page cache loses data) kubectl exec "pod/${GDCS_MODEL_LOADER_POD_NAME}" \ --namespace="${GDCS_WORKLOAD_NAMESPACE}" \ -- sync EOF chmod u+x "${GDCS_IMPLEMENTATION_HOME}/model_weight_upload.sh"
部署模型
- Gemma 模型與 vLLM 使用手冊：Gemma 4 E2B、E4B、12B、26B-A4B 和 31B，以及 GDC 實體隔離方案的 GPU 選項
- Mistral 模型與 vLLM 使用手冊：Mistral 7B Instruct、Mistral NeMo 12B、Mistral Small 3.2 24B、Ministral 8B、Mixtral 8x7B 和 Magistral Small 1.2，搭配 GDC 實體隔離方案的 GPU 選項