This document provides step-by-step instructions for deploying open weight
models from Mistral AI on Google Distributed Cloud (GDC) air-gapped environments. It covers
using vLLM for high-throughput serving, using the GDC
platform's capabilities, including Kubernetes, Harbor, and GPU resources. Select
the model and the GPU option in the Model section; every other step is the same
for all models.

## Supported models

| Model | Hugging Face repository | Parameters | License | GPU options |
|---|---|---|---|---|
| Mistral 7B Instruct | [`mistralai/Mistral-7B-Instruct-v0.2`](https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.2) | 7.2B (dense) | Apache 2.0 | 1 x any GPU |
| Mistral NeMo 12B Instruct | [`mistralai/Mistral-Nemo-Instruct-2407`](https://huggingface.co/mistralai/Mistral-Nemo-Instruct-2407) | 12B (dense) | Apache 2.0 | 1 x any GPU |
| Mistral Small 3.2 24B Instruct | [`mistralai/Mistral-Small-3.2-24B-Instruct-2506`](https://huggingface.co/mistralai/Mistral-Small-3.2-24B-Instruct-2506) | 24B (dense, with vision encoder) | Apache 2.0 | 1 x NVIDIA A100 80 GB 1 x NVIDIA H100 94 GB 1 x NVIDIA H200 141 GB or B300 288 GB |
| Ministral 8B Instruct | [`mistralai/Ministral-8B-Instruct-2410`](https://huggingface.co/mistralai/Ministral-8B-Instruct-2410) | 8B (dense) | Mistral AI Research License (research and non-commercial use only) | 1 x any GPU |
| Mixtral 8x7B Instruct | [`mistralai/Mixtral-8x7B-Instruct-v0.1`](https://huggingface.co/mistralai/Mixtral-8x7B-Instruct-v0.1) | 46.7B total / 12.9B active (mixture of experts, 8x7B) | Apache 2.0 | 2 x NVIDIA A100 80 GB 2 x NVIDIA H100 94 GB 1 x NVIDIA A100 80 GB or H100 94 GB with `--quantization=fp8` 1 x NVIDIA H200 141 GB or B300 288 GB |
| Magistral Small 1.2 24B | [`mistralai/Magistral-Small-2509`](https://huggingface.co/mistralai/Magistral-Small-2509) | 24B (dense, reasoning, with vision encoder) | Apache 2.0 | 1 x NVIDIA A100 80 GB 1 x NVIDIA H100 94 GB 1 x NVIDIA H200 141 GB or B300 288 GB |

> [!NOTE]
> **Note:** For gated Hugging Face repositories, you must accept the license terms on the model's Hugging Face repository page with the account whose access token is configured in `${HOME}/gdcag-solutions/open-weight-models/vllm/secrets/huggingface_token` before downloading the weights.

## Architecture

The solution involves deploying a containerized serving backend (vLLM) as a
Deployment within a standard cluster. Model weights are stored on Persistent
Volumes, populated from files downloaded locally and uploaded using a loader
pod. A Kubernetes `Service` of type `LoadBalancer` exposes the backend API, and
`NetworkPolicy` / `ProjectNetworkPolicy` objects secure access to it.

> [!NOTE]
> **Note:** Without a ReadOnlyMany StorageClass, this solution can deploy only one replica.

## Before you begin

> [!NOTE]
> **Note:** This guide requires GDC air-gapped 1.16.2-hf1 (1.16.2-gdch.1353-1) or later, and a standard cluster.

Complete the [Open Weight Models with vLLM reference
implementation](https://docs.cloud.google.com/distributed-cloud/hosted/docs/latest/gdcag/solutions/open-weight-models/vllm/reference-implementation)
on the workstation, and make sure you have access to the
`${HOME}/gdcag-solutions` directory.

### Identity and Access Management

Ensure that the necessary IAM accounts, roles, and permissions
are properly configured.

**GDC User** roles on **organization**:

- Optional: Standard Cluster Admin (View clusters from the console)

**GDC User** roles on **project**:

- Standard Cluster Admin (`standard-cluster-admin`)

**GDC User** role on the **standard cluster**:

- `StandardClusterRoleBinding` to the `StandardClusterRole` `cluster-admin` (created by an IAM Project Admin, see the *Identity and Access Management* section of the reference implementation)

### Workstation

Run all commands in this guide from a workstation with the required tools and
connectivity to the environment and internet, unless a step states otherwise.

The environment configuration is layered:

- `${HOME}/gdcag-solutions/env.sh` loads the platform and registry settings.
- `${HOME}/gdcag-solutions/open-weight-models/vllm/env.sh` adds the vLLM
  settings.

- This user guide adds `env.d/model.sh` (the model) and
  `env.d/accelerator.sh` (the GPU option and runtime settings), which you write in
  the Model section; and `env.d/model-common.sh` (the values derived from them).

1. Create the user guide directory structure:

   ```bash
   mkdir -p ${HOME}/gdcag-solutions/open-weight-models/vllm/mistral/env.d
   ```
2. Create the common model environment configuration file. It derives every
   path and resource name from `GDCS_HF_MODEL`, so it is the same for all
   models:

   ```bash
   cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/mistral/env.d/model-common.sh && echo "Successfully created." || echo "Failed to create!"
   # Model (derived from env.d/model.sh)
   export GDCS_HF_DOWNLOAD_PATH="${GDCS_USER_GUIDE_HOME}/hf-models"
   export GDCS_HF_MODEL_LOWERCASE="${GDCS_HF_MODEL,,}"
   export GDCS_HF_MODEL_DOWNLOAD_DIR="${GDCS_HF_DOWNLOAD_PATH}/${GDCS_HF_MODEL_LOWERCASE}"
   export GDCS_HF_PV_MOUNT_PATH="/mnt/hf-model"

   GDCS_HF_MODEL_KUBERNETES="${GDCS_HF_MODEL_LOWERCASE//\//-}"
   export GDCS_HF_MODEL_KUBERNETES="${GDCS_HF_MODEL_KUBERNETES//./-}"

   # Workload
   export GDCS_WORKLOAD_NAMESPACE="vllm-mistral"
   export GDCS_WORKLOAD_NAME="${GDCS_HF_MODEL_KUBERNETES}"
   export GDCS_MODEL_LOADER_POD_NAME="loader-${GDCS_HF_MODEL_KUBERNETES}"
   EOF
   ```
3. Create the user guide environment loader file:

   ```bash
   cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/mistral/env.sh && echo "Successfully created." || echo "Failed to create!"
   source "${HOME}/gdcag-solutions/open-weight-models/vllm/env.sh"

   export GDCS_USER_GUIDE_HOME="${HOME}/gdcag-solutions/open-weight-models/vllm/mistral"
   echo "GDCS_USER_GUIDE_HOME=${GDCS_USER_GUIDE_HOME}"

   # Sourced in dependency order: the model selection, the accelerator and runtime settings, then the values derived from them
   source "${GDCS_USER_GUIDE_HOME}/env.d/model.sh"
   source "${GDCS_USER_GUIDE_HOME}/env.d/accelerator.sh"
   source "${GDCS_USER_GUIDE_HOME}/env.d/model-common.sh"
   echo "GDCS_HF_MODEL=${GDCS_HF_MODEL} (${GDCS_WORKLOAD_ACCELERATOR_COUNT} x ${GDCS_WORKLOAD_ACCELERATOR_TYPE:-unset})"
   EOF
   ```

### Model

Select the model and the GPU option that matches the GPU nodes of your standard
cluster. GDC air-gapped offers the following GPUs (see
[Cluster node machine
types](https://docs.cloud.google.com/distributed-cloud/hosted/docs/latest/gdcag/platform/pa-user/cluster-node-machines)); the accelerator
resource type is the name under which the nodes expose the GPU to Kubernetes.

| GPU | Memory | Machine types | Accelerator resource type |
|---|---|---|---|
| NVIDIA A100 | 40 GB | `a2-highgpu-1g-gdc` | `nvidia.com/gpu-pod-NVIDIA_A100_PCIE_40GB` |
| NVIDIA A100 | 80 GB | `a2-ultragpu-1g-gdc`, `a2-ultragpu-2g-gdc` | `nvidia.com/gpu-pod-NVIDIA_A100_80GB_PCIE` |
| NVIDIA H100 | 94 GB | `a3-highgpu-1g-gdc` (Preview), `a3-highgpu-2g-gdc` (Preview), `a3-highgpu-4g-gdc` | `nvidia.com/gpu-pod-NVIDIA_H100_NVL` |
| NVIDIA H200 | 141 GB | `a3-ultragpu-8g-gdc` | `nvidia.com/gpu-pod-NVIDIA_H200` |
| NVIDIA B300 | 288 GB | `a4-ultragpu-8g-gdc` | `nvidia.com/gpu-pod-NVIDIA_B300_SXM6_AC` |

Each tab lists the model facts, the command that writes the model environment
file `env.d/model.sh`, and nested tabs that write the accelerator environment
file `env.d/accelerator.sh` for your chosen GPU option with pre-filled resource
types and vLLM settings. To switch models or GPU options later, write different
environment files, source the environment again, and repeat the Implementation
section; the workloads of several models can run side by side in the same
namespace because every resource name is derived from the model name.

1. Create the model and accelerator environment files for the model and GPU
   option you want to deploy.

   ### Mistral 7B Instruct

   | Attribute | Value |
   |---|---|
   | **Hugging Face repository** | [`mistralai/Mistral-7B-Instruct-v0.2`](https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.2) |
   | **Parameters** | 7.2B (dense) |
   | **License** | Apache 2.0 |
   | **Weights used by vLLM** | Hugging Face format (`model-*.safetensors`), about 14.5 GB |
   | **Context length** | 32k tokens; this guide serves `--max-model-len 32768` |

   Create the model environment file:

   ```bash
   cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/mistral/env.d/model.sh && echo "Successfully created." || echo "Failed to create!"
   # Model: Mistral 7B Instruct
   export GDCS_HF_MODEL="mistralai/Mistral-7B-Instruct-v0.2"
   export GDCS_HF_MODEL_REVISION="main"  # Hugging Face branch, tag, or commit to download
   export GDCS_HF_DOWNLOAD_EXCLUDE="consolidated* *.bin"
   EOF
   ```

   Create the accelerator environment file for your GPU option:

   ### NVIDIA A100 40 GB

   ```bash
   cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/mistral/env.d/accelerator.sh && echo "Successfully created." || echo "Failed to create!"
   # Accelerator: NVIDIA A100 40 GB
   export GDCS_WORKLOAD_ACCELERATOR_TYPE="nvidia.com/gpu-pod-NVIDIA_A100_PCIE_40GB"
   export GDCS_WORKLOAD_ACCELERATOR_COUNT="1"

   # vLLM
   export GDCS_VLLM_MAX_MODEL_LEN="32768"
   export GDCS_VLLM_EXTRA_ARGS=""
   export GDCS_VLLM_STARTUP_FAILURE_THRESHOLD="60"
   export GDCS_VLLM_TEST_MAX_TOKENS="128"
   EOF
   ```

   ### NVIDIA A100 80 GB

   ```bash
   cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/mistral/env.d/accelerator.sh && echo "Successfully created." || echo "Failed to create!"
   # Accelerator: NVIDIA A100 80 GB
   export GDCS_WORKLOAD_ACCELERATOR_TYPE="nvidia.com/gpu-pod-NVIDIA_A100_80GB_PCIE"
   export GDCS_WORKLOAD_ACCELERATOR_COUNT="1"

   # vLLM
   export GDCS_VLLM_MAX_MODEL_LEN="32768"
   export GDCS_VLLM_EXTRA_ARGS=""
   export GDCS_VLLM_STARTUP_FAILURE_THRESHOLD="60"
   export GDCS_VLLM_TEST_MAX_TOKENS="128"
   EOF
   ```

   ### NVIDIA H100 94 GB

   ```bash
   cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/mistral/env.d/accelerator.sh && echo "Successfully created." || echo "Failed to create!"
   # Accelerator: NVIDIA H100 94 GB
   export GDCS_WORKLOAD_ACCELERATOR_TYPE="nvidia.com/gpu-pod-NVIDIA_H100_NVL"
   export GDCS_WORKLOAD_ACCELERATOR_COUNT="1"

   # vLLM
   export GDCS_VLLM_MAX_MODEL_LEN="32768"
   export GDCS_VLLM_EXTRA_ARGS=""
   export GDCS_VLLM_STARTUP_FAILURE_THRESHOLD="60"
   export GDCS_VLLM_TEST_MAX_TOKENS="128"
   EOF
   ```

   ### NVIDIA H200 141 GB

   ```bash
   cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/mistral/env.d/accelerator.sh && echo "Successfully created." || echo "Failed to create!"
   # Accelerator: NVIDIA H200 141 GB
   export GDCS_WORKLOAD_ACCELERATOR_TYPE="nvidia.com/gpu-pod-NVIDIA_H200"
   export GDCS_WORKLOAD_ACCELERATOR_COUNT="1"

   # vLLM
   export GDCS_VLLM_MAX_MODEL_LEN="32768"
   export GDCS_VLLM_EXTRA_ARGS=""
   export GDCS_VLLM_STARTUP_FAILURE_THRESHOLD="60"
   export GDCS_VLLM_TEST_MAX_TOKENS="128"
   EOF
   ```

   ### NVIDIA B300 288 GB

   ```bash
   cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/mistral/env.d/accelerator.sh && echo "Successfully created." || echo "Failed to create!"
   # Accelerator: NVIDIA B300 288 GB
   export GDCS_WORKLOAD_ACCELERATOR_TYPE="nvidia.com/gpu-pod-NVIDIA_B300_SXM6_AC"
   export GDCS_WORKLOAD_ACCELERATOR_COUNT="1"

   # vLLM
   export GDCS_VLLM_MAX_MODEL_LEN="32768"
   export GDCS_VLLM_EXTRA_ARGS=""
   export GDCS_VLLM_STARTUP_FAILURE_THRESHOLD="60"
   export GDCS_VLLM_TEST_MAX_TOKENS="128"
   EOF
   ```

   ### Mistral NeMo 12B Instruct

   | Attribute | Value |
   |---|---|
   | **Hugging Face repository** | [`mistralai/Mistral-Nemo-Instruct-2407`](https://huggingface.co/mistralai/Mistral-Nemo-Instruct-2407) |
   | **Parameters** | 12B (dense) |
   | **License** | Apache 2.0 |
   | **Weights used by vLLM** | Mistral format (`consolidated.safetensors`, `params.json`, `tekken.json`), about 24.5 GB |
   | **Context length** | 128k tokens; this guide serves `--max-model-len 32768` |

   > [!NOTE]
   > **Note:** vLLM detects Mistral format (`consolidated.safetensors`, `params.json`, `tekken.json`) automatically, so the download skips the Hugging Face `model-*.safetensors` shards. To serve longer prompts on GPUs with more memory, raise `GDCS_VLLM_MAX_MODEL_LEN`.

   Create the model environment file:

   ```bash
   cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/mistral/env.d/model.sh && echo "Successfully created." || echo "Failed to create!"
   # Model: Mistral NeMo 12B Instruct
   export GDCS_HF_MODEL="mistralai/Mistral-Nemo-Instruct-2407"
   export GDCS_HF_MODEL_REVISION="main"  # Hugging Face branch, tag, or commit to download
   export GDCS_HF_DOWNLOAD_EXCLUDE="model-*.safetensors *.bin"
   EOF
   ```

   Create the accelerator environment file for your GPU option:

   ### NVIDIA A100 40 GB

   ```bash
   cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/mistral/env.d/accelerator.sh && echo "Successfully created." || echo "Failed to create!"
   # Accelerator: NVIDIA A100 40 GB
   export GDCS_WORKLOAD_ACCELERATOR_TYPE="nvidia.com/gpu-pod-NVIDIA_A100_PCIE_40GB"
   export GDCS_WORKLOAD_ACCELERATOR_COUNT="1"

   # vLLM
   export GDCS_VLLM_MAX_MODEL_LEN="32768"
   export GDCS_VLLM_EXTRA_ARGS=""
   export GDCS_VLLM_STARTUP_FAILURE_THRESHOLD="180"
   export GDCS_VLLM_TEST_MAX_TOKENS="128"
   EOF
   ```

   ### NVIDIA A100 80 GB

   ```bash
   cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/mistral/env.d/accelerator.sh && echo "Successfully created." || echo "Failed to create!"
   # Accelerator: NVIDIA A100 80 GB
   export GDCS_WORKLOAD_ACCELERATOR_TYPE="nvidia.com/gpu-pod-NVIDIA_A100_80GB_PCIE"
   export GDCS_WORKLOAD_ACCELERATOR_COUNT="1"

   # vLLM
   export GDCS_VLLM_MAX_MODEL_LEN="32768"
   export GDCS_VLLM_EXTRA_ARGS=""
   export GDCS_VLLM_STARTUP_FAILURE_THRESHOLD="180"
   export GDCS_VLLM_TEST_MAX_TOKENS="128"
   EOF
   ```

   ### NVIDIA H100 94 GB

   ```bash
   cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/mistral/env.d/accelerator.sh && echo "Successfully created." || echo "Failed to create!"
   # Accelerator: NVIDIA H100 94 GB
   export GDCS_WORKLOAD_ACCELERATOR_TYPE="nvidia.com/gpu-pod-NVIDIA_H100_NVL"
   export GDCS_WORKLOAD_ACCELERATOR_COUNT="1"

   # vLLM
   export GDCS_VLLM_MAX_MODEL_LEN="32768"
   export GDCS_VLLM_EXTRA_ARGS=""
   export GDCS_VLLM_STARTUP_FAILURE_THRESHOLD="180"
   export GDCS_VLLM_TEST_MAX_TOKENS="128"
   EOF
   ```

   ### NVIDIA H200 141 GB

   ```bash
   cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/mistral/env.d/accelerator.sh && echo "Successfully created." || echo "Failed to create!"
   # Accelerator: NVIDIA H200 141 GB
   export GDCS_WORKLOAD_ACCELERATOR_TYPE="nvidia.com/gpu-pod-NVIDIA_H200"
   export GDCS_WORKLOAD_ACCELERATOR_COUNT="1"

   # vLLM
   export GDCS_VLLM_MAX_MODEL_LEN="32768"
   export GDCS_VLLM_EXTRA_ARGS=""
   export GDCS_VLLM_STARTUP_FAILURE_THRESHOLD="180"
   export GDCS_VLLM_TEST_MAX_TOKENS="128"
   EOF
   ```

   ### NVIDIA B300 288 GB

   ```bash
   cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/mistral/env.d/accelerator.sh && echo "Successfully created." || echo "Failed to create!"
   # Accelerator: NVIDIA B300 288 GB
   export GDCS_WORKLOAD_ACCELERATOR_TYPE="nvidia.com/gpu-pod-NVIDIA_B300_SXM6_AC"
   export GDCS_WORKLOAD_ACCELERATOR_COUNT="1"

   # vLLM
   export GDCS_VLLM_MAX_MODEL_LEN="32768"
   export GDCS_VLLM_EXTRA_ARGS=""
   export GDCS_VLLM_STARTUP_FAILURE_THRESHOLD="180"
   export GDCS_VLLM_TEST_MAX_TOKENS="128"
   EOF
   ```

   ### Mistral Small 3.2 24B Instruct

   | Attribute | Value |
   |---|---|
   | **Hugging Face repository** | [`mistralai/Mistral-Small-3.2-24B-Instruct-2506`](https://huggingface.co/mistralai/Mistral-Small-3.2-24B-Instruct-2506) |
   | **Parameters** | 24B (dense, with vision encoder) |
   | **License** | Apache 2.0 |
   | **Weights used by vLLM** | Mistral format (`consolidated.safetensors`, `params.json`, `tekken.json`), about 48 GB |
   | **Context length** | 128k tokens; this guide serves `--max-model-len 32768` |

   > [!NOTE]
   > **Note:** vLLM detects Mistral format (`consolidated.safetensors`, `params.json`, `tekken.json`) automatically, so the download skips the Hugging Face `model-*.safetensors` shards.

   Create the model environment file:

   ```bash
   cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/mistral/env.d/model.sh && echo "Successfully created." || echo "Failed to create!"
   # Model: Mistral Small 3.2 24B Instruct
   export GDCS_HF_MODEL="mistralai/Mistral-Small-3.2-24B-Instruct-2506"
   export GDCS_HF_MODEL_REVISION="main"  # Hugging Face branch, tag, or commit to download
   export GDCS_HF_DOWNLOAD_EXCLUDE="model-*.safetensors *.bin"
   EOF
   ```

   Create the accelerator environment file for your GPU option:

   ### NVIDIA A100 80 GB

   ```bash
   cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/mistral/env.d/accelerator.sh && echo "Successfully created." || echo "Failed to create!"
   # Accelerator: NVIDIA A100 80 GB
   export GDCS_WORKLOAD_ACCELERATOR_TYPE="nvidia.com/gpu-pod-NVIDIA_A100_80GB_PCIE"
   export GDCS_WORKLOAD_ACCELERATOR_COUNT="1"

   # vLLM
   export GDCS_VLLM_MAX_MODEL_LEN="32768"
   export GDCS_VLLM_EXTRA_ARGS=""
   export GDCS_VLLM_STARTUP_FAILURE_THRESHOLD="300"
   export GDCS_VLLM_TEST_MAX_TOKENS="128"
   EOF
   ```

   ### NVIDIA H100 94 GB

   ```bash
   cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/mistral/env.d/accelerator.sh && echo "Successfully created." || echo "Failed to create!"
   # Accelerator: NVIDIA H100 94 GB
   export GDCS_WORKLOAD_ACCELERATOR_TYPE="nvidia.com/gpu-pod-NVIDIA_H100_NVL"
   export GDCS_WORKLOAD_ACCELERATOR_COUNT="1"

   # vLLM
   export GDCS_VLLM_MAX_MODEL_LEN="32768"
   export GDCS_VLLM_EXTRA_ARGS=""
   export GDCS_VLLM_STARTUP_FAILURE_THRESHOLD="300"
   export GDCS_VLLM_TEST_MAX_TOKENS="128"
   EOF
   ```

   ### NVIDIA H200 141 GB

   ```bash
   cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/mistral/env.d/accelerator.sh && echo "Successfully created." || echo "Failed to create!"
   # Accelerator: NVIDIA H200 141 GB
   export GDCS_WORKLOAD_ACCELERATOR_TYPE="nvidia.com/gpu-pod-NVIDIA_H200"
   export GDCS_WORKLOAD_ACCELERATOR_COUNT="1"

   # vLLM
   export GDCS_VLLM_MAX_MODEL_LEN="32768"
   export GDCS_VLLM_EXTRA_ARGS=""
   export GDCS_VLLM_STARTUP_FAILURE_THRESHOLD="300"
   export GDCS_VLLM_TEST_MAX_TOKENS="128"
   EOF
   ```

   ### NVIDIA B300 288 GB

   ```bash
   cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/mistral/env.d/accelerator.sh && echo "Successfully created." || echo "Failed to create!"
   # Accelerator: NVIDIA B300 288 GB
   export GDCS_WORKLOAD_ACCELERATOR_TYPE="nvidia.com/gpu-pod-NVIDIA_B300_SXM6_AC"
   export GDCS_WORKLOAD_ACCELERATOR_COUNT="1"

   # vLLM
   export GDCS_VLLM_MAX_MODEL_LEN="32768"
   export GDCS_VLLM_EXTRA_ARGS=""
   export GDCS_VLLM_STARTUP_FAILURE_THRESHOLD="300"
   export GDCS_VLLM_TEST_MAX_TOKENS="128"
   EOF
   ```

   ### Ministral 8B Instruct

   | Attribute | Value |
   |---|---|
   | **Hugging Face repository** | [`mistralai/Ministral-8B-Instruct-2410`](https://huggingface.co/mistralai/Ministral-8B-Instruct-2410) |
   | **Parameters** | 8B (dense) |
   | **License** | Mistral AI Research License (research and non-commercial use only) |
   | **Weights used by vLLM** | Mistral format (`consolidated.safetensors`, `params.json`, `tekken.json`), about 16 GB |
   | **Context length** | 128k tokens; this guide serves `--max-model-len 32768` |

   > [!NOTE]
   > **Note:** vLLM detects Mistral format (`consolidated.safetensors`, `params.json`, `tekken.json`) automatically, so the download skips the Hugging Face `model-*.safetensors` shards.

   > [!NOTE]
   > **Note:** The Mistral AI Research License permits research and non-commercial use only. Get a commercial license from Mistral AI before you use Ministral 8B in production.

   Create the model environment file:

   ```bash
   cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/mistral/env.d/model.sh && echo "Successfully created." || echo "Failed to create!"
   # Model: Ministral 8B Instruct
   export GDCS_HF_MODEL="mistralai/Ministral-8B-Instruct-2410"
   export GDCS_HF_MODEL_REVISION="main"  # Hugging Face branch, tag, or commit to download
   export GDCS_HF_DOWNLOAD_EXCLUDE="model-*.safetensors *.bin"
   EOF
   ```

   Create the accelerator environment file for your GPU option:

   ### NVIDIA A100 40 GB

   ```bash
   cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/mistral/env.d/accelerator.sh && echo "Successfully created." || echo "Failed to create!"
   # Accelerator: NVIDIA A100 40 GB
   export GDCS_WORKLOAD_ACCELERATOR_TYPE="nvidia.com/gpu-pod-NVIDIA_A100_PCIE_40GB"
   export GDCS_WORKLOAD_ACCELERATOR_COUNT="1"

   # vLLM
   export GDCS_VLLM_MAX_MODEL_LEN="32768"
   export GDCS_VLLM_EXTRA_ARGS=""
   export GDCS_VLLM_STARTUP_FAILURE_THRESHOLD="120"
   export GDCS_VLLM_TEST_MAX_TOKENS="128"
   EOF
   ```

   ### NVIDIA A100 80 GB

   ```bash
   cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/mistral/env.d/accelerator.sh && echo "Successfully created." || echo "Failed to create!"
   # Accelerator: NVIDIA A100 80 GB
   export GDCS_WORKLOAD_ACCELERATOR_TYPE="nvidia.com/gpu-pod-NVIDIA_A100_80GB_PCIE"
   export GDCS_WORKLOAD_ACCELERATOR_COUNT="1"

   # vLLM
   export GDCS_VLLM_MAX_MODEL_LEN="32768"
   export GDCS_VLLM_EXTRA_ARGS=""
   export GDCS_VLLM_STARTUP_FAILURE_THRESHOLD="120"
   export GDCS_VLLM_TEST_MAX_TOKENS="128"
   EOF
   ```

   ### NVIDIA H100 94 GB

   ```bash
   cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/mistral/env.d/accelerator.sh && echo "Successfully created." || echo "Failed to create!"
   # Accelerator: NVIDIA H100 94 GB
   export GDCS_WORKLOAD_ACCELERATOR_TYPE="nvidia.com/gpu-pod-NVIDIA_H100_NVL"
   export GDCS_WORKLOAD_ACCELERATOR_COUNT="1"

   # vLLM
   export GDCS_VLLM_MAX_MODEL_LEN="32768"
   export GDCS_VLLM_EXTRA_ARGS=""
   export GDCS_VLLM_STARTUP_FAILURE_THRESHOLD="120"
   export GDCS_VLLM_TEST_MAX_TOKENS="128"
   EOF
   ```

   ### NVIDIA H200 141 GB

   ```bash
   cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/mistral/env.d/accelerator.sh && echo "Successfully created." || echo "Failed to create!"
   # Accelerator: NVIDIA H200 141 GB
   export GDCS_WORKLOAD_ACCELERATOR_TYPE="nvidia.com/gpu-pod-NVIDIA_H200"
   export GDCS_WORKLOAD_ACCELERATOR_COUNT="1"

   # vLLM
   export GDCS_VLLM_MAX_MODEL_LEN="32768"
   export GDCS_VLLM_EXTRA_ARGS=""
   export GDCS_VLLM_STARTUP_FAILURE_THRESHOLD="120"
   export GDCS_VLLM_TEST_MAX_TOKENS="128"
   EOF
   ```

   ### NVIDIA B300 288 GB

   ```bash
   cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/mistral/env.d/accelerator.sh && echo "Successfully created." || echo "Failed to create!"
   # Accelerator: NVIDIA B300 288 GB
   export GDCS_WORKLOAD_ACCELERATOR_TYPE="nvidia.com/gpu-pod-NVIDIA_B300_SXM6_AC"
   export GDCS_WORKLOAD_ACCELERATOR_COUNT="1"

   # vLLM
   export GDCS_VLLM_MAX_MODEL_LEN="32768"
   export GDCS_VLLM_EXTRA_ARGS=""
   export GDCS_VLLM_STARTUP_FAILURE_THRESHOLD="120"
   export GDCS_VLLM_TEST_MAX_TOKENS="128"
   EOF
   ```

   ### Mixtral 8x7B Instruct

   | Attribute | Value |
   |---|---|
   | **Hugging Face repository** | [`mistralai/Mixtral-8x7B-Instruct-v0.1`](https://huggingface.co/mistralai/Mixtral-8x7B-Instruct-v0.1) |
   | **Parameters** | 46.7B total / 12.9B active (mixture of experts, 8x7B) |
   | **License** | Apache 2.0 |
   | **Weights used by vLLM** | Hugging Face format (`model-*.safetensors`), about 93 GB (the repository also carries `consolidated.*.pt` PyTorch files, excluded) |
   | **Context length** | 32k tokens; this guide serves `--max-model-len 32768` |

   > [!NOTE]
   > **Note:** Mixtral is a sparse mixture-of-experts model (8 experts, 2 active per token): about 12.9B parameters are active per token, but all 46.7B must be resident in GPU memory. The bf16 weights (about 93 GB) don't fit one 80 GB or 94 GB GPU: use two GPUs (tensor parallel), FP8 weight-only quantization on one GPU, or one H200 or B300 GPU.

   Create the model environment file:

   ```bash
   cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/mistral/env.d/model.sh && echo "Successfully created." || echo "Failed to create!"
   # Model: Mixtral 8x7B Instruct
   export GDCS_HF_MODEL="mistralai/Mixtral-8x7B-Instruct-v0.1"
   export GDCS_HF_MODEL_REVISION="main"  # Hugging Face branch, tag, or commit to download
   export GDCS_HF_DOWNLOAD_EXCLUDE="consolidated* *.bin"
   EOF
   ```

   Create the accelerator environment file for your GPU option:

   ### NVIDIA A100 80 GB (2 GPUs)

   Requires a node with two A100 80 GB GPUs (`a2-ultragpu-2g-gdc`);
   vLLM runs with `--tensor-parallel-size 2`.

   ```bash
   cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/mistral/env.d/accelerator.sh && echo "Successfully created." || echo "Failed to create!"
   # Accelerator: NVIDIA A100 80 GB (2 GPUs)
   export GDCS_WORKLOAD_ACCELERATOR_TYPE="nvidia.com/gpu-pod-NVIDIA_A100_80GB_PCIE"
   export GDCS_WORKLOAD_ACCELERATOR_COUNT="2"

   # vLLM
   export GDCS_VLLM_MAX_MODEL_LEN="32768"
   export GDCS_VLLM_EXTRA_ARGS=""
   export GDCS_VLLM_STARTUP_FAILURE_THRESHOLD="480"
   export GDCS_VLLM_TEST_MAX_TOKENS="128"
   EOF
   ```

   ### NVIDIA H100 94 GB (2 GPUs)

   Requires a node with at least two H100 GPUs (`a3-highgpu-2g-gdc` or
   `a3-highgpu-4g-gdc`); vLLM runs with `--tensor-parallel-size 2`.

   ```bash
   cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/mistral/env.d/accelerator.sh && echo "Successfully created." || echo "Failed to create!"
   # Accelerator: NVIDIA H100 94 GB (2 GPUs)
   export GDCS_WORKLOAD_ACCELERATOR_TYPE="nvidia.com/gpu-pod-NVIDIA_H100_NVL"
   export GDCS_WORKLOAD_ACCELERATOR_COUNT="2"

   # vLLM
   export GDCS_VLLM_MAX_MODEL_LEN="32768"
   export GDCS_VLLM_EXTRA_ARGS=""
   export GDCS_VLLM_STARTUP_FAILURE_THRESHOLD="480"
   export GDCS_VLLM_TEST_MAX_TOKENS="128"
   EOF
   ```

   ### NVIDIA A100 80 GB (FP8)

   `--quantization=fp8` quantizes the weights to FP8 while loading
   (about 47 GB instead of 93 GB), with slightly lower output quality.

   ```bash
   cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/mistral/env.d/accelerator.sh && echo "Successfully created." || echo "Failed to create!"
   # Accelerator: NVIDIA A100 80 GB (FP8)
   export GDCS_WORKLOAD_ACCELERATOR_TYPE="nvidia.com/gpu-pod-NVIDIA_A100_80GB_PCIE"
   export GDCS_WORKLOAD_ACCELERATOR_COUNT="1"

   # vLLM
   export GDCS_VLLM_MAX_MODEL_LEN="32768"
   export GDCS_VLLM_EXTRA_ARGS="--quantization=fp8"
   export GDCS_VLLM_STARTUP_FAILURE_THRESHOLD="480"
   export GDCS_VLLM_TEST_MAX_TOKENS="128"
   EOF
   ```

   ### NVIDIA H100 94 GB (FP8)

   `--quantization=fp8` quantizes the weights to FP8 while loading
   (about 47 GB instead of 93 GB), with slightly lower output quality.

   ```bash
   cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/mistral/env.d/accelerator.sh && echo "Successfully created." || echo "Failed to create!"
   # Accelerator: NVIDIA H100 94 GB (FP8)
   export GDCS_WORKLOAD_ACCELERATOR_TYPE="nvidia.com/gpu-pod-NVIDIA_H100_NVL"
   export GDCS_WORKLOAD_ACCELERATOR_COUNT="1"

   # vLLM
   export GDCS_VLLM_MAX_MODEL_LEN="32768"
   export GDCS_VLLM_EXTRA_ARGS="--quantization=fp8"
   export GDCS_VLLM_STARTUP_FAILURE_THRESHOLD="480"
   export GDCS_VLLM_TEST_MAX_TOKENS="128"
   EOF
   ```

   ### NVIDIA H200 141 GB

   ```bash
   cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/mistral/env.d/accelerator.sh && echo "Successfully created." || echo "Failed to create!"
   # Accelerator: NVIDIA H200 141 GB
   export GDCS_WORKLOAD_ACCELERATOR_TYPE="nvidia.com/gpu-pod-NVIDIA_H200"
   export GDCS_WORKLOAD_ACCELERATOR_COUNT="1"

   # vLLM
   export GDCS_VLLM_MAX_MODEL_LEN="32768"
   export GDCS_VLLM_EXTRA_ARGS=""
   export GDCS_VLLM_STARTUP_FAILURE_THRESHOLD="480"
   export GDCS_VLLM_TEST_MAX_TOKENS="128"
   EOF
   ```

   ### NVIDIA B300 288 GB

   ```bash
   cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/mistral/env.d/accelerator.sh && echo "Successfully created." || echo "Failed to create!"
   # Accelerator: NVIDIA B300 288 GB
   export GDCS_WORKLOAD_ACCELERATOR_TYPE="nvidia.com/gpu-pod-NVIDIA_B300_SXM6_AC"
   export GDCS_WORKLOAD_ACCELERATOR_COUNT="1"

   # vLLM
   export GDCS_VLLM_MAX_MODEL_LEN="32768"
   export GDCS_VLLM_EXTRA_ARGS=""
   export GDCS_VLLM_STARTUP_FAILURE_THRESHOLD="480"
   export GDCS_VLLM_TEST_MAX_TOKENS="128"
   EOF
   ```

   ### Magistral Small 1.2 24B

   | Attribute | Value |
   |---|---|
   | **Hugging Face repository** | [`mistralai/Magistral-Small-2509`](https://huggingface.co/mistralai/Magistral-Small-2509) |
   | **Parameters** | 24B (dense, reasoning, with vision encoder) |
   | **License** | Apache 2.0 |
   | **Weights used by vLLM** | Mistral format (`consolidated.safetensors`, `params.json`, `tekken.json`), about 48 GB |
   | **Context length** | 128k tokens (40k recommended by Mistral); this guide serves `--max-model-len 40960` |

   > [!NOTE]
   > **Note:** vLLM detects Mistral format (`consolidated.safetensors`, `params.json`, `tekken.json`) automatically, so the download skips the Hugging Face `model-*.safetensors` shards.

   > [!NOTE]
   > **Note:** Magistral Small 1.2 is a reasoning model. It reasons only when the request includes Mistral's recommended system prompt (`SYSTEM_PROMPT.txt`, uploaded to the PersistentVolume with the weights); vLLM then returns the reasoning in `message.reasoning_content` and the answer in `message.content`. Reasoning consumes output tokens, so this guide sets a larger `GDCS_VLLM_TEST_MAX_TOKENS`.

   Create the model environment file:

   ```bash
   cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/mistral/env.d/model.sh && echo "Successfully created." || echo "Failed to create!"
   # Model: Magistral Small 1.2 24B
   export GDCS_HF_MODEL="mistralai/Magistral-Small-2509"
   export GDCS_HF_MODEL_REVISION="main"  # Hugging Face branch, tag, or commit to download
   export GDCS_HF_DOWNLOAD_EXCLUDE="model-*.safetensors *.bin"
   EOF
   ```

   Create the accelerator environment file for your GPU option:

   ### NVIDIA A100 80 GB

   ```bash
   cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/mistral/env.d/accelerator.sh && echo "Successfully created." || echo "Failed to create!"
   # Accelerator: NVIDIA A100 80 GB
   export GDCS_WORKLOAD_ACCELERATOR_TYPE="nvidia.com/gpu-pod-NVIDIA_A100_80GB_PCIE"
   export GDCS_WORKLOAD_ACCELERATOR_COUNT="1"

   # vLLM
   export GDCS_VLLM_MAX_MODEL_LEN="40960"
   export GDCS_VLLM_EXTRA_ARGS="--reasoning-parser=mistral"
   export GDCS_VLLM_STARTUP_FAILURE_THRESHOLD="300"
   export GDCS_VLLM_TEST_MAX_TOKENS="1024"
   EOF
   ```

   ### NVIDIA H100 94 GB

   ```bash
   cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/mistral/env.d/accelerator.sh && echo "Successfully created." || echo "Failed to create!"
   # Accelerator: NVIDIA H100 94 GB
   export GDCS_WORKLOAD_ACCELERATOR_TYPE="nvidia.com/gpu-pod-NVIDIA_H100_NVL"
   export GDCS_WORKLOAD_ACCELERATOR_COUNT="1"

   # vLLM
   export GDCS_VLLM_MAX_MODEL_LEN="40960"
   export GDCS_VLLM_EXTRA_ARGS="--reasoning-parser=mistral"
   export GDCS_VLLM_STARTUP_FAILURE_THRESHOLD="300"
   export GDCS_VLLM_TEST_MAX_TOKENS="1024"
   EOF
   ```

   ### NVIDIA H200 141 GB

   ```bash
   cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/mistral/env.d/accelerator.sh && echo "Successfully created." || echo "Failed to create!"
   # Accelerator: NVIDIA H200 141 GB
   export GDCS_WORKLOAD_ACCELERATOR_TYPE="nvidia.com/gpu-pod-NVIDIA_H200"
   export GDCS_WORKLOAD_ACCELERATOR_COUNT="1"

   # vLLM
   export GDCS_VLLM_MAX_MODEL_LEN="40960"
   export GDCS_VLLM_EXTRA_ARGS="--reasoning-parser=mistral"
   export GDCS_VLLM_STARTUP_FAILURE_THRESHOLD="300"
   export GDCS_VLLM_TEST_MAX_TOKENS="1024"
   EOF
   ```

   ### NVIDIA B300 288 GB

   ```bash
   cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/mistral/env.d/accelerator.sh && echo "Successfully created." || echo "Failed to create!"
   # Accelerator: NVIDIA B300 288 GB
   export GDCS_WORKLOAD_ACCELERATOR_TYPE="nvidia.com/gpu-pod-NVIDIA_B300_SXM6_AC"
   export GDCS_WORKLOAD_ACCELERATOR_COUNT="1"

   # vLLM
   export GDCS_VLLM_MAX_MODEL_LEN="40960"
   export GDCS_VLLM_EXTRA_ARGS="--reasoning-parser=mistral"
   export GDCS_VLLM_STARTUP_FAILURE_THRESHOLD="300"
   export GDCS_VLLM_TEST_MAX_TOKENS="1024"
   EOF
   ```
2. Source the environment file:

   ```bash
   source ${HOME}/gdcag-solutions/open-weight-models/vllm/mistral/env.sh
   ```

   The output is similar to the following:

       GDCS_ROOT_HOME=HOME_DIRECTORY_PATH/gdcag-solutions
       GDCS_IMPLEMENTATION_HOME=HOME_DIRECTORY_PATH/gdcag-solutions/open-weight-models/vllm
       GDCS_USER_GUIDE_HOME=HOME_DIRECTORY_PATH/gdcag-solutions/open-weight-models/vllm/mistral
       GDCS_HF_MODEL=HUGGING_FACE_REPOSITORY (ACCELERATOR_COUNT x ACCELERATOR_RESOURCE_TYPE)

> [!NOTE]
> **Note:** Any time a new terminal or shell is started, the environment file needs to be sourced.

### Cluster

1. Configure `gdcloud`:

   ```bash
   gdcloud config set core/account "default-user"
   gdcloud config set core/organization_console_url "${GDC_ZONAL_CONSOLE_URL}"
   gdcloud config set core/project "${GDC_PROJECT}"
   gdcloud config set core/zone "${GDC_ZONE}"
   ```
2. Authenticate to the GDC environment:

   ```bash
   gdcloud auth login
   ```
3. Retrieve cluster credentials:

   ```bash
   gdcloud clusters get-credentials "${GDC_STANDARD_CLUSTER_NAME}" \
   --project="${GDC_PROJECT}" \
   --standard \
   --zone="${GDC_ZONE}"
   ```
4. Verify connectivity to the cluster:

   ```bash
   kubectl get nodes -L node.cluster.private.gdc.goog/machine-class
   ```
5. Verify that the accelerator resource type of the accelerator environment
   file is available in the cluster. The command lists the accelerator resource
   types that the nodes expose (the name depends on the installed GPU model; it
   is empty on clusters without GPU nodes) and the configured value:

   ```bash
   kubectl get nodes --output=json | jq -r '[.items[].status.allocatable | keys[] | select(startswith("nvidia.com/"))] | unique[]'
   echo "GDCS_WORKLOAD_ACCELERATOR_TYPE=${GDCS_WORKLOAD_ACCELERATOR_TYPE}"
   ```

   > [!NOTE]
   > **Note:** If the configured value isn't in the list, set `GDCS_WORKLOAD_ACCELERATOR_TYPE` in `env.d/accelerator.sh` to one of the listed resource types and source the environment file again.

## Implementation

### Namespace

1. Create the namespace:

   ```bash
   kubectl create namespace "${GDCS_WORKLOAD_NAMESPACE}" \
   --dry-run=client \
   --output=yaml | kubectl apply -f -
   ```
2. Add the `imagePullSecret`:

   ```bash
   kubectl create secret docker-registry "${GDCS_HARBOR_K8S_PULL_SECRET}" \
   --dry-run=client \
   --from-file=.dockerconfigjson=${GDCS_HARBOR_K8S_DOCKER_CONFIG}/config.json \
   --namespace="${GDCS_WORKLOAD_NAMESPACE}" \
   --output=yaml | kubectl apply -f -
   ```

### Model weights

1. Download the model weights (only the files vLLM needs for the selected
   model, `GDCS_HF_DOWNLOAD_EXCLUDE` skips the rest) and remove the `.cache`
   directory:

   ```bash
   EXCLUDE_ARGS=()
   if [[ -n "${GDCS_HF_DOWNLOAD_EXCLUDE:-}" ]]; then
     read -r -a EXCLUDE_PATTERNS <<< "${GDCS_HF_DOWNLOAD_EXCLUDE}"
     EXCLUDE_ARGS=(--exclude "${EXCLUDE_PATTERNS[@]}")
   fi
   hf download "${GDCS_HF_MODEL}" --revision "${GDCS_HF_MODEL_REVISION}" --local-dir "${GDCS_HF_MODEL_DOWNLOAD_DIR}" "${EXCLUDE_ARGS[@]}" && \
   rm -rf "${GDCS_HF_MODEL_DOWNLOAD_DIR}/.cache"
   ```
2. Calculate the required `PersistentVolumeClaim` size from the downloaded
   model weights with 25% overhead (rounded up to the next whole GiB):

   ```bash
   MODEL_KIB=$(du -sk "${GDCS_HF_MODEL_DOWNLOAD_DIR}" | awk '{print $1}')
   export GDCS_MODEL_PVC_SIZE="$(( (MODEL_KIB * 125 / 100 + 1048575) / 1048576 ))Gi"
   echo "GDCS_MODEL_PVC_SIZE=${GDCS_MODEL_PVC_SIZE}"
   ```
3. Create the manifest for the ReadWriteOnce PersistentVolumeClaim for the
   model weights:

   ```bash
   cat <<EOF > "${GDCS_USER_GUIDE_HOME}/${GDCS_WORKLOAD_NAME}-rwo-pvc.yaml"
   apiVersion: v1
   kind: PersistentVolumeClaim
   metadata:
     name: ${GDCS_WORKLOAD_NAME}-rwo
   spec:
     accessModes:
       - ReadWriteOnce
     resources:
       requests:
         storage: ${GDCS_MODEL_PVC_SIZE}
     storageClassName: standard-rwo
     volumeMode: Filesystem
   EOF
   ```
4. Apply the manifest for the PersistentVolumeClaim for the model weights:

   ```bash
   kubectl apply \
   --filename="${GDCS_USER_GUIDE_HOME}/${GDCS_WORKLOAD_NAME}-rwo-pvc.yaml" \
   --namespace="${GDCS_WORKLOAD_NAMESPACE}"
   ```
5. Wait for the `PersistentVolumeClaim` to bind and print the values that the
   Infrastructure Operator needs to adjust the I/O quality of service (QoS)
   policy:

   ```bash
   kubectl wait \
   --for=jsonpath='{.status.phase}'=Bound \
   "pvc/${GDCS_WORKLOAD_NAME}-rwo" \
   --namespace="${GDCS_WORKLOAD_NAMESPACE}" \
   --timeout=120s
   BACKING_PVC=$(kubectl get pvc "${GDCS_WORKLOAD_NAME}-rwo" \
   --namespace="${GDCS_WORKLOAD_NAMESPACE}" \
   --output=jsonpath='{.spec.volumeName}')
   echo "BACKING_PVC=${BACKING_PVC} ORG=${GDC_ORG}"
   ```
6. If `performance-rwo` isn't available on your cluster, ask your
   Infrastructure Operator to apply the high-performance QoS policy
   (`system-performance-<ORG>-admin`, 30 IOPS per GiB with a 6,148 IOPS
   minimum floor) to the volume that backs the `PersistentVolumeClaim`.
   Provide the `BACKING_PVC` and `ORG` values that the previous step printed.
   The Infrastructure Operator applies the policy as described in
   the FILE-R0097 runbook in the Infrastructure Operator documentation;
   it takes effect on the live volume, so you can continue with the next steps.

   > [!NOTE]
   > **Note:** If high-performance block storage is enabled on your cluster, you can set `storageClassName: performance-rwo` in the `PersistentVolumeClaim` manifest instead of requesting the QoS policy. If `kubectl get storageclass performance-rwo` returns nothing, the Infrastructure Operator has not enabled high-performance block storage.

7. Create the manifest for the model loader Pod:

   ```bash
   cat <<EOF > "${GDCS_USER_GUIDE_HOME}/${GDCS_MODEL_LOADER_POD_NAME}-pod.yaml"
   apiVersion: v1
   kind: Pod
   metadata:
     name: ${GDCS_MODEL_LOADER_POD_NAME}
   spec:
     containers:
       - name: uploader
         image: ${GDCS_HARBOR_PROJECT_URI}/library/busybox:${GDCS_BUSYBOX_IMAGE_TAG}
         command: ["sleep", "infinity"]
         volumeMounts:
           - name: model-data
             mountPath: ${GDCS_HF_PV_MOUNT_PATH}
     imagePullSecrets:
       - name: ${GDCS_HARBOR_K8S_PULL_SECRET}
     volumes:
       - name: model-data
         persistentVolumeClaim:
           claimName: ${GDCS_WORKLOAD_NAME}-rwo
   EOF
   ```
8. Apply the manifest for the model loader Pod:

   ```bash
   kubectl apply \
   --filename="${GDCS_USER_GUIDE_HOME}/${GDCS_MODEL_LOADER_POD_NAME}-pod.yaml" \
   --namespace="${GDCS_WORKLOAD_NAMESPACE}"
   ```
9. Wait for the model loader Pod to be Running:

   ```bash
   watch --color --interval 5 --no-title \
   "kubectl get pod/${GDCS_MODEL_LOADER_POD_NAME} \
   --namespace=${GDCS_WORKLOAD_NAMESPACE} | GREP_COLORS='mt=01;92' grep -E --color=always -e '^' -e 'Running'"
   ```
10. Upload the model weights to the `PersistentVolume` (the script skips files
    that already exist with the same size, so it can be re-run):

    ```bash
    export LOCAL_MODEL_DIR="${GDCS_HF_MODEL_DOWNLOAD_DIR}"
    export REMOTE_MODEL_DIR="${GDCS_HF_PV_MOUNT_PATH}/${GDCS_HF_MODEL_LOWERCASE}"
    export REMOTE_MOUNT_DIR="${GDCS_HF_PV_MOUNT_PATH}"

    ${GDCS_IMPLEMENTATION_HOME}/model_weight_upload.sh
    ```
11. Delete the model loader Pod:

    ```bash
    kubectl delete pod/${GDCS_MODEL_LOADER_POD_NAME} \
    --namespace="${GDCS_WORKLOAD_NAMESPACE}"
    ```

### Inference server

1. Create the manifest for the vLLM Deployment. The ConfigMap carries the
   model-specific vLLM settings from the environment, so the manifest is the
   same for every model and GPU option:

   ```bash
   cat <<EOF > "${GDCS_USER_GUIDE_HOME}/${GDCS_WORKLOAD_NAME}-deployment.yaml"
   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: ${GDCS_WORKLOAD_NAME}
   data:
     MAX_MODEL_LEN: "${GDCS_VLLM_MAX_MODEL_LEN}"
     MODEL: "${GDCS_HF_PV_MOUNT_PATH}/${GDCS_HF_MODEL_LOWERCASE}"
     PORT: "8000"
     TENSOR_PARALLEL_SIZE: "${GDCS_WORKLOAD_ACCELERATOR_COUNT}"
     VLLM_EXTRA_ARGS: "${GDCS_VLLM_EXTRA_ARGS}"
   ---
   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: ${GDCS_WORKLOAD_NAME}
     labels:
       app: ${GDCS_WORKLOAD_NAME}
   spec:
     replicas: 1
     selector:
       matchLabels:
         app: ${GDCS_WORKLOAD_NAME}
     strategy:
       type: Recreate
     template:
       metadata:
         labels:
           app: ${GDCS_WORKLOAD_NAME}
       spec:
         containers:
           - command:
               - /bin/sh
               - -c
             args:
               - exec vllm serve "\$MODEL" --port="\$PORT" --tensor-parallel-size="\$TENSOR_PARALLEL_SIZE" --max-model-len="\$MAX_MODEL_LEN" \$VLLM_EXTRA_ARGS
             envFrom:
               - configMapRef:
                   name: ${GDCS_WORKLOAD_NAME}
             image: ${GDCS_HARBOR_PROJECT_URI}/vllm/vllm-openai:${GDCS_VLLM_IMAGE_TAG}-fips
             livenessProbe:
               failureThreshold: 3
               httpGet:
                 path: /health
                 port: 8000
               periodSeconds: 20
               timeoutSeconds: 5
             name: vllm
             ports:
               - containerPort: 8000
             readinessProbe:
               failureThreshold: 3
               httpGet:
                 path: /health
                 port: 8000
               periodSeconds: 10
               successThreshold: 1
               timeoutSeconds: 5
             resources:
               limits:
                 ${GDCS_WORKLOAD_ACCELERATOR_TYPE}: "${GDCS_WORKLOAD_ACCELERATOR_COUNT}"
               requests:
                 ${GDCS_WORKLOAD_ACCELERATOR_TYPE}: "${GDCS_WORKLOAD_ACCELERATOR_COUNT}"
             startupProbe:
               httpGet:
                 path: /health
                 port: 8000
               failureThreshold: ${GDCS_VLLM_STARTUP_FAILURE_THRESHOLD}
               periodSeconds: 10
               timeoutSeconds: 5
             volumeMounts:
               - name: model-weights
                 mountPath: ${GDCS_HF_PV_MOUNT_PATH}
               - name: shm
                 mountPath: /dev/shm
         securityContext:
           seLinuxOptions:
             type: spc_t
         imagePullSecrets:
           - name: ${GDCS_HARBOR_K8S_PULL_SECRET}
         volumes:
           - name: model-weights
             persistentVolumeClaim:
               claimName: ${GDCS_WORKLOAD_NAME}-rwo
           - name: shm
             emptyDir:
               medium: Memory
               sizeLimit: "8Gi"
   ---
   apiVersion: v1
   kind: Service
   metadata:
     name: ${GDCS_WORKLOAD_NAME}
   spec:
     ports:
       - name: vllm-http
         port: 80
         protocol: TCP
         targetPort: 8000
     selector:
       app: ${GDCS_WORKLOAD_NAME}
     type: LoadBalancer
   EOF
   ```

   > [!NOTE]
   > **Note:** The startup probe allows `GDCS_VLLM_STARTUP_FAILURE_THRESHOLD` x 10 seconds for vLLM to load the weights and compile the CUDA graphs; the accelerator environment file (`env.d/accelerator.sh`) sets a value that matches the size of the model. The `exec vllm serve` form appends the space-separated `VLLM_EXTRA_ARGS` as individual arguments, so the values must not contain quotes; use vLLM's dotted form for JSON-typed flags, for example `--limit-mm-per-prompt.image=0`.

2. Apply the manifest for the vLLM Deployment:

   ```bash
   kubectl apply \
   --filename="${GDCS_USER_GUIDE_HOME}/${GDCS_WORKLOAD_NAME}-deployment.yaml" \
   --namespace="${GDCS_WORKLOAD_NAMESPACE}"
   ```
3. Create the manifest for the vLLM `NetworkPolicy`:

   ```bash
   cat << EOF > "${GDCS_USER_GUIDE_HOME}/vllm-networkpolicy.yaml" && echo "Successfully created." || echo "Failed to create!"
   apiVersion: networking.k8s.io/v1
   kind: NetworkPolicy
   metadata:
     name: allow-vllm-ingress
     namespace: ${GDCS_WORKLOAD_NAMESPACE}
   spec:
     ingress:
     - from:
       - ipBlock:
           cidr: 0.0.0.0/0
       - namespaceSelector: {}
       ports:
       - port: 8000
         protocol: TCP
     podSelector: {}
     policyTypes:
     - Ingress
   EOF
   ```
4. Apply the manifest for the vLLM `NetworkPolicy`:

   ```bash
   kubectl apply \
   --filename="${GDCS_USER_GUIDE_HOME}/vllm-networkpolicy.yaml"
   ```

> [!NOTE]
> **Note:** The `NetworkPolicy` admits traffic from any source to the workload port inside the cluster. Access to the LoadBalancer IP address from outside the project additionally requires a `ProjectNetworkPolicy` (`networking.gdc.goog/v1`) created in the project namespace by a Project NetworkPolicy Admin, see [Configure project network
> policies](https://docs.cloud.google.com/distributed-cloud/hosted/docs/latest/gdcag/platform/pa-user/pnp/pnp-overview).

1. Wait for the Pod to be Running:

   ```bash
   watch --color --interval 5 --no-title \
   "kubectl get pods --selector=app=${GDCS_WORKLOAD_NAME} \
   --namespace=${GDCS_WORKLOAD_NAMESPACE} | GREP_COLORS='mt=01;92' grep -E --color=always -e '^' -e 'Running'"
   ```
2. Wait for the Deployment to be Ready:

   ```bash
   watch --color --interval 5 --no-title \
   "kubectl get deployment/${GDCS_WORKLOAD_NAME} \
   --namespace=${GDCS_WORKLOAD_NAMESPACE} | GREP_COLORS='mt=01;92' grep -E --color=always -e '^' -e '1/1     1            1'"
   ```
3. Optional: Stream the container logs to monitor startup progress:

   ```bash
   kubectl logs deployment/${GDCS_WORKLOAD_NAME} \
   --follow \
   --namespace="${GDCS_WORKLOAD_NAMESPACE}"
   ```

## Validation

1. Retrieve the LoadBalancer IP address for vLLM:

   ```bash
   export VLLM_LB_IP=$(kubectl get service/${GDCS_WORKLOAD_NAME} --namespace="${GDCS_WORKLOAD_NAMESPACE}" --output=jsonpath='{.status.loadBalancer.ingress[*].ip}')
   echo "VLLM_LB_IP=${VLLM_LB_IP}"
   ```
2. Send a test request to vLLM. The `max_tokens` value is
   `GDCS_VLLM_TEST_MAX_TOKENS` from `env.d/accelerator.sh`; reasoning models
   need a larger budget because the reasoning phase consumes output tokens:

   ```bash
   curl http://${VLLM_LB_IP}/v1/models

   curl http://${VLLM_LB_IP}/v1/chat/completions \
   -H "Content-Type: application/json" \
   -d '{
     "model": "'${GDCS_HF_PV_MOUNT_PATH}/${GDCS_HF_MODEL_LOWERCASE}'",
     "messages": [
       {"role": "user", "content": "Explain Google Distributed Cloud air-gapped briefly."}
     ],
     "max_tokens": '${GDCS_VLLM_TEST_MAX_TOKENS}'
   }'
   ```

   > [!NOTE]
   > **Note:** If you send Mistral's system prompt to Magistral Small 1.2 and `content` is empty with `finish_reason: length`, raise `max_tokens`.

3. If the LoadBalancer IP address isn't reachable from the workstation, for
   example because no `ProjectNetworkPolicy` admits it yet, send the requests
   through port forwarding:

   ```bash
   kubectl port-forward service/${GDCS_WORKLOAD_NAME} \
   --namespace="${GDCS_WORKLOAD_NAMESPACE}" 8000:80 &
   PF_PID=$!

   sleep 1

   curl http://127.0.0.1:8000/v1/models

   kill ${PF_PID}
   ```