This tutorial shows you how to deploy and serve the mixture of experts model
[Qwen3-30B-A3B (`Qwen/Qwen3-30B-A3B-FP8`)](https://huggingface.co/Qwen/Qwen3-30B-A3B-FP8)
on Google Kubernetes Engine (GKE) using a single-host
[TPU v6e (Trillium)](https://docs.cloud.google.com/tpu/docs/v6e.md.txt) accelerator and the
[vLLM TPU serving framework](https://github.com/vllm-project/vllm-torchtpu)
with TorchTPU (`torchtpu`). You run the workload on a single
`ct6e-standard-4t` node, which contains four TPU v6e chips in a `2x2` topology,
with data parallelism and expert parallelism enabled.

To run a similar single-host vLLM TorchTPU serving workflow on
[TPU7x (Ironwood)](https://docs.cloud.google.com/tpu/docs/tpu7x.md.txt) accelerators with the larger Qwen3.5-397B
model, see
[Serve Qwen3.5-397B on GKE with vLLM TorchTPU on TPU7x accelerators](https://docs.cloud.google.com/ai-hypercomputer/docs/tutorials/tpu/serve-qwen3-5-397b-ironwood-vllm-torchtpu.md.txt).

## Objectives

- Set up your GKE cluster and single-host TPU v6e node pool.
- Deploy the single-host vLLM TorchTPU server on GKE.
- Send an inference request to the vLLM TorchTPU server.
- Run a serving benchmark workload.
- Clean up.

## Costs

In this document, you use the following billable components of
Google Cloud:

- [GKE](https://docs.cloud.google.com/kubernetes-engine/pricing)
- [Cloud TPU](https://docs.cloud.google.com/tpu/pricing)
- [Compute Engine](https://docs.cloud.google.com/compute/all-pricing)

To generate a cost estimate based on your projected usage, use the
[pricing calculator](https://cloud.google.com/products/calculator).

When you finish the tasks that are described in this document, you can avoid
continued billing by deleting the resources that you created. For more
information, see [Clean up](https://docs.cloud.google.com/ai-hypercomputer/docs/tutorials/tpu/serve-qwen3-30b-v6e-vllm-torchtpu.md.txt#clean-up).

## Before you begin

Set up your environment for the tutorial:

- Enable the Kubernetes Engine API in your Google Cloud project:

  [Enable the Kubernetes Engine API](https://console.cloud.google.com/apis/enableflow?apiid=container.googleapis.com)
- Enable the Artifact Registry API in your Google Cloud project:

  [Enable the Artifact Registry API](https://console.cloud.google.com/apis/enableflow?apiid=artifactregistry.googleapis.com)
- [Install](https://docs.cloud.google.com/sdk/docs/install) and [initialize](https://docs.cloud.google.com/sdk/docs/initialize.md.txt) the
  Google Cloud CLI. If you're using Cloud Shell, the Google Cloud CLI is
  preinstalled.

- Install [`kubectl`](https://kubernetes.io/docs/tasks/tools/#kubectl),
  `gke-gcloud-auth-plugin`, `jq`, and `envsubst`. If you're using
  Cloud Shell, these tools are preinstalled.

- Ensure that your project has sufficient
  [Google Cloud Hyperdisk Balanced quota and external IP address quota](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/tpus.md.txt#additional-quotas)
  in the region where your TPU capacity is located:

  1. In the Google Cloud console, go to the **Quotas \& System Limits** page.

     [Go to Quotas \& System Limits](https://console.cloud.google.com/iam-admin/quotas)
  2. Filter by **Hyperdisk Balanced Capacity (GB)** and verify that you have
     at least 600 GB of available quota for the model cache volume (500 GB)
     and TPU node boot disk (100 GB).

  3. Filter by **In-use regional external IPv4 addresses** and verify that
     you have sufficient IP address quota in your target region.

> [!IMPORTANT]
> **Important:** To complete this tutorial, you must have quota for TPU v6e chips in your Google Cloud project in a [region and zone where TPU v6e is available](https://docs.cloud.google.com/tpu/docs/regions-zones.md.txt). This tutorial uses a TPU v6e reservation, but you can also use Spot capacity or Flex-start VMs if available. To learn more about reserving Cloud TPU capacity, see [About Cloud TPU reservations](https://docs.cloud.google.com/tpu/docs/about-tpu-reservations.md.txt).

### Get a Hugging Face access token

You need a Hugging Face access token to use this tutorial. You can sign up for a
free account at [Hugging Face](http://huggingface.co/login). After you have an
account, generate an access token:

1. On the [Welcome to Hugging Face](https://huggingface.co/welcome) page, click your account avatar and select **Access tokens**.
2. On the **Access tokens** page, click **Create new token**.
3. Select the **Read** token type and enter a name for your token.
4. Your access token is displayed. Save the token in a safe place.

### Required roles


To get the permissions that
you need to complete this tutorial,

ask your administrator to grant you the
following IAM roles on your project:

- [Kubernetes Engine Admin](https://docs.cloud.google.com/iam/docs/roles-permissions/container.md.txt#container.admin) (`roles/container.admin`)
- [Compute Admin](https://docs.cloud.google.com/iam/docs/roles-permissions/compute.md.txt#compute.admin) (`roles/compute.admin`)
- [Service Account User](https://docs.cloud.google.com/iam/docs/roles-permissions/iam.md.txt#iam.serviceAccountUser) (`roles/iam.serviceAccountUser`)
- [Service Usage Consumer](https://docs.cloud.google.com/iam/docs/roles-permissions/serviceusage.md.txt#serviceusage.serviceUsageConsumer) (`roles/serviceusage.serviceUsageConsumer`)


For more information about granting roles, see [Manage access to projects, folders, and organizations](https://docs.cloud.google.com/iam/docs/granting-changing-revoking-access.md.txt).


You might also be able to get
the required permissions through [custom
roles](https://docs.cloud.google.com/iam/docs/creating-custom-roles) or other [predefined
roles](https://docs.cloud.google.com/iam/docs/roles-overview#predefined).

## Set up your environment variables

Set up your authentication:

```bash
gcloud auth login
gcloud auth application-default login
```

Set up your environment variables:

```bash
export PROJECT_ID="YOUR_PROJECT_ID"
export REGION="YOUR_REGION" # for example, us-central1
export ZONE="YOUR_ZONE" # for example, us-central1-b
export CLUSTER_NAME="YOUR_CLUSTER_NAME" # maximum 40 chars; start with a-z; only a-z, 0-9, -
export MACHINE_TYPE="ct6e-standard-4t"
export NODE_POOL_NAME="np-0"
export RESERVATION_NAME="YOUR_RESERVATION_NAME"
export HF_TOKEN="YOUR_HF_TOKEN"
export NETWORK="YOUR_NETWORK_NAME"
export SUBNETWORK="YOUR_SUBNETWORK_NAME"
export VLLM_IMAGE="vllm/vllm-torchtpu:v0.29.0@sha256:3976ccc1a69ffd361e0a7d2831fe317f6b685636014f426321a39ae02f8da0dc"

gcloud config set project "${PROJECT_ID}"
gcloud config set billing/quota_project "${PROJECT_ID}"
```

Replace the following:

- `YOUR_PROJECT_ID`: the ID of your Google Cloud project.
- `YOUR_REGION`: the region where your capacity is located and where you want to deploy your cluster, such as `us-central1`.
- `YOUR_ZONE`: the zone where your capacity is located and where you want to deploy your TPU v6e node pool, such as `us-central1-c`.
- `YOUR_CLUSTER_NAME`: the name of your GKE cluster, such as `tpu-v6e-tutorial-cluster`. The name must be at most 40 characters, start with a lowercase letter, and contain only lowercase letters, numbers, and hyphens.
- `YOUR_RESERVATION_NAME`: if you're using a reservation, the name of your TPU v6e reservation.
- `YOUR_HF_TOKEN`: your Hugging Face access token.
- `YOUR_NETWORK_NAME`: the name of your Google Cloud network. If you [created a custom network](https://docs.cloud.google.com/vpc/docs/create-modify-vpc-networks.md.txt), then specify that network. Otherwise, specify `default`.
- `YOUR_SUBNETWORK_NAME`: the name of your Google Cloud subnetwork. If you created a custom subnetwork, then specify that subnetwork. Otherwise, specify `default`.

If you're using a custom vLLM TorchTPU container image, replace the default
`vllm/vllm-torchtpu` image URI in the `VLLM_IMAGE` variable with the URI of
your image. For public images, see the
[vLLM TorchTPU](https://github.com/vllm-project/vllm-torchtpu) repository.

## Create the GKE cluster

To create a GKE cluster in Standard mode and attach a
single-host TPU v6e node pool, complete the following steps:

1. Create a regional GKE cluster:

   ```bash
   gcloud container clusters create "${CLUSTER_NAME}" \
       --project="${PROJECT_ID}" \
       --location="${REGION}" \
       --machine-type=e2-standard-4 \
       --network="${NETWORK}" \
       --subnetwork="${SUBNETWORK}"
   ```

   The default machine type for GKE clusters (`e2-medium`)
   doesn't have sufficient CPU capacity for workload controllers, so this
   command specifies `e2-standard-4`.

   The output is similar to the following:

       NAME                      LOCATION     MASTER_VERSION      MASTER_IP      MACHINE_TYPE   NODE_VERSION        NUM_NODES  STATUS
       tpu-v6e-tutorial-cluster  us-central1  1.35.7-gke.1027000  34.63.222.246  e2-standard-4  1.35.7-gke.1027000  3          RUNNING

2. Create a single-host TPU v6e node pool with one `ct6e-standard-4t` node
   (four TPU v6e chips in a `2x2` topology) using your reservation:

   ```bash
   gcloud container node-pools create "${NODE_POOL_NAME}" \
       --cluster="${CLUSTER_NAME}" \
       --machine-type="${MACHINE_TYPE}" \
       --location="${REGION}" \
       --node-locations="${ZONE}" \
       --project="${PROJECT_ID}" \
       --num-nodes=1 \
       --reservation-affinity=specific \
       --reservation="${RESERVATION_NAME}"
   ```

   Instead of using a specific reservation, you can also create the node pool
   using Spot capacity or Flex-start VMs by replacing the
   `--reservation-affinity` and `--reservation` flags with the following
   flags:
   - **Spot VMs** : To provision the node pool using [Spot](https://docs.cloud.google.com/compute/docs/instances/spot.md.txt) capacity, pass the `--spot` flag. Instances that use Spot capacity might take time to be created after the node pool is created, and they might be preempted during workload runs. If your Pods take time to be created, or workloads stop in the middle of running, this might be the cause.
   - **Flex-start VMs** : to provision the node pool using the [Flex-start](https://docs.cloud.google.com/kubernetes-engine/docs/concepts/dws.md.txt) consumption option, pass the `--flex-start` flag. For more information, see [Run a small batch workload with TPUs and Flex-start VMs](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/dws-flex-start-training-tpu.md.txt).

   The output is similar to the following:

       NAME  MACHINE_TYPE      DISK_SIZE_GB  NODE_VERSION
       np-0  ct6e-standard-4t  100           1.35.7-gke.1027000

3. Configure `kubectl` credentials to communicate with your cluster:

   ```bash
   gcloud container clusters get-credentials "${CLUSTER_NAME}" \
       --location="${REGION}" \
       --project="${PROJECT_ID}"
   ```

## Configure credentials

Create a Kubernetes Secret named `hf-secret` to store your Hugging Face access
token:

```bash
kubectl create secret generic hf-secret \
    --from-literal=hf_api_token="${HF_TOKEN}" \
    --dry-run=client -o yaml | kubectl apply -f -
```

## Deploy the vLLM TorchTPU server

The workload manifest (`qwen3_server.yaml`) defines the following resources:

- A StorageClass (`hyperdisk-balanced-qwen3`) and a 500 GiB
  PersistentVolumeClaim (`hd-claim-qwen3`) backed by
  Google Cloud Hyperdisk Balanced storage to store the downloaded model weights.

- A Deployment (`vllm-qwen3`) scheduled on a single TPU v6e node with a `2x2`
  topology (`google.com/tpu: '4'`). The container runs `vllm serve` for
  `Qwen/Qwen3-30B-A3B-FP8` with the following flags:

  - `--data-parallel-size=4`
  - `--tensor-parallel-size=1`
  - `--enable-expert-parallel`
  - `--kv-cache-dtype=fp8`
  - `--max-model-len=9236`
  - `--max-num-seqs=64`
- A `ClusterIP` Service (`vllm-qwen3-service`) that exposes port `8000`.

To deploy and monitor the workload, complete the following steps:

1. Fetch the server manifest:

   ```bash
   curl -fLO https://raw.githubusercontent.com/GoogleCloudSamples/AIHypercomputerSamples/refs/heads/main/tpu/inference/qwen3-5-397b/qwen3_server.yaml
   ```

   Alternatively, you can choose to create a file named `qwen3_server.yaml`
   with the following content:

   ```yaml
   apiVersion: storage.k8s.io/v1
   kind: StorageClass
   metadata:
     name: hyperdisk-balanced-qwen3
   provisioner: pd.csi.storage.gke.io
   parameters:
     type: hyperdisk-balanced
   reclaimPolicy: Delete
   volumeBindingMode: WaitForFirstConsumer
   allowVolumeExpansion: true
   ---
   apiVersion: v1
   kind: PersistentVolumeClaim
   metadata:
     name: hd-claim-qwen3
     namespace: default
   spec:
     storageClassName: hyperdisk-balanced-qwen3
     accessModes:
       - ReadWriteOnce
     resources:
       requests:
         storage: 500Gi
   ---
   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: vllm-qwen3
     namespace: default
   spec:
     replicas: 1
     selector:
       matchLabels:
         app: vllm-qwen3
     template:
       metadata:
         labels:
           app: vllm-qwen3
       spec:
         enableServiceLinks: false
         nodeSelector:
           cloud.google.com/gke-tpu-accelerator: tpu-v6e-slice
           cloud.google.com/gke-tpu-topology: 2x2
         tolerations:
         - key: "cloud.google.com/gke-spot"
           operator: "Equal"
           value: "true"
           effect: "NoSchedule"
         containers:
         - name: vllm-tpu
           image: $VLLM_IMAGE
           command: ["vllm", "serve"]
           args:
           - --host=0.0.0.0
           - --port=8000
           - --model=Qwen/Qwen3-30B-A3B-FP8
           - --tokenizer=Qwen/Qwen3-30B-A3B-FP8
           - --max-model-len=9236
           - --max-num-batched-tokens=1024
           - --max-num-seqs=64
           - --no-enable-prefix-caching
           - --gpu-memory-utilization=0.9
           - --data-parallel-size=4
           - --tensor-parallel-size=1
           - --reasoning-parser=qwen3
           - --async-scheduling
           - --kv-cache-dtype=fp8
           - --enable-expert-parallel
           - --download-dir=/data
           - --block-size=128
           env:
           - name: MODEL_IMPL_TYPE
             value: "vllm"
           - name: DP_SCHED_ENABLED
             value: "1"
           - name: VLLM_ENGINE_READY_TIMEOUT_S
             value: "9600"
           - name: VLLM_ENGINE_ITERATION_TIMEOUT_S
             value: "1800"
           - name: USE_MOE_SPARSE_CORE
             value: "1"
           - name: TPU_MOE_SKIP_PADDED_TOKENS
             value: "1"
           - name: VLLM_CACHE_ROOT
             value: "/data/vllm_cache"
           - name: TORCHINDUCTOR_CACHE_DIR
             value: "/data/torch_compile_cache"
           - name: XLA_CACHE_DIR
             value: "/data/xla_cache"
           - name: HF_HOME
             value: /data
           - name: HF_TOKEN
             valueFrom:
               secretKeyRef:
                 name: hf-secret
                 key: hf_api_token
           ports:
           - containerPort: 8000
           resources:
             limits:
               google.com/tpu: '4'
             requests:
               google.com/tpu: '4'
           startupProbe:
             httpGet:
               path: "/health"
               port: 8000
             periodSeconds: 15
             failureThreshold: 120
           readinessProbe:
             tcpSocket:
               port: 8000
             initialDelaySeconds: 15
             periodSeconds: 10
           volumeMounts:
           - mountPath: "/data"
             name: data-volume
           - mountPath: /dev/shm
             name: dshm
         volumes:
         - emptyDir:
             medium: Memory
           name: dshm
         - name: data-volume
           persistentVolumeClaim:
             claimName: hd-claim-qwen3
   ---
   apiVersion: v1
   kind: Service
   metadata:
     name: vllm-qwen3-service
     namespace: default
   spec:
     selector:
       app: vllm-qwen3
     type: ClusterIP
     ports:
     - name: http
       protocol: TCP
       port: 8000
       targetPort: 8000
   ```
2. Apply the vLLM TorchTPU server manifest to your cluster:

   ```bash
   envsubst '$VLLM_IMAGE' < "qwen3_server.yaml" | kubectl apply -f -
   ```
3. Monitor the server deployment logs until application startup completes:

   ```bash
   kubectl wait --for=condition=Ready pod -l app=vllm-qwen3 --timeout=1800s
   kubectl logs deployment/vllm-qwen3 -c vllm-tpu -f
   ```

   To stop streaming logs at any time, press <kbd>Control+C</kbd>. Initial
   service startup might take approximately 15 to 30 minutes while the Pod
   downloads the model weights and compiles the model.

   If you're using preemptive capacity like Spot or Flex-start, it might take
   longer after node pool creation for your TPU slice nodes to be created, and
   they can be preempted during workload runs. If your Pods are taking longer
   than expected to start, or your workloads stop in the middle, this could be
   the cause. For more information, see
   [Preemption of Spot VMs](https://docs.cloud.google.com/compute/docs/instances/spot.md.txt#preemption). To check
   if your TPU slice nodes have been created, see
   [Check the node's status and conditions](https://docs.cloud.google.com/kubernetes-engine/docs/troubleshooting/node-notready.md.txt#check-node-status-conditions).

   Wait until the logs report `Application startup complete` before you send
   traffic:

       (APIServer pid=1) INFO:     Started server process [1]
       (APIServer pid=1) INFO:     Waiting for application startup.
       (APIServer pid=1) INFO:     Application startup complete.

4. Open a second terminal session.

5. In your second terminal session, forward local port `8000` to
   `vllm-qwen3-service`:

   ```bash
   kubectl port-forward service/vllm-qwen3-service 8000:8000
   ```

   Leave port forwarding running in this terminal session.

## Send inference requests

Return to your first terminal session and send a chat completion request to the
`/v1/chat/completions` endpoint:

```bash
curl http://localhost:8000/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
        "model": "Qwen/Qwen3-30B-A3B-FP8",
        "messages": [
            {"role": "user", "content": "Hello! Tell me a joke."}
        ],
        "max_tokens": 512,
        "temperature": 0.7
    }' | jq .
```

## Run a benchmark workload

To benchmark the deployed Qwen3-30B-A3B model using an 8,192-token input and
1,024-token output (8k/1k) workload with the
[InferenceX benchmark suite](https://github.com/SemiAnalysisAI/InferenceX),
complete the following steps:

1. Fetch the benchmark manifest:

   ```bash
   curl -fLO https://raw.githubusercontent.com/GoogleCloudSamples/AIHypercomputerSamples/refs/heads/main/tpu/inference/qwen3-5-397b/qwen3-benchmark-8k1k.yaml
   ```

   Alternatively, create a file named `qwen3-benchmark-8k1k.yaml` with the
   following content:

   ```yaml
   apiVersion: v1
   kind: Pod
   metadata:
     name: qwen3-bench-8k1k
     namespace: default
   spec:
     restartPolicy: OnFailure
     terminationGracePeriodSeconds: 60
     containers:
     - name: vllm-bench
       image: $VLLM_IMAGE
       command: ["/bin/bash", "-c"]
       args:
       - |
         while ! curl -sf http://vllm-qwen3-service:8000/health > /dev/null; do sleep 30 && echo 'Waiting for server...'; done
         if ! command -v git >/dev/null 2>&1; then apt-get update && apt-get install -y git; fi && \
         git clone https://github.com/SemiAnalysisAI/InferenceX.git /ubench/inferencex && \
         cd /ubench/inferencex && \
         git checkout 89ce6098ef2bc4576a735c43f39c7d972b091cfc && \
         python3 /ubench/inferencex/utils/bench_serving/benchmark_serving.py \
           --backend=vllm \
           --percentile-metrics='ttft,tpot,itl,e2el' \
           --host=vllm-qwen3-service \
           --port=8000 \
           --model=Qwen/Qwen3-30B-A3B-FP8 \
           --tokenizer=Qwen/Qwen3-30B-A3B-FP8 \
           --dataset-name=random \
           --random-input-len=8192 \
           --random-output-len=1024 \
           --random-range-ratio=1.0 \
           --num-prompts=2560 \
           --num-warmups=512 \
           --max-concurrency=256 \
           --ignore-eos
       env:
       - name: VLLM_PLUGINS
         value: ""
       - name: VLLM_TARGET_DEVICE
         value: "cpu"
       - name: JAX_PLATFORMS
         value: "cpu"
       - name: PYTHONUNBUFFERED
         value: "1"
       - name: HUGGING_FACE_HUB_TOKEN
         valueFrom:
           secretKeyRef:
             key: hf_api_token
             name: hf-secret
   ```
2. Apply the benchmark Pod manifest:

   ```bash
   envsubst '$VLLM_IMAGE' < "qwen3-benchmark-8k1k.yaml" | kubectl apply -f -
   ```
3. Monitor the benchmark logs:

   ```bash
   kubectl wait --for=jsonpath='{.status.phase}'=Running pod/qwen3-bench-8k1k --timeout=600s
   kubectl logs -f qwen3-bench-8k1k
   ```

   The benchmark takes approximately 25 to 30 minutes to complete. After the
   benchmark workload begins to run, it might not generate output logs until
   it has completed its run.
4. After the benchmark has completed, look for output that contains metrics
   such as the following:

   - `Output token throughput`
   - `Total token throughput`
   - `TTFT`
   - `TPOT`

## Clean up

To avoid incurring additional ongoing charges to your Google Cloud account
for the resources used in this tutorial, delete the Kubernetes workloads, the
TPU node pool, and the GKE cluster:

1. Delete the vLLM TorchTPU server Deployment, Service, PersistentVolumeClaim,
   and Hugging Face Secret:

   ```bash
   kubectl delete -f qwen3-benchmark-8k1k.yaml --ignore-not-found || true
   kubectl delete -f qwen3_server.yaml --ignore-not-found || true
   kubectl wait --for=delete pvc/hd-claim-qwen3 --timeout=120s || true
   kubectl delete secret hf-secret --ignore-not-found || true
   ```
2. Delete the TPU v6e node pool and the GKE cluster:

   ```bash
   gcloud container node-pools delete "${NODE_POOL_NAME}" \
       --cluster="${CLUSTER_NAME}" \
       --location="${REGION}" \
       --project="${PROJECT_ID}" \
       --quiet || true

   gcloud container clusters delete "${CLUSTER_NAME}" \
       --location="${REGION}" \
       --project="${PROJECT_ID}" \
       --quiet || true
   ```

## What's next

- Learn more about [vLLM on Cloud TPU](https://github.com/vllm-project/vllm-torchtpu).
- Learn more about [TPU v6e (Trillium)](https://docs.cloud.google.com/tpu/docs/v6e.md.txt).
- Learn more about [Cloud TPU](https://docs.cloud.google.com/tpu/docs/intro-to-tpu.md.txt).