This page describes how you can use GKE Inference Quickstart (GIQ) to simplify
the deployment of AI/ML inference workloads on Google Kubernetes Engine (GKE).
**Inference Quickstart** is a utility that lets you specify your
inference business requirements and get optimized Kubernetes configurations
based on best practices and Google's benchmarks for models, model servers,
accelerators ([GPUs](https://docs.cloud.google.com/kubernetes-engine/docs/concepts/gpus.md.txt),
[TPUs](https://docs.cloud.google.com/kubernetes-engine/docs/concepts/tpus.md.txt)), scaling, and storage.
This helps you avoid the time-consuming process of manually adjusting and
testing configurations.

This page is for Machine learning (ML) engineers,
Platform admins and operators, and for Data and AI specialists who want to
understand how to efficiently manage and optimize GKE for AI/ML
inference. To learn more about common roles and example tasks that we reference
in Google Cloud content, see
[Common GKE user roles and tasks](https://docs.cloud.google.com/anthos/docs/concepts/roles-tasks).

To learn more about model serving concepts and terminology, and how GKE Gen AI capabilities can enhance and support your model serving performance, see
[About model inference on GKE](https://docs.cloud.google.com/kubernetes-engine/docs/concepts/machine-learning/inference.md.txt).

Before reading this page, ensure you're familiar with [Kubernetes](https://docs.cloud.google.com/kubernetes-engine/docs/learn/get-started-with-kubernetes.md.txt), [GKE](https://docs.cloud.google.com/kubernetes-engine/docs/concepts/kubernetes-engine-overview.md.txt#how_works), and
[model serving](https://www.cloudskillsboost.google/paths/17).

### Using Inference Quickstart

Inference Quickstart lets you analyze the performance and
cost-efficiency of your inference workloads, and make data-driven
decisions about resource allocation and model deployment strategies.

The high-level steps to use Inference Quickstart are as follows:

1. [**Analyze performance and cost**](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/machine-learning/inference/inference-quickstart.md.txt#view): explore available configurations
   and filter them based on your performance and cost requirements, by using the
   `gcloud container ai profiles list` command. To view the complete set of
   benchmarking data for a specific configuration, use the
   `gcloud container ai profiles benchmarks list` command. This command lets you
   identify the most cost-effective hardware for your specific performance requirements.

   Additionally, you can filter Inference Quickstart
   recommendations based on workload characteristics like use case and
   input/output sizes, and by serving stack. A *serving stack*
   is the complete set of technologies used to host a model, handle inference,
   and route user requests to deliver predictions. For example, the `llm-d`
   serving stack uses vLLM as its underlying model server, adding a layer of
   orchestration and routing on top of the core inference engine.
2. [**Deploy manifests**](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/machine-learning/inference/inference-quickstart.md.txt#deploy): after your analysis, you can generate an optimized
   Kubernetes manifest and deploy it. You can optionally enable optimizations for
   storage and autoscaling. You can deploy from the Google Cloud console or by using
   the `kubectl apply` command. Before you deploy, you must ensure you have
   sufficient accelerator quota for the selected GPUs or TPUs in your Google Cloud project.

3. [**(Optional) Run your own benchmarks**](https://github.com/kubernetes-sigs/inference-perf/):
   the configurations and performance data provided by
   Inference Quickstart are based on benchmarks generated by the [`inference-perf` tool](https://github.com/kubernetes-sigs/inference-perf/).
   Because your workload's performance might vary from this baseline, we recommend that you use the `inference-perf` tool to measure
   your model's performance with a dataset that best represents your use case.

> [!TIP]
> **Tip:** To interactively explore and visualize the trade-offs between throughput, latency, and cost, you can use the [Inference Quickstart Colab notebook](https://colab.research.google.com/github/GoogleCloudPlatform/kubernetes-engine-samples/blob/main/ai-ml/notebooks/giq_visualizations.ipynb). This notebook provides an interactive guide to using the Inference Quickstart recommender API to analyze benchmarking data.

### Benefits

Inference Quickstart helps you save time and resources by
providing optimized configurations. These optimizations improve performance and
reduce infrastructure costs, in the following ways:

- You receive detailed tailored best practices for setting the accelerator (GPU and TPU), model server, and scaling configurations. GKE routinely updates Inference Quickstart with the latest fixes, images, and performance benchmarks.
- You can specify your workload's latency and throughput requirements by using the Google Cloud console UI or a command-line interface, and get detailed tailored best practices as Kubernetes deployment manifests.

### How it works

Inference Quickstart provides tailored best practices based on
Google's exhaustive internal benchmarks of single-replica performance for model,
model server, and accelerator topology combinations. These benchmarks graph
latency versus throughput, including queue size and KV cache metrics, which map
out performance curves for each combination.

#### How tailored best practices are generated

We measure latency in *Normalized Time per Output Token* (NTPOT) and
*Time to First Token* (TTFT) in milliseconds,
and *throughput* in output tokens per second, by saturating accelerators. To
learn more about these performance metrics, see [About model inference on GKE](https://docs.cloud.google.com/kubernetes-engine/docs/concepts/machine-learning/inference.md.txt#performance).

The following example latency profile illustrates the inflection point where
throughput plateaus (green), the post-inflection point where latency worsens
(red), and the ideal zone (blue) for optimal throughput at the latency target.
Inference Quickstart provides performance data and configurations for this ideal zone.

![Latency profile with green marker less than 2000 output tokens per second and red marker higher than 2000 output tokens per second](https://docs.cloud.google.com/static/kubernetes-engine/images/inference-recommender-latency.png)

Based on an inference application's latency requirements, Inference Quickstart
identifies suitable combinations and determines the optimal operating point on
the latency-throughput curve. This point sets the [Horizontal Pod Autoscaler](https://docs.cloud.google.com/kubernetes-engine/docs/concepts/horizontalpodautoscaler.md.txt) (HPA) threshold, with a buffer to account for
scale-up latency. The overall threshold also informs the initial number of
replicas needed, though the HPA dynamically adjusts this number based on workload.

#### Cost estimation

To estimate per-token costs of your accelerator VMs, Inference Quickstart uses a configurable
output-to-input cost ratio. For example, if this ratio is set to 4, it's assumed
that each output token costs four times as much as an input token. The following
equations are used to calculate the cost-per-token metrics:

\\\[
\\$/\\text{output token} = \\frac{\\text{GPU \\$/s}}{(\\frac{1}{\\text{output-to-input-cost-ratio}} \\cdot \\text{input tokens/s} + \\text{output tokens/s})}
\\\]

where

\\\[
\\$/\\text{input token} = \\frac{\\text{\\$/output token}}{\\text{output-to-input-cost-ratio}}
\\\]

> [!NOTE]
> **Note:** This formula helps you to estimate your per-token costs based on the parameters that you configure. Your actual billing costs are subject to [GKE pricing](https://cloud.google.com/kubernetes-engine/pricing) and might be different from these estimates.

#### Benchmarking

The provided configurations and performance data are based on benchmarks generated by the [`inference-perf` tool](https://github.com/kubernetes-sigs/inference-perf/) to send traffic with the following input and output distribution.

| Input Tokens |||||| Output Tokens ||||||
| Min | Median | Mean | P90 | P99 | Max | Min | Median | Mean | P90 | P99 | Max |
|---|---|---|---|---|---|---|---|---|---|---|---|
| 4 | 108 | 226 | 635 | 887 | 1024 | 1 | 132 | 195 | 488 | 778 | 1024 |

> [!NOTE]
> **Note:** Performance for your workloads might vary from our baseline. We strongly recommend doing further testing and adjustment of manifests by using the `inference-perf` tool, especially for prefill or decode-heavy workloads.

## Before you begin


Before you start, make sure that you have performed the following tasks:

- Enable the Google Kubernetes Engine API.
[Enable Google Kubernetes Engine API](https://console.cloud.google.com/apis/enableflow?apiid=container.googleapis.com)
- To use the Google Cloud CLI for this task, [install](https://docs.cloud.google.com/sdk/docs/install) and then [initialize](https://docs.cloud.google.com/sdk/docs/initialize.md.txt) the gcloud CLI. If you previously installed the gcloud CLI, get the latest version by running the `gcloud components update` command. Earlier gcloud CLI versions might not support running the commands in this document.

  > [!NOTE]
  > **Note:** For existing gcloud CLI installations, make sure to set the `compute/region` [property](https://docs.cloud.google.com/sdk/docs/properties.md.txt#setting_properties). If you use primarily zonal clusters, set the `compute/zone` instead. By setting a default location, you can avoid errors in the gcloud CLI like the following: `One of [--zone, --region] must be supplied: Please specify location`. You might need to specify the location in certain commands if the location of your cluster differs from the default that you set.

<!-- -->

- In the Google Cloud console, on the project selector page, select or create a Google Cloud project.

  > [!NOTE]
  > **Note:** If you don't plan to keep the resources that you create in this procedure, create a project instead of selecting an existing project. After you finish these steps, you can delete the project, removing all resources associated with the project.

- [Make sure that billing is enabled for your Google Cloud project](https://docs.cloud.google.com/billing/docs/how-to/verify-billing-enabled.md.txt#confirm_billing_is_enabled_on_a_project).

- Ensure you have sufficient accelerator capacity for your project:

  - If you use GPUs: Check the [Quotas page](https://console.cloud.google.com/iam-admin/quotas).
  - If you use TPUs: Refer to [Ensure quota for TPUs and other GKE resources](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/tpus-autopilot.md.txt#ensure-tpu-quota).

### Prepare to use the GKE AI/ML user interface

If you use the Google Cloud console, you also need to create an Autopilot cluster, if one
is not already created in your project. Follow the instructions in [Create an Autopilot cluster](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/creating-an-autopilot-cluster.md.txt#gke_autopilot_create-).

### Prepare to use the command line interface

If you use the gcloud CLI to run Inference Quickstart,
you also need to run these additional commands:

1. Enable the `gkerecommender.googleapis.com` API:

   ```bash
   gcloud services enable gkerecommender.googleapis.com
   ```
2. Set the billing quota project that you use for API calls:

   ```bash
   gcloud config set billing/quota_project PROJECT_ID
   ```
3. Check that your gcloud CLI version is at least 536.0.1. If not,
   run the following:

   ```bash
   gcloud components update
   ```

## Limitations

Be aware of the following limitations before you start using Inference Quickstart:

- Google Cloud console model deployment supports deployment only to Autopilot clusters.
- Inference Quickstart does not provide profiles for all models supported by a given model server.
- If you don't set the `HF_HOME` environment variable when you use a generated manifest for a large model (90 GiB or greater) from Hugging Face, you must either use a cluster with larger-than-default boot disks or modify the manifest to set `HF_HOME` to `/dev/shm/hf_cache`. This will use RAM for the cache instead of the node's boot disk. For more information, see the [Troubleshooting](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/machine-learning/inference/inference-quickstart.md.txt#low-ephemeral-storage) section.
- [Loading models from Cloud Storage](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/machine-learning/inference/inference-quickstart.md.txt#storage-optimized) requires Workload Identity Federation for GKE. The Cloud Storage FUSE CSI driver is also required for all TPU workloads and as a fallback for specific model server versions earlier than `v0.11.1`. Workload Identity Federation for GKE and the Cloud Storage FUSE CSI driver are enabled by default in Autopilot clusters. For more details, see [Set up the Cloud Storage FUSE CSI driver for GKE](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/cloud-storage-fuse-csi-driver-setup.md.txt).

## Analyze and view optimized configurations for model inference

This section describes how to explore and analyze configuration recommendations
by using Google Cloud CLI.

Use the [`gcloud container ai profiles`](https://docs.cloud.google.com/sdk/gcloud/reference/container/ai/profiles)
command to explore and analyze optimized profiles (combinations of model, model server,
model server version, and accelerators):

### Models

To explore and select a model, use the [`models`](https://docs.cloud.google.com/sdk/gcloud/reference/container/ai/profiles/models) option.

> [!NOTE]
> **Note:** Some of the model families we support include Gemma, Llama, Mistral, DeepSeek, and Qwen. Use this command to get the most up-to-date list of supported models, as we routinely update them.

```bash
  gcloud container ai profiles models list
```

### Profiles

Use the [`list`](https://docs.cloud.google.com/sdk/gcloud/reference/container/ai/profiles/list)
command to explore generated profiles and filter them based on your
performance and cost requirements. For example:

```bash
gcloud container ai profiles list \
    --model=openai/gpt-oss-20b \
    --pricing-model=on-demand \
    --target-ttft-milliseconds=300
```

The output shows supported profiles with performance metrics like
throughput, latency, and cost per million tokens at the inflection point.
It looks similar to the following:

      Instance Type Accelerator      Cost/M Input Tokens Cost/M Output Tokens Output Tokens/s NTPOT(ms) TTFT(ms) Model Server Model Server Version Model
      a3-highgpu-1g nvidia-h100-80gb 0.009               0.035                13335           67        297      vllm         gptoss               openai/gpt-oss-20b

The values represent the performance observed at the point where
throughput stops increasing and latency starts
dramatically increasing (that is, the inflection or saturation point) for
a given profile with this accelerator type. To learn more about these
performance metrics, see
[About model inference on GKE](https://docs.cloud.google.com/kubernetes-engine/docs/concepts/machine-learning/inference.md.txt#performance).

For the full list of flags you can set, see the
[`list`](https://docs.cloud.google.com/sdk/gcloud/reference/container/ai/profiles/list) command documentation.

All pricing information is available in USD currency only and defaults to the
`us-east5` region, except for configurations that use A3 machines, which
default to the `us-central1` region.

### Benchmarks

To get all benchmarking data for a specific profile, use the
[`benchmarks list`](https://docs.cloud.google.com/sdk/gcloud/reference/container/ai/profiles/benchmarks/list) command.

For example:

```bash
gcloud container ai profiles benchmarks list \
    --model=deepseek-ai/DeepSeek-R1-Distill-Qwen-7B \
    --model-server=vllm
```

The output contains a list of performance metrics from benchmarks run
at different request rates.

The command displays the output in CSV format. To store the output as a
file, use output redirection. For example:
`gcloud container ai profiles benchmarks list > profiles.csv`.

For the full list of flags you can set, see the
[`benchmarks list`](https://docs.cloud.google.com/sdk/gcloud/reference/container/ai/profiles/benchmarks/list) command documentation.

After you choose a model, model server, model server version, and accelerator,
you can proceed to create a deployment manifest.

### View recommendations by use case

Inference Quickstart provides recommendations for various use
cases that represent common AI/ML workloads. These use cases are characterized
by a range of input and output token lengths based on real-world customer
workloads, which can impact the performance of the underlying inference serving stack.

The following table provides a summary of the available use cases and their characteristics:

| Use case | Input tokens | Output tokens | Ratio | Description |
|---|---|---|---|---|
| Advanced Customer Support | 8,192 | 256 | 32:1 | Resolve complex customer issues with comprehensive, synthesized answers to scale your support operations and improve user satisfaction. |
| Code Completion | 512 | 32 | 16:1 | Accelerate software development by automatically suggesting and completing code. |
| Text Summarization | 1,024 | 128 | 8:1 | Condense long documents, articles, or conversations into concise summaries. |
| Chatbot (ShareGPT) | 128 | 128 | 1:1 | Provide instant, conversational assistance to answer questions, perform tasks, or offer guidance. |
| Text Generation | 512 | 2,048 | 1:4 | Automate the creation of new written content, from drafting emails and reports to generating creative stories. |
| Deep Research | 256 | 4,096 | 1:16 | Enable detailed investigations and studies by analyzing and synthesizing vast amounts of data to find specific information, identify patterns, and uncover insights. |

To discover the full list of use cases that are supported by
Inference Quickstart, use the
[`gcloud container ai profiles use-cases list`]() command.

To filter recommendations by use case, use the `--use-case` flag. When you create a manifest, if you don't specify this flag, it defaults to `Chatbot`.

For example:

```bash
gcloud container ai profiles list \
    --use-case="Text Summarization"
```

## Deploy recommended configurations

This section describes how to generate and deploy configuration recommendations
by using either the Google Cloud console or the command line.

### Console

1. In the Google Cloud console, go to the GKE AI/ML page. [Open the GKE AI/ML page](https://console.cloud.google.com/kubernetes/aiml/overview)

2. Click **Deploy Models**.
3. Select a model you want to deploy. Models that are supported by
   Inference Quickstart are displayed with the **Optimized** tag.

   - If you selected a foundation model, it opens a model page. Click **Deploy**. You can still modify the configuration before actual deployment.
   - You are prompted to create an Autopilot cluster, if there isn't one in your project. Follow the instructions in [Create an Autopilot cluster](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/creating-an-autopilot-cluster.md.txt#gke_autopilot_create-). After creating the cluster, return to the GKE AI/ML page in the Google Cloud console to select a model.
   - To deploy gated models like Gemma or Llama, you must first create a Hugging Face token and add it as a Kubernetes Secret. Failing to do so might raise a `"Does not have minimum availability"` error that prevents the Deployment from being created. For detailed instructions, open the **gcloud** tab in this documentation page.

   The model deployment page prepopulates with your selected model
   as well as the recommended model server and accelerator. You can
   also configure settings like maximum latency and model source.
4. (Optional) To view the manifest with the recommended configuration, click **View YAML**.

5. To deploy the manifest with the recommended configuration, click **Deploy**.
   It might take several minutes for the deployment operation to complete.

   > [!NOTE]
   > **Note:** Before deployment, GKE validates the selected environment for potential issues, such as deployment name or label collisions. We recommend using a new namespace for each model deployment.

To view your deployment, go to the **Kubernetes Engine \> Workloads** page.

### gcloud

1. **Prepare to load models from your model registry**: Inference Quickstart
   supports loading models from Hugging Face or Cloud Storage.

   > [!IMPORTANT]
   > **Best practice** : Model weights are typically sourced
   > from public repositories such as Hugging Face.
   > To improve performance, we recommend loading model weights from
   > Cloud Storage and [enabling optimal storage configuration](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/machine-learning/inference/inference-quickstart.md.txt#storage-optimized).
   > This automatically utilizes the high-performance Run:ai Model Streamer for supported vLLM versions, or the [Cloud Storage FUSE CSI driver](https://docs.cloud.google.com/kubernetes-engine/docs/concepts/cloud-storage-fuse-csi-driver.md.txt) for other configurations.

   ### Hugging Face


   If you don't have one already, [generate a
   Hugging Face access token](https://huggingface.co/docs/hub/security-tokens)
   and a corresponding Kubernetes Secret.

   To create a Kubernetes Secret that contains the Hugging Face token, run
   the following command:

   ```bash
   kubectl create secret generic hf-secret \
       --from-literal=hf_api_token=HUGGING_FACE_TOKEN \
       --namespace=NAMESPACE
   ```

   Replace the following values:
   - <var translate="no">HUGGING_FACE_TOKEN</var>: the Hugging Face token you created earlier.
   - <var translate="no">NAMESPACE</var>: the Kubernetes namespace where you want to deploy your model server.

   Some models might also require you to accept and sign their consent license agreement.

   ### Cloud Storage

   You can [load supported models from Cloud Storage](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/machine-learning/inference/inference-quickstart.md.txt#storage-optimized)
   with a tuned Cloud Storage FUSE setup. To do this, you first need to load the
   model from Hugging Face to your Cloud Storage bucket.

   > [!NOTE]
   > **Note:** The following Kubernetes Job is experimental.

   You can deploy this [Kubernetes Job](https://gke-ai-labs.dev/docs/tutorials/storage/hf-gcs-transfer/)
   to transfer the model, changing `MODEL_ID` to the Inference Quickstart
   supported model.
2. **Generate manifests**: you have these options for generating manifests:

   - **Base configuration**: generates the standard Kubernetes Deployment, Service, and PodMonitoring manifests for deploying a single-replica inference server.
   - **(Optional) Storage-optimized configuration** : generates a manifest optimized for Cloud Storage loading. This configuration prioritizes Run:ai Model Streamer for improved performance on supported vLLM versions, while using a tuned Cloud Storage FUSE setup for earlier versions. You enable this configuration by using the `--model-bucket-uri` flag. These optimizations can improve LLM Pod startup time by more than 7x.
   - **(Optional) Autoscaling-optimized configuration** : generates a
     manifest with a Horizontal Pod Autoscaler (HPA) to automatically adjust
     the number of model server replicas based on traffic. You enable this
     configuration by specifying a latency target with flags like
     `--target-ntpot-milliseconds`.

     > [!NOTE]
     > **Note:** You can combine flags; for example, you can set both the `--model-bucket-uri` and the `--target-ntpot-milliseconds` flags to generate a manifest with a tuned Cloud Storage FUSE setup and an HPA.

   ### Base configuration

   In the terminal, use the [`manifests`](https://docs.cloud.google.com/sdk/gcloud/reference/container/ai/profiles/manifests)
   option to generate Deployment, Service, and PodMonitoring manifests:

   ```bash
   gcloud container ai profiles manifests create
   ```

   Use the required `--model`, `--model-server`, and
   `--accelerator-type` parameters to customize your manifest.

   Optionally, you can set these parameters:
   - `--target-ntpot-milliseconds`: set this parameter to specify your HPA
     threshold. This parameter lets you define a scaling threshold to keep
     the Normalized Time Per Output Token (NTPOT) P50 latency, which is measured
     at the 50th percentile, below the specified value. Choose a value above the minimum
     latency of your accelerator. The HPA is configured for maximum throughput
     if you specify an NTPOT value above the maximum latency of your accelerator.
     For example:

     ```bash
     gcloud container ai profiles manifests create \
         --model=google/gemma-4-E4B-it \
         --model-server=vllm \
         --model-server-version=v0.7.2 \
         --accelerator-type=nvidia-l4 \
         --target-ntpot-milliseconds=200
     ```
   - `--target-ttft-milliseconds`: the target time to first token (TTFT)
     in milliseconds. If this is set, the manifest includes Horizontal Pod
     Autoscaler (HPA) resources to keep p50 TTFT below the specified threshold.

   - `--output-path`: if specified, the output is saved to the provided
     path, rather than printed to the terminal so you can edit the output
     before deploying it. For example, you can use this with the `--output=manifest`
     option if you want to save your manifest in a YAML file. For example:

     ```bash
     gcloud container ai profiles manifests create \
         --model deepseek-ai/DeepSeek-R1-Distill-Qwen-7B \
         --model-server vllm \
         --accelerator-type=nvidia-tesla-a100 \
         --output=manifest \
         --output-path /tmp/manifests.yaml
     ```
     - `--target-itl-milliseconds`: the target inter-token latency (ITL) in milliseconds. If this is set, the manifest includes Horizontal Pod Autoscaler (HPA) resources to keep p50 ITL below the specified threshold.
     - `--use-case`: the manifest will be optimized for this use case. If not specified, this flag defaults to `Chatbot`. For more information, see [View recommendations by use case](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/machine-learning/inference/inference-quickstart.md.txt#use-case).

   For the full list of flags you can set, see the
   [`manifests create`](https://docs.cloud.google.com/sdk/gcloud/reference/container/ai/profiles/manifests/create)
   command documentation.

   ### Storage-optimized

   You can improve Pod startup time by loading models from
   Cloud Storage. Inference Quickstart automatically selects
   the highest-performance storage backend for your configuration:
   - **Run:ai Model Streamer**: a streaming solution that's used automatically for vLLM version 0.11.1 and later in GPUs and 0.14.0 and later in TPUs.
   - **Cloud Storage FUSE CSI driver**: The standard storage optimization, used for vLLM versions earlier than 0.11.1 and for other model servers.

   Loading from Cloud Storage requires
   GKE versions 1.29.6-gke.1254000, 1.30.2-gke.1394000, or
   later

   To do this, follow these steps:
   1. [Load the model from the Hugging Face repository to your Cloud Storage bucket](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/machine-learning/inference/inference-quickstart.md.txt#cloud-storage).
   2. Set the `--model-bucket-uri` flag when generating your manifest.
      This configures the model to load from a Cloud Storage bucket.
      The model automatically uses the high-performance Run:ai Model Streamer
      for supported vLLM versions, falling back to the Cloud Storage FUSE CSI driver
      for others. The URI must point to the path containing the
      model's `config.json` file and weights. You can specify a path to a directory
      within the bucket by appending it to the bucket URI.

      For example:

      ```bash
      gcloud container ai profiles manifests create \
          --model=openai/gpt-oss-120b \
          --model-server=vllm \
          --model-server-version=v0.11.2 \
          --accelerator-type=nvidia-h100-80gb \
          --model-bucket-uri=gs://BUCKET_NAME \
          --output-path=manifests.yaml
      ```

      Replace `BUCKET_NAME` with the name of your Cloud Storage bucket.
   3. Before applying the manifest, you must run the `gcloud storage buckets add-iam-policy-binding`
      command found in the manifest's comments. This command is required to grant the
      GKE service account permission to access the Cloud Storage
      bucket using Workload Identity Federation for GKE.

      If you intend to scale your Deployment to more than one replica on TPUs, you must
      choose one of the following options to prevent concurrent write errors to
      the XLA cache path (`VLLM_XLA_CACHE_PATH`):
      - Option 1 (Recommended): First, scale the Deployment to 1 replica. Wait for the Pod to become ready, which allows it to write to the XLA cache. Then, scale up to the number of replicas you want. The subsequent replicas will read from the populated cache without write conflicts.
      - Option 2: Remove the `VLLM_XLA_CACHE_PATH` environment variable from the manifest entirely. This approach is simpler but disables caching for all replicas.

      On TPU accelerator types, this cache path is used to store the XLA
      compilation cache, which accelerates model preparation for repeated deployments.

   For more tips about improving performance, see
   [Optimize Cloud Storage FUSE CSI driver for GKE performance](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/cloud-storage-fuse-csi-driver-perf.md.txt).

   ### Autoscaling-optimized

   You can configure the Horizontal Pod Autoscaler (HPA) to automatically adjust
   the number of model server replicas based on the load. This helps your
   model servers to efficiently handle varying loads by scaling up or down as needed.
   The HPA configuration follows the autoscaling best practices for [GPUs](https://docs.cloud.google.com/kubernetes-engine/docs/best-practices/machine-learning/inference/autoscaling.md.txt) and [TPUs](https://docs.cloud.google.com/kubernetes-engine/docs/best-practices/machine-learning/inference/autoscaling-tpu.md.txt) guides.

   To include HPA configurations when generating manifests, use one or both of
   the `--target-ntpot-milliseconds` and `--target-ttft-milliseconds` flags.
   These parameters define a scaling threshold for the HPA to keep the P50
   latency for either NTPOT or TTFT below the specified
   value. If you set only one of these flags, only that metric will be taken into
   account for scaling.

   Choose a value above the minimum latency of your accelerator. The HPA is
   configured for maximum throughput if you specify a value above the maximum
   latency of your accelerator.

   For example:

   ```bash
   gcloud container ai profiles manifests create \
       --model=google/gemma-4-E4B-it \
       --accelerator-type=nvidia-l4 \
       --target-ntpot-milliseconds=250
   ```

   > [!TIP]
   > **Tip:** The `gcloud container ai profiles manifests create` command prints the manifest to standard output. You should save it to a file (for example, by using output redirection `>`) to apply it later.

3. **Create a cluster** : You can serve your model on GKE
   Autopilot or Standard clusters. We recommend that you use
   an Autopilot cluster for a fully managed Kubernetes experience. To
   choose the GKE mode of operation that is the best fit for your workloads,
   see [Choose a GKE mode of operation](https://docs.cloud.google.com/kubernetes-engine/docs/concepts/choose-cluster-mode.md.txt).

   If you don't have an existing cluster, follow these steps:

   ### Autopilot

   Follow these instructions to [create an Autopilot cluster](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/creating-an-autopilot-cluster.md.txt). GKE handles provisioning
   the nodes with GPU or TPU capacity based on the deployment manifests, if
   you have the necessary quota in your project.

   ### Standard

   1. Create a [zonal](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/creating-a-zonal-cluster.md.txt) or [regional](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/creating-a-regional-cluster.md.txt) cluster.
   2. Create a node pool with the appropriate accelerators. Follow these
      steps based on your chosen accelerator type:

      - **GPUs** : First, check the [Quotas page](https://console.cloud.google.com/iam-admin/quotas) in the Google Cloud console to ensure you have sufficient GPU capacity. Then, follow the instructions in [Create a GPU node pool](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/gpus.md.txt#create-gpu-pool-auto-drivers).
      - **TPUs** : First, ensure you have sufficient TPU by following the instructions in [Ensure quota for TPUs and other GKE resources](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/tpus-autopilot.md.txt#ensure-tpu-quota). Then, proceed to [Create a TPU node pool](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/tpus-autopilot.md.txt#create-tpu-pool).
4. **(Optional, but recommended) Enable observability features**: In the
   comments section of the generated manifest, additional commands are
   provided to enable suggested observability features. Enabling these features
   provides more insights to help you monitor the performance
   and status of workloads and the underlying infrastructure.

   The following is an example of a command to enable observability features:

   ```bash
   gcloud container clusters update $CLUSTER_NAME \
       --project=$PROJECT_ID \
       --location=$LOCATION \
       --enable-managed-prometheus \
       --logging=SYSTEM,WORKLOAD \
       --monitoring=SYSTEM,DEPLOYMENT,HPA,POD,DCGM \
       --auto-monitoring-scope=ALL
   ```

   For more information, see [Monitor your inference workloads](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/machine-learning/inference/inference-quickstart.md.txt#monitor).
5. **(HPA only) Deploy a metrics adapter** : A metrics adapter, such as the [Custom Metrics Stackdriver Adapter](https://github.com/GoogleCloudPlatform/k8s-stackdriver/blob/master/custom-metrics-stackdriver-adapter/README.md),
   is necessary if HPA resources were generated in the deployment manifests.
   The metrics adapter enables the HPA to access model server metrics that use the
   [kube external metrics API](https://kubernetes.io/docs/reference/external-api/external-metrics.v1beta1/). To deploy the adapter, refer to the [adapter documentation on GitHub](https://github.com/GoogleCloudPlatform/k8s-stackdriver/blob/master/custom-metrics-stackdriver-adapter/README.md).

6. **Deploy the manifests** : run the `kubectl apply` command and pass in
   the YAML file for your manifests. For example:

   ```bash
   kubectl apply -f ./manifests.yaml
   ```

### Deploy recommended configurations for the llm-d serving stack

llm-d is a Kubernetes-native distributed inference serving stack that
provides tested and benchmarked ["well-lit paths"](https://llm-d.ai/docs/guide)
to help you serve large generative AI models at scale with a high level of
performance. For more information, see the
[llm-d documentation](https://llm-d.ai/docs/architecture).

To generate and deploy a recommended configuration for the llm-d serving stack
by using the Inference Quickstart, use the
[`--serving-stack`](https://docs.cloud.google.com/sdk/gcloud/reference/container/ai/profiles/manifests/create#--serving-stack)
flag with the value set to `llm-d`. For example:

```bash
gcloud container ai profiles manifests create \
    --accelerator-type=nvidia-h100-80gb \
    --model=openai/gpt-oss-120b \
    --model-server=vllm \
    --serving-stack=llm-d \
    --use-case 'Multi Agent Large Document Summarization'
```

Check the `manifests create` output for additional cluster creation and
dependency installation instructions to correctly set up your environment for
llm-d.

## Test your deployment endpoints

After deploying the manifest, the service is exposed at
`http://SERVICE_NAME:8000`, where
<var translate="no">SERVICE_NAME</var> is the name of your deployment.

To confirm the service name, you can list the services in your namespace
(for example, `default`):

```bash
kubectl get services --namespace NAMESPACE
```

To test your deployment, use the `kubectl port-forward` command to forward a local port to
the service port. In a separate terminal, run the following command:

```bash
kubectl port-forward service/SERVICE_NAME 8000:8000
```

You can then send requests to `http://localhost:8000`. For examples of how to
build and send requests to your endpoint, see the
[vLLM](https://docs.vllm.ai/en/latest/getting_started/quickstart.html#openai-completions-api-with-vllm) documentation.

## Manifest versioning

Inference Quickstart provides the latest manifests that were validated
on recent GKE cluster versions. The manifest returned for a
profile might change over time so that you receive an optimized
configuration at deployment. If you need a stable manifest, save and
store it separately.

The manifest includes comments and a `recommender.ai.gke.io/version` annotation
in the following format:

    # Generated on DATE using:
    # GKE cluster CLUSTER_VERSION
    # GPU_DRIVER_VERSION GPU driver for node version NODE_VERSION
    # Model server MODEL_SERVER MODEL_SERVER_VERSION

The previous annotation has the following values:

- <var translate="no">DATE</var>: the date the manifest was generated.
- <var translate="no">CLUSTER_VERSION</var>: the GKE cluster version used for validation.
- <var translate="no">NODE_VERSION</var>: the GKE node version used for validation.
- <var translate="no">GPU_DRIVER_VERSION</var>: (GPU only) the GPU driver version used for validation.
- <var translate="no">MODEL_SERVER</var>: the model server used in the manifest.
- <var translate="no">MODEL_SERVER_VERSION</var>: the model server version used in the manifest.

## Monitor your inference workloads

To monitor your deployed inference workloads, go to the [Metrics Explorer in the Google Cloud console](https://console.cloud.google.com/monitoring/metrics-explorer).

### Enable auto-monitoring

GKE includes an auto-monitoring feature that is part of the
broader observability features. This feature scans the cluster for workloads
that run on supported model servers and deploys the [PodMonitoring resources](https://docs.cloud.google.com/stackdriver/docs/managed-prometheus/setup-managed.md.txt#gmp-pod-monitoring) that enable these workload metrics to be visible in
Cloud Monitoring. For more information about enabling and configuring auto-monitoring, see [Configure automatic application monitoring for workloads](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/configure-automatic-application-monitoring.md.txt).

After enabling the feature, GKE installs prebuilt
dashboards for monitoring applications for [supported workloads](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/configure-automatic-application-monitoring.md.txt#supported-workloads).

> [!IMPORTANT]
> **Important:** If `PodMonitoring` resources have already been deployed to collect metrics from [supported workloads](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/configure-automatic-application-monitoring.md.txt#supported-workloads), remove them before enabling the auto-monitoring feature to avoid double collection.

If you deploy from the GKE AI/ML page in the Google Cloud console,
PodMonitoring and HPA resources are automatically created for you by using the
`targetNtpot` configuration.

## Troubleshooting

- If you set the latency too low, Inference Quickstart might not generate a recommendation. To fix this issue, select a latency target between the minimum and maximum latency that was observed for your selected accelerators.
- Inference Quickstart exists independently of GKE components, so your cluster version is not directly relevant for using the service. However, we recommend using a fresh or up-to-date cluster to avoid any discrepancies in performance.
- If you get a `PERMISSION_DENIED` error for `gkerecommender.googleapis.com` commands that says a quota project is missing, you need to set it manually. Run `gcloud config set billing/quota_project PROJECT_ID` to fix this.

### Pod evicted due to low ephemeral storage

When deploying a large model (90 GiB or more) from Hugging Face, your Pod might be evicted with an error message similar to this:

    Fails because inference server consumes too much ephemeral storage, and gets evicted low resources:  Warning  Evicted              3m24s                   kubelet                                The node was low on resource: ephemeral-storage. Threshold quantity: 10120387530, available: 303108Ki. Container inference-server was using 92343412Ki, request is 0, has larger consumption of ephemeral-storage..,

This error occurs because the model is cached on the node's boot disk, a form of [ephemeral storage](https://docs.cloud.google.com/kubernetes-engine/docs/concepts/plan-node-sizes.md.txt#ephemeral_storage_backed_by_node_boot_disk). Boot disk is used for ephemeral storage when the deployment manifest does not set the `HF_HOME` environment variable to a directory in the node's RAM.

- By default, GKE nodes have a 100 GiB boot disk.
- GKE reserves 10% of the boot disk for system overhead, leaving 90 GiB for your workloads.
- If the model size is 90 GiB or larger, and run on a default sized boot disk, kubelet evicts the Pod to free up ephemeral storage.

To resolve this issue, choose one of the following options:

- **Use RAM for model caching** : In your deployment manifest, set the `HF_HOME` environment variable to `/dev/shm/hf_cache`. This uses the node's RAM to cache the model instead of the boot disk.
- **Increase the boot disk size** :
  - **GKE Standard** : Increase the boot disk size when you [create a cluster](https://cloud.google.com/sdk/gcloud/reference/container/clusters/create#--disk-size), [create a node pool](https://cloud.google.com/sdk/gcloud/reference/container/node-pools/create#--disk-size), or [update a node pool](https://cloud.google.com/sdk/gcloud/reference/container/node-pools/update#--disk-size).
  - **Autopilot** : To request a larger boot disk, create a [Custom Compute class](https://docs.cloud.google.com/kubernetes-engine/docs/concepts/about-custom-compute-classes.md.txt#machinetype-rule) and set the `bootDiskSize` field in the `machineType` rule.

### Pod enters a crash loop when loading models from Cloud Storage

After you deploy a manifest that was generated with the `--model-bucket-uri` flag,
the Deployment may get stuck and the Pod enters a `CrashLoopBackOff` state.
Checking the logs for the `inference-server` container might show a misleading
error, such as `huggingface_hub.errors.HFValidationError`. For example:

    huggingface_hub.errors.HFValidationError: Repo id must use alphanumeric chars or '-', '_', '.', '--' and '..' are forbidden, '-' and '.' cannot start or end the name, max length is 96: '/data'.

This error typically occurs when the Cloud Storage path provided in
the `--model-bucket-uri` flag is incorrect. The inference server, such as vLLM,
cannot find the required model files (like `config.json`) at the mounted path.
When it fails to find the local files, the server falls back to assuming the
path is a Hugging Face Hub repository ID. Because the path is not a valid
repository ID, the server fails with a validation error and enters a crash loop.

To resolve this issue, verify that the path you provided to the `--model-bucket-uri`
flag points to the exact directory in your Cloud Storage bucket that
contains the model's `config.json` file and all associated model weights.

## What's next

- Visit the [AI/ML orchestration on GKE portal](https://docs.cloud.google.com/kubernetes-engine/docs/integrations/ai-infra.md.txt) to explore our official guides, tutorials, and use cases for running AI/ML workloads on GKE.
- For more information about model serving optimization, see [Best practices for optimizing large language model inference with GPUs](https://docs.cloud.google.com/kubernetes-engine/docs/best-practices/machine-learning/inference/llm-optimization.md.txt). It covers best practices for LLM serving with GPUs on GKE, like quantization, tensor parallelism, and memory management.
- For more information about best practices for autoscaling, see these guides:
  - [Best practices for autoscaling large language model (LLM) inference workloads with GPUs](https://docs.cloud.google.com/kubernetes-engine/docs/best-practices/machine-learning/inference/autoscaling.md.txt)
  - [Best practices for autoscaling large language model (LLM) inference workloads with TPUs](https://docs.cloud.google.com/kubernetes-engine/docs/best-practices/machine-learning/inference/autoscaling-tpu.md.txt)
- For information on storage best practices, see [Optimize Cloud Storage FUSE CSI driver for GKE performance](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/cloud-storage-fuse-csi-driver-perf.md.txt).
- Explore experimental samples for leveraging GKE to accelerate your AI/ML initiatives in [GKE AI Labs](https://gke-ai-labs.dev/).