Run and profile vLLM workloads with ML Diagnostics

vLLM is an open-source library for large language model (LLM) inference and serving. Integrating ML Diagnostics with vLLM lets you track serving configurations, monitor workload telemetry, and capture XProf profiles on Google Cloud. This document describes the ML Diagnostics wrapper command for vLLM and shows you how to containerize, run, and profile a vLLM workload.

Wrapper command-line arguments

ML Diagnostics provides a built-in wrapper command, google_cloud_mldiagnostics.commands.run_vllm, to launch vLLM serving and inference workloads and automatically configure telemetry and profiling. You can set the run_vllm wrapper as the entrypoint in your vLLM Docker container or run it directly from the command line on Google Kubernetes Engine (GKE) or Compute Engine.

When you start a workload with the run_vllm command, the wrapper performs the following actions:

  • Register the run: Creates and registers a managed MLRun resource configured for vLLM serving workloads.
  • Configure profiling: Sets profiler environment variables and updates profiler output paths in Cloud Storage to organize traces in a directory for your ML run.
  • Forward arguments: Passes all remaining command-line arguments directly to the vllm command.

The following table describes the command-line arguments that the run_vllm wrapper accepts:

Argument Description
--mlrun_name Required. Unique identifier for the ML run (MLRun) resource.
--mlrun_gcs_path Required. Cloud Storage bucket path that stores profiler traces and artifacts. For example, gs://my-bucket/vllm-profiles.
--project Optional. Your Google Cloud project ID.
--region Optional. Region that stores the MLRun metadata. Default: us-central1.
--run_group Optional. Identifier that groups multiple related runs.
--configs Optional. User-defined configurations that you pass as a JSON object string. For example, '{"user":"my-team", "running_on":"TPU"}'.
--framework Optional. Underlying ML framework for the workload (pytorch or jax). Default: pytorch.
--jax_profiler_port Optional. JAX profiler server port. Default: 9999.
--phased_profiling_dir Optional. Base Cloud Storage directory for JAX phased profiling traces. The wrapper appends the RUN_NAME/plugins/profile/ path and exports the result as the PHASED_PROFILING_DIR environment variable to the vLLM process. Use this argument to enable phased profiling.

Phased profiling in multi-host setups

To enable JAX phased profiling in multi-host setups, such as a Ray backend on a GKE LeaderWorkerSet, pass the --phased_profiling_dir argument and omit the PHASED_PROFILING_DIR environment variable in your operating system environment.

If you set the PHASED_PROFILING_DIR environment variable without passing the --phased_profiling_dir argument, phased profiles might not appear in XProf in the Google Cloud console for the following reasons:

  • The run_vllm wrapper detects the PHASED_PROFILING_DIR variable in the operating system environment, appends the run-scoped suffix, forwards the path to the vllm serve command, and logs a warning.
  • vLLM's Ray executor applies driver environment variables on remote workers by using the os.environ.setdefault method.
  • Remote workers that already have the PHASED_PROFILING_DIR variable in their environment keep their existing path instead of adopting the run-scoped path, which writes their traces outside the MLRun directory.

Run and profile vLLM workloads

To serve a model with vLLM and ML Diagnostics, complete the following steps:

  1. Build a Docker container image that packages vLLM with the ML Diagnostics SDK and configures the run_vllm wrapper as the entrypoint.
  2. Run the container on GKE or Compute Engine by passing your MLRun arguments alongside your vllm serve arguments.
  3. Capture performance traces while the server processes inference requests by using phased, manual, or on-demand profiling, and view the results in the Google Cloud console.

Prepare a Docker image for vLLM

To use the run_vllm wrapper in a container, build a Docker image that installs the ML Diagnostics SDK, the xprof package, and logging dependencies on top of the vLLM TPU base image. In the Dockerfile, set the google_cloud_mldiagnostics.commands.run_vllm module as the container entrypoint.

The following Dockerfile example packages a vLLM TPU image with the ML Diagnostics SDK:

FROM vllm/vllm-tpu:latest
WORKDIR /app
RUN pip install --no-cache-dir --upgrade pip && \
    pip install --no-cache-dir \
        google-cloud-mldiagnostics \
        xprof \
        google-cloud-logging \
        packaging \
        tensorflow-cpu
EXPOSE 8000 9999
ENTRYPOINT ["python3", "-m", "google_cloud_mldiagnostics.commands.run_vllm"]
CMD ["--help"]

Run vLLM with ML Diagnostics

After you build the Docker image, you can run the container on GKE or Compute Engine. When you start the container, configure the profiler environment variables and pass the MLRun arguments along with your standard vllm serve arguments.

Run the vLLM container with ML Diagnostics enabled:

docker run --name vllm_diagnostics -d --privileged --net=host \
  -e HF_HUB_DISABLE_XET=1 \
  -e USE_MOE_EP_KERNEL=1 \
  -e PJRT_DEVICE=TPU \
  -e ENABLE_GOOGLE_DIAGON_ML_DIAGNOSTICS=True \
  -e VLLM_LOGGING_LEVEL=DEBUG \
  -e VLLM_TORCH_PROFILER_DIR="gs://BUCKET_NAME/vllm_profiles" \
  -e PHASED_PROFILING_DIR="gs://BUCKET_NAME/vllm_profiles" \
  -e PYTHONUNBUFFERED=1 \
  -e PHASED_PROFILER_NUM_STEPS_TO_PROFILE_FOR=100 \
  -e HF_TOKEN="HF_TOKEN" \
  -e HUGGING_FACE_HUB_TOKEN="HF_TOKEN" \
  VLLM_IMAGE \
  --mlrun_name="vllm-tpu-run" \
  --run_group="vllm-benchmark" \
  --configs='{"user":"ml-team", "running_on":"TPU"}' \
  --mlrun_gcs_path="gs://BUCKET_NAME/vllm_profiles" \
  --project="PROJECT_ID" \
  --region="us-central1" \
  -- serve Qwen/Qwen2-1.5B \
  --kv-cache-dtype=fp8 \
  --tensor-parallel-size=1 \
  --data-parallel-size=1 \
  --max-model-len=2048 \
  --max-num-batched-tokens=2048 \
  --max-num-seqs=64 \
  --no-enable-prefix-caching \
  --block-size=256 \
  --gpu-memory-utilization=0.95 \
  --port=8000 \
  --profiler-config.profiler=torch \
  --profiler-config.torch_profiler_dir="gs://BUCKET_NAME/vllm_profiles"

Replace the following:

  • BUCKET_NAME: the name of your Cloud Storage bucket for storing profiler traces and artifacts
  • HF_TOKEN: your Hugging Face access token
  • VLLM_IMAGE: the name or URI of your vLLM Docker image
  • PROJECT_ID: your Google Cloud project ID

Profile vLLM workloads

When you start your container with the run_vllm command, the wrapper configures the profiler environment and saves all profiler traces in Cloud Storage under the MLRUN_GCS_PATH/RUN_NAME/plugins/profile/ directory:

  • MLRUN_GCS_PATH: the Cloud Storage bucket path that you specify with the --mlrun_gcs_path argument
  • RUN_NAME: the unique identifier for your MLRun resource

While your vLLM container serves inference requests, you can capture profiler traces by using phased, manual, or on-demand profiling.

Phased profiling

Phased profiling automatically captures traces across different inference workload phases, such as PREFILL_ONLY, PREFILL_HEAVY, BALANCED, and DECODE_HEAVY. When you provide a base Cloud Storage path, the run_vllm wrapper appends the RUN_NAME/plugins/profile/ subdirectory and exports the resulting path to the PHASED_PROFILING_DIR environment variable.

To enable phased profiling when you start the container, configure the following settings:

  • Set the ENABLE_GOOGLE_DIAGON_ML_DIAGNOSTICS=True environment variable.
  • Set the PHASED_PROFILER_NUM_STEPS_TO_PROFILE_FOR environment variable to the number of steps to profile.
  • Pass your base Cloud Storage path using the --phased_profiling_dir wrapper argument (or set the PHASED_PROFILING_DIR environment variable in single-host setups).

Manual (API-triggered) profiling

Manual profiling lets you start and stop profiling sessions programmatically for specific windows of inference traffic. When you start the container, the run_vllm wrapper updates the VLLM_TORCH_PROFILER_DIR environment variable and the --profiler-config.torch_profiler_dir argument to save traces in the profile directory for your ML run.

To capture a manual profile:

  1. When you start the container, set the VLLM_TORCH_PROFILER_DIR environment variable and pass the --profiler-config.profiler=torch and --profiler-config.torch_profiler_dir arguments to the vllm serve command.
  2. While generating load against the server, send HTTP requests to vLLM's /start_profile and /stop_profile API endpoints to start and stop the profiling session.

On-demand profiling in the Google Cloud console

On-demand profiling lets you trigger live profile captures interactively from the Google Cloud console without restarting your workload or sending API requests. When you start the container, the run_vllm wrapper automatically starts the JAX profiler server by setting the USE_JAX_PROFILER_SERVER=True and JAX_PROFILER_SERVER_PORT environment variables.

To capture an on-demand profile while your workload is serving requests, complete the following steps:

  1. In the Google Cloud console, go to the Run Diagnostics tab and select your MLRun resource:
    • Cluster Director: Go to the Cluster Director page and click the Run Diagnostics tab.
    • GKE: Go to the Kubernetes page, click AI/ML in the navigation menu, and click the Run Diagnostics tab.
  2. Select the Profiles tab.
  3. Start a capture session.