llm-d

This document describes how your Google Kubernetes Engine deployment can use Google Cloud Managed Service for Prometheus to collect metrics from llm-d. llm-d consists of the llm-d Router (Endpoint Picker, deployed in Standalone Mode or Gateway Mode with GKE Gateway) and model servers such as vLLM.

For information about collecting metrics from the llm-d Router and vLLM, see the following:

  • llm-d Router (EPP): When deploying the router with the llm-d Helm chart, include the monitoring-gke.values.yaml recipe (router.monitoring.provider.name=gmp and router.monitoring.prometheus.enabled=true) to automatically create the PodMonitoring resource and metrics-reader Secret in your workload namespace. For standalone GKE Inference Gateway deployments or llm-d versions earlier than v0.8.0, see GKE Inference Gateway.
  • vLLM: Enable automatic application monitoring or use the PodMonitoring configuration described in this document to map the llm-d.ai/role pod label for Prefill/Decode (P/D) disaggregation.

The instructions in these documents apply only if you are using managed collection with Managed Service for Prometheus. If you are using self-deployed collection, then see the llm-d documentation.

After you configure the llm-d Router and vLLM, you can access a predefined dashboard in Cloud Monitoring to view the metrics.

Prerequisites

To collect metrics from llm-d by using Managed Service for Prometheus and managed collection, your deployment must meet the following requirements:

  • Your cluster must be running Google Kubernetes Engine version 1.28.15-gke.2475000 or later.
  • You must be running Managed Service for Prometheus with managed collection enabled. For more information, see Get started with managed collection.

To group vLLM metrics by the llm-d.ai/role pod label (for example, prefill and decode in Prefill/Decode disaggregation), apply the following PodMonitoring resource in your workload namespace:

apiVersion: monitoring.googleapis.com/v1
kind: PodMonitoring
metadata:
  name: llm-d-metrics
spec:
  selector:
    matchLabels:
      llm-d.ai/InferenceServing: "true"
  endpoints:
  - port: 8000
    interval: 15s
    path: /metrics
  targetLabels:
    fromPod:
    - from: llm-d.ai/role
      to: role
    metadata:
    - pod
    - container
    - node
    - top_level_controller_name
    - top_level_controller_type

View dashboards

The Cloud Monitoring integration includes the following unified dashboards for both Standalone Mode and Gateway Mode (the Application Load Balancer section is populated when the llm-d proxy uses GKE Inference Gateway):

  • llm-d Overview: For llm-d version v0.8.0 and later (queries llm_d_epp_* EPP metrics alongside Application Load Balancer, GPU networking, and vLLM metrics).
  • llm-d Overview (< v0.8.0): For llm-d versions earlier than v0.8.0 (queries inference_objective_* metrics).

Dashboards are automatically installed when you configure the integration. You can also view static previews of dashboards without installing the integration.

To view an installed dashboard, do the following:

  1. In the Google Cloud console, go to the  Dashboards page:

    Go to Dashboards

    If you use the search bar to find this page, then select the result whose subheading is Monitoring.

  2. Select the Dashboard List tab.
  3. Choose the Integrations category.
  4. Click the name of the dashboard you want to view.

To view a static preview of the dashboard, do the following:

  1. In the Google Cloud console, go to the  Integrations page:

    Go to Integrations

    If you use the search bar to find this page, then select the result whose subheading is Monitoring.

  2. Click the Kubernetes Engine deployment-platform filter.
  3. Locate the llm-d integration and click View Details.
  4. Select the Dashboards tab.