Configure model routing

This page describes how to configure, deploy, and test model routing in API Gateway using OpenAPI 3.x specifications.

Before you begin

Before configuring model routing, check that your environment meets the following prerequisites:

  1. Check IAM permissions: Check that you have access to the API Gateway Management Plane and Vertex AI Model Garden. You must have the API Gateway Admin (roles/apigateway.admin) role to create API configs and gateways. In addition, the service account used by your API gateway—either the default Compute Engine service account or a user-managed service account specified when creating the API config—must be granted the Vertex AI User (roles/aiplatform.user) role to access target models.
  2. Check model availability and endpoint access: Check that your routable models are pre-deployed open models for Model as a Service (MaaS) in Vertex AI Model Garden. All models referenced by a single router must share the exact same hostname. Pick either the global endpoint (aiplatform.googleapis.com) or a single regional endpoint (for example, us-central1-aiplatform.googleapis.com) for every model referenced within that router.
  3. Check gateway deployment eligibility: You cannot update an existing gateway deployed without model routing to enable model routing, nor can you update a gateway deployed with model routing to disable or remove model routing. To switch routing modes, you must create and deploy a new API config and gateway instance.
  4. Check VPC Service Controls and endpoint compatibility: Model routing gateways do not support VPC Service Controls or Private Service Connect (PSC) endpoint configurations. Check that your target project and API Gateway instances are not restricted by VPC Service Controls perimeters, and that your models use public regional or global endpoints.

Configuration validation

When you deploy an API config, the API Gateway management plane validates your OpenAPI specification. The management plane rejects invalid configurations during deployment with an informational validation error. The validation process enforces the following rules:

Structural and location checks

  • The x-google-api-management extension and its associated blocks (backends, ai.models.routing.routers, individual routers, and rules) must be well-formed. Keys must match their expected data types (map, list, or string). The management plane rejects type mismatches with an expected map/list/string error.
  • The x-google-api-management extension must contain a valid backends block when model routing is enabled.
  • The x-google-model-router extension is supported only in OpenAPI 3.x specifications (it isn't supported in OpenAPI 2.0 / Swagger).
  • The x-google-model-router extension can only be specified at the operation level. The management plane explicitly rejects x-google-model-router definitions placed at the path level or root (top) level.
  • The ai.models.routing.routers block must be defined inside x-google-api-management whenever any operation references x-google-model-router.
  • You cannot specify both x-google-model-router and x-google-backend on the same API operation.
  • An OpenAPI specification cannot contain a mix of model routing and non-model routing operations. You cannot specify standard routing extensions (such as x-google-backend) on some operations while using x-google-model-router on other operations within the same API specification.

HTTP method check

  • The x-google-model-router extension can only be applied to operations using the POST HTTP method. The management plane rejects model routing on any other HTTP method (such as GET, PUT, or DELETE).

Backend validity

  • Every backend defined under x-google-api-management.backends must include a non-empty address field.
  • The backend address must be a valid URL using the http or https scheme. To protect prompt payloads and authentication credentials in transit across public or remote endpoints, always specify the https scheme when defining the address field.
  • Every backend defined under x-google-api-management.backends and referenced by a model router must use pathTranslation: CONSTANT_ADDRESS. The management plane rejects configurations using pathTranslation: APPEND_PATH_TO_ADDRESS for model routing backends because path translation is ignored in the model router's runtime path.
  • Model routing backends don't support VPC Service Controls or Private Service Connect (PSC) endpoint configurations. All backend address fields must point to public regional or global MaaS open model endpoints.

Router reference resolution

  • The router name referenced by an operation's x-google-model-router must match a valid router key defined under ai.models.routing.routers.
  • The backend referenced by a router's defaultModel must match a valid backend defined under x-google-api-management.backends.
  • The backend referenced by each rule in a router must match a valid backend defined under x-google-api-management.backends.

Router contents

  • Each router must define a defaultModel.
  • The defaultModel must include a valid backend field.
  • The defaultModel must include a non-empty targetModel field.
  • Each entry under rules must include a non-empty model field. The string value default is reserved and cannot be used as a rule's model value.
  • Each entry under rules must include a non-empty targetModel field.
  • The model values defined across all rules within a single router must be unique. The management plane rejects duplicate model values within the same router.

Backend host and scheme consistency

  • All backends referenced by a single router (including defaultModel.backend and every rule's backend) must share the identical hostname and URL scheme. The management plane rejects configurations with differing hostnames or inconsistent schemes (http versus https) within the same router, ensuring the router dispatches all requests to a consistent upstream service endpoint.

Target model validation

  • The <provider> portion of the targetModel string (google, openai, or anthropic) and the <provider>/<model> identifier format are both validated at config-create (deploy) time. The management plane rejects a targetModel that isn't formatted as <provider>/<model> or whose provider isn't google, openai, or anthropic with an InvalidArgument: unsupported publisher error during deployment.

Step 1: Identify target models

Identify the target foundation models and their corresponding Vertex AI endpoint URLs. All routable models within a router must share a single hostname (for MaaS open models, this hostname is aiplatform.googleapis.com).

Endpoint URL paths vary based on the model provider:

  • Google Gemini: Uses the :generateContent method.
  • Anthropic Claude: Uses the :rawPredict method.
  • OpenAI: Uses the /endpoints/openapi/chat/completions endpoint path.

The following table lists the MaaS endpoints used in the OpenAPI specification example later in this section:

Model Endpoint URL
google/gemini-3.5-flash-lite https://aiplatform.googleapis.com/v1/projects/YOUR_PROJECT_ID/locations/global/publishers/google/models/gemini-3.5-flash-lite:generateContent
anthropic/claude-opus-4-7 https://aiplatform.googleapis.com/v1/projects/YOUR_PROJECT_ID/locations/global/publishers/anthropic/models/claude-opus-4-7:rawPredict
openai/gpt-oss-120b-maas https://aiplatform.googleapis.com/v1/projects/YOUR_PROJECT_ID/locations/global/endpoints/openapi/chat/completions

Replace YOUR_PROJECT_ID with your Google Cloud project ID.

Step 2: Configure the OpenAPI 3.x specification

Create or update your OpenAPI 3.x specification to define your backend endpoints and model routing configurations.

The following example demonstrates an OpenAPI 3.0.3 specification defining two distinct model routers. To prevent horizontal scrolling, long backend address URLs use YAML double-quoted multi-line string continuation (``):

openapi: 3.0.3

info:
  title: OpenAPI 3.x spec using Model Routing
  description: Using Model Routing in an OAS 3.x spec
  version: 1.0.0

x-google-api-management:
  backends:
    gemini-35-flashlite:
      address: "https://aiplatform.googleapis.com/v1/projects/\
        YOUR_PROJECT_ID/locations/global/publishers/google/\
        models/gemini-3.5-flash-lite:generateContent"
      deadline: 60.0
      pathTranslation: CONSTANT_ADDRESS

    anthropic-claude-opus-47:
      address: "https://aiplatform.googleapis.com/v1/projects/\
        YOUR_PROJECT_ID/locations/global/publishers/anthropic/\
        models/claude-opus-4-7:rawPredict"
      deadline: 60.0
      pathTranslation: CONSTANT_ADDRESS

    openai-gpt-oss-120b:
      address: "https://aiplatform.googleapis.com/v1/projects/\
        YOUR_PROJECT_ID/locations/global/endpoints/openapi/\
        chat/completions"
      deadline: 60.0
      pathTranslation: CONSTANT_ADDRESS

  ai:
    models:
      routing:
        routers:
          # Router 1: route between Gemini (default) and Claude.
          gemini-claude-router:
            defaultModel:
              backend: gemini-35-flashlite
              targetModel: google/gemini-3.5-flash-lite
            rules:
              - model: "claude-opus-4-7"
                backend: anthropic-claude-opus-47
                targetModel: anthropic/claude-opus-4-7

          # Router 2: route between OpenAI GPT (default) and Gemini.
          openai-gemini-router:
            defaultModel:
              backend: openai-gpt-oss-120b
              targetModel: openai/gpt-oss-120b-maas
            rules:
              - model: "gemini-3.5-flash-lite"
                backend: gemini-35-flashlite
                targetModel: google/gemini-3.5-flash-lite

servers:
  - url: "https://my-gateway-url.com"

paths:
  /v1/chat/gemini-claude:
    post:
      summary: "Endpoint:defaults to Gemini & Claude as an option."
      operationId: "chatGeminiClaude"
      x-google-model-router: gemini-claude-router
      responses:
        '200':
          description: "OK"

  /v1/chat/openai-gemini:
    post:
      summary: "Endpoint:defaults to OpenAI & Gemini as an option."
      operationId: "chatOpenAIGemini"
      x-google-model-router: openai-gemini-router
      responses:
        '200':
          description: "OK"

Configuration properties

  1. backends: The backends object under x-google-api-management defines all routable model endpoints. Each backend name represents a symbolic model name (for example, gemini-35-flashlite) containing the destination address. The backends field is an existing Google OpenAPI extension.
  2. ai.models.routing: The model routing configuration resides under x-google-api-management as ai.models.routing, containing a map of named routers. Each map entry defines one model router, where the key represents the router's name (for example, gemini-claude-router) and the value contains:
    • defaultModel: The required fallback model destination used when an incoming request payload doesn't match any explicit rule. It shares the exact structure of a rule entry but omits the model matching field. For OpenAI-compatible routes, when a request falls back to defaultModel, the value of targetModel is forwarded as the outgoing model attribute in the request body sent to Vertex AI.
    • rules: An optional array where each element maps a client-payload model string to a destination backend and target model.
  3. Rule properties: Each entry within rules (and the defaultModel) defines the following properties:
    • model (rules only): The string value matched against the model attribute within the client's incoming JSON prompt payload. The router compares the incoming payload's model value to this string. If no rule matches, the router selects the defaultModel. For OpenAI-compatible routes (where the destination backend is /openapi/chat/completions), this string is forwarded directly as the outgoing model attribute in the request body sent to Vertex AI. Therefore, for OpenAI-compatible routes, the model selector must itself be a valid publisher model identifier (for example, openai/gpt-oss-120b-maas); using an alias such as gpt-oss results in a 400 Malformed publisher model error from Vertex AI.
    • backend: The symbolic backend name defined under x-google-api-management.backends where the gateway sends the prompt.
    • targetModel: The target model identifier formatted as <provider>/<model-id>. The model router uses this string to translate requests and responses for the destination model. The <provider> prefix must be exactly google, openai, or anthropic. The <model-id> must be a valid Vertex AI Model Garden publisher model identifier. The gateway echoes this string back within the model field of the response returned to the client. Example values include:
      • google/gemini-3.5-flash-lite
      • google/gemini-2.5-pro
      • openai/gpt-oss-120b-maas
      • anthropic/claude-opus-4-7
  4. x-google-model-router: To attach a model router to an API operation path, specify the router name using the x-google-model-router attribute. In the previous example, a POST request sent to /v1/chat/gemini-claude invokes gemini-claude-router, which routes the prompt based on the model name specified in the JSON payload.

Step 3: Create and deploy the API config

Create an API config using your authored OpenAPI 3.x specification and deploy the config to your API Gateway instance as described in Deploying an API to a gateway.

The API Gateway management plane processes your model routing configuration and activates the routing layer. When your gateway deployment completes, the gateway is ready to receive prompt requests formatted as OpenAI-compatible JSON payloads.

Step 4: Test routing behavior

Before testing your gateway, wait for the gateway to reach the ACTIVE state, and then retrieve its URL:

gcloud api-gateway gateways describe GATEWAY_ID \
  --location=GATEWAY_LOCATION \
  --project=PROJECT_ID \
  --format='value(defaultHostname)'

During Public Preview, model routing gateways return a *.run.app hostname. Retrieve the hostname only after the gateway is ACTIVE; the value reported while the gateway is still being created is not the final URL.

Test your gateway's routing behavior using curl to send OpenAI-compatible prompt requests to your gateway URL (https://GATEWAY_URL). In the following examples, $TOKEN represents a valid authentication token obtained using any of the methods described in Choosing an Authentication Method.

Test explicit rule routing

Send a prompt requesting the Claude model anthropic/claude-opus-4-7:

curl https://GATEWAY_URL/v1/chat/gemini-claude \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $TOKEN" \
  -d '{
    "model": "claude-opus-4-7",
    "messages": [
      {
        "role": "system",
        "content": "You are a helpful assistant."
      },
      {
        "role": "user",
        "content": "Explain the concept of recursion in one sentence."
      }
    ]
  }'

Sending the request to /v1/chat/gemini-claude invokes gemini-claude-router. The attribute "model": "claude-opus-4-7" within the JSON payload matches the explicit rule in gemini-claude-router, directing the gateway to route the request to the anthropic-claude-opus-47 backend.

Test default model fallback

Send a prompt specifying an unmatched model name to test fallback routing:

curl https://GATEWAY_URL/v1/chat/gemini-claude \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $TOKEN" \
  -d '{
    "model": "unrecognized-model",
    "messages": [
      {
        "role": "user",
        "content": "Write a short poem about the ocean."
      }
    ],
    "stream": true
  }'

Sending the request to /v1/chat/gemini-claude invokes gemini-claude-router. Because the attribute "model": "unrecognized-model" doesn't match any explicit rule, the gateway dispatches the request to the router's configured defaultModel—the gemini-35-flashlite backend.

Test alternative router path

Send a prompt requesting Gemini through the secondary router endpoint:

curl https://GATEWAY_URL/v1/chat/openai-gemini \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $TOKEN" \
  -d '{
    "model": "gemini-3.5-flash-lite",
    "messages": [
      {
        "role": "user",
        "content": "List the three largest cities in the world."
      }
    ]
  }'

Sending the request to /v1/chat/openai-gemini invokes openai-gemini-router. The attribute "model": "gemini-3.5-flash-lite" matches the explicit rule in that router, directing the gateway to route the request to the gemini-35-flashlite backend. A single backend can be referenced by multiple routers; in this configuration, gemini-35-flashlite serves as an explicit rule target in openai-gemini-router and as the fallback defaultModel in gemini-claude-router.

Observability

The model router is instrumented so you can verify your gateway is serving traffic, inspect per-request metadata using Cloud Logging, and diagnose failures using Cloud Monitoring.

Cloud Logging

Every request routed through the gateway generates an entry in the standard API Gateway request log located in your Google Cloud project at:

projects/YOUR_PROJECT_ID/logs/apigateway.googleapis.com%2Frequests

Each log entry includes the following fields:

  • httpRequest.requestUrl, httpRequest.status, httpRequest.latency
  • api, apiConfig, apiMethod
  • backendRequest.hostname: The hostname of the Vertex AI backend to which the request was proxied.
  • responseDetails: Populated with a branded error category on model router failures (see Troubleshooting model router failures right below).

To find recent requests sent to a specific gateway, use the following Cloud Logging query filter:

(resource.type="apigateway.googleapis.com/Gateway" OR resource.type="api")
logName="projects/YOUR_PROJECT_ID/logs/apigateway.googleapis.com%2Frequests"

Cloud Monitoring

The standard API Gateway metric apigateway.googleapis.com/proxy/request_count (BETA) reports gateway traffic volume broken down by:

  • response_code_class: One of 2xx, 3xx, 4xx, or 5xx.
  • api_config: The API config name the gateway is using.

This metric lets you verify overall traffic volume and error rates. Model-router-specific metrics (such as per-router or per-target-model breakdowns) will be added in a future release.

To track aggregated request latency, you can create a log-based metric from the httpRequest.latency field in the request log.

Troubleshooting model router failures

When a request routed through the model router fails, the responseDetails field on the corresponding request log entry indicates whether the failure occurred within the model router layer. The model router surfaces four branded categories:

responseDetails value Meaning Typical fix
model_router_application_error The request couldn't be routed. This usually indicates a missing rule, a payload containing a model value that doesn't match any rule (without a configured defaultModel), or a malformed request payload. Customer side: Verify that your payload's model parameter matches one of the rule.model strings in your router configuration or that a defaultModel fallback is defined. Check that the request body is valid OpenAI-compatible JSON and explicitly includes a model attribute (during Public Preview, a missing model attribute in the request payload is incorrectly processed instead of being rejected).
model_router_timeout The model router exceeded the per-request timeout. The request might be unusually large or complex, or there might be a capacity bottleneck. Check request complexity and timeout settings across backends. If the issue persists across normal payloads, contact Google Cloud Support with the request timestamp and a log sample.
model_router_upstream_error The upstream target model returned an HTTP error to the gateway. Upstream service side: Check the status code and payload from the target Vertex AI service endpoint. If this is unexpected for valid requests, open a support case.
model_router_unavailable The model router was unreachable from the gateway due to a transport or connectivity failure. Platform side: Open a support case with Google Cloud Support.

What's next