*This page
applies to **Apigee** and **Apigee hybrid**.*


*View [Apigee Edge](https://docs.apigee.com/api-platform/get-started/what-apigee-edge) documentation.*
![](https://fonts.gstatic.com/s/i/googlematerialicons/help/v13/gm_grey-24dp/1x/gm_help_gm_grey_24dp.png)

This page describes how to troubleshoot and resolve issues with the deployment of
MCP discovery proxies. You can use this page to understand the asynchronous provisioning lifecycle of MCP deployments,
resolve error codes, and verify runtime connectivity.

> [!NOTE]
> **Note (Apigee hybrid):** All error status codes, response body semantics, CORS behavior, and API-style constraints on this page apply identically to Apigee hybrid. Apigee hybrid also has cluster-side failure modes that require in-cluster diagnosis with `kubectl`. See [Cluster-side diagnosis (Apigee hybrid only)](https://docs.cloud.google.com/apigee/docs/api-platform/apigee-mcp/troubleshoot-mcp-deployments.md.txt#hybrid-cluster-side-diagnosis) below.

## Understand MCP deployment

The deployment of an MCP Discovery Proxy is an asynchronous, multi-step process that ensures the configuration
is fully propagated across the Apigee and Google Cloud infrastructure before proxy deployment is complete.

While these downstream provisioning steps are taking place, the UI shows the proxy in a **Provisioning** status.
When the process is complete, the status changes to **Deployed**. This status confirms
that all downstream components are fully provisioned and the proxy is ready to handle traffic.

> [!NOTE]
> **Note (Apigee hybrid):** In Apigee hybrid, the "downstream components" include the in-cluster `apigee-mcp-server` pods and the `apigee-mcp-server-config` sidecar that pulls the MCP configuration from the Apigee control plane. If your MCP Discovery Proxy does not reach **Deployed** , or the endpoint does not respond, use [Cluster-side diagnosis](https://docs.cloud.google.com/apigee/docs/api-platform/apigee-mcp/troubleshoot-mcp-deployments.md.txt#hybrid-cluster-side-diagnosis) to check the in-cluster components.

## Troubleshooting

The following sections describe errors and known issues you might encounter while using
MCP in Apigee.

### Backend does not receive a tool argument in the expected location

**Issue:** A backend reads a tool argument from one request location
--- the query string, a header, or a cookie --- but the OpenAPI
specification declares the parameter `in` a different location, so
the argument arrives somewhere else on the backend request. The call fails or
behaves as though the argument were absent.

**Symptoms:** The `tools/call` request succeeds at the MCP
layer and the MCP client supplies the argument, but the backend responds as
though the value were missing --- for example a `400` for a
required value, or a default being applied in place of the value sent.

**Cause:** The parameter is declared `in: header` or
`in: cookie` in the OpenAPI specification, so Apigee sends
its value as a request header or in the `Cookie` header rather than
in the query string. A backend that reads the value only from the query string
does not see it. See
[Tool
argument placement on the backend request](https://docs.cloud.google.com/apigee/docs/api-platform/apigee-mcp/apigee-mcp-overview#tool-argument-placement).

**Resolution:** Decide where the value belongs and make the specification
and the backend agree:

- To keep the value in the query string, declare the parameter `in: query`.
- To send it as a header or a cookie, leave the declaration as it is and read the value from that location in your backend.

### Backend stops receiving a cookie that the MCP client's request carried

**Issue:** A cookie that arrives on the HTTP request the MCP client makes
--- a session cookie from a browser-based client, for example --- stops
reaching the backend on some tool calls.

**Symptoms:** The backend behaves as though the caller were not
authenticated or not in session, and the behavior is inconsistent: the same
operation works on some calls and not others.

**Cause:** The operation declares one or more `in: cookie`
parameters, and the affected calls supplied at least one of them. The
`Cookie` header is rewritten as a whole rather than appended to, so
on those calls it carries the declared cookie parameters only and every other
cookie is dropped. Calls that supplied none of the cookie parameters pass the
header through untouched, which is why only some calls fail. See
[Cookies
on the request to your backend](https://docs.cloud.google.com/apigee/docs/api-platform/apigee-mcp/apigee-mcp-overview#cookie-header-replacement).

**Resolution:** Declare every cookie the backend needs as an
`in: cookie` parameter on each operation that needs it, so its
value is always supplied by the tool call rather than inherited from the
client's request. There is no setting that preserves an undeclared cookie.

### Error response status codes for tool calls and metadata

Apigee MCP endpoints return HTTP status codes that follow the
[Model Context Protocol (MCP) specification](https://modelcontextprotocol.io/specification/2025-06-18/server/tools#error-handling)
and related OAuth standards. Understanding this behavior helps you correctly handle errors in your
MCP client or AI agent.

The following table summarizes the status codes returned by Apigee MCP endpoints:

| Scenario | HTTP status code | Response body |
|---|---|---|
| A tool execution fails on `tools/call` (for example, the backend API returns a `4xx` or `5xx` error, such as a validation error or a missing required field). | `200 OK` | A `CallToolResult` with `result.isError` set to `true` and the error details in `content`. This lets the MCP client and the LLM agent read the error and recover (for example, by re-prompting for missing input and retrying) instead of terminating the session. |
| The OAuth Protected Resource Metadata (PRM) is not configured or cannot be found at `/.well-known/oauth-protected-resource/mcp`. | `404 Not Found` | Indicates that no protected resource metadata is available at the requested location. |
| The access token is missing, invalid, or expired. | `401 Unauthorized` | Includes a `WWW-Authenticate` challenge to drive the OAuth discovery flow. |
| The access token has insufficient scope. | `403 Forbidden` | Includes a `WWW-Authenticate` challenge to drive the step-up authorization flow. |

#### Example: tool execution error response body

When a tool call fails, Apigee MCP returns `HTTP 200 OK` and
reports the failure in the JSON-RPC `result` object, with
`isError` set to `true`. The original error returned by your
backend is **preserved** : the backend response body is placed verbatim (as a string)
in `result.content[].text`. For example, if the backend returns
`HTTP 404` with the body `{"error":"User 42 not found"}`, the MCP
response is:

```json
{
  "jsonrpc": "2.0",
  "id": 1,
  "result": {
    "content": [
      {
        "type": "text",
        "text": "{\"error\":\"User 42 not found\"}"
      }
    ],
    "isError": true
  }
}
```

The `id` matches the `id` of your `tools/call` request.
If the backend returns no response body, `text` contains a generated summary
instead, such as `Operation failed (HTTP 500).`

Your MCP client should inspect `result.isError` (not the HTTP status code)
to detect a failed tool call. **If your client code specifically expects a
`4xx` or `5xx` HTTP status to signal a tool execution error, it
does not detect the error** , because the response is
`HTTP 200 OK`. Update such clients to read `result.isError` and
the `content` array.

Protocol-level errors (for example, a malformed JSON-RPC request) are distinct from
tool execution errors: they continue to be returned in the JSON-RPC `error`
object (for example, `{"jsonrpc": "2.0", "error": {"code": -32700, ...}}`)
rather than in `result.isError`.

**Recommendation:** Standards-compliant MCP clients handle these status codes
through their default logic, so no changes are required. If you previously added a
client-side workaround to handle non-standard status codes---for example, parsing
a non-`2xx` response body for `tools/call` errors, or treating a
`500` response from the PRM endpoint as "metadata not configured"---we
recommend removing that workaround. Otherwise, your client might continue to use
obsolete handling logic instead of processing the corrected status codes normally.

### CORS policy for browser-based MCP calls

A Cross-Origin Resource Sharing (CORS) policy is necessary when Apigee MCP calls are made directly from a browser
(for example, when the MCP inspector uses the "Direct" setting instead of "Via Proxy"). The MCP Discovery Proxy template includes the Apigee [CORS
policy](https://docs.cloud.google.com/apigee/docs/api-platform/reference/policies/cors-policy) in the Proxy Endpoint **Request PreFlow** of the proxy bundle by default to enable browser-based calls.

If you encounter CORS-related issues when making calls directly from a browser, you may need to adjust the CORS policy
settings in the Proxy Endpoint **Request PreFlow** of your MCP Discovery proxy bundle.

### JSON parse error when sending request to MCP endpoint

You may receive an error message similar to the following (or with a different error code and message) when sending a request to your MCP endpoint:

```
{
  "error": {
    "code": -32700,
    "message": "JSON parse error"
  },
  "id": null,
  "jsonrpc": "2.0"
}
```

In that case, we recommend confirming the following information:

- You have entered the correct MCP endpoint URL.
- Your request is formatted correctly.
- You are accessing a supported method.
- You enabled and configured your MCP Discovery Proxy in a [supported region](https://docs.cloud.google.com/apigee/docs/api-platform/apigee-mcp/apigee-mcp-overview.md.txt#limitations). For a list of regions that may have capacity limits, see [deployment failures](https://docs.cloud.google.com/apigee/docs/api-platform/apigee-mcp/troubleshoot-mcp-deployments.md.txt#deployment-failures).

### Deployment failures

If deployment of your MCP Discovery Proxy fails with a **Failed** status in the UI, check
the error message for more details. Common failures include:

- **Networking configuration failed** or **Unsupported Schema in OAS** : These errors typically indicate a problem with your OpenAPI specification. Ensure your specification is valid and uses a [supported version](https://docs.cloud.google.com/apigee/docs/api-platform/apigee-mcp/apigee-mcp-overview.md.txt#limitations).
- **Region capacity limits (Apigee only)** : If you see an error related to load balancer provisioning failure, or the provisioning status never changes to **Deployed** , it might be due to temporary infrastructure capacity limits in one of the following regions:
  - `asia-east2`
  - `asia-northeast3`
  - `asia-southeast2`
  - `australia-southeast1`
  - `europe-central2`
  - `europe-west12`
  - `europe-west9`
  - `me-central2`
  - `us-central2`

  To resolve the error, try deploying the proxy to an environment in a different region. This
  failure mode does not apply to Apigee hybrid, where the runtime infrastructure is your
  own cluster.
- **MCP not enabled in cluster (Apigee hybrid only)** : If your MCP proxy deployment stays in **Provisioning** status indefinitely and never reaches **Deployed** , verify that MCP is enabled in your cluster. See [Cluster-side diagnosis](https://docs.cloud.google.com/apigee/docs/api-platform/apigee-mcp/troubleshoot-mcp-deployments.md.txt#hybrid-cluster-side-diagnosis) below.

### Cluster-side diagnosis (Apigee hybrid only)

Because the Apigee hybrid MCP data plane runs inside your own Kubernetes cluster,
in-cluster failure modes are diagnosed with `kubectl` against the cluster.
Work through the following diagnostic checklist in order.

> [!NOTE]
> **Placeholder conventions in this section.** Every `kubectl` and `helm` command below uses the following placeholders. Replace them with values from your installation:
>
> - `APIGEE_NAMESPACE` --- the Kubernetes namespace where Apigee hybrid is installed (default `apigee`). Look up the `namespace` field in your `overrides.yaml`.
> - `APIGEE_ORG_RELEASE_NAME` --- the Helm release name for the `apigee-org` chart in your cluster (customer-chosen at install time). Confirm with `helm list -n APIGEE_NAMESPACE | grep apigee-org`. The **NAME** column of the matching row is the value to use, for example:
>
>   ```
>   NAME                        NAMESPACE  REVISION  ...  CHART
>   my-hybrid-org               apigee     3         ...  apigee-org-1.17.0
>   ```
>   Here, `APIGEE_ORG_RELEASE_NAME` is `my-hybrid-org`.
> - `ORG_CR_SUFFIX` --- the suffix that the Apigee operator appends to MCP resource names. The suffix is a truncated form of your Apigee organization name followed by a short hash (for example, `my-org-abc12de`). Look up the exact suffix from the `ApigeeOrganization` custom resource: `kubectl get apigeeorganization -n APIGEE_NAMESPACE -o
>   jsonpath='{.items[0].metadata.name}'`. The trailing token after `apigee-organization-` is the suffix.

#### Step 1: Verify MCP is enabled in the cluster

MCP is deployed as an `ApigeeDeployment` custom resource (managed by the
Apigee operator), not as a vanilla Kubernetes `Deployment`. Check for
the MCP `ApigeeDeployment`:

```bash
kubectl -n APIGEE_NAMESPACE get apigeedeployment -l app=apigee-mcp-server
```

Expected output (resource names include an organization-derived suffix, shown here as
`ORG_CR_SUFFIX`):

```
NAME                                     STATE     NESTEDSTATE   AGE
apigee-mcp-server-default-ORG_CR_SUFFIX       running                 1h
```

If the command returns `No resources found in APIGEE_NAMESPACE namespace.`,
MCP is not enabled. Return to
[Enable MCP in Apigee hybrid](https://docs.cloud.google.com/apigee/docs/api-platform/apigee-mcp/enable-mcp.md.txt) and
verify that `enableMcpServer: true` is set as a top-level key in your
`overrides.yaml`, and that you upgraded the `apigee-operator`
Helm release before the `apigee-org` release. See
[Step 3](https://docs.cloud.google.com/apigee/docs/api-platform/apigee-mcp/enable-mcp.md.txt#step-3-helm-upgrade-operator)
and [Step 4](https://docs.cloud.google.com/apigee/docs/api-platform/apigee-mcp/enable-mcp.md.txt#step-4-helm-upgrade-org)
of the enable procedure for the correct `helm` commands.

> [!CAUTION]
> **Caution:** Do not use `kubectl get deploy -l app=apigee-mcp-server`. That command returns no results because MCP is not a vanilla Kubernetes `Deployment`; it is an `ApigeeDeployment` custom resource. Use the `kubectl get apigeedeployment` command shown above.

#### Step 2: Verify MCP pods are running

```bash
kubectl -n APIGEE_NAMESPACE get pods -l app=apigee-mcp-server
```

Expected output (pod suffixes will differ):

```
NAME                                                              READY   STATUS    RESTARTS   AGE
apigee-mcp-server-default-ORG_CR_SUFFIX-1170-abcd1234   2/2     Running   0          1h
apigee-mcp-server-default-ORG_CR_SUFFIX-1170-efgh5678   2/2     Running   0          1h
```

Pod names follow the template
`apigee-mcp-server-default-ORG_CR_SUFFIX-REV-POD_HASH`,
where <var translate="no">REV</var> is the four-digit hybrid release revision (for example, `1170`
for hybrid 1.17.0) and <var translate="no">POD_HASH</var> is an eight-character alphanumeric suffix that
differs for each pod.

Every pod must show `2/2` in the `READY` column and
`Running` in the `STATUS` column. If pods are not `Running`,
`kubectl -n APIGEE_NAMESPACE describe pod POD_NAME` shows
the reason. Common causes are image-pull failures (verify egress from your cluster to the
image registry configured in your `overrides.yaml`) and insufficient CPU or
memory headroom (raise `mcpServer.resources.requests` in
`overrides.yaml`).

#### Step 3: Check the sidecar configuration pull status

The `apigee-mcp-server-config` sidecar container in each MCP pod
periodically pulls MCP configuration from the Apigee control plane. Each pull retrieves
the current configuration reference from `apigee.googleapis.com`, then
fetches the referenced configuration bundle from Cloud Storage. The MCP data plane
starts serving MCP tool traffic on `https://mcp.apigee.internal/mcp` once a
pull completes successfully with a valid configuration.

Before that first successful pull with a valid configuration (either because no MCP
Discovery Proxy has been deployed for your organization yet, or because the sidecar
cannot reach the control plane), the MCP pods stay `Ready` but requests to
`https://mcp.apigee.internal/mcp` fail with **connection refused**.

Inspect the sidecar logs to determine which case you are in:

```bash
kubectl -n APIGEE_NAMESPACE logs -l app=apigee-mcp-server -c apigee-mcp-server-config --tail=100
```

- **Repeating `info` line indicating no MCP configuration from the
  control plane:**

  ```
  {"level":"info","ts":1787582542.68,"caller":"mcpsidecar/sidecar.go:397","msg":"mcpsidecar: no MCP config from CP yet; skipping tick, pod stays Ready via seed","org":"YOUR_ORG","mcp_server_id":"default"}
  ```
  The Apigee control plane returned a successful response but no MCP Discovery Proxy has been deployed for your organization yet. This is the expected steady state during onboarding. To transition the MCP data plane into serving state, deploy an MCP Discovery Proxy per the [MCP quickstart](https://docs.cloud.google.com/apigee/docs/api-platform/apigee-mcp/apigee-mcp-quickstart.md.txt).
- **`error` line with a `consecutive_failures` counter:**

  ```
  {"level":"error","ts":1787613247.74,"caller":"mcpsidecar/sidecar.go:...","msg":"mcpsidecar: ...","org":"YOUR_ORG","mcp_server_id":"default","consecutive_failures":185,"error":"<wrapped error text>"}
  ```
  Sidecar logs are structured JSON. When you see `"level":"error"` with a rising `consecutive_failures` counter, the sidecar could not complete a configuration pull. The sidecar retries automatically with backoff and resets the counter to 0 on the next successful pull.

  Common root causes to check when this counter keeps rising:
  - The `apigee-watcher` Google Cloud service account is not in the `watcher_identities` list on your Apigee organization's `controlPlaneAccess` resource. See [Step
    2: Grant the watcher identity access to the MCP configuration on the control
    plane](https://docs.cloud.google.com/apigee/docs/api-platform/apigee-mcp/enable-mcp#step-2-grant-watcher-cp-access) in the enable procedure to add the grant and verify it landed.
  - Egress from your cluster to `apigee.googleapis.com` or `storage.googleapis.com` is blocked. Both endpoints must be reachable from the MCP pods (see [Outbound
    network requirements](https://docs.cloud.google.com/apigee/docs/api-platform/apigee-mcp/enable-mcp#outbound-network) in the enable procedure).
  - The `apigee-watcher` service account cannot obtain Google Cloud access tokens from this cluster. If existing Apigee hybrid components in the same namespace are also reporting authentication failures, address the shared authentication problem first.
  - A transient Apigee control-plane or Cloud Storage error. If the counter keeps rising for an extended period after ruling out the causes above, contact [Apigee
    Support](https://cloud.google.com/apigee/docs/api-platform/contact-support).

#### Step 4: Verify Message Processor hostAliases were injected

When you enable MCP, the Apigee operator injects one `hostAliases` entry into the
Message Processor (MP) pod spec. The entry contains one IP (the MCP Service ClusterIP) and two
hostnames: `mcp.apigee.internal` and the lowercased
`ORG_NAME.mcp.apigee.internal`. Verify the injection succeeded:

```bash
kubectl -n APIGEE_NAMESPACE get pods -l app=apigee-runtime \
  -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.hostAliases}{"\n"}{end}'
```

Every listed MP pod must have a `hostAliases` entry containing both
`mcp.apigee.internal` and the lowercased
`ORG_NAME.mcp.apigee.internal` pointing at the MCP Service ClusterIP. Any
MP pod that shows an empty or missing entry was created before MCP was enabled. Delete
the affected MP pod so the `ApigeeDeployment` controller recreates it from the
current pod spec (which includes the injected `hostAliases` entry):

```bash
kubectl -n APIGEE_NAMESPACE delete pod -l app=apigee-runtime
```

> [!CAUTION]
> **Caution:** Message Processor is deployed as an `ApigeeDeployment` custom resource, not a vanilla Kubernetes `Deployment`. Do not use `kubectl rollout restart deploy -l app=apigee-runtime`; the command silently does nothing on an `ApigeeDeployment`. Use `kubectl delete pod` as shown above so the `ApigeeDeployment` controller recreates the pod from the current pod spec.

#### Step 5: End-to-end verification

For an end-to-end test that sends an actual MCP `initialize` request through
the MCP data plane, deploy an MCP Discovery Proxy and follow the
[MCP quickstart](https://docs.cloud.google.com/apigee/docs/api-platform/apigee-mcp/apigee-mcp-quickstart.md.txt).
Before an MCP Discovery Proxy is deployed for your organization, requests to
`https://mcp.apigee.internal/mcp` fail with **connection refused**
regardless of whether Message Processor can reach the MCP pods (see Step 3).

#### Step 6: Check mTLS to the ingress gateway (if configured)

This step applies only if you configured
[`virtualhosts[].mcpMtls`](https://docs.cloud.google.com/apigee/docs/api-platform/apigee-mcp/enable-mcp.md.txt#mcp-mtls)
for an environment group (Apigee hybrid `1.17.4` and later). The typical
symptom of an mTLS problem is that MCP tool calls to **one** environment group fail
(usually `503 Service Unavailable`) while calls to other environment groups
succeed, because the MCP data plane keeps its certificate configuration per environment
group.

1. **Confirm the operator accepted the field.** The environment group's routing resource must carry the `mcpMtls` block:

   ```bash
   kubectl -n APIGEE_NAMESPACE get apigeeroutes -l envGroupId=ENV_GROUP_NAME \
     -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.mcpMtls}{"\n"}{end}'
   ```
   An empty second column, several minutes after the `apigee-virtualhost` upgrade, usually means the Apigee custom resource definitions in the cluster are older than `1.17.4` and do not declare `mcpMtls`, so the API server dropped the field. (Less often, the `apigee-org` chart that supplies the watcher image has not been upgraded to `1.17.4`.) This happens when an upgrade ran `helm upgrade` for the operator chart but skipped the [Install the updated Apigee
   CRDs](https://docs.cloud.google.com/apigee/docs/hybrid/v1.17/upgrade#prepare) step. The MCP data plane is then still using one-way TLS, and the ingress gateway in `MUTUAL` mode rejects it; the same stale CRDs also prevent the `AIS_MCP_MTLS_*` issues in the next step from being reported. Check and fix with:

   ```bash
   kubectl get crd apigeeroutes.apigee.cloud.google.com -o yaml | grep -c mcpMtls   # 0 means the CRDs are stale
   kubectl apply -k apigee-operator/etc/crds/default/ --server-side --force-conflicts --validate=false
   ```
   Run the `kubectl apply` from the Helm charts directory, then re-run the `apigee-virtualhost` upgrade for the environment group.
2. **Check for certificate issues reported by the cluster.** The watcher scans the configured Secrets and creates an `ApigeeIssue` for each problem:

   ```bash
   kubectl -n APIGEE_NAMESPACE get apigeeissues -l 'ais-reason in (AIS_MCP_MTLS_CLIENT_CERT_INVALID,AIS_MCP_MTLS_CLIENT_CERT_EXPIRY,AIS_MCP_MTLS_CA_CERT_INVALID,AIS_MCP_MTLS_CA_CERT_EXPIRY)'
   kubectl -n APIGEE_NAMESPACE describe apigeeissue ISSUE_NAME
   ```
   The `Spec.Details` field names the Secret and the `ApigeeRouteConfig` (named `ORG_NAME-ENV_GROUP_NAME`) the problem belongs to, and the specific defect.

   | Reason | Severity | Meaning | Fix |
   |---|---|---|---|
   | `AIS_MCP_MTLS_CLIENT_CERT_INVALID` | `ERROR` | The client certificate Secret does not exist (`secret not found`), is missing a `cert` or `key` entry, an entry is empty or not PEM, the certificate and key do not match, or the certificate's extended key usage omits `clientAuth`. The MCP data plane does not load it. | Create or fix the Secret (Option 2) or the files followed by `helm upgrade` (Option 1); if the extended key usage is the problem, reissue the certificate with `clientAuth`. |
   | `AIS_MCP_MTLS_CLIENT_CERT_EXPIRY` | `WARNING` within 14 days of expiry; `ERROR` after | The client certificate is about to expire or has expired. An expired certificate is still presented; the ingress gateway rejects the handshake. | Rotate the certificate as described in [Rotate certificates](https://docs.cloud.google.com/apigee/docs/api-platform/apigee-mcp/enable-mcp.md.txt#mcp-mtls-rotate). |
   | `AIS_MCP_MTLS_CA_CERT_INVALID` | `ERROR` | The `-cacert` Secret does not exist (`secret not found`; with `sslSecret` you create it yourself), is missing its `cacert` entry, the entry is not PEM, a block is not a certificate, or a certificate in it is not a CA certificate. | Create the Secret if needed and put the CA certificate (or chain) that signed the ingress gateway's certificate in the `cacert` entry. |
   | `AIS_MCP_MTLS_CA_CERT_EXPIRY` | `WARNING` within 14 days of expiry; `ERROR` after | A CA certificate in the `-cacert` Secret is about to expire or has expired. After expiry the MCP data plane no longer trusts the ingress gateway's certificate chain. | Replace the CA certificate in the `-cacert` Secret before it expires, together with the ingress gateway's own certificate if that is also being reissued. |

   Issues are withdrawn automatically on the next scan after the Secret is fixed.
3. **Read the sidecar's mTLS messages.** The `apigee-mcp-server-config` container logs an `error` line whenever it cannot apply an environment group's certificates. Each line typically includes fields such as `env_group`, `secret` (and a trust-Secret field when chain validation is on), a `reason` code, and the wrapped error text. Filter on the `reason` code value, which is the stable identifier; do not rely on the other field names:

   ```bash
   kubectl -n APIGEE_NAMESPACE logs -l app=apigee-mcp-server -c apigee-mcp-server-config \
     | grep MCP_MTLS_
   ```
   Search the whole log rather than a recent window: a faulty Secret is reported when the sidecar first sees it (and again when it changes), not on every refresh.

   The following reason codes are stable for the current release. If you see a code
   that is not listed, contact [Google Cloud Customer Care](https://cloud.google.com/support-hub/).

   | `reason` | Meaning | Action |
   |---|---|---|
   | `MCP_MTLS_SECRET_NOT_FOUND` | A referenced Secret does not exist in the Apigee namespace. The `error` text names it and whether it is the client or the trust Secret. | Create the Secret with the exact name (remember the `-cacert` suffix for the trust CA). The sidecar notices the new Secret on its own. |
   | `MCP_MTLS_MATERIAL_UNUSABLE` | The Secret exists but its content is rejected: a missing or empty data key, content that is not PEM, a certificate and key that do not pair, a client certificate without `clientAuth`, or a trust Secret with no CA certificate. The same defect is reported as an `ApigeeIssue` (previous step). | Fix the Secret content. Editing the Secret triggers a new attempt. |
   | `MCP_MTLS_SECRET_READ_FAILED`, `MCP_MTLS_SECRET_WATCH_FAILED`, `MCP_MTLS_SECRET_ACCESS_FAILED` | The sidecar could not read or watch Secrets through the Kubernetes API (API server unavailable, or the MCP Role lost its `secrets` permission). An `MCP_MTLS_SECRET_READ_FAILED` line includes retry context, and the sidecar retries with backoff; the other two report the state of the sidecar's Secret watch, and `MCP_MTLS_SECRET_READER_RECOVERED` is logged when access returns. | Check that the MCP Role still grants `get`, `list`, `watch` on `secrets` (see [Trust boundary](https://docs.cloud.google.com/apigee/docs/api-platform/apigee-mcp/enable-mcp.md.txt#trust-boundary)), and that other components in the namespace can reach the API server. The Role name ends in a generated suffix, so find it with `kubectl -n APIGEE_NAMESPACE get role | grep apigee-mcp-server`. |
   | `MCP_MTLS_PUBLISH_FAILED` | The certificates were valid but could not be written to the MCP pod's in-memory volume. The sidecar retries with backoff and logs `MCP_MTLS_PUBLISH_RECOVERED` when the write succeeds. | This is not expected in normal operation. If it repeats, collect the sidecar logs and contact [Google Cloud Customer Care](https://cloud.google.com/support-hub/). |
   | `MCP_MTLS_UNKEYABLE_ENV_GROUP` | The environment group has `mcpMtls` configured, but its `envGroupId` label value isn't a valid path element, so the MCP data plane serves the group over one-way TLS and an ingress gateway in `MUTUAL` mode rejects its tool calls. Kubernetes label rules normally prevent such values, so a group in this state came from a routing resource applied directly to the cluster. No `ApigeeIssue` reports this. | Confirm the routing resource carries a valid label: `kubectl -n APIGEE_NAMESPACE get apigeeroutes --show-labels`. Rerun the `apigee-virtualhost` upgrade for the environment group so the resource is recreated from the chart. |
   | `MCP_MTLS_SAN_WITHOUT_CHAIN_VALIDATION` | The environment group set `disableChainValidation: true` and `disableSanValidation: false` together. SAN matching happens inside the validation that chain validation sets up, so with chain validation off there is nothing to match against and the setting is ignored. The chart rejects this pairing, so a group in this state came from a custom resource applied directly to the cluster. | Remove `disableSanValidation` from the environment group's `mcpMtls` block, or set `disableChainValidation` back to `false` and supply a trust CA (see [`disableSanValidation`](https://docs.cloud.google.com/apigee/docs/hybrid/v1.17/config-prop-ref.md.txt#virtualhosts-mcpmtls-disablesanvalidation)). |
   | `MCP_MTLS_CONFLICTING_CONFIG` | Two routing resources in the same environment group carry different `mcpMtls` blocks, so which one the MCP data plane uses depends on the order the routes are processed. The group can end up presenting a different client certificate from the one you configured, and nothing downstream records which block won. | Re-run the `apigee-virtualhost` upgrade for the environment group, then confirm the blocks match with the `get apigeeroutes` command in step 1. Every route in a group derives from that group's single `virtualhosts[]` entry, so a disagreement means the resources were applied directly to the cluster, or an earlier upgrade propagated only part way. |

   Except for `MCP_MTLS_UNKEYABLE_ENV_GROUP`, which drops the environment group to one-way TLS, the environment group continues to use the last certificates the sidecar loaded successfully. If it never loaded any (first enablement), MCP tool calls to that environment group fail until the problem is fixed; other environment groups are not affected. No pod restart is needed after the fix.

   The remaining `MCP_MTLS_` codes need no action on their own.
   `MCP_MTLS_CHAIN_VALIDATION_DISABLED` and
   `MCP_MTLS_SAN_VALIDATION_DISABLED` record that an environment group turned
   a validation off, so the choice stays findable across a fleet.
   `MCP_MTLS_MATERIAL_REMOVED` is the routine cleanup after a group opts out,
   and `MCP_MTLS_COLLECTION_FAILED` means that cleanup did not complete and
   the group's certificate files are still on the pod; the next refresh retries the
   cleanup.
4. **If the cluster reports no issue and the sidecar logs are clean** but tool calls to the environment group still fail, the handshake is being rejected by one side:
   - The ingress gateway rejects the client certificate when its issuer is not in the CA the ingress gateway trusts for that environment group (`virtualhosts[].caCertPath` or `INGRESS_SECRET_NAME-cacert`). Verify with `openssl verify -CAfile INGRESS_CA_FILE MCP_CLIENT_CERT_FILE`.
   - The MCP data plane rejects the ingress gateway's certificate when it does not chain to the CA in `mcpMtls.caCertPath` or `MCP_CLIENT_SECRET_NAME-cacert`, or when its Subject Alternative Names do not include the environment group's host alias. Verify with `openssl verify -CAfile MCP_TRUST_CA_FILE INGRESS_CERT_FILE` and inspect the SANs with `openssl x509 -in INGRESS_CERT_FILE -noout -ext subjectAltName`. Prefer reissuing the ingress gateway's certificate so that it covers the host alias. Only if that is not possible, set `mcpMtls.disableSanValidation: true` for that environment group, which stops the MCP data plane from checking the name on the ingress gateway's certificate.

#### Support cases

If you open a support case that involves the Apigee hybrid MCP data plane, attach the following
output to accelerate diagnosis:

- `kubectl -n APIGEE_NAMESPACE logs -l app=apigee-mcp-server --all-containers --tail=1000`
- `kubectl -n APIGEE_NAMESPACE describe pod -l app=apigee-mcp-server`
- `kubectl -n APIGEE_NAMESPACE get pods -l app=apigee-mcp-server -o wide` and `kubectl -n APIGEE_NAMESPACE get svc -l app=apigee-mcp-server`.
- Recent `overrides.yaml` and `helm history APIGEE_ORG_RELEASE_NAME -n APIGEE_NAMESPACE` output (where `APIGEE_ORG_RELEASE_NAME` is the Helm release name for the `apigee-org` chart; find it with `helm list -n APIGEE_NAMESPACE | grep apigee-org`). Before sharing `overrides.yaml`, redact secret material, including:
  - Any inline service account keys under `gcp.workloadIdentity`, `envs[].serviceAccountPaths`, or `mcpServer.serviceAccountPath`.
  - Any TLS certificates or private keys under `virtualhosts`.
  - The `cassandra.auth` password fields.
  - Any inline credentials that your overrides pass to the operator.
- The full `curl` command and response reproducing the issue.

### Cannot change API style to MCP in API Hub

If an API resource with existing API operations is already present in Apigee API hub,
you cannot change the **API style** attribute for that resource to **MCP** .
To register an API with Apigee API hub as an MCP API, you must select
the **MCP** style when first registering the API, or ensure no operations are present on the
API resource before changing the style to MCP.

If you encounter any other issues, see [Using debug](https://docs.cloud.google.com/apigee/docs/api-platform/debug/trace.md.txt)
for detailed information on using the Debug tool in the Google Cloud console to analyze requests and responses to and from your
MCP Discovery Proxy.

<br />