*This page
applies to **Apigee hybrid** , but not to **Apigee**.*

*There is no equivalent
[Apigee Edge](https://docs.apigee.com/api-platform/get-started/get-started) documentation for this topic.*

## Symptom


An `ApigeeIssue` with `Reason` `AIS_MCP_MTLS_CA_CERT_INVALID`
is shown when viewing resources in the `apigee` namespace.


This issue is raised only for environment groups whose `mcpMtls` block has chain
validation enabled (the default). For a group with
`mcpMtls.disableChainValidation: true`, the MCP data plane does not use a CA
certificate and the `-cacert` secret is not scanned.

## Error messages


After running `kubectl -n apigee get apigeeissues`, an error like the following is
displayed:

```
NAME                                        SEVERITY   REASON                         DOCUMENTATION                                                                              AGE
mcp-ca-cert-invalid-my-org-my-virtualhost   ERROR      AIS_MCP_MTLS_CA_CERT_INVALID   https://cloud.google.com/apigee/docs/hybrid/troubleshooting/AIS_MCP_MTLS_CA_CERT_INVALID   5h18m
```

## Possible causes

The following table lists possible causes of this issue:

| Cause | Description |
|---|---|
| [MCP mTLS CA certificate secret or entry is missing](https://docs.cloud.google.com/apigee/docs/api-platform/troubleshoot/playbooks/mcp-ca-cert-invalid.md.txt#missing) | The `-cacert` secret does not exist, or it has no usable `cacert` entry. |
| [MCP mTLS CA certificate is not PEM-encoded](https://docs.cloud.google.com/apigee/docs/api-platform/troubleshoot/playbooks/mcp-ca-cert-invalid.md.txt#not-pem) | The `cacert` entry holds no PEM certificate block, or holds bytes outside the PEM blocks. |
| [MCP mTLS CA certificate is malformed or is not a CA certificate](https://docs.cloud.google.com/apigee/docs/api-platform/troubleshoot/playbooks/mcp-ca-cert-invalid.md.txt#malformed) | A block in the `cacert` entry is not a certificate, does not parse, or is not a CA certificate. |

## Cause: MCP mTLS CA certificate secret or entry is missing


The Kubernetes secret that should hold the CA certificate the MCP data plane validates the
ingress gateway against does not exist in the `apigee` namespace, or it exists
without a non-empty `cacert` entry. Its name is the client certificate secret's name
with `-cacert` appended. The chart creates this secret only when you use certificate
files; with `sslSecret` you must create it yourself, and forgetting it is the most
common cause of this issue.

### Diagnosis


Run the following command:

```
kubectl -n apigee describe apigeeissue ISSUE_NAME
```


Replace `ISSUE_NAME` with the name of the issue, for example
`mcp-ca-cert-invalid-my-org-my-virtualhost`.


The output should look similar to the following:

```
Name:         mcp-ca-cert-invalid-my-org-my-virtualhost
Namespace:    apigee
Labels:       ais-reason=AIS_MCP_MTLS_CA_CERT_INVALID
Annotations:  <none>
API Version:  apigee.cloud.google.com/v1alpha1
Kind:         ApigeeIssue
Metadata:
  Creation Timestamp:  2026-11-12T17:03:43Z
  Generation:          1
  Owner References:
    API Version:     apigee.cloud.google.com/v1alpha2
    Kind:            ApigeeOrganization
    Name:            my-org
    UID:             7e83a52c-ce00-4bed-98be-55835ada1817
  Resource Version:  3281563
  UID:               adc775c2-376d-4bf9-9860-500b2b2b8273
Spec:
  Details:        MCP mTLS CA certificate in secret "my-org-my-virtualhost-mcp-cacert" for ApigeeRouteConfig "my-org-my-virtualhost" is not usable: secret not found: secrets "my-org-my-virtualhost-mcp-cacert" not found
  Documentation:  https://cloud.google.com/apigee/docs/hybrid/troubleshooting/AIS_MCP_MTLS_CA_CERT_INVALID
  Reason:         AIS_MCP_MTLS_CA_CERT_INVALID
  Severity:       ERROR
Events:           <none>
```

```
Spec:
  Details:        MCP mTLS CA certificate in secret "my-org-my-virtualhost-mcp-cacert" for ApigeeRouteConfig "my-org-my-virtualhost" is not usable: secret not found: secrets "my-org-my-virtualhost-mcp-cacert" not found
```


The contents of `Spec.Details` name the secret, the
`ApigeeRouteConfig`, and what is wrong. For example:

- `secret not found`
- `secret has no usable "cacert" entry`

### Resolution

1.
   Use `kubectl describe` to display the name of the Kubernetes secret and
   `ApigeeRouteConfig` the issue belongs to:

   ```
   kubectl -n apigee describe apigeeissue ISSUE_NAME
   ```


   Replace `ISSUE_NAME` with the name of the issue, for example
   `mcp-ca-cert-invalid-my-org-my-virtualhost`.


   In this example, the `ApigeeRouteConfig` `my-org-my-virtualhost` is
   referencing the secret `my-org-my-virtualhost-mcp-cacert`.
2.
   Determine the `virtualhost` name from the `ApigeeRouteConfig`.


   The `ApigeeRouteConfig`'s name is in the format:
   `<Apigee organization>-<virtualhost name>`.


   In this example, `my-org` is the Apigee organization and the virtualhost
   name is `my-virtualhost`.
3.
   Find the corresponding virtualhost in your `overrides.yaml` and check which form
   its `mcpMtls` block uses (see
   [Configure mTLS between the MCP data plane and the ingress gateway](https://docs.cloud.google.com/apigee/docs/api-platform/apigee-mcp/enable-mcp.md.txt#mcp-mtls)).
   Select the tab that matches, and follow the steps in that tab.

### Certificate files


When you use file paths (`sslCertPath`, `sslKeyPath`,
`caCertPath`), the virtualhost chart creates and manages the secrets from the
file contents.

```
virtualhosts:
...
- name: my-virtualhost
  selector:
    app: apigee-ingressgateway
  tlsMode: MUTUAL
  caCertPath: ./certs/ca.pem
  sslCertPath: ./certs/vhost.pem
  sslKeyPath: ./certs/vhost.key
  mcpMtls:
    caCertPath: ./certs/mcp-ca.pem
    sslCertPath: ./certs/mcp-client.pem
    sslKeyPath: ./certs/mcp-client.key
...
```

> [!CAUTION]
> **Caution:** Do not directly edit the secrets that the chart manages. The next `helm upgrade` of the virtualhost overwrites them.

4.
   Check whether the secret exists and has a `cacert` entry, where
   <var translate="no">MCP_CA_SECRET_NAME</var> is the secret named in `Spec.Details`:

   ```
   kubectl -n apigee describe secret MCP_CA_SECRET_NAME
   ```


   If the secret is missing or lacks an entry, make sure that the file referenced
   by `mcpMtls.caCertPath` exists and is not empty. The next step
   restores the secret from it. If you intend to run without chain validation, set
   `mcpMtls.disableChainValidation: true` and remove
   `mcpMtls.caCertPath` (the chart rejects the two together). Be aware
   that this turns off verification of the ingress gateway's certificate by the MCP
   data plane.
5. Upgrade the virtualhost so that the chart rewrites the secret:

   ```
   helm upgrade ENV_GROUP_NAME apigee-virtualhost/ \
     --namespace apigee \
     --set envgroup=ENV_GROUP_NAME \
     -f overrides.yaml
   ```

### Kubernetes secrets


When you use a Kubernetes secret reference (`sslSecret`), the MCP data plane
reads the certificate values directly from the secrets that you created, and updates
apply automatically. No `helm upgrade` is needed.

```
virtualhosts:
...
- name: my-virtualhost
  selector:
    app: apigee-ingressgateway
  tlsMode: MUTUAL
  sslSecret: my-ingress-secret
  mcpMtls:
    sslSecret: my-mcp-client-secret
...
```

4.
   Confirm the secret exists under exactly that name with a `cacert`
   entry, where <var translate="no">MCP_CLIENT_SECRET_NAME</var> is the client certificate secret
   named by `mcpMtls.sslSecret`:

   ```
   kubectl -n apigee describe secret MCP_CLIENT_SECRET_NAME-cacert
   ```


   `helm template` does not check that the secrets exist; only
   `helm install` and `helm upgrade` do, so a clean
   `helm template` run is not evidence that this secret is present. If
   you intend to run without chain validation, omitting the secret is not enough.
   Set `mcpMtls.disableChainValidation: true` in
   `overrides.yaml` and upgrade the virtualhost. Be aware that this
   turns off verification of the ingress gateway's certificate by the MCP data
   plane.
5.
   Create or update the CA certificate secret, where
   <var translate="no">MCP_CLIENT_SECRET_NAME</var> is the value of
   `mcpMtls.sslSecret`, the CA certificate secret is that name with
   `-cacert` appended, and <var translate="no">MCP_TRUST_CA_FILE</var> is the CA
   certificate file:

   ```
   kubectl -n apigee create secret generic MCP_CLIENT_SECRET_NAME-cacert \
     --from-file=cacert=MCP_TRUST_CA_FILE \
     --dry-run=client -o yaml | kubectl apply -f -
   ```


The MCP data plane picks up the corrected secret on its next periodic refresh, typically within a
few minutes, without a pod restart, and the `ApigeeIssue` is withdrawn on the next
scan.

## Cause: MCP mTLS CA certificate is not PEM-encoded


The `cacert` entry is missing the PEM-encoding text blocks, or it has content after
the last `---END CERTIFICATE---` line that is not PEM. The MCP data plane does
not load it.

### Diagnosis


Run the following command:

```
kubectl -n apigee describe apigeeissue ISSUE_NAME
```


Replace `ISSUE_NAME` with the name of the issue, for example
`mcp-ca-cert-invalid-my-org-my-virtualhost`.


The output should look similar to the following:

```
Name:         mcp-ca-cert-invalid-my-org-my-virtualhost
Namespace:    apigee
Labels:       ais-reason=AIS_MCP_MTLS_CA_CERT_INVALID
Annotations:  <none>
API Version:  apigee.cloud.google.com/v1alpha1
Kind:         ApigeeIssue
Metadata:
  Creation Timestamp:  2026-11-12T17:03:43Z
  Generation:          1
  Owner References:
    API Version:     apigee.cloud.google.com/v1alpha2
    Kind:            ApigeeOrganization
    Name:            my-org
    UID:             7e83a52c-ce00-4bed-98be-55835ada1817
  Resource Version:  3281563
  UID:               adc775c2-376d-4bf9-9860-500b2b2b8273
Spec:
  Details:        MCP mTLS CA certificate in secret "my-org-my-virtualhost-mcp-cacert" for ApigeeRouteConfig "my-org-my-virtualhost" is not usable: holds 1289 trailing bytes that are not PEM
  Documentation:  https://cloud.google.com/apigee/docs/hybrid/troubleshooting/AIS_MCP_MTLS_CA_CERT_INVALID
  Reason:         AIS_MCP_MTLS_CA_CERT_INVALID
  Severity:       ERROR
Events:           <none>
```

```
Spec:
  Details:        MCP mTLS CA certificate in secret "my-org-my-virtualhost-mcp-cacert" for ApigeeRouteConfig "my-org-my-virtualhost" is not usable: holds 1289 trailing bytes that are not PEM
```


The contents of `Spec.Details` name the secret, the
`ApigeeRouteConfig`, and what is wrong. For example:

- `entry holds no PEM encoded certificate`
- `holds N trailing bytes that are not PEM`

### Resolution

1.
   Use `kubectl describe` to display the name of the Kubernetes secret and
   `ApigeeRouteConfig` the issue belongs to:

   ```
   kubectl -n apigee describe apigeeissue ISSUE_NAME
   ```


   Replace `ISSUE_NAME` with the name of the issue, for example
   `mcp-ca-cert-invalid-my-org-my-virtualhost`.


   In this example, the `ApigeeRouteConfig` `my-org-my-virtualhost` is
   referencing the secret `my-org-my-virtualhost-mcp-cacert`.
2.
   Determine the `virtualhost` name from the `ApigeeRouteConfig`.


   The `ApigeeRouteConfig`'s name is in the format:
   `<Apigee organization>-<virtualhost name>`.


   In this example, `my-org` is the Apigee organization and the virtualhost
   name is `my-virtualhost`.
3.
   Find the corresponding virtualhost in your `overrides.yaml` and check which form
   its `mcpMtls` block uses (see
   [Configure mTLS between the MCP data plane and the ingress gateway](https://docs.cloud.google.com/apigee/docs/api-platform/apigee-mcp/enable-mcp.md.txt#mcp-mtls)).
   Select the tab that matches, and follow the steps in that tab.

### Certificate files


When you use file paths (`sslCertPath`, `sslKeyPath`,
`caCertPath`), the virtualhost chart creates and manages the secrets from the
file contents.

```
virtualhosts:
...
- name: my-virtualhost
  selector:
    app: apigee-ingressgateway
  tlsMode: MUTUAL
  caCertPath: ./certs/ca.pem
  sslCertPath: ./certs/vhost.pem
  sslKeyPath: ./certs/vhost.key
  mcpMtls:
    caCertPath: ./certs/mcp-ca.pem
    sslCertPath: ./certs/mcp-client.pem
    sslKeyPath: ./certs/mcp-client.key
...
```

> [!CAUTION]
> **Caution:** Do not directly edit the secrets that the chart manages. The next `helm upgrade` of the virtualhost overwrites them.

4.
   Validate the contents of the file referenced by `mcpMtls.caCertPath`,
   and correct them if needed. Ensure the content contains only complete
   certificate blocks, each with its starting and ending lines. For example:

   ```
   ---BEGIN CERTIFICATE---
   MIIDYTCCAkmgAwIBAgIUSXeU0pQYRFzYlqZpKhNNJdBLFBIwDQYJKoZIhvcNAQEL
   ... <contents omitted> ...
   QUJDREVGR0hJSktMTU5PUFFSU1RVVldYWVowMTIzNDU2Nzg5QUJDREVGR0hJSktM
   MMM=
   ---END CERTIFICATE---
   ```

   > [!NOTE]
   > **Tip:** `openssl storeutl -noout -certs
   > MCP_TRUST_CA_FILE`, where <var translate="no">MCP_TRUST_CA_FILE</var> is the file referenced by `mcpMtls.caCertPath`, must list every block without error; remove anything it rejects and any text outside the `---BEGIN CERTIFICATE---` / `---END
   > CERTIFICATE---` lines.

5. Upgrade the virtualhost so that the chart rewrites the secret:

   ```
   helm upgrade ENV_GROUP_NAME apigee-virtualhost/ \
     --namespace apigee \
     --set envgroup=ENV_GROUP_NAME \
     -f overrides.yaml
   ```

### Kubernetes secrets


When you use a Kubernetes secret reference (`sslSecret`), the MCP data plane
reads the certificate values directly from the secrets that you created, and updates
apply automatically. No `helm upgrade` is needed.

```
virtualhosts:
...
- name: my-virtualhost
  selector:
    app: apigee-ingressgateway
  tlsMode: MUTUAL
  sslSecret: my-ingress-secret
  mcpMtls:
    sslSecret: my-mcp-client-secret
...
```

4.
   Validate the contents of the CA certificate file (<var translate="no">MCP_TRUST_CA_FILE</var>)
   that you store in the `cacert` entry of the secret, and correct them
   if needed. Ensure the content contains only complete certificate blocks, each
   with its starting and ending lines. For example:

   ```
   ---BEGIN CERTIFICATE---
   MIIDYTCCAkmgAwIBAgIUSXeU0pQYRFzYlqZpKhNNJdBLFBIwDQYJKoZIhvcNAQEL
   ... <contents omitted> ...
   QUJDREVGR0hJSktMTU5PUFFSU1RVVldYWVowMTIzNDU2Nzg5QUJDREVGR0hJSktM
   MMM=
   ---END CERTIFICATE---
   ```

   > [!NOTE]
   > **Tip:** `openssl storeutl -noout -certs
   > MCP_TRUST_CA_FILE` must list every block without error; remove anything it rejects and any text outside the `---BEGIN
   > CERTIFICATE---` / `---END CERTIFICATE---` lines.

5.
   Update the CA certificate secret in place, where
   <var translate="no">MCP_CLIENT_SECRET_NAME</var> is the value of
   `mcpMtls.sslSecret`, the CA certificate secret is that name with
   `-cacert` appended, and <var translate="no">MCP_TRUST_CA_FILE</var> is the CA
   certificate file:

   ```
   kubectl -n apigee create secret generic MCP_CLIENT_SECRET_NAME-cacert \
     --from-file=cacert=MCP_TRUST_CA_FILE \
     --dry-run=client -o yaml | kubectl apply -f -
   ```


The MCP data plane picks up the corrected secret on its next periodic refresh, typically within a
few minutes, without a pod restart, and the `ApigeeIssue` is withdrawn on the next
scan.

## Cause: MCP mTLS CA certificate is malformed or is not a CA certificate


Every PEM block in the `cacert` entry must be a `CERTIFICATE` block that
parses and has the CA basic constraint. A private key block, a leaf certificate (for example the
ingress gateway's own server certificate), or a corrupted block makes the entry unusable, and
the MCP data plane does not load it.

### Diagnosis


Run the following command:

```
kubectl -n apigee describe apigeeissue ISSUE_NAME
```


Replace `ISSUE_NAME` with the name of the issue, for example
`mcp-ca-cert-invalid-my-org-my-virtualhost`.


The output should look similar to the following:

```
Name:         mcp-ca-cert-invalid-my-org-my-virtualhost
Namespace:    apigee
Labels:       ais-reason=AIS_MCP_MTLS_CA_CERT_INVALID
Annotations:  <none>
API Version:  apigee.cloud.google.com/v1alpha1
Kind:         ApigeeIssue
Metadata:
  Creation Timestamp:  2026-11-12T17:03:43Z
  Generation:          1
  Owner References:
    API Version:     apigee.cloud.google.com/v1alpha2
    Kind:            ApigeeOrganization
    Name:            my-org
    UID:             7e83a52c-ce00-4bed-98be-55835ada1817
  Resource Version:  3281563
  UID:               adc775c2-376d-4bf9-9860-500b2b2b8273
Spec:
  Details:        MCP mTLS CA certificate in secret "my-org-my-virtualhost-mcp-cacert" for ApigeeRouteConfig "my-org-my-virtualhost" is not usable: block 1 is not a CA certificate
  Documentation:  https://cloud.google.com/apigee/docs/hybrid/troubleshooting/AIS_MCP_MTLS_CA_CERT_INVALID
  Reason:         AIS_MCP_MTLS_CA_CERT_INVALID
  Severity:       ERROR
Events:           <none>
```

```
Spec:
  Details:        MCP mTLS CA certificate in secret "my-org-my-virtualhost-mcp-cacert" for ApigeeRouteConfig "my-org-my-virtualhost" is not usable: block 1 is not a CA certificate
```


The contents of `Spec.Details` name the secret, the
`ApigeeRouteConfig`, and what is wrong. For example:

- `no PEM block parsed as a certificate: ...`: none of the certificate blocks can be parsed (for example, the only certificate is corrupted).
- `block N is a "..." block, want CERTIFICATE`: block N is not a certificate (for example, a private key).
- `block N does not parse`: block N is a corrupted certificate.
- `block N is not a CA certificate`: block N is a certificate, but not a CA (for example, a leaf certificate).


Blocks are numbered from 0.

### Resolution

1.
   Use `kubectl describe` to display the name of the Kubernetes secret and
   `ApigeeRouteConfig` the issue belongs to:

   ```
   kubectl -n apigee describe apigeeissue ISSUE_NAME
   ```


   Replace `ISSUE_NAME` with the name of the issue, for example
   `mcp-ca-cert-invalid-my-org-my-virtualhost`.


   In this example, the `ApigeeRouteConfig` `my-org-my-virtualhost` is
   referencing the secret `my-org-my-virtualhost-mcp-cacert`.
2.
   Determine the `virtualhost` name from the `ApigeeRouteConfig`.


   The `ApigeeRouteConfig`'s name is in the format:
   `<Apigee organization>-<virtualhost name>`.


   In this example, `my-org` is the Apigee organization and the virtualhost
   name is `my-virtualhost`.
3.
   Find the corresponding virtualhost in your `overrides.yaml` and check which form
   its `mcpMtls` block uses (see
   [Configure mTLS between the MCP data plane and the ingress gateway](https://docs.cloud.google.com/apigee/docs/api-platform/apigee-mcp/enable-mcp.md.txt#mcp-mtls)).
   Select the tab that matches, and follow the steps in that tab.

### Certificate files


When you use file paths (`sslCertPath`, `sslKeyPath`,
`caCertPath`), the virtualhost chart creates and manages the secrets from the
file contents.

```
virtualhosts:
...
- name: my-virtualhost
  selector:
    app: apigee-ingressgateway
  tlsMode: MUTUAL
  caCertPath: ./certs/ca.pem
  sslCertPath: ./certs/vhost.pem
  sslKeyPath: ./certs/vhost.key
  mcpMtls:
    caCertPath: ./certs/mcp-ca.pem
    sslCertPath: ./certs/mcp-client.pem
    sslKeyPath: ./certs/mcp-client.key
...
```

> [!CAUTION]
> **Caution:** Do not directly edit the secrets that the chart manages. The next `helm upgrade` of the virtualhost overwrites them.

4.
   Rebuild the file referenced by `mcpMtls.caCertPath` so that it holds
   only the CA certificate (or chain) that signed the ingress gateway's
   certificate. Then confirm that every block is a CA certificate, where
   <var translate="no">MCP_TRUST_CA_FILE</var> is that file:

   ```
   openssl storeutl -noout -text -certs MCP_TRUST_CA_FILE | grep -E 'Subject:|CA:'
   # every certificate must show CA:TRUE
   ```
5. Upgrade the virtualhost so that the chart rewrites the secret:

   ```
   helm upgrade ENV_GROUP_NAME apigee-virtualhost/ \
     --namespace apigee \
     --set envgroup=ENV_GROUP_NAME \
     -f overrides.yaml
   ```

### Kubernetes secrets


When you use a Kubernetes secret reference (`sslSecret`), the MCP data plane
reads the certificate values directly from the secrets that you created, and updates
apply automatically. No `helm upgrade` is needed.

```
virtualhosts:
...
- name: my-virtualhost
  selector:
    app: apigee-ingressgateway
  tlsMode: MUTUAL
  sslSecret: my-ingress-secret
  mcpMtls:
    sslSecret: my-mcp-client-secret
...
```

4.
   Rebuild the CA certificate file (<var translate="no">MCP_TRUST_CA_FILE</var>) so that it holds
   only the CA certificate (or chain) that signed the ingress gateway's
   certificate. Then confirm that every block is a CA certificate:

   ```
   openssl storeutl -noout -text -certs MCP_TRUST_CA_FILE | grep -E 'Subject:|CA:'
   # every certificate must show CA:TRUE
   ```
5.
   Update the CA certificate secret in place, where
   <var translate="no">MCP_CLIENT_SECRET_NAME</var> is the value of
   `mcpMtls.sslSecret`, the CA certificate secret is that name with
   `-cacert` appended, and <var translate="no">MCP_TRUST_CA_FILE</var> is the CA
   certificate file:

   ```
   kubectl -n apigee create secret generic MCP_CLIENT_SECRET_NAME-cacert \
     --from-file=cacert=MCP_TRUST_CA_FILE \
     --dry-run=client -o yaml | kubectl apply -f -
   ```


The MCP data plane picks up the corrected secret on its next periodic refresh, typically within a
few minutes, without a pod restart, and the `ApigeeIssue` is withdrawn on the next
scan.

## Must gather diagnostic information


If the problem persists even after following the preceding instructions, gather
the following diagnostic information and then contact [Google Cloud Customer Care](https://cloud.google.com/support-hub/):

1. The Google Cloud project ID.
2. The name of the Apigee hybrid organization.
3. The `Spec.Details` field of the `ApigeeIssue`.
4. (optional) The certificate in the file referenced by `mcpMtls.caCertPath`, or in the `cacert` entry of the secret, for the affected virtualhost.