MCP mTLS CA certificate invalid

This page applies to Apigee hybrid, but not to Apigee.

There is no equivalent Apigee Edge documentation for this topic.

Symptom

An ApigeeIssue with Reason AIS_MCP_MTLS_CA_CERT_INVALID is shown when viewing resources in the apigee namespace.

This issue is raised only for environment groups whose mcpMtls block has chain validation enabled (the default). For a group with mcpMtls.disableChainValidation: true, the MCP data plane does not use a CA certificate and the -cacert secret is not scanned.

Error messages

After running kubectl -n apigee get apigeeissues, an error like the following is displayed:

NAME                                        SEVERITY   REASON                         DOCUMENTATION                                                                              AGE
mcp-ca-cert-invalid-my-org-my-virtualhost   ERROR      AIS_MCP_MTLS_CA_CERT_INVALID   https://cloud.google.com/apigee/docs/hybrid/troubleshooting/AIS_MCP_MTLS_CA_CERT_INVALID   5h18m

Possible causes

The following table lists possible causes of this issue:

Cause Description
MCP mTLS CA certificate secret or entry is missing The -cacert secret does not exist, or it has no usable cacert entry.
MCP mTLS CA certificate is not PEM-encoded The cacert entry holds no PEM certificate block, or holds bytes outside the PEM blocks.
MCP mTLS CA certificate is malformed or is not a CA certificate A block in the cacert entry is not a certificate, does not parse, or is not a CA certificate.

Cause: MCP mTLS CA certificate secret or entry is missing

The Kubernetes secret that should hold the CA certificate the MCP data plane validates the ingress gateway against does not exist in the apigee namespace, or it exists without a non-empty cacert entry. Its name is the client certificate secret's name with -cacert appended. The chart creates this secret only when you use certificate files; with sslSecret you must create it yourself, and forgetting it is the most common cause of this issue.

Diagnosis

Run the following command:

kubectl -n apigee describe apigeeissue ISSUE_NAME

Replace ISSUE_NAME with the name of the issue, for example mcp-ca-cert-invalid-my-org-my-virtualhost.

The output should look similar to the following:

Name:         mcp-ca-cert-invalid-my-org-my-virtualhost
Namespace:    apigee
Labels:       ais-reason=AIS_MCP_MTLS_CA_CERT_INVALID
Annotations:  <none>
API Version:  apigee.cloud.google.com/v1alpha1
Kind:         ApigeeIssue
Metadata:
  Creation Timestamp:  2026-11-12T17:03:43Z
  Generation:          1
  Owner References:
    API Version:     apigee.cloud.google.com/v1alpha2
    Kind:            ApigeeOrganization
    Name:            my-org
    UID:             7e83a52c-ce00-4bed-98be-55835ada1817
  Resource Version:  3281563
  UID:               adc775c2-376d-4bf9-9860-500b2b2b8273
Spec:
  Details:        MCP mTLS CA certificate in secret "my-org-my-virtualhost-mcp-cacert" for ApigeeRouteConfig "my-org-my-virtualhost" is not usable: secret not found: secrets "my-org-my-virtualhost-mcp-cacert" not found
  Documentation:  https://cloud.google.com/apigee/docs/hybrid/troubleshooting/AIS_MCP_MTLS_CA_CERT_INVALID
  Reason:         AIS_MCP_MTLS_CA_CERT_INVALID
  Severity:       ERROR
Events:           <none>
Spec:
  Details:        MCP mTLS CA certificate in secret "my-org-my-virtualhost-mcp-cacert" for ApigeeRouteConfig "my-org-my-virtualhost" is not usable: secret not found: secrets "my-org-my-virtualhost-mcp-cacert" not found

The contents of Spec.Details name the secret, the ApigeeRouteConfig, and what is wrong. For example:

  • secret not found
  • secret has no usable "cacert" entry

Resolution

  1. Use kubectl describe to display the name of the Kubernetes secret and ApigeeRouteConfig the issue belongs to:

    kubectl -n apigee describe apigeeissue ISSUE_NAME

    Replace ISSUE_NAME with the name of the issue, for example mcp-ca-cert-invalid-my-org-my-virtualhost.

    In this example, the ApigeeRouteConfig my-org-my-virtualhost is referencing the secret my-org-my-virtualhost-mcp-cacert.

  2. Determine the virtualhost name from the ApigeeRouteConfig.

    The ApigeeRouteConfig's name is in the format: <Apigee organization>-<virtualhost name>.

    In this example, my-org is the Apigee organization and the virtualhost name is my-virtualhost.

  3. Find the corresponding virtualhost in your overrides.yaml and check which form its mcpMtls block uses (see Configure mTLS between the MCP data plane and the ingress gateway). Select the tab that matches, and follow the steps in that tab.

Certificate files

When you use file paths (sslCertPath, sslKeyPath, caCertPath), the virtualhost chart creates and manages the secrets from the file contents.

virtualhosts:
...
- name: my-virtualhost
  selector:
    app: apigee-ingressgateway
  tlsMode: MUTUAL
  caCertPath: ./certs/ca.pem
  sslCertPath: ./certs/vhost.pem
  sslKeyPath: ./certs/vhost.key
  mcpMtls:
    caCertPath: ./certs/mcp-ca.pem
    sslCertPath: ./certs/mcp-client.pem
    sslKeyPath: ./certs/mcp-client.key
...
  1. Check whether the secret exists and has a cacert entry, where MCP_CA_SECRET_NAME is the secret named in Spec.Details:

    kubectl -n apigee describe secret MCP_CA_SECRET_NAME

    If the secret is missing or lacks an entry, make sure that the file referenced by mcpMtls.caCertPath exists and is not empty. The next step restores the secret from it. If you intend to run without chain validation, set mcpMtls.disableChainValidation: true and remove mcpMtls.caCertPath (the chart rejects the two together). Be aware that this turns off verification of the ingress gateway's certificate by the MCP data plane.

  2. Upgrade the virtualhost so that the chart rewrites the secret:

    helm upgrade ENV_GROUP_NAME apigee-virtualhost/ \
      --namespace apigee \
      --set envgroup=ENV_GROUP_NAME \
      -f overrides.yaml

Kubernetes secrets

When you use a Kubernetes secret reference (sslSecret), the MCP data plane reads the certificate values directly from the secrets that you created, and updates apply automatically. No helm upgrade is needed.

virtualhosts:
...
- name: my-virtualhost
  selector:
    app: apigee-ingressgateway
  tlsMode: MUTUAL
  sslSecret: my-ingress-secret
  mcpMtls:
    sslSecret: my-mcp-client-secret
...
  1. Confirm the secret exists under exactly that name with a cacert entry, where MCP_CLIENT_SECRET_NAME is the client certificate secret named by mcpMtls.sslSecret:

    kubectl -n apigee describe secret MCP_CLIENT_SECRET_NAME-cacert

    helm template does not check that the secrets exist; only helm install and helm upgrade do, so a clean helm template run is not evidence that this secret is present. If you intend to run without chain validation, omitting the secret is not enough. Set mcpMtls.disableChainValidation: true in overrides.yaml and upgrade the virtualhost. Be aware that this turns off verification of the ingress gateway's certificate by the MCP data plane.

  2. Create or update the CA certificate secret, where MCP_CLIENT_SECRET_NAME is the value of mcpMtls.sslSecret, the CA certificate secret is that name with -cacert appended, and MCP_TRUST_CA_FILE is the CA certificate file:

    kubectl -n apigee create secret generic MCP_CLIENT_SECRET_NAME-cacert \
      --from-file=cacert=MCP_TRUST_CA_FILE \
      --dry-run=client -o yaml | kubectl apply -f -

The MCP data plane picks up the corrected secret on its next periodic refresh, typically within a few minutes, without a pod restart, and the ApigeeIssue is withdrawn on the next scan.

Cause: MCP mTLS CA certificate is not PEM-encoded

The cacert entry is missing the PEM-encoding text blocks, or it has content after the last -----END CERTIFICATE----- line that is not PEM. The MCP data plane does not load it.

Diagnosis

Run the following command:

kubectl -n apigee describe apigeeissue ISSUE_NAME

Replace ISSUE_NAME with the name of the issue, for example mcp-ca-cert-invalid-my-org-my-virtualhost.

The output should look similar to the following:

Name:         mcp-ca-cert-invalid-my-org-my-virtualhost
Namespace:    apigee
Labels:       ais-reason=AIS_MCP_MTLS_CA_CERT_INVALID
Annotations:  <none>
API Version:  apigee.cloud.google.com/v1alpha1
Kind:         ApigeeIssue
Metadata:
  Creation Timestamp:  2026-11-12T17:03:43Z
  Generation:          1
  Owner References:
    API Version:     apigee.cloud.google.com/v1alpha2
    Kind:            ApigeeOrganization
    Name:            my-org
    UID:             7e83a52c-ce00-4bed-98be-55835ada1817
  Resource Version:  3281563
  UID:               adc775c2-376d-4bf9-9860-500b2b2b8273
Spec:
  Details:        MCP mTLS CA certificate in secret "my-org-my-virtualhost-mcp-cacert" for ApigeeRouteConfig "my-org-my-virtualhost" is not usable: holds 1289 trailing bytes that are not PEM
  Documentation:  https://cloud.google.com/apigee/docs/hybrid/troubleshooting/AIS_MCP_MTLS_CA_CERT_INVALID
  Reason:         AIS_MCP_MTLS_CA_CERT_INVALID
  Severity:       ERROR
Events:           <none>
Spec:
  Details:        MCP mTLS CA certificate in secret "my-org-my-virtualhost-mcp-cacert" for ApigeeRouteConfig "my-org-my-virtualhost" is not usable: holds 1289 trailing bytes that are not PEM

The contents of Spec.Details name the secret, the ApigeeRouteConfig, and what is wrong. For example:

  • entry holds no PEM encoded certificate
  • holds N trailing bytes that are not PEM

Resolution

  1. Use kubectl describe to display the name of the Kubernetes secret and ApigeeRouteConfig the issue belongs to:

    kubectl -n apigee describe apigeeissue ISSUE_NAME

    Replace ISSUE_NAME with the name of the issue, for example mcp-ca-cert-invalid-my-org-my-virtualhost.

    In this example, the ApigeeRouteConfig my-org-my-virtualhost is referencing the secret my-org-my-virtualhost-mcp-cacert.

  2. Determine the virtualhost name from the ApigeeRouteConfig.

    The ApigeeRouteConfig's name is in the format: <Apigee organization>-<virtualhost name>.

    In this example, my-org is the Apigee organization and the virtualhost name is my-virtualhost.

  3. Find the corresponding virtualhost in your overrides.yaml and check which form its mcpMtls block uses (see Configure mTLS between the MCP data plane and the ingress gateway). Select the tab that matches, and follow the steps in that tab.

Certificate files

When you use file paths (sslCertPath, sslKeyPath, caCertPath), the virtualhost chart creates and manages the secrets from the file contents.

virtualhosts:
...
- name: my-virtualhost
  selector:
    app: apigee-ingressgateway
  tlsMode: MUTUAL
  caCertPath: ./certs/ca.pem
  sslCertPath: ./certs/vhost.pem
  sslKeyPath: ./certs/vhost.key
  mcpMtls:
    caCertPath: ./certs/mcp-ca.pem
    sslCertPath: ./certs/mcp-client.pem
    sslKeyPath: ./certs/mcp-client.key
...
  1. Validate the contents of the file referenced by mcpMtls.caCertPath, and correct them if needed. Ensure the content contains only complete certificate blocks, each with its starting and ending lines. For example:

    -----BEGIN CERTIFICATE-----
    MIIDYTCCAkmgAwIBAgIUSXeU0pQYRFzYlqZpKhNNJdBLFBIwDQYJKoZIhvcNAQEL
    ... <contents omitted> ...
    QUJDREVGR0hJSktMTU5PUFFSU1RVVldYWVowMTIzNDU2Nzg5QUJDREVGR0hJSktM
    MMM=
    -----END CERTIFICATE-----
  2. Upgrade the virtualhost so that the chart rewrites the secret:

    helm upgrade ENV_GROUP_NAME apigee-virtualhost/ \
      --namespace apigee \
      --set envgroup=ENV_GROUP_NAME \
      -f overrides.yaml

Kubernetes secrets

When you use a Kubernetes secret reference (sslSecret), the MCP data plane reads the certificate values directly from the secrets that you created, and updates apply automatically. No helm upgrade is needed.

virtualhosts:
...
- name: my-virtualhost
  selector:
    app: apigee-ingressgateway
  tlsMode: MUTUAL
  sslSecret: my-ingress-secret
  mcpMtls:
    sslSecret: my-mcp-client-secret
...
  1. Validate the contents of the CA certificate file (MCP_TRUST_CA_FILE) that you store in the cacert entry of the secret, and correct them if needed. Ensure the content contains only complete certificate blocks, each with its starting and ending lines. For example:

    -----BEGIN CERTIFICATE-----
    MIIDYTCCAkmgAwIBAgIUSXeU0pQYRFzYlqZpKhNNJdBLFBIwDQYJKoZIhvcNAQEL
    ... <contents omitted> ...
    QUJDREVGR0hJSktMTU5PUFFSU1RVVldYWVowMTIzNDU2Nzg5QUJDREVGR0hJSktM
    MMM=
    -----END CERTIFICATE-----
  2. Update the CA certificate secret in place, where MCP_CLIENT_SECRET_NAME is the value of mcpMtls.sslSecret, the CA certificate secret is that name with -cacert appended, and MCP_TRUST_CA_FILE is the CA certificate file:

    kubectl -n apigee create secret generic MCP_CLIENT_SECRET_NAME-cacert \
      --from-file=cacert=MCP_TRUST_CA_FILE \
      --dry-run=client -o yaml | kubectl apply -f -

The MCP data plane picks up the corrected secret on its next periodic refresh, typically within a few minutes, without a pod restart, and the ApigeeIssue is withdrawn on the next scan.

Cause: MCP mTLS CA certificate is malformed or is not a CA certificate

Every PEM block in the cacert entry must be a CERTIFICATE block that parses and has the CA basic constraint. A private key block, a leaf certificate (for example the ingress gateway's own server certificate), or a corrupted block makes the entry unusable, and the MCP data plane does not load it.

Diagnosis

Run the following command:

kubectl -n apigee describe apigeeissue ISSUE_NAME

Replace ISSUE_NAME with the name of the issue, for example mcp-ca-cert-invalid-my-org-my-virtualhost.

The output should look similar to the following:

Name:         mcp-ca-cert-invalid-my-org-my-virtualhost
Namespace:    apigee
Labels:       ais-reason=AIS_MCP_MTLS_CA_CERT_INVALID
Annotations:  <none>
API Version:  apigee.cloud.google.com/v1alpha1
Kind:         ApigeeIssue
Metadata:
  Creation Timestamp:  2026-11-12T17:03:43Z
  Generation:          1
  Owner References:
    API Version:     apigee.cloud.google.com/v1alpha2
    Kind:            ApigeeOrganization
    Name:            my-org
    UID:             7e83a52c-ce00-4bed-98be-55835ada1817
  Resource Version:  3281563
  UID:               adc775c2-376d-4bf9-9860-500b2b2b8273
Spec:
  Details:        MCP mTLS CA certificate in secret "my-org-my-virtualhost-mcp-cacert" for ApigeeRouteConfig "my-org-my-virtualhost" is not usable: block 1 is not a CA certificate
  Documentation:  https://cloud.google.com/apigee/docs/hybrid/troubleshooting/AIS_MCP_MTLS_CA_CERT_INVALID
  Reason:         AIS_MCP_MTLS_CA_CERT_INVALID
  Severity:       ERROR
Events:           <none>
Spec:
  Details:        MCP mTLS CA certificate in secret "my-org-my-virtualhost-mcp-cacert" for ApigeeRouteConfig "my-org-my-virtualhost" is not usable: block 1 is not a CA certificate

The contents of Spec.Details name the secret, the ApigeeRouteConfig, and what is wrong. For example:

  • no PEM block parsed as a certificate: ...: none of the certificate blocks can be parsed (for example, the only certificate is corrupted).
  • block N is a "..." block, want CERTIFICATE: block N is not a certificate (for example, a private key).
  • block N does not parse: block N is a corrupted certificate.
  • block N is not a CA certificate: block N is a certificate, but not a CA (for example, a leaf certificate).

Blocks are numbered from 0.

Resolution

  1. Use kubectl describe to display the name of the Kubernetes secret and ApigeeRouteConfig the issue belongs to:

    kubectl -n apigee describe apigeeissue ISSUE_NAME

    Replace ISSUE_NAME with the name of the issue, for example mcp-ca-cert-invalid-my-org-my-virtualhost.

    In this example, the ApigeeRouteConfig my-org-my-virtualhost is referencing the secret my-org-my-virtualhost-mcp-cacert.

  2. Determine the virtualhost name from the ApigeeRouteConfig.

    The ApigeeRouteConfig's name is in the format: <Apigee organization>-<virtualhost name>.

    In this example, my-org is the Apigee organization and the virtualhost name is my-virtualhost.

  3. Find the corresponding virtualhost in your overrides.yaml and check which form its mcpMtls block uses (see Configure mTLS between the MCP data plane and the ingress gateway). Select the tab that matches, and follow the steps in that tab.

Certificate files

When you use file paths (sslCertPath, sslKeyPath, caCertPath), the virtualhost chart creates and manages the secrets from the file contents.

virtualhosts:
...
- name: my-virtualhost
  selector:
    app: apigee-ingressgateway
  tlsMode: MUTUAL
  caCertPath: ./certs/ca.pem
  sslCertPath: ./certs/vhost.pem
  sslKeyPath: ./certs/vhost.key
  mcpMtls:
    caCertPath: ./certs/mcp-ca.pem
    sslCertPath: ./certs/mcp-client.pem
    sslKeyPath: ./certs/mcp-client.key
...
  1. Rebuild the file referenced by mcpMtls.caCertPath so that it holds only the CA certificate (or chain) that signed the ingress gateway's certificate. Then confirm that every block is a CA certificate, where MCP_TRUST_CA_FILE is that file:

    openssl storeutl -noout -text -certs MCP_TRUST_CA_FILE | grep -E 'Subject:|CA:'
    # every certificate must show CA:TRUE
  2. Upgrade the virtualhost so that the chart rewrites the secret:

    helm upgrade ENV_GROUP_NAME apigee-virtualhost/ \
      --namespace apigee \
      --set envgroup=ENV_GROUP_NAME \
      -f overrides.yaml

Kubernetes secrets

When you use a Kubernetes secret reference (sslSecret), the MCP data plane reads the certificate values directly from the secrets that you created, and updates apply automatically. No helm upgrade is needed.

virtualhosts:
...
- name: my-virtualhost
  selector:
    app: apigee-ingressgateway
  tlsMode: MUTUAL
  sslSecret: my-ingress-secret
  mcpMtls:
    sslSecret: my-mcp-client-secret
...
  1. Rebuild the CA certificate file (MCP_TRUST_CA_FILE) so that it holds only the CA certificate (or chain) that signed the ingress gateway's certificate. Then confirm that every block is a CA certificate:

    openssl storeutl -noout -text -certs MCP_TRUST_CA_FILE | grep -E 'Subject:|CA:'
    # every certificate must show CA:TRUE
  2. Update the CA certificate secret in place, where MCP_CLIENT_SECRET_NAME is the value of mcpMtls.sslSecret, the CA certificate secret is that name with -cacert appended, and MCP_TRUST_CA_FILE is the CA certificate file:

    kubectl -n apigee create secret generic MCP_CLIENT_SECRET_NAME-cacert \
      --from-file=cacert=MCP_TRUST_CA_FILE \
      --dry-run=client -o yaml | kubectl apply -f -

The MCP data plane picks up the corrected secret on its next periodic refresh, typically within a few minutes, without a pod restart, and the ApigeeIssue is withdrawn on the next scan.

Must gather diagnostic information

If the problem persists even after following the preceding instructions, gather the following diagnostic information and then contact Google Cloud Customer Care:

  1. The Google Cloud project ID.
  2. The name of the Apigee hybrid organization.
  3. The Spec.Details field of the ApigeeIssue.
  4. (optional) The certificate in the file referenced by mcpMtls.caCertPath, or in the cacert entry of the secret, for the affected virtualhost.