MCP mTLS client certificate invalid

This page applies to Apigee hybrid, but not to Apigee.

There is no equivalent Apigee Edge documentation for this topic.

Symptom

An ApigeeIssue with Reason AIS_MCP_MTLS_CLIENT_CERT_INVALID is shown when viewing resources in the apigee namespace.

Error messages

After running kubectl -n apigee get apigeeissues, an error like the following is displayed:

NAME                                            SEVERITY   REASON                             DOCUMENTATION                                                                                  AGE
mcp-client-cert-invalid-my-org-my-virtualhost   ERROR      AIS_MCP_MTLS_CLIENT_CERT_INVALID   https://cloud.google.com/apigee/docs/hybrid/troubleshooting/AIS_MCP_MTLS_CLIENT_CERT_INVALID   5h18m

Possible causes

The following table lists possible causes of this issue:

Cause Description
MCP mTLS client certificate secret or entry is missing The secret named by mcpMtls does not exist, or it has no usable cert or key entry.
MCP mTLS client certificate or key is malformed The cert entry is not PEM-encoded, or the key entry is not the matching private key.
MCP mTLS client certificate omits the clientAuth extended key usage The certificate parses, but its extended key usage does not include clientAuth.

Cause: MCP mTLS client certificate secret or entry is missing

The Kubernetes secret that should hold the MCP mTLS client certificate does not exist in the apigee namespace, or it exists without a non-empty cert entry or without a key entry. The MCP data plane has no client certificate to present for this environment group, so MCP tool calls to the group fail at the TLS handshake.

Diagnosis

Run the following command:

kubectl -n apigee describe apigeeissue ISSUE_NAME

Replace ISSUE_NAME with the name of the issue, for example mcp-client-cert-invalid-my-org-my-virtualhost.

The output should look similar to the following:

Name:         mcp-client-cert-invalid-my-org-my-virtualhost
Namespace:    apigee
Labels:       ais-reason=AIS_MCP_MTLS_CLIENT_CERT_INVALID
Annotations:  <none>
API Version:  apigee.cloud.google.com/v1alpha1
Kind:         ApigeeIssue
Metadata:
  Creation Timestamp:  2026-11-12T17:03:43Z
  Generation:          1
  Owner References:
    API Version:     apigee.cloud.google.com/v1alpha2
    Kind:            ApigeeOrganization
    Name:            my-org
    UID:             7e83a52c-ce00-4bed-98be-55835ada1817
  Resource Version:  3281563
  UID:               adc775c2-376d-4bf9-9860-500b2b2b8273
Spec:
  Details:        MCP mTLS client certificate in secret "my-org-my-virtualhost-mcp" for ApigeeRouteConfig "my-org-my-virtualhost" is not usable: secret not found: secrets "my-org-my-virtualhost-mcp" not found
  Documentation:  https://cloud.google.com/apigee/docs/hybrid/troubleshooting/AIS_MCP_MTLS_CLIENT_CERT_INVALID
  Reason:         AIS_MCP_MTLS_CLIENT_CERT_INVALID
  Severity:       ERROR
Events:           <none>
Spec:
  Details:        MCP mTLS client certificate in secret "my-org-my-virtualhost-mcp" for ApigeeRouteConfig "my-org-my-virtualhost" is not usable: secret not found: secrets "my-org-my-virtualhost-mcp" not found

The contents of Spec.Details name the secret, the ApigeeRouteConfig, and what is wrong. For example:

  • secret not found
  • secret has no usable "cert" entry
  • secret has no "key" entry

Resolution

  1. Use kubectl describe to display the name of the Kubernetes secret and ApigeeRouteConfig the issue belongs to:

    kubectl -n apigee describe apigeeissue ISSUE_NAME

    Replace ISSUE_NAME with the name of the issue, for example mcp-client-cert-invalid-my-org-my-virtualhost.

    In this example, the ApigeeRouteConfig my-org-my-virtualhost is referencing the secret my-org-my-virtualhost-mcp.

  2. Determine the virtualhost name from the ApigeeRouteConfig.

    The ApigeeRouteConfig's name is in the format: <Apigee organization>-<virtualhost name>.

    In this example, my-org is the Apigee organization and the virtualhost name is my-virtualhost.

  3. Find the corresponding virtualhost in your overrides.yaml and check which form its mcpMtls block uses (see Configure mTLS between the MCP data plane and the ingress gateway). Select the tab that matches, and follow the steps in that tab.

Certificate files

When you use file paths (sslCertPath, sslKeyPath, caCertPath), the virtualhost chart creates and manages the secrets from the file contents.

virtualhosts:
...
- name: my-virtualhost
  selector:
    app: apigee-ingressgateway
  tlsMode: MUTUAL
  caCertPath: ./certs/ca.pem
  sslCertPath: ./certs/vhost.pem
  sslKeyPath: ./certs/vhost.key
  mcpMtls:
    caCertPath: ./certs/mcp-ca.pem
    sslCertPath: ./certs/mcp-client.pem
    sslKeyPath: ./certs/mcp-client.key
...
  1. Check whether the secret exists and has both entries, where MCP_CLIENT_SECRET_NAME is the secret named in Spec.Details. The Data section must list cert and key:

    kubectl -n apigee describe secret MCP_CLIENT_SECRET_NAME

    If the secret is missing or lacks an entry, make sure that the files referenced by mcpMtls.sslCertPath and mcpMtls.sslKeyPath exist and are not empty. The next step restores the secret from them.

  2. Upgrade the virtualhost so that the chart rewrites the secret:

    helm upgrade ENV_GROUP_NAME apigee-virtualhost/ \
      --namespace apigee \
      --set envgroup=ENV_GROUP_NAME \
      -f overrides.yaml

Kubernetes secrets

When you use a Kubernetes secret reference (sslSecret), the MCP data plane reads the certificate values directly from the secrets that you created, and updates apply automatically. No helm upgrade is needed.

virtualhosts:
...
- name: my-virtualhost
  selector:
    app: apigee-ingressgateway
  tlsMode: MUTUAL
  sslSecret: my-ingress-secret
  mcpMtls:
    sslSecret: my-mcp-client-secret
...
  1. Confirm the secret exists and has both entries, where MCP_CLIENT_SECRET_NAME is the secret named in Spec.Details. The Data section must list cert and key. Check that the name in mcpMtls.sslSecret matches the secret that you created and that the secret is in the apigee namespace:

    kubectl -n apigee describe secret MCP_CLIENT_SECRET_NAME
    kubectl get secret -A --field-selector metadata.name=MCP_CLIENT_SECRET_NAME   # if the first command finds nothing

    helm template does not check that the secrets exist; only helm install and helm upgrade do, so a clean helm template run is not evidence that this secret is present.

  2. Create or update the secret with the certificate and its private key together, where MCP_CLIENT_SECRET_NAME is the secret named in Spec.Details and MCP_CLIENT_CERT_FILE and MCP_CLIENT_KEY_FILE are the certificate and its private key:

    kubectl -n apigee create secret generic MCP_CLIENT_SECRET_NAME \
      --from-file=cert=MCP_CLIENT_CERT_FILE --from-file=key=MCP_CLIENT_KEY_FILE \
      --dry-run=client -o yaml | kubectl apply -f -

The MCP data plane picks up the corrected secret on its next periodic refresh, typically within a few minutes, without a pod restart, and the ApigeeIssue is withdrawn on the next scan.

Cause: MCP mTLS client certificate or key is malformed

The cert entry does not contain a PEM-encoded X.509 certificate, or the key entry is not PEM-encoded or is not the private key for that certificate. The MCP data plane cannot load the pair.

Diagnosis

Run the following command:

kubectl -n apigee describe apigeeissue ISSUE_NAME

Replace ISSUE_NAME with the name of the issue, for example mcp-client-cert-invalid-my-org-my-virtualhost.

The output should look similar to the following:

Name:         mcp-client-cert-invalid-my-org-my-virtualhost
Namespace:    apigee
Labels:       ais-reason=AIS_MCP_MTLS_CLIENT_CERT_INVALID
Annotations:  <none>
API Version:  apigee.cloud.google.com/v1alpha1
Kind:         ApigeeIssue
Metadata:
  Creation Timestamp:  2026-11-12T17:03:43Z
  Generation:          1
  Owner References:
    API Version:     apigee.cloud.google.com/v1alpha2
    Kind:            ApigeeOrganization
    Name:            my-org
    UID:             7e83a52c-ce00-4bed-98be-55835ada1817
  Resource Version:  3281563
  UID:               adc775c2-376d-4bf9-9860-500b2b2b8273
Spec:
  Details:        MCP mTLS client certificate in secret "my-org-my-virtualhost-mcp" for ApigeeRouteConfig "my-org-my-virtualhost" is not usable: "cert" and "key" are not a usable certificate and key: tls: private key does not match public key
  Documentation:  https://cloud.google.com/apigee/docs/hybrid/troubleshooting/AIS_MCP_MTLS_CLIENT_CERT_INVALID
  Reason:         AIS_MCP_MTLS_CLIENT_CERT_INVALID
  Severity:       ERROR
Events:           <none>
Spec:
  Details:        MCP mTLS client certificate in secret "my-org-my-virtualhost-mcp" for ApigeeRouteConfig "my-org-my-virtualhost" is not usable: "cert" and "key" are not a usable certificate and key: tls: private key does not match public key

The contents of Spec.Details name the secret, the ApigeeRouteConfig, and what is wrong. For example:

  • entry holds no PEM encoded certificate: the cert entry is not PEM.
  • tls: failed to find any PEM data in key input: the key entry is not PEM.
  • tls: private key does not match public key: the cert and key entries do not pair.
  • tls: failed to parse private key: the key cannot be parsed, usually because it is encrypted. The key entry must be an unencrypted PEM private key.

Resolution

  1. Use kubectl describe to display the name of the Kubernetes secret and ApigeeRouteConfig the issue belongs to:

    kubectl -n apigee describe apigeeissue ISSUE_NAME

    Replace ISSUE_NAME with the name of the issue, for example mcp-client-cert-invalid-my-org-my-virtualhost.

    In this example, the ApigeeRouteConfig my-org-my-virtualhost is referencing the secret my-org-my-virtualhost-mcp.

  2. Determine the virtualhost name from the ApigeeRouteConfig.

    The ApigeeRouteConfig's name is in the format: <Apigee organization>-<virtualhost name>.

    In this example, my-org is the Apigee organization and the virtualhost name is my-virtualhost.

  3. Find the corresponding virtualhost in your overrides.yaml and check which form its mcpMtls block uses (see Configure mTLS between the MCP data plane and the ingress gateway). Select the tab that matches, and follow the steps in that tab.

Certificate files

When you use file paths (sslCertPath, sslKeyPath, caCertPath), the virtualhost chart creates and manages the secrets from the file contents.

virtualhosts:
...
- name: my-virtualhost
  selector:
    app: apigee-ingressgateway
  tlsMode: MUTUAL
  caCertPath: ./certs/ca.pem
  sslCertPath: ./certs/vhost.pem
  sslKeyPath: ./certs/vhost.key
  mcpMtls:
    caCertPath: ./certs/mcp-ca.pem
    sslCertPath: ./certs/mcp-client.pem
    sslKeyPath: ./certs/mcp-client.key
...
  1. Validate the certificate file and the key file referenced by mcpMtls.sslCertPath and mcpMtls.sslKeyPath, and correct them if needed. The certificate must contain the starting and ending certificate lines, and the key must be the private key for it. For example:

    -----BEGIN CERTIFICATE-----
    MIIDYTCCAkmgAwIBAgIUSXeU0pQYRFzYlqZpKhNNJdBLFBIwDQYJKoZIhvcNAQEL
    ... <contents omitted> ...
    QUJDREVGR0hJSktMTU5PUFFSU1RVVldYWVowMTIzNDU2Nzg5QUJDREVGR0hJSktM
    MMM=
    -----END CERTIFICATE-----
  2. Upgrade the virtualhost so that the chart rewrites the secret:

    helm upgrade ENV_GROUP_NAME apigee-virtualhost/ \
      --namespace apigee \
      --set envgroup=ENV_GROUP_NAME \
      -f overrides.yaml

Kubernetes secrets

When you use a Kubernetes secret reference (sslSecret), the MCP data plane reads the certificate values directly from the secrets that you created, and updates apply automatically. No helm upgrade is needed.

virtualhosts:
...
- name: my-virtualhost
  selector:
    app: apigee-ingressgateway
  tlsMode: MUTUAL
  sslSecret: my-ingress-secret
  mcpMtls:
    sslSecret: my-mcp-client-secret
...
  1. Validate the certificate file (MCP_CLIENT_CERT_FILE) and the key file (MCP_CLIENT_KEY_FILE) that you store in the cert and key entries of the secret, and correct them if needed. The certificate must contain the starting and ending certificate lines, and the key must be the private key for it. For example:

    -----BEGIN CERTIFICATE-----
    MIIDYTCCAkmgAwIBAgIUSXeU0pQYRFzYlqZpKhNNJdBLFBIwDQYJKoZIhvcNAQEL
    ... <contents omitted> ...
    QUJDREVGR0hJSktMTU5PUFFSU1RVVldYWVowMTIzNDU2Nzg5QUJDREVGR0hJSktM
    MMM=
    -----END CERTIFICATE-----
  2. Update the secret in place with the certificate and its private key together, where MCP_CLIENT_SECRET_NAME is the secret named in Spec.Details and MCP_CLIENT_CERT_FILE and MCP_CLIENT_KEY_FILE are the certificate and its private key:

    kubectl -n apigee create secret generic MCP_CLIENT_SECRET_NAME \
      --from-file=cert=MCP_CLIENT_CERT_FILE --from-file=key=MCP_CLIENT_KEY_FILE \
      --dry-run=client -o yaml | kubectl apply -f -

The MCP data plane picks up the corrected secret on its next periodic refresh, typically within a few minutes, without a pod restart, and the ApigeeIssue is withdrawn on the next scan.

Cause: MCP mTLS client certificate omits the clientAuth extended key usage

The certificate is well formed and pairs with its key, but its extended key usage (EKU) does not include clientAuth, so it cannot be used for client authentication and the MCP data plane does not present it. This is common when a server certificate, or a certificate from a public CA profile that no longer issues clientAuth, is reused.

Diagnosis

Run the following command:

kubectl -n apigee describe apigeeissue ISSUE_NAME

Replace ISSUE_NAME with the name of the issue, for example mcp-client-cert-invalid-my-org-my-virtualhost.

The output should look similar to the following:

Name:         mcp-client-cert-invalid-my-org-my-virtualhost
Namespace:    apigee
Labels:       ais-reason=AIS_MCP_MTLS_CLIENT_CERT_INVALID
Annotations:  <none>
API Version:  apigee.cloud.google.com/v1alpha1
Kind:         ApigeeIssue
Metadata:
  Creation Timestamp:  2026-11-12T17:03:43Z
  Generation:          1
  Owner References:
    API Version:     apigee.cloud.google.com/v1alpha2
    Kind:            ApigeeOrganization
    Name:            my-org
    UID:             7e83a52c-ce00-4bed-98be-55835ada1817
  Resource Version:  3281563
  UID:               adc775c2-376d-4bf9-9860-500b2b2b8273
Spec:
  Details:        MCP mTLS client certificate in secret "my-org-my-virtualhost-mcp" for ApigeeRouteConfig "my-org-my-virtualhost" is not usable: "cert" has extended key usage [serverAuth], which omits clientAuth
  Documentation:  https://cloud.google.com/apigee/docs/hybrid/troubleshooting/AIS_MCP_MTLS_CLIENT_CERT_INVALID
  Reason:         AIS_MCP_MTLS_CLIENT_CERT_INVALID
  Severity:       ERROR
Events:           <none>
Spec:
  Details:        MCP mTLS client certificate in secret "my-org-my-virtualhost-mcp" for ApigeeRouteConfig "my-org-my-virtualhost" is not usable: "cert" has extended key usage [serverAuth], which omits clientAuth

The contents of Spec.Details name the secret and the ApigeeRouteConfig, and list the extended key usages the certificate does carry.

Resolution

  1. Use kubectl describe to display the name of the Kubernetes secret and ApigeeRouteConfig the issue belongs to:

    kubectl -n apigee describe apigeeissue ISSUE_NAME

    Replace ISSUE_NAME with the name of the issue, for example mcp-client-cert-invalid-my-org-my-virtualhost.

    In this example, the ApigeeRouteConfig my-org-my-virtualhost is referencing the secret my-org-my-virtualhost-mcp.

  2. Determine the virtualhost name from the ApigeeRouteConfig.

    The ApigeeRouteConfig's name is in the format: <Apigee organization>-<virtualhost name>.

    In this example, my-org is the Apigee organization and the virtualhost name is my-virtualhost.

  3. Find the corresponding virtualhost in your overrides.yaml and check which form its mcpMtls block uses (see Configure mTLS between the MCP data plane and the ingress gateway). Select the tab that matches, and follow the steps in that tab.

Certificate files

When you use file paths (sslCertPath, sslKeyPath, caCertPath), the virtualhost chart creates and manages the secrets from the file contents.

virtualhosts:
...
- name: my-virtualhost
  selector:
    app: apigee-ingressgateway
  tlsMode: MUTUAL
  caCertPath: ./certs/ca.pem
  sslCertPath: ./certs/vhost.pem
  sslKeyPath: ./certs/vhost.key
  mcpMtls:
    caCertPath: ./certs/mcp-ca.pem
    sslCertPath: ./certs/mcp-client.pem
    sslKeyPath: ./certs/mcp-client.key
...
  1. Reissue the client certificate from your private CA with a profile that includes the clientAuth extended key usage. Confirm it, where MCP_CLIENT_CERT_FILE is the reissued certificate, and then replace the files referenced by mcpMtls.sslCertPath and mcpMtls.sslKeyPath:

    openssl x509 -in MCP_CLIENT_CERT_FILE -noout -ext extendedKeyUsage
    # must list: TLS Web Client Authentication
  2. Upgrade the virtualhost so that the chart rewrites the secret:

    helm upgrade ENV_GROUP_NAME apigee-virtualhost/ \
      --namespace apigee \
      --set envgroup=ENV_GROUP_NAME \
      -f overrides.yaml

Kubernetes secrets

When you use a Kubernetes secret reference (sslSecret), the MCP data plane reads the certificate values directly from the secrets that you created, and updates apply automatically. No helm upgrade is needed.

virtualhosts:
...
- name: my-virtualhost
  selector:
    app: apigee-ingressgateway
  tlsMode: MUTUAL
  sslSecret: my-ingress-secret
  mcpMtls:
    sslSecret: my-mcp-client-secret
...
  1. Reissue the client certificate from your private CA with a profile that includes the clientAuth extended key usage. Confirm it before you update the secret, where MCP_CLIENT_CERT_FILE is the reissued certificate:

    openssl x509 -in MCP_CLIENT_CERT_FILE -noout -ext extendedKeyUsage
    # must list: TLS Web Client Authentication
  2. Update the secret in place with the certificate and its private key together, where MCP_CLIENT_SECRET_NAME is the secret named in Spec.Details and MCP_CLIENT_CERT_FILE and MCP_CLIENT_KEY_FILE are the certificate and its private key:

    kubectl -n apigee create secret generic MCP_CLIENT_SECRET_NAME \
      --from-file=cert=MCP_CLIENT_CERT_FILE --from-file=key=MCP_CLIENT_KEY_FILE \
      --dry-run=client -o yaml | kubectl apply -f -

The MCP data plane picks up the corrected secret on its next periodic refresh, typically within a few minutes, without a pod restart, and the ApigeeIssue is withdrawn on the next scan.

Must gather diagnostic information

If the problem persists even after following the preceding instructions, gather the following diagnostic information and then contact Google Cloud Customer Care:

  1. The Google Cloud project ID.
  2. The name of the Apigee hybrid organization.
  3. The Spec.Details field of the ApigeeIssue.
  4. (optional) The certificate in the file referenced by mcpMtls.sslCertPath, or in the cert entry of the secret, for the affected virtualhost.