This page applies to Apigee hybrid, but not to Apigee.
There is no equivalent Apigee Edge documentation for this topic.
Symptom
An ApigeeIssue with Reason AIS_MCP_MTLS_CLIENT_CERT_INVALID
is shown when viewing resources in the apigee namespace.
Error messages
After running kubectl -n apigee get apigeeissues, an error like the following is
displayed:
NAME SEVERITY REASON DOCUMENTATION AGE mcp-client-cert-invalid-my-org-my-virtualhost ERROR AIS_MCP_MTLS_CLIENT_CERT_INVALID https://cloud.google.com/apigee/docs/hybrid/troubleshooting/AIS_MCP_MTLS_CLIENT_CERT_INVALID 5h18m
Possible causes
The following table lists possible causes of this issue:
| Cause | Description |
|---|---|
| MCP mTLS client certificate secret or entry is missing |
The secret named by mcpMtls does not exist, or it has no usable
cert or key entry.
|
| MCP mTLS client certificate or key is malformed |
The cert entry is not PEM-encoded, or the key entry is not
the matching private key.
|
| MCP mTLS client certificate omits the clientAuth extended key usage |
The certificate parses, but its extended key usage does not include
clientAuth.
|
Cause: MCP mTLS client certificate secret or entry is missing
The Kubernetes secret that should hold the MCP mTLS client certificate does not exist in the
apigee namespace, or it exists without a non-empty cert entry or
without a key entry. The MCP data plane has no client certificate to present for
this environment group, so MCP tool calls to the group fail at the TLS handshake.
Diagnosis
Run the following command:
kubectl -n apigee describe apigeeissue ISSUE_NAME
Replace ISSUE_NAME with the name of the issue, for example
mcp-client-cert-invalid-my-org-my-virtualhost.
The output should look similar to the following:
Name: mcp-client-cert-invalid-my-org-my-virtualhost Namespace: apigee Labels: ais-reason=AIS_MCP_MTLS_CLIENT_CERT_INVALID Annotations: <none> API Version: apigee.cloud.google.com/v1alpha1 Kind: ApigeeIssue Metadata: Creation Timestamp: 2026-11-12T17:03:43Z Generation: 1 Owner References: API Version: apigee.cloud.google.com/v1alpha2 Kind: ApigeeOrganization Name: my-org UID: 7e83a52c-ce00-4bed-98be-55835ada1817 Resource Version: 3281563 UID: adc775c2-376d-4bf9-9860-500b2b2b8273 Spec: Details: MCP mTLS client certificate in secret "my-org-my-virtualhost-mcp" for ApigeeRouteConfig "my-org-my-virtualhost" is not usable: secret not found: secrets "my-org-my-virtualhost-mcp" not found Documentation: https://cloud.google.com/apigee/docs/hybrid/troubleshooting/AIS_MCP_MTLS_CLIENT_CERT_INVALID Reason: AIS_MCP_MTLS_CLIENT_CERT_INVALID Severity: ERROR Events: <none>
Spec: Details: MCP mTLS client certificate in secret "my-org-my-virtualhost-mcp" for ApigeeRouteConfig "my-org-my-virtualhost" is not usable: secret not found: secrets "my-org-my-virtualhost-mcp" not found
The contents of Spec.Details name the secret, the
ApigeeRouteConfig, and what is wrong. For example:
secret not foundsecret has no usable "cert" entrysecret has no "key" entry
Resolution
-
Use
kubectl describeto display the name of the Kubernetes secret andApigeeRouteConfigthe issue belongs to:kubectl -n apigee describe apigeeissue ISSUE_NAME
Replace
ISSUE_NAMEwith the name of the issue, for examplemcp-client-cert-invalid-my-org-my-virtualhost.In this example, the
ApigeeRouteConfigmy-org-my-virtualhostis referencing the secretmy-org-my-virtualhost-mcp. -
Determine the
virtualhostname from theApigeeRouteConfig.The
ApigeeRouteConfig's name is in the format:<Apigee organization>-<virtualhost name>.In this example,
my-orgis the Apigee organization and the virtualhost name ismy-virtualhost. -
Find the corresponding virtualhost in your
overrides.yamland check which form itsmcpMtlsblock uses (see Configure mTLS between the MCP data plane and the ingress gateway). Select the tab that matches, and follow the steps in that tab.
Certificate files
When you use file paths (sslCertPath, sslKeyPath,
caCertPath), the virtualhost chart creates and manages the secrets from the
file contents.
virtualhosts:
...
- name: my-virtualhost
selector:
app: apigee-ingressgateway
tlsMode: MUTUAL
caCertPath: ./certs/ca.pem
sslCertPath: ./certs/vhost.pem
sslKeyPath: ./certs/vhost.key
mcpMtls:
caCertPath: ./certs/mcp-ca.pem
sslCertPath: ./certs/mcp-client.pem
sslKeyPath: ./certs/mcp-client.key
...-
Check whether the secret exists and has both entries, where MCP_CLIENT_SECRET_NAME is the secret named in
Spec.Details. TheDatasection must listcertandkey:kubectl -n apigee describe secret MCP_CLIENT_SECRET_NAME
If the secret is missing or lacks an entry, make sure that the files referenced by
mcpMtls.sslCertPathandmcpMtls.sslKeyPathexist and are not empty. The next step restores the secret from them. -
Upgrade the virtualhost so that the chart rewrites the secret:
helm upgrade ENV_GROUP_NAME apigee-virtualhost/ \ --namespace apigee \ --set envgroup=ENV_GROUP_NAME \ -f overrides.yaml
Kubernetes secrets
When you use a Kubernetes secret reference (sslSecret), the MCP data plane
reads the certificate values directly from the secrets that you created, and updates
apply automatically. No helm upgrade is needed.
virtualhosts:
...
- name: my-virtualhost
selector:
app: apigee-ingressgateway
tlsMode: MUTUAL
sslSecret: my-ingress-secret
mcpMtls:
sslSecret: my-mcp-client-secret
...-
Confirm the secret exists and has both entries, where MCP_CLIENT_SECRET_NAME is the secret named in
Spec.Details. TheDatasection must listcertandkey. Check that the name inmcpMtls.sslSecretmatches the secret that you created and that the secret is in theapigeenamespace:kubectl -n apigee describe secret MCP_CLIENT_SECRET_NAME kubectl get secret -A --field-selector metadata.name=MCP_CLIENT_SECRET_NAME # if the first command finds nothing
helm templatedoes not check that the secrets exist; onlyhelm installandhelm upgradedo, so a cleanhelm templaterun is not evidence that this secret is present. -
Create or update the secret with the certificate and its private key together, where MCP_CLIENT_SECRET_NAME is the secret named in
Spec.Detailsand MCP_CLIENT_CERT_FILE and MCP_CLIENT_KEY_FILE are the certificate and its private key:kubectl -n apigee create secret generic MCP_CLIENT_SECRET_NAME \ --from-file=cert=MCP_CLIENT_CERT_FILE --from-file=key=MCP_CLIENT_KEY_FILE \ --dry-run=client -o yaml | kubectl apply -f -
The MCP data plane picks up the corrected secret on its next periodic refresh, typically within a
few minutes, without a pod restart, and the ApigeeIssue is withdrawn on the next
scan.
Cause: MCP mTLS client certificate or key is malformed
The cert entry does not contain a PEM-encoded X.509 certificate, or the
key entry is not PEM-encoded or is not the private key for that certificate. The
MCP data plane cannot load the pair.
Diagnosis
Run the following command:
kubectl -n apigee describe apigeeissue ISSUE_NAME
Replace ISSUE_NAME with the name of the issue, for example
mcp-client-cert-invalid-my-org-my-virtualhost.
The output should look similar to the following:
Name: mcp-client-cert-invalid-my-org-my-virtualhost Namespace: apigee Labels: ais-reason=AIS_MCP_MTLS_CLIENT_CERT_INVALID Annotations: <none> API Version: apigee.cloud.google.com/v1alpha1 Kind: ApigeeIssue Metadata: Creation Timestamp: 2026-11-12T17:03:43Z Generation: 1 Owner References: API Version: apigee.cloud.google.com/v1alpha2 Kind: ApigeeOrganization Name: my-org UID: 7e83a52c-ce00-4bed-98be-55835ada1817 Resource Version: 3281563 UID: adc775c2-376d-4bf9-9860-500b2b2b8273 Spec: Details: MCP mTLS client certificate in secret "my-org-my-virtualhost-mcp" for ApigeeRouteConfig "my-org-my-virtualhost" is not usable: "cert" and "key" are not a usable certificate and key: tls: private key does not match public key Documentation: https://cloud.google.com/apigee/docs/hybrid/troubleshooting/AIS_MCP_MTLS_CLIENT_CERT_INVALID Reason: AIS_MCP_MTLS_CLIENT_CERT_INVALID Severity: ERROR Events: <none>
Spec: Details: MCP mTLS client certificate in secret "my-org-my-virtualhost-mcp" for ApigeeRouteConfig "my-org-my-virtualhost" is not usable: "cert" and "key" are not a usable certificate and key: tls: private key does not match public key
The contents of Spec.Details name the secret, the
ApigeeRouteConfig, and what is wrong. For example:
-
entry holds no PEM encoded certificate: thecertentry is not PEM. -
tls: failed to find any PEM data in key input: thekeyentry is not PEM. -
tls: private key does not match public key: thecertandkeyentries do not pair. -
tls: failed to parse private key: the key cannot be parsed, usually because it is encrypted. Thekeyentry must be an unencrypted PEM private key.
Resolution
-
Use
kubectl describeto display the name of the Kubernetes secret andApigeeRouteConfigthe issue belongs to:kubectl -n apigee describe apigeeissue ISSUE_NAME
Replace
ISSUE_NAMEwith the name of the issue, for examplemcp-client-cert-invalid-my-org-my-virtualhost.In this example, the
ApigeeRouteConfigmy-org-my-virtualhostis referencing the secretmy-org-my-virtualhost-mcp. -
Determine the
virtualhostname from theApigeeRouteConfig.The
ApigeeRouteConfig's name is in the format:<Apigee organization>-<virtualhost name>.In this example,
my-orgis the Apigee organization and the virtualhost name ismy-virtualhost. -
Find the corresponding virtualhost in your
overrides.yamland check which form itsmcpMtlsblock uses (see Configure mTLS between the MCP data plane and the ingress gateway). Select the tab that matches, and follow the steps in that tab.
Certificate files
When you use file paths (sslCertPath, sslKeyPath,
caCertPath), the virtualhost chart creates and manages the secrets from the
file contents.
virtualhosts:
...
- name: my-virtualhost
selector:
app: apigee-ingressgateway
tlsMode: MUTUAL
caCertPath: ./certs/ca.pem
sslCertPath: ./certs/vhost.pem
sslKeyPath: ./certs/vhost.key
mcpMtls:
caCertPath: ./certs/mcp-ca.pem
sslCertPath: ./certs/mcp-client.pem
sslKeyPath: ./certs/mcp-client.key
...-
Validate the certificate file and the key file referenced by
mcpMtls.sslCertPathandmcpMtls.sslKeyPath, and correct them if needed. The certificate must contain the starting and ending certificate lines, and the key must be the private key for it. For example:-----BEGIN CERTIFICATE----- MIIDYTCCAkmgAwIBAgIUSXeU0pQYRFzYlqZpKhNNJdBLFBIwDQYJKoZIhvcNAQEL ... <contents omitted> ... QUJDREVGR0hJSktMTU5PUFFSU1RVVldYWVowMTIzNDU2Nzg5QUJDREVGR0hJSktM MMM= -----END CERTIFICATE-----
-
Upgrade the virtualhost so that the chart rewrites the secret:
helm upgrade ENV_GROUP_NAME apigee-virtualhost/ \ --namespace apigee \ --set envgroup=ENV_GROUP_NAME \ -f overrides.yaml
Kubernetes secrets
When you use a Kubernetes secret reference (sslSecret), the MCP data plane
reads the certificate values directly from the secrets that you created, and updates
apply automatically. No helm upgrade is needed.
virtualhosts:
...
- name: my-virtualhost
selector:
app: apigee-ingressgateway
tlsMode: MUTUAL
sslSecret: my-ingress-secret
mcpMtls:
sslSecret: my-mcp-client-secret
...-
Validate the certificate file (MCP_CLIENT_CERT_FILE) and the key file (MCP_CLIENT_KEY_FILE) that you store in the
certandkeyentries of the secret, and correct them if needed. The certificate must contain the starting and ending certificate lines, and the key must be the private key for it. For example:-----BEGIN CERTIFICATE----- MIIDYTCCAkmgAwIBAgIUSXeU0pQYRFzYlqZpKhNNJdBLFBIwDQYJKoZIhvcNAQEL ... <contents omitted> ... QUJDREVGR0hJSktMTU5PUFFSU1RVVldYWVowMTIzNDU2Nzg5QUJDREVGR0hJSktM MMM= -----END CERTIFICATE-----
-
Update the secret in place with the certificate and its private key together, where MCP_CLIENT_SECRET_NAME is the secret named in
Spec.Detailsand MCP_CLIENT_CERT_FILE and MCP_CLIENT_KEY_FILE are the certificate and its private key:kubectl -n apigee create secret generic MCP_CLIENT_SECRET_NAME \ --from-file=cert=MCP_CLIENT_CERT_FILE --from-file=key=MCP_CLIENT_KEY_FILE \ --dry-run=client -o yaml | kubectl apply -f -
The MCP data plane picks up the corrected secret on its next periodic refresh, typically within a
few minutes, without a pod restart, and the ApigeeIssue is withdrawn on the next
scan.
Cause: MCP mTLS client certificate omits the clientAuth extended key usage
The certificate is well formed and pairs with its key, but its extended key usage (EKU) does not
include clientAuth, so it cannot be used for client authentication and the MCP data
plane does not present it. This is common when a server certificate, or a certificate from a
public CA profile that no longer issues clientAuth, is reused.
Diagnosis
Run the following command:
kubectl -n apigee describe apigeeissue ISSUE_NAME
Replace ISSUE_NAME with the name of the issue, for example
mcp-client-cert-invalid-my-org-my-virtualhost.
The output should look similar to the following:
Name: mcp-client-cert-invalid-my-org-my-virtualhost Namespace: apigee Labels: ais-reason=AIS_MCP_MTLS_CLIENT_CERT_INVALID Annotations: <none> API Version: apigee.cloud.google.com/v1alpha1 Kind: ApigeeIssue Metadata: Creation Timestamp: 2026-11-12T17:03:43Z Generation: 1 Owner References: API Version: apigee.cloud.google.com/v1alpha2 Kind: ApigeeOrganization Name: my-org UID: 7e83a52c-ce00-4bed-98be-55835ada1817 Resource Version: 3281563 UID: adc775c2-376d-4bf9-9860-500b2b2b8273 Spec: Details: MCP mTLS client certificate in secret "my-org-my-virtualhost-mcp" for ApigeeRouteConfig "my-org-my-virtualhost" is not usable: "cert" has extended key usage [serverAuth], which omits clientAuth Documentation: https://cloud.google.com/apigee/docs/hybrid/troubleshooting/AIS_MCP_MTLS_CLIENT_CERT_INVALID Reason: AIS_MCP_MTLS_CLIENT_CERT_INVALID Severity: ERROR Events: <none>
Spec: Details: MCP mTLS client certificate in secret "my-org-my-virtualhost-mcp" for ApigeeRouteConfig "my-org-my-virtualhost" is not usable: "cert" has extended key usage [serverAuth], which omits clientAuth
The contents of Spec.Details name the secret and the
ApigeeRouteConfig, and list the extended key usages the certificate does carry.
Resolution
-
Use
kubectl describeto display the name of the Kubernetes secret andApigeeRouteConfigthe issue belongs to:kubectl -n apigee describe apigeeissue ISSUE_NAME
Replace
ISSUE_NAMEwith the name of the issue, for examplemcp-client-cert-invalid-my-org-my-virtualhost.In this example, the
ApigeeRouteConfigmy-org-my-virtualhostis referencing the secretmy-org-my-virtualhost-mcp. -
Determine the
virtualhostname from theApigeeRouteConfig.The
ApigeeRouteConfig's name is in the format:<Apigee organization>-<virtualhost name>.In this example,
my-orgis the Apigee organization and the virtualhost name ismy-virtualhost. -
Find the corresponding virtualhost in your
overrides.yamland check which form itsmcpMtlsblock uses (see Configure mTLS between the MCP data plane and the ingress gateway). Select the tab that matches, and follow the steps in that tab.
Certificate files
When you use file paths (sslCertPath, sslKeyPath,
caCertPath), the virtualhost chart creates and manages the secrets from the
file contents.
virtualhosts:
...
- name: my-virtualhost
selector:
app: apigee-ingressgateway
tlsMode: MUTUAL
caCertPath: ./certs/ca.pem
sslCertPath: ./certs/vhost.pem
sslKeyPath: ./certs/vhost.key
mcpMtls:
caCertPath: ./certs/mcp-ca.pem
sslCertPath: ./certs/mcp-client.pem
sslKeyPath: ./certs/mcp-client.key
...-
Reissue the client certificate from your private CA with a profile that includes the
clientAuthextended key usage. Confirm it, where MCP_CLIENT_CERT_FILE is the reissued certificate, and then replace the files referenced bymcpMtls.sslCertPathandmcpMtls.sslKeyPath:openssl x509 -in MCP_CLIENT_CERT_FILE -noout -ext extendedKeyUsage # must list: TLS Web Client Authentication
-
Upgrade the virtualhost so that the chart rewrites the secret:
helm upgrade ENV_GROUP_NAME apigee-virtualhost/ \ --namespace apigee \ --set envgroup=ENV_GROUP_NAME \ -f overrides.yaml
Kubernetes secrets
When you use a Kubernetes secret reference (sslSecret), the MCP data plane
reads the certificate values directly from the secrets that you created, and updates
apply automatically. No helm upgrade is needed.
virtualhosts:
...
- name: my-virtualhost
selector:
app: apigee-ingressgateway
tlsMode: MUTUAL
sslSecret: my-ingress-secret
mcpMtls:
sslSecret: my-mcp-client-secret
...-
Reissue the client certificate from your private CA with a profile that includes the
clientAuthextended key usage. Confirm it before you update the secret, where MCP_CLIENT_CERT_FILE is the reissued certificate:openssl x509 -in MCP_CLIENT_CERT_FILE -noout -ext extendedKeyUsage # must list: TLS Web Client Authentication
-
Update the secret in place with the certificate and its private key together, where MCP_CLIENT_SECRET_NAME is the secret named in
Spec.Detailsand MCP_CLIENT_CERT_FILE and MCP_CLIENT_KEY_FILE are the certificate and its private key:kubectl -n apigee create secret generic MCP_CLIENT_SECRET_NAME \ --from-file=cert=MCP_CLIENT_CERT_FILE --from-file=key=MCP_CLIENT_KEY_FILE \ --dry-run=client -o yaml | kubectl apply -f -
The MCP data plane picks up the corrected secret on its next periodic refresh, typically within a
few minutes, without a pod restart, and the ApigeeIssue is withdrawn on the next
scan.
Must gather diagnostic information
If the problem persists even after following the preceding instructions, gather the following diagnostic information and then contact Google Cloud Customer Care:
- The Google Cloud project ID.
- The name of the Apigee hybrid organization.
- The
Spec.Detailsfield of theApigeeIssue. -
(optional) The certificate in the file referenced by
mcpMtls.sslCertPath, or in thecertentry of the secret, for the affected virtualhost.