This page applies to Apigee hybrid, but not to Apigee.
There is no equivalent Apigee Edge documentation for this topic.
Symptom
An ApigeeIssue with Reason AIS_MCP_MTLS_CA_CERT_INVALID
is shown when viewing resources in the apigee namespace.
This issue is raised only for environment groups whose mcpMtls block has chain
validation enabled (the default). For a group with
mcpMtls.disableChainValidation: true, the MCP data plane does not use a CA
certificate and the -cacert secret is not scanned.
Error messages
After running kubectl -n apigee get apigeeissues, an error like the following is
displayed:
NAME SEVERITY REASON DOCUMENTATION AGE mcp-ca-cert-invalid-my-org-my-virtualhost ERROR AIS_MCP_MTLS_CA_CERT_INVALID https://cloud.google.com/apigee/docs/hybrid/troubleshooting/AIS_MCP_MTLS_CA_CERT_INVALID 5h18m
Possible causes
The following table lists possible causes of this issue:
| Cause | Description |
|---|---|
| MCP mTLS CA certificate secret or entry is missing |
The -cacert secret does not exist, or it has no usable
cacert entry.
|
| MCP mTLS CA certificate is not PEM-encoded |
The cacert entry holds no PEM certificate block, or holds bytes outside
the PEM blocks.
|
| MCP mTLS CA certificate is malformed or is not a CA certificate |
A block in the cacert entry is not a certificate, does not parse, or is
not a CA certificate.
|
Cause: MCP mTLS CA certificate secret or entry is missing
The Kubernetes secret that should hold the CA certificate the MCP data plane validates the
ingress gateway against does not exist in the apigee namespace, or it exists
without a non-empty cacert entry. Its name is the client certificate secret's name
with -cacert appended. The chart creates this secret only when you use certificate
files; with sslSecret you must create it yourself, and forgetting it is the most
common cause of this issue.
Diagnosis
Run the following command:
kubectl -n apigee describe apigeeissue ISSUE_NAME
Replace ISSUE_NAME with the name of the issue, for example
mcp-ca-cert-invalid-my-org-my-virtualhost.
The output should look similar to the following:
Name: mcp-ca-cert-invalid-my-org-my-virtualhost Namespace: apigee Labels: ais-reason=AIS_MCP_MTLS_CA_CERT_INVALID Annotations: <none> API Version: apigee.cloud.google.com/v1alpha1 Kind: ApigeeIssue Metadata: Creation Timestamp: 2026-11-12T17:03:43Z Generation: 1 Owner References: API Version: apigee.cloud.google.com/v1alpha2 Kind: ApigeeOrganization Name: my-org UID: 7e83a52c-ce00-4bed-98be-55835ada1817 Resource Version: 3281563 UID: adc775c2-376d-4bf9-9860-500b2b2b8273 Spec: Details: MCP mTLS CA certificate in secret "my-org-my-virtualhost-mcp-cacert" for ApigeeRouteConfig "my-org-my-virtualhost" is not usable: secret not found: secrets "my-org-my-virtualhost-mcp-cacert" not found Documentation: https://cloud.google.com/apigee/docs/hybrid/troubleshooting/AIS_MCP_MTLS_CA_CERT_INVALID Reason: AIS_MCP_MTLS_CA_CERT_INVALID Severity: ERROR Events: <none>
Spec: Details: MCP mTLS CA certificate in secret "my-org-my-virtualhost-mcp-cacert" for ApigeeRouteConfig "my-org-my-virtualhost" is not usable: secret not found: secrets "my-org-my-virtualhost-mcp-cacert" not found
The contents of Spec.Details name the secret, the
ApigeeRouteConfig, and what is wrong. For example:
secret not foundsecret has no usable "cacert" entry
Resolution
-
Use
kubectl describeto display the name of the Kubernetes secret andApigeeRouteConfigthe issue belongs to:kubectl -n apigee describe apigeeissue ISSUE_NAME
Replace
ISSUE_NAMEwith the name of the issue, for examplemcp-ca-cert-invalid-my-org-my-virtualhost.In this example, the
ApigeeRouteConfigmy-org-my-virtualhostis referencing the secretmy-org-my-virtualhost-mcp-cacert. -
Determine the
virtualhostname from theApigeeRouteConfig.The
ApigeeRouteConfig's name is in the format:<Apigee organization>-<virtualhost name>.In this example,
my-orgis the Apigee organization and the virtualhost name ismy-virtualhost. -
Find the corresponding virtualhost in your
overrides.yamland check which form itsmcpMtlsblock uses (see Configure mTLS between the MCP data plane and the ingress gateway). Select the tab that matches, and follow the steps in that tab.
Certificate files
When you use file paths (sslCertPath, sslKeyPath,
caCertPath), the virtualhost chart creates and manages the secrets from the
file contents.
virtualhosts:
...
- name: my-virtualhost
selector:
app: apigee-ingressgateway
tlsMode: MUTUAL
caCertPath: ./certs/ca.pem
sslCertPath: ./certs/vhost.pem
sslKeyPath: ./certs/vhost.key
mcpMtls:
caCertPath: ./certs/mcp-ca.pem
sslCertPath: ./certs/mcp-client.pem
sslKeyPath: ./certs/mcp-client.key
...-
Check whether the secret exists and has a
cacertentry, where MCP_CA_SECRET_NAME is the secret named inSpec.Details:kubectl -n apigee describe secret MCP_CA_SECRET_NAME
If the secret is missing or lacks an entry, make sure that the file referenced by
mcpMtls.caCertPathexists and is not empty. The next step restores the secret from it. If you intend to run without chain validation, setmcpMtls.disableChainValidation: trueand removemcpMtls.caCertPath(the chart rejects the two together). Be aware that this turns off verification of the ingress gateway's certificate by the MCP data plane. -
Upgrade the virtualhost so that the chart rewrites the secret:
helm upgrade ENV_GROUP_NAME apigee-virtualhost/ \ --namespace apigee \ --set envgroup=ENV_GROUP_NAME \ -f overrides.yaml
Kubernetes secrets
When you use a Kubernetes secret reference (sslSecret), the MCP data plane
reads the certificate values directly from the secrets that you created, and updates
apply automatically. No helm upgrade is needed.
virtualhosts:
...
- name: my-virtualhost
selector:
app: apigee-ingressgateway
tlsMode: MUTUAL
sslSecret: my-ingress-secret
mcpMtls:
sslSecret: my-mcp-client-secret
...-
Confirm the secret exists under exactly that name with a
cacertentry, where MCP_CLIENT_SECRET_NAME is the client certificate secret named bymcpMtls.sslSecret:kubectl -n apigee describe secret MCP_CLIENT_SECRET_NAME-cacert
helm templatedoes not check that the secrets exist; onlyhelm installandhelm upgradedo, so a cleanhelm templaterun is not evidence that this secret is present. If you intend to run without chain validation, omitting the secret is not enough. SetmcpMtls.disableChainValidation: trueinoverrides.yamland upgrade the virtualhost. Be aware that this turns off verification of the ingress gateway's certificate by the MCP data plane. -
Create or update the CA certificate secret, where MCP_CLIENT_SECRET_NAME is the value of
mcpMtls.sslSecret, the CA certificate secret is that name with-cacertappended, and MCP_TRUST_CA_FILE is the CA certificate file:kubectl -n apigee create secret generic MCP_CLIENT_SECRET_NAME-cacert \ --from-file=cacert=MCP_TRUST_CA_FILE \ --dry-run=client -o yaml | kubectl apply -f -
The MCP data plane picks up the corrected secret on its next periodic refresh, typically within a
few minutes, without a pod restart, and the ApigeeIssue is withdrawn on the next
scan.
Cause: MCP mTLS CA certificate is not PEM-encoded
The cacert entry is missing the PEM-encoding text blocks, or it has content after
the last -----END CERTIFICATE----- line that is not PEM. The MCP data plane does
not load it.
Diagnosis
Run the following command:
kubectl -n apigee describe apigeeissue ISSUE_NAME
Replace ISSUE_NAME with the name of the issue, for example
mcp-ca-cert-invalid-my-org-my-virtualhost.
The output should look similar to the following:
Name: mcp-ca-cert-invalid-my-org-my-virtualhost Namespace: apigee Labels: ais-reason=AIS_MCP_MTLS_CA_CERT_INVALID Annotations: <none> API Version: apigee.cloud.google.com/v1alpha1 Kind: ApigeeIssue Metadata: Creation Timestamp: 2026-11-12T17:03:43Z Generation: 1 Owner References: API Version: apigee.cloud.google.com/v1alpha2 Kind: ApigeeOrganization Name: my-org UID: 7e83a52c-ce00-4bed-98be-55835ada1817 Resource Version: 3281563 UID: adc775c2-376d-4bf9-9860-500b2b2b8273 Spec: Details: MCP mTLS CA certificate in secret "my-org-my-virtualhost-mcp-cacert" for ApigeeRouteConfig "my-org-my-virtualhost" is not usable: holds 1289 trailing bytes that are not PEM Documentation: https://cloud.google.com/apigee/docs/hybrid/troubleshooting/AIS_MCP_MTLS_CA_CERT_INVALID Reason: AIS_MCP_MTLS_CA_CERT_INVALID Severity: ERROR Events: <none>
Spec: Details: MCP mTLS CA certificate in secret "my-org-my-virtualhost-mcp-cacert" for ApigeeRouteConfig "my-org-my-virtualhost" is not usable: holds 1289 trailing bytes that are not PEM
The contents of Spec.Details name the secret, the
ApigeeRouteConfig, and what is wrong. For example:
entry holds no PEM encoded certificateholds N trailing bytes that are not PEM
Resolution
-
Use
kubectl describeto display the name of the Kubernetes secret andApigeeRouteConfigthe issue belongs to:kubectl -n apigee describe apigeeissue ISSUE_NAME
Replace
ISSUE_NAMEwith the name of the issue, for examplemcp-ca-cert-invalid-my-org-my-virtualhost.In this example, the
ApigeeRouteConfigmy-org-my-virtualhostis referencing the secretmy-org-my-virtualhost-mcp-cacert. -
Determine the
virtualhostname from theApigeeRouteConfig.The
ApigeeRouteConfig's name is in the format:<Apigee organization>-<virtualhost name>.In this example,
my-orgis the Apigee organization and the virtualhost name ismy-virtualhost. -
Find the corresponding virtualhost in your
overrides.yamland check which form itsmcpMtlsblock uses (see Configure mTLS between the MCP data plane and the ingress gateway). Select the tab that matches, and follow the steps in that tab.
Certificate files
When you use file paths (sslCertPath, sslKeyPath,
caCertPath), the virtualhost chart creates and manages the secrets from the
file contents.
virtualhosts:
...
- name: my-virtualhost
selector:
app: apigee-ingressgateway
tlsMode: MUTUAL
caCertPath: ./certs/ca.pem
sslCertPath: ./certs/vhost.pem
sslKeyPath: ./certs/vhost.key
mcpMtls:
caCertPath: ./certs/mcp-ca.pem
sslCertPath: ./certs/mcp-client.pem
sslKeyPath: ./certs/mcp-client.key
...-
Validate the contents of the file referenced by
mcpMtls.caCertPath, and correct them if needed. Ensure the content contains only complete certificate blocks, each with its starting and ending lines. For example:-----BEGIN CERTIFICATE----- MIIDYTCCAkmgAwIBAgIUSXeU0pQYRFzYlqZpKhNNJdBLFBIwDQYJKoZIhvcNAQEL ... <contents omitted> ... QUJDREVGR0hJSktMTU5PUFFSU1RVVldYWVowMTIzNDU2Nzg5QUJDREVGR0hJSktM MMM= -----END CERTIFICATE-----
-
Upgrade the virtualhost so that the chart rewrites the secret:
helm upgrade ENV_GROUP_NAME apigee-virtualhost/ \ --namespace apigee \ --set envgroup=ENV_GROUP_NAME \ -f overrides.yaml
Kubernetes secrets
When you use a Kubernetes secret reference (sslSecret), the MCP data plane
reads the certificate values directly from the secrets that you created, and updates
apply automatically. No helm upgrade is needed.
virtualhosts:
...
- name: my-virtualhost
selector:
app: apigee-ingressgateway
tlsMode: MUTUAL
sslSecret: my-ingress-secret
mcpMtls:
sslSecret: my-mcp-client-secret
...-
Validate the contents of the CA certificate file (MCP_TRUST_CA_FILE) that you store in the
cacertentry of the secret, and correct them if needed. Ensure the content contains only complete certificate blocks, each with its starting and ending lines. For example:-----BEGIN CERTIFICATE----- MIIDYTCCAkmgAwIBAgIUSXeU0pQYRFzYlqZpKhNNJdBLFBIwDQYJKoZIhvcNAQEL ... <contents omitted> ... QUJDREVGR0hJSktMTU5PUFFSU1RVVldYWVowMTIzNDU2Nzg5QUJDREVGR0hJSktM MMM= -----END CERTIFICATE-----
-
Update the CA certificate secret in place, where MCP_CLIENT_SECRET_NAME is the value of
mcpMtls.sslSecret, the CA certificate secret is that name with-cacertappended, and MCP_TRUST_CA_FILE is the CA certificate file:kubectl -n apigee create secret generic MCP_CLIENT_SECRET_NAME-cacert \ --from-file=cacert=MCP_TRUST_CA_FILE \ --dry-run=client -o yaml | kubectl apply -f -
The MCP data plane picks up the corrected secret on its next periodic refresh, typically within a
few minutes, without a pod restart, and the ApigeeIssue is withdrawn on the next
scan.
Cause: MCP mTLS CA certificate is malformed or is not a CA certificate
Every PEM block in the cacert entry must be a CERTIFICATE block that
parses and has the CA basic constraint. A private key block, a leaf certificate (for example the
ingress gateway's own server certificate), or a corrupted block makes the entry unusable, and
the MCP data plane does not load it.
Diagnosis
Run the following command:
kubectl -n apigee describe apigeeissue ISSUE_NAME
Replace ISSUE_NAME with the name of the issue, for example
mcp-ca-cert-invalid-my-org-my-virtualhost.
The output should look similar to the following:
Name: mcp-ca-cert-invalid-my-org-my-virtualhost Namespace: apigee Labels: ais-reason=AIS_MCP_MTLS_CA_CERT_INVALID Annotations: <none> API Version: apigee.cloud.google.com/v1alpha1 Kind: ApigeeIssue Metadata: Creation Timestamp: 2026-11-12T17:03:43Z Generation: 1 Owner References: API Version: apigee.cloud.google.com/v1alpha2 Kind: ApigeeOrganization Name: my-org UID: 7e83a52c-ce00-4bed-98be-55835ada1817 Resource Version: 3281563 UID: adc775c2-376d-4bf9-9860-500b2b2b8273 Spec: Details: MCP mTLS CA certificate in secret "my-org-my-virtualhost-mcp-cacert" for ApigeeRouteConfig "my-org-my-virtualhost" is not usable: block 1 is not a CA certificate Documentation: https://cloud.google.com/apigee/docs/hybrid/troubleshooting/AIS_MCP_MTLS_CA_CERT_INVALID Reason: AIS_MCP_MTLS_CA_CERT_INVALID Severity: ERROR Events: <none>
Spec: Details: MCP mTLS CA certificate in secret "my-org-my-virtualhost-mcp-cacert" for ApigeeRouteConfig "my-org-my-virtualhost" is not usable: block 1 is not a CA certificate
The contents of Spec.Details name the secret, the
ApigeeRouteConfig, and what is wrong. For example:
-
no PEM block parsed as a certificate: ...: none of the certificate blocks can be parsed (for example, the only certificate is corrupted). -
block N is a "..." block, want CERTIFICATE: block N is not a certificate (for example, a private key). -
block N does not parse: block N is a corrupted certificate. -
block N is not a CA certificate: block N is a certificate, but not a CA (for example, a leaf certificate).
Blocks are numbered from 0.
Resolution
-
Use
kubectl describeto display the name of the Kubernetes secret andApigeeRouteConfigthe issue belongs to:kubectl -n apigee describe apigeeissue ISSUE_NAME
Replace
ISSUE_NAMEwith the name of the issue, for examplemcp-ca-cert-invalid-my-org-my-virtualhost.In this example, the
ApigeeRouteConfigmy-org-my-virtualhostis referencing the secretmy-org-my-virtualhost-mcp-cacert. -
Determine the
virtualhostname from theApigeeRouteConfig.The
ApigeeRouteConfig's name is in the format:<Apigee organization>-<virtualhost name>.In this example,
my-orgis the Apigee organization and the virtualhost name ismy-virtualhost. -
Find the corresponding virtualhost in your
overrides.yamland check which form itsmcpMtlsblock uses (see Configure mTLS between the MCP data plane and the ingress gateway). Select the tab that matches, and follow the steps in that tab.
Certificate files
When you use file paths (sslCertPath, sslKeyPath,
caCertPath), the virtualhost chart creates and manages the secrets from the
file contents.
virtualhosts:
...
- name: my-virtualhost
selector:
app: apigee-ingressgateway
tlsMode: MUTUAL
caCertPath: ./certs/ca.pem
sslCertPath: ./certs/vhost.pem
sslKeyPath: ./certs/vhost.key
mcpMtls:
caCertPath: ./certs/mcp-ca.pem
sslCertPath: ./certs/mcp-client.pem
sslKeyPath: ./certs/mcp-client.key
...-
Rebuild the file referenced by
mcpMtls.caCertPathso that it holds only the CA certificate (or chain) that signed the ingress gateway's certificate. Then confirm that every block is a CA certificate, where MCP_TRUST_CA_FILE is that file:openssl storeutl -noout -text -certs MCP_TRUST_CA_FILE | grep -E 'Subject:|CA:' # every certificate must show CA:TRUE
-
Upgrade the virtualhost so that the chart rewrites the secret:
helm upgrade ENV_GROUP_NAME apigee-virtualhost/ \ --namespace apigee \ --set envgroup=ENV_GROUP_NAME \ -f overrides.yaml
Kubernetes secrets
When you use a Kubernetes secret reference (sslSecret), the MCP data plane
reads the certificate values directly from the secrets that you created, and updates
apply automatically. No helm upgrade is needed.
virtualhosts:
...
- name: my-virtualhost
selector:
app: apigee-ingressgateway
tlsMode: MUTUAL
sslSecret: my-ingress-secret
mcpMtls:
sslSecret: my-mcp-client-secret
...-
Rebuild the CA certificate file (MCP_TRUST_CA_FILE) so that it holds only the CA certificate (or chain) that signed the ingress gateway's certificate. Then confirm that every block is a CA certificate:
openssl storeutl -noout -text -certs MCP_TRUST_CA_FILE | grep -E 'Subject:|CA:' # every certificate must show CA:TRUE
-
Update the CA certificate secret in place, where MCP_CLIENT_SECRET_NAME is the value of
mcpMtls.sslSecret, the CA certificate secret is that name with-cacertappended, and MCP_TRUST_CA_FILE is the CA certificate file:kubectl -n apigee create secret generic MCP_CLIENT_SECRET_NAME-cacert \ --from-file=cacert=MCP_TRUST_CA_FILE \ --dry-run=client -o yaml | kubectl apply -f -
The MCP data plane picks up the corrected secret on its next periodic refresh, typically within a
few minutes, without a pod restart, and the ApigeeIssue is withdrawn on the next
scan.
Must gather diagnostic information
If the problem persists even after following the preceding instructions, gather the following diagnostic information and then contact Google Cloud Customer Care:
- The Google Cloud project ID.
- The name of the Apigee hybrid organization.
- The
Spec.Detailsfield of theApigeeIssue. -
(optional) The certificate in the file referenced by
mcpMtls.caCertPath, or in thecacertentry of the secret, for the affected virtualhost.