MCP mTLS CA certificate expiry

This page applies to Apigee hybrid, but not to Apigee.

There is no equivalent Apigee Edge documentation for this topic.

Symptom

An ApigeeIssue with Reason AIS_MCP_MTLS_CA_CERT_EXPIRY is shown when viewing resources in the apigee namespace.

This issue is raised only for environment groups whose mcpMtls block has chain validation enabled (the default). For a group with mcpMtls.disableChainValidation: true, the MCP data plane does not use a CA certificate and the -cacert secret is not scanned.

Error messages

After running kubectl -n apigee get apigeeissues, an issue like the following is displayed:

NAME                                        SEVERITY   REASON                        DOCUMENTATION                                                                             AGE
mcp-ca-cert-expired-my-org-my-virtualhost   ERROR      AIS_MCP_MTLS_CA_CERT_EXPIRY   https://cloud.google.com/apigee/docs/hybrid/troubleshooting/AIS_MCP_MTLS_CA_CERT_EXPIRY   5h18m

Possible causes

The following table lists possible causes of this issue:

Cause Description
MCP mTLS CA certificate is expiring soon The configured MCP mTLS CA certificate is expiring within 14 days.
MCP mTLS CA certificate is expired The configured MCP mTLS CA certificate is expired.

Cause: MCP mTLS CA certificate is expiring soon

The MCP mTLS CA certificate stored in the Kubernetes secret referenced by an ApigeeRouteConfig is expiring within the next 14 days.

Diagnosis

Run the following command:

kubectl -n apigee describe apigeeissue ISSUE_NAME

Replace ISSUE_NAME with the name of the issue, for example mcp-ca-cert-expiring-soon-my-org-my-virtualhost.

The output should look similar to the following:

Name:         mcp-ca-cert-expiring-soon-my-org-my-virtualhost
Namespace:    apigee
Labels:       ais-reason=AIS_MCP_MTLS_CA_CERT_EXPIRY
Annotations:  <none>
API Version:  apigee.cloud.google.com/v1alpha1
Kind:         ApigeeIssue
Metadata:
  Creation Timestamp:  2026-11-12T17:03:43Z
  Generation:          1
  Owner References:
    API Version:     apigee.cloud.google.com/v1alpha2
    Kind:            ApigeeOrganization
    Name:            my-org
    UID:             7e83a52c-ce00-4bed-98be-55835ada1817
  Resource Version:  3281563
  UID:               adc775c2-376d-4bf9-9860-500b2b2b8273
Spec:
  Details:        MCP mTLS CA certificate in secret "my-org-my-virtualhost-mcp-cacert" for ApigeeRouteConfig "my-org-my-virtualhost" is expiring in 335h12m4s at 2026-11-26 16:15:47 +0000 UTC
  Documentation:  https://cloud.google.com/apigee/docs/hybrid/troubleshooting/AIS_MCP_MTLS_CA_CERT_EXPIRY
  Reason:         AIS_MCP_MTLS_CA_CERT_EXPIRY
  Severity:       WARNING
Events:           <none>
Spec:
  Details:        MCP mTLS CA certificate in secret "my-org-my-virtualhost-mcp-cacert" for ApigeeRouteConfig "my-org-my-virtualhost" is expiring in 335h12m4s at 2026-11-26 16:15:47 +0000 UTC

The contents of Spec.Details explain the name of the ApigeeRouteConfig that refers to the Kubernetes secret containing the MCP mTLS CA certificate which is expiring soon.

Resolution

  1. Use kubectl describe to display the name of the Kubernetes secret and ApigeeRouteConfig the issue belongs to:

    kubectl -n apigee describe apigeeissue ISSUE_NAME

    Replace ISSUE_NAME with the name of the issue, for example mcp-ca-cert-expiring-soon-my-org-my-virtualhost.

    In this example, the ApigeeRouteConfig my-org-my-virtualhost is referencing the secret my-org-my-virtualhost-mcp-cacert.

  2. Determine the virtualhost name from the ApigeeRouteConfig.

    The ApigeeRouteConfig's name is in the format: <Apigee organization>-<virtualhost name>.

    In this example, my-org is the Apigee organization and the virtualhost name is my-virtualhost.

  3. Find the corresponding virtualhost in your overrides.yaml and check which form its mcpMtls block uses (see Configure mTLS between the MCP data plane and the ingress gateway). Select the tab that matches, and follow the steps in that tab.

Certificate files

When you use file paths (sslCertPath, sslKeyPath, caCertPath), the virtualhost chart creates and manages the secrets from the file contents.

virtualhosts:
...
- name: my-virtualhost
  selector:
    app: apigee-ingressgateway
  tlsMode: MUTUAL
  caCertPath: ./certs/ca.pem
  sslCertPath: ./certs/vhost.pem
  sslKeyPath: ./certs/vhost.key
  mcpMtls:
    caCertPath: ./certs/mcp-ca.pem
    sslCertPath: ./certs/mcp-client.pem
    sslKeyPath: ./certs/mcp-client.key
...
  1. Obtain the renewed CA certificate and put it in the file referenced by mcpMtls.caCertPath. If the ingress gateway's certificate is still signed by the old CA, keep both the old and the new CA certificate in the file, and remove the old one only after the ingress gateway's certificate has been reissued under the new CA.

  2. Upgrade the virtualhost so that the chart rewrites the secret:

    helm upgrade ENV_GROUP_NAME apigee-virtualhost/ \
      --namespace apigee \
      --set envgroup=ENV_GROUP_NAME \
      -f overrides.yaml

Kubernetes secrets

When you use a Kubernetes secret reference (sslSecret), the MCP data plane reads the certificate values directly from the secrets that you created, and updates apply automatically. No helm upgrade is needed.

virtualhosts:
...
- name: my-virtualhost
  selector:
    app: apigee-ingressgateway
  tlsMode: MUTUAL
  sslSecret: my-ingress-secret
  mcpMtls:
    sslSecret: my-mcp-client-secret
...
  1. Obtain the renewed CA certificate. If the ingress gateway's certificate is still signed by the old CA, put both the old and the new CA certificate in the CA certificate file, and remove the old one only after the ingress gateway's certificate has been reissued under the new CA.

  2. Update the CA certificate secret in place, where MCP_CLIENT_SECRET_NAME is the value of mcpMtls.sslSecret, the CA certificate secret is that name with -cacert appended, and MCP_TRUST_CA_FILE is the CA certificate file:

    kubectl -n apigee create secret generic MCP_CLIENT_SECRET_NAME-cacert \
      --from-file=cacert=MCP_TRUST_CA_FILE \
      --dry-run=client -o yaml | kubectl apply -f -

The MCP data plane picks up the updated secret on its next periodic refresh, typically within a few minutes, without a pod restart, and the ApigeeIssue is withdrawn on the next scan.

Cause: MCP mTLS CA certificate is expired

The MCP mTLS CA certificate stored in the Kubernetes secret referenced by an ApigeeRouteConfig has expired.

Diagnosis

Run the following command:

kubectl -n apigee describe apigeeissue ISSUE_NAME

Replace ISSUE_NAME with the name of the issue, for example mcp-ca-cert-expired-my-org-my-virtualhost.

The output should look similar to the following:

Name:         mcp-ca-cert-expired-my-org-my-virtualhost
Namespace:    apigee
Labels:       ais-reason=AIS_MCP_MTLS_CA_CERT_EXPIRY
Annotations:  <none>
API Version:  apigee.cloud.google.com/v1alpha1
Kind:         ApigeeIssue
Metadata:
  Creation Timestamp:  2026-11-12T17:03:43Z
  Generation:          1
  Owner References:
    API Version:     apigee.cloud.google.com/v1alpha2
    Kind:            ApigeeOrganization
    Name:            my-org
    UID:             7e83a52c-ce00-4bed-98be-55835ada1817
  Resource Version:  3281563
  UID:               adc775c2-376d-4bf9-9860-500b2b2b8273
Spec:
  Details:        MCP mTLS CA certificate in secret "my-org-my-virtualhost-mcp-cacert" for ApigeeRouteConfig "my-org-my-virtualhost" expired at: 2026-11-10 17:03:43 +0000 UTC
  Documentation:  https://cloud.google.com/apigee/docs/hybrid/troubleshooting/AIS_MCP_MTLS_CA_CERT_EXPIRY
  Reason:         AIS_MCP_MTLS_CA_CERT_EXPIRY
  Severity:       ERROR
Events:           <none>
Spec:
  Details:        MCP mTLS CA certificate in secret "my-org-my-virtualhost-mcp-cacert" for ApigeeRouteConfig "my-org-my-virtualhost" expired at: 2026-11-10 17:03:43 +0000 UTC

The contents of Spec.Details explain the name of the ApigeeRouteConfig that refers to the Kubernetes secret containing the MCP mTLS CA certificate which has expired.

Resolution

  1. Use kubectl describe to display the name of the Kubernetes secret and ApigeeRouteConfig the issue belongs to:

    kubectl -n apigee describe apigeeissue ISSUE_NAME

    Replace ISSUE_NAME with the name of the issue, for example mcp-ca-cert-expired-my-org-my-virtualhost.

    In this example, the ApigeeRouteConfig my-org-my-virtualhost is referencing the secret my-org-my-virtualhost-mcp-cacert.

  2. Determine the virtualhost name from the ApigeeRouteConfig.

    The ApigeeRouteConfig's name is in the format: <Apigee organization>-<virtualhost name>.

    In this example, my-org is the Apigee organization and the virtualhost name is my-virtualhost.

  3. Find the corresponding virtualhost in your overrides.yaml and check which form its mcpMtls block uses (see Configure mTLS between the MCP data plane and the ingress gateway). Select the tab that matches, and follow the steps in that tab.

Certificate files

When you use file paths (sslCertPath, sslKeyPath, caCertPath), the virtualhost chart creates and manages the secrets from the file contents.

virtualhosts:
...
- name: my-virtualhost
  selector:
    app: apigee-ingressgateway
  tlsMode: MUTUAL
  caCertPath: ./certs/ca.pem
  sslCertPath: ./certs/vhost.pem
  sslKeyPath: ./certs/vhost.key
  mcpMtls:
    caCertPath: ./certs/mcp-ca.pem
    sslCertPath: ./certs/mcp-client.pem
    sslKeyPath: ./certs/mcp-client.key
...
  1. Obtain the renewed CA certificate and put only it in the file referenced by mcpMtls.caCertPath. Because the old CA has expired, the MCP data plane cannot validate the ingress gateway until the ingress gateway's certificate has also been reissued under the new CA; do that first if it has not been done.

  2. Upgrade the virtualhost so that the chart rewrites the secret:

    helm upgrade ENV_GROUP_NAME apigee-virtualhost/ \
      --namespace apigee \
      --set envgroup=ENV_GROUP_NAME \
      -f overrides.yaml

Kubernetes secrets

When you use a Kubernetes secret reference (sslSecret), the MCP data plane reads the certificate values directly from the secrets that you created, and updates apply automatically. No helm upgrade is needed.

virtualhosts:
...
- name: my-virtualhost
  selector:
    app: apigee-ingressgateway
  tlsMode: MUTUAL
  sslSecret: my-ingress-secret
  mcpMtls:
    sslSecret: my-mcp-client-secret
...
  1. Obtain the renewed CA certificate and put only it in the CA certificate file. Because the old CA has expired, the MCP data plane cannot validate the ingress gateway until the ingress gateway's certificate has also been reissued under the new CA; do that first if it has not been done.

  2. Update the CA certificate secret in place, where MCP_CLIENT_SECRET_NAME is the value of mcpMtls.sslSecret, the CA certificate secret is that name with -cacert appended, and MCP_TRUST_CA_FILE is the CA certificate file:

    kubectl -n apigee create secret generic MCP_CLIENT_SECRET_NAME-cacert \
      --from-file=cacert=MCP_TRUST_CA_FILE \
      --dry-run=client -o yaml | kubectl apply -f -

The MCP data plane picks up the updated secret on its next periodic refresh, typically within a few minutes, without a pod restart, and the ApigeeIssue is withdrawn on the next scan.

Must gather diagnostic information

If the problem persists even after following the preceding instructions, gather the following diagnostic information and then contact Google Cloud Customer Care:

  1. The Google Cloud project ID.
  2. The name of the Apigee hybrid organization.
  3. The Spec.Details field of the ApigeeIssue.
  4. (optional) The certificate in the file referenced by mcpMtls.caCertPath, or in the cacert entry of the secret, for the affected virtualhost.