MCP mTLS client certificate expiry

This page applies to Apigee hybrid, but not to Apigee.

There is no equivalent Apigee Edge documentation for this topic.

Symptom

An ApigeeIssue with Reason AIS_MCP_MTLS_CLIENT_CERT_EXPIRY is shown when viewing resources in the apigee namespace.

Error messages

After running kubectl -n apigee get apigeeissues, an issue like the following is displayed:

NAME                                            SEVERITY   REASON                            DOCUMENTATION                                                                                 AGE
mcp-client-cert-expired-my-org-my-virtualhost   ERROR      AIS_MCP_MTLS_CLIENT_CERT_EXPIRY   https://cloud.google.com/apigee/docs/hybrid/troubleshooting/AIS_MCP_MTLS_CLIENT_CERT_EXPIRY   5h18m

Possible causes

The following table lists possible causes of this issue:

Cause Description
MCP mTLS client certificate is expiring soon The configured MCP mTLS client certificate is expiring within 14 days.
MCP mTLS client certificate is expired The configured MCP mTLS client certificate is expired.

Cause: MCP mTLS client certificate is expiring soon

The MCP mTLS client certificate stored in the Kubernetes secret referenced by an ApigeeRouteConfig is expiring within the next 14 days.

Diagnosis

Run the following command:

kubectl -n apigee describe apigeeissue ISSUE_NAME

Replace ISSUE_NAME with the name of the issue, for example mcp-client-cert-expiring-soon-my-org-my-virtualhost.

The output should look similar to the following:

Name:         mcp-client-cert-expiring-soon-my-org-my-virtualhost
Namespace:    apigee
Labels:       ais-reason=AIS_MCP_MTLS_CLIENT_CERT_EXPIRY
Annotations:  <none>
API Version:  apigee.cloud.google.com/v1alpha1
Kind:         ApigeeIssue
Metadata:
  Creation Timestamp:  2026-11-12T17:03:43Z
  Generation:          1
  Owner References:
    API Version:     apigee.cloud.google.com/v1alpha2
    Kind:            ApigeeOrganization
    Name:            my-org
    UID:             7e83a52c-ce00-4bed-98be-55835ada1817
  Resource Version:  3281563
  UID:               adc775c2-376d-4bf9-9860-500b2b2b8273
Spec:
  Details:        MCP mTLS client certificate in secret "my-org-my-virtualhost-mcp" for ApigeeRouteConfig "my-org-my-virtualhost" is expiring in 335h12m4s at 2026-11-26 16:15:47 +0000 UTC
  Documentation:  https://cloud.google.com/apigee/docs/hybrid/troubleshooting/AIS_MCP_MTLS_CLIENT_CERT_EXPIRY
  Reason:         AIS_MCP_MTLS_CLIENT_CERT_EXPIRY
  Severity:       WARNING
Events:           <none>
Spec:
  Details:        MCP mTLS client certificate in secret "my-org-my-virtualhost-mcp" for ApigeeRouteConfig "my-org-my-virtualhost" is expiring in 335h12m4s at 2026-11-26 16:15:47 +0000 UTC

The contents of Spec.Details explain the name of the ApigeeRouteConfig that refers to the Kubernetes secret containing the MCP mTLS client certificate which is expiring soon.

Resolution

  1. Use kubectl describe to display the name of the Kubernetes secret and ApigeeRouteConfig the issue belongs to:

    kubectl -n apigee describe apigeeissue ISSUE_NAME

    Replace ISSUE_NAME with the name of the issue, for example mcp-client-cert-expiring-soon-my-org-my-virtualhost.

    In this example, the ApigeeRouteConfig my-org-my-virtualhost is referencing the secret my-org-my-virtualhost-mcp.

  2. Determine the virtualhost name from the ApigeeRouteConfig.

    The ApigeeRouteConfig's name is in the format: <Apigee organization>-<virtualhost name>.

    In this example, my-org is the Apigee organization and the virtualhost name is my-virtualhost.

  3. Find the corresponding virtualhost in your overrides.yaml and check which form its mcpMtls block uses (see Configure mTLS between the MCP data plane and the ingress gateway). Select the tab that matches, and follow the steps in that tab.

Certificate files

When you use file paths (sslCertPath, sslKeyPath, caCertPath), the virtualhost chart creates and manages the secrets from the file contents.

virtualhosts:
...
- name: my-virtualhost
  selector:
    app: apigee-ingressgateway
  tlsMode: MUTUAL
  caCertPath: ./certs/ca.pem
  sslCertPath: ./certs/vhost.pem
  sslKeyPath: ./certs/vhost.key
  mcpMtls:
    caCertPath: ./certs/mcp-ca.pem
    sslCertPath: ./certs/mcp-client.pem
    sslKeyPath: ./certs/mcp-client.key
...
  1. Obtain a renewed MCP mTLS client certificate from your certificate authority, and replace the certificate and its private key together in the files referenced by mcpMtls.sslCertPath and mcpMtls.sslKeyPath.

  2. Upgrade the virtualhost so that the chart rewrites the secret:

    helm upgrade ENV_GROUP_NAME apigee-virtualhost/ \
      --namespace apigee \
      --set envgroup=ENV_GROUP_NAME \
      -f overrides.yaml

Kubernetes secrets

When you use a Kubernetes secret reference (sslSecret), the MCP data plane reads the certificate values directly from the secrets that you created, and updates apply automatically. No helm upgrade is needed.

virtualhosts:
...
- name: my-virtualhost
  selector:
    app: apigee-ingressgateway
  tlsMode: MUTUAL
  sslSecret: my-ingress-secret
  mcpMtls:
    sslSecret: my-mcp-client-secret
...
  1. Obtain a renewed MCP mTLS client certificate and its private key from your certificate authority.

  2. Update the secret in place with the certificate and its private key together, where MCP_CLIENT_SECRET_NAME is the secret named in Spec.Details and MCP_CLIENT_CERT_FILE and MCP_CLIENT_KEY_FILE are the certificate and its private key:

    kubectl -n apigee create secret generic MCP_CLIENT_SECRET_NAME \
      --from-file=cert=MCP_CLIENT_CERT_FILE --from-file=key=MCP_CLIENT_KEY_FILE \
      --dry-run=client -o yaml | kubectl apply -f -

The MCP data plane picks up the updated secret on its next periodic refresh, typically within a few minutes, without a pod restart, and the ApigeeIssue is withdrawn on the next scan.

Cause: MCP mTLS client certificate is expired

The MCP mTLS client certificate stored in the Kubernetes secret referenced by an ApigeeRouteConfig has expired.

Diagnosis

Run the following command:

kubectl -n apigee describe apigeeissue ISSUE_NAME

Replace ISSUE_NAME with the name of the issue, for example mcp-client-cert-expired-my-org-my-virtualhost.

The output should look similar to the following:

Name:         mcp-client-cert-expired-my-org-my-virtualhost
Namespace:    apigee
Labels:       ais-reason=AIS_MCP_MTLS_CLIENT_CERT_EXPIRY
Annotations:  <none>
API Version:  apigee.cloud.google.com/v1alpha1
Kind:         ApigeeIssue
Metadata:
  Creation Timestamp:  2026-11-12T17:03:43Z
  Generation:          1
  Owner References:
    API Version:     apigee.cloud.google.com/v1alpha2
    Kind:            ApigeeOrganization
    Name:            my-org
    UID:             7e83a52c-ce00-4bed-98be-55835ada1817
  Resource Version:  3281563
  UID:               adc775c2-376d-4bf9-9860-500b2b2b8273
Spec:
  Details:        MCP mTLS client certificate in secret "my-org-my-virtualhost-mcp" for ApigeeRouteConfig "my-org-my-virtualhost" expired at: 2026-11-10 17:03:43 +0000 UTC
  Documentation:  https://cloud.google.com/apigee/docs/hybrid/troubleshooting/AIS_MCP_MTLS_CLIENT_CERT_EXPIRY
  Reason:         AIS_MCP_MTLS_CLIENT_CERT_EXPIRY
  Severity:       ERROR
Events:           <none>
Spec:
  Details:        MCP mTLS client certificate in secret "my-org-my-virtualhost-mcp" for ApigeeRouteConfig "my-org-my-virtualhost" expired at: 2026-11-10 17:03:43 +0000 UTC

The contents of Spec.Details explain the name of the ApigeeRouteConfig that refers to the Kubernetes secret containing the MCP mTLS client certificate which has expired.

Resolution

  1. Use kubectl describe to display the name of the Kubernetes secret and ApigeeRouteConfig the issue belongs to:

    kubectl -n apigee describe apigeeissue ISSUE_NAME

    Replace ISSUE_NAME with the name of the issue, for example mcp-client-cert-expired-my-org-my-virtualhost.

    In this example, the ApigeeRouteConfig my-org-my-virtualhost is referencing the secret my-org-my-virtualhost-mcp.

  2. Determine the virtualhost name from the ApigeeRouteConfig.

    The ApigeeRouteConfig's name is in the format: <Apigee organization>-<virtualhost name>.

    In this example, my-org is the Apigee organization and the virtualhost name is my-virtualhost.

  3. Find the corresponding virtualhost in your overrides.yaml and check which form its mcpMtls block uses (see Configure mTLS between the MCP data plane and the ingress gateway). Select the tab that matches, and follow the steps in that tab.

Certificate files

When you use file paths (sslCertPath, sslKeyPath, caCertPath), the virtualhost chart creates and manages the secrets from the file contents.

virtualhosts:
...
- name: my-virtualhost
  selector:
    app: apigee-ingressgateway
  tlsMode: MUTUAL
  caCertPath: ./certs/ca.pem
  sslCertPath: ./certs/vhost.pem
  sslKeyPath: ./certs/vhost.key
  mcpMtls:
    caCertPath: ./certs/mcp-ca.pem
    sslCertPath: ./certs/mcp-client.pem
    sslKeyPath: ./certs/mcp-client.key
...
  1. Obtain a renewed MCP mTLS client certificate from your certificate authority, and replace the certificate and its private key together in the files referenced by mcpMtls.sslCertPath and mcpMtls.sslKeyPath.

  2. Upgrade the virtualhost so that the chart rewrites the secret:

    helm upgrade ENV_GROUP_NAME apigee-virtualhost/ \
      --namespace apigee \
      --set envgroup=ENV_GROUP_NAME \
      -f overrides.yaml

Kubernetes secrets

When you use a Kubernetes secret reference (sslSecret), the MCP data plane reads the certificate values directly from the secrets that you created, and updates apply automatically. No helm upgrade is needed.

virtualhosts:
...
- name: my-virtualhost
  selector:
    app: apigee-ingressgateway
  tlsMode: MUTUAL
  sslSecret: my-ingress-secret
  mcpMtls:
    sslSecret: my-mcp-client-secret
...
  1. Obtain a renewed MCP mTLS client certificate and its private key from your certificate authority.

  2. Update the secret in place with the certificate and its private key together, where MCP_CLIENT_SECRET_NAME is the secret named in Spec.Details and MCP_CLIENT_CERT_FILE and MCP_CLIENT_KEY_FILE are the certificate and its private key:

    kubectl -n apigee create secret generic MCP_CLIENT_SECRET_NAME \
      --from-file=cert=MCP_CLIENT_CERT_FILE --from-file=key=MCP_CLIENT_KEY_FILE \
      --dry-run=client -o yaml | kubectl apply -f -

The MCP data plane picks up the updated secret on its next periodic refresh, typically within a few minutes, without a pod restart, and the ApigeeIssue is withdrawn on the next scan.

Must gather diagnostic information

If the problem persists even after following the preceding instructions, gather the following diagnostic information and then contact Google Cloud Customer Care:

  1. The Google Cloud project ID.
  2. The name of the Apigee hybrid organization.
  3. The Spec.Details field of the ApigeeIssue.
  4. (optional) The certificate in the file referenced by mcpMtls.sslCertPath, or in the cert entry of the secret, for the affected virtualhost.