*This page
applies to **Apigee** and **Apigee hybrid**.*

*View [Apigee Edge](https://docs.apigee.com/api-platform/troubleshoot/troubleshooting-introduction) documentation.*


The act of troubleshooting is both an art and a science. The constant effort
of Apigee technical support teams has been to demystify the art and expose
the science behind problem identification and resolution.

## What are playbooks?


Developed in collaboration with ApigeeTechnical Support teams, Apigee
troubleshooting **playbooks** are designed to provide quick and effective
solutions to errors or other issues that you may encounter when working with
Apigee products.

## Audience


Troubleshooting playbooks are intended for readers with a high-level
understanding of Apigee and its architecture, as well as some
understanding of basic concepts such as policies and analytics.


Some problems can be diagnosed and solved only by Apigee hybrid users
and may require knowledge of internal components such as Cassandra datastores, Message
Processors, and Synchronizers.


If you are on Apigee, then we clearly specify when you can perform the
indicated troubleshooting steps and when you need to contact [Google Cloud Customer Care](https://cloud.google.com/support-hub/)
for assistance.

## Playbooks

This section describes the current playbooks.


To filter this table, do one or more of the following: select a category,
select a product, type a search term, or click a column heading to sort.
Select category Apigee hybrid installation and upgrade Automated issue surfacing Cassandra Deployment Ingressgateway Logging Management/UI Network configuration Runtime Other Select product Apigee and Apigee hybrid Apigee hybrid only <br />

| Category | Playbook/Problem description | Error message | Playbook applicable for |
|---|---|---|---|
| Cassandra | [Cassandra TLS certificate verification failure](https://docs.cloud.google.com/apigee/docs/api-platform/troubleshoot/playbooks/cassandra/cassandra-tls-certificate-verification-failure.md.txt) | If the Apigee CA certificate does not match across clusters, then TLS certificate verification in Cassandra could fail. | Apigee hybrid only |
| Cassandra | [Troubleshooting Cassandra restore](https://docs.cloud.google.com/apigee/docs/api-platform/troubleshoot/playbooks/cassandra/ts-cassandra-restore.md.txt) | During the Cassandra restoration in Apigee hybrid, you may encounter errors in the restore logs. | Apigee hybrid only |
| Automated issue surfacing | [No network connectivity between runtime plane and control plane](https://docs.cloud.google.com/apigee/docs/api-platform/troubleshoot/playbooks/no-network-connectivity.md.txt) | Apigee API management requests fail: - API products, Developers, Apps do not get populated on the Apigee UI. - API proxy deployments do not complete. - Apigee API management requests fail. | Apigee hybrid only |
| Automated issue surfacing | [Virtual host missing environment group](https://docs.cloud.google.com/apigee/docs/api-platform/troubleshoot/playbooks/vh-missing-env-group.md.txt) | After running `kubectl -n apigee get apigeeissues`, the AIS_VIRTUALHOST_MISSING_ENVGROUP error is displayed. | Apigee hybrid only |
| Automated issue surfacing | [Virtual host missing selector](https://docs.cloud.google.com/apigee/docs/api-platform/troubleshoot/playbooks/vh-missing-selector.md.txt) | After running `kubectl -n apigee get apigeeissues`, the AIS_VIRTUALHOST_MISSING_SELECTOR error is displayed. | Apigee hybrid only |
| Automated issue surfacing | [Ingress cert mismatch](https://docs.cloud.google.com/apigee/docs/api-platform/troubleshoot/playbooks/ingress-cert-mismatch.md.txt) | After running `kubectl -n apigee get apigeeissues`, the AIS_INGRESS_CERT_MISMATCH error is displayed. | Apigee hybrid only |
| Automated issue surfacing | [Ingress cert expiry](https://docs.cloud.google.com/apigee/docs/api-platform/troubleshoot/playbooks/ingress-cert-expiry.md.txt) | After running `kubectl -n apigee get apigeeissues`, the AIS_INGRESS_CERT_EXPIREY error is displayed. | Apigee hybrid only |
| Automated issue surfacing | [Ingress mTLS CA cert expiry](https://docs.cloud.google.com/apigee/docs/api-platform/troubleshoot/playbooks/ca-cert-expiry.md.txt) | After running `kubectl -n apigee get apigeeissues`, the AIS_INGRESS_MTLS_CA_CERT_EXPIREY error is displayed. | Apigee hybrid only |
| Automated issue surfacing | [Ingress mTLS CA cert invalid](https://docs.cloud.google.com/apigee/docs/api-platform/troubleshoot/playbooks/ca-cert-invalid.md.txt) | After running `kubectl -n apigee get apigeeissues`, the AIS_INGRESS_MTLS_CA_CERT_INVALID error is displayed. | Apigee hybrid only |
| Cassandra | [Cassandra data replication failure](https://docs.cloud.google.com/apigee/docs/api-platform/troubleshoot/playbooks/cassandra/cassandra-data-replication-failure.md.txt) | When replicating data during a multi-region expansion, the `CassandraDataReplication` status may show an error state and data replication may fail. | Apigee hybrid only |
| Cassandra | [Cassandra Java heap space issues](https://docs.cloud.google.com/apigee/docs/api-platform/troubleshoot/playbooks/cassandra/cassandra-java-heap-space-issues.md.txt) | Cassandra heap issues may cause slowness in the Apigee hybrid proxy execution or even `Datastore` errors. Sometimes logs are an early indicator, even before the onset of symptoms. | Apigee hybrid only |
| Cassandra | [Cassandra pods not starting in the secondary region](https://docs.cloud.google.com/apigee/docs/api-platform/troubleshoot/playbooks/cassandra/cassandra-pods-not-starting-secondary-region.md.txt) | Cassandra pods fail to start in one of the regions in a multi-region Hybrid setup. You may see a `node already exists` error message in the Cassandra pod logs, or a `FailedPreStopHook` warning in the Cassandra pod status. | Apigee hybrid only |
| Cassandra | [Cassandra troubleshooting guide](https://docs.cloud.google.com/apigee/docs/api-platform/troubleshoot/playbooks/cassandra/ts-cassandra.md.txt) | When you use `kubectl` to view the pod states, you see that one or more Cassandra pods are stuck. This guide describes the diagnosis and resolution for problems with the Cassandra datastore. | Apigee hybrid only |
| Deployment | [API proxy deployments fail with no active runtime pods warning](https://docs.cloud.google.com/apigee/docs/api-platform/troubleshoot/playbooks/deployment/no-active-runtime-pods-warning.md.txt) | The **No active runtime pods** warning is displayed in the **Details** dialog next to the error message **Deployment issues on <var translate="no">ENVIRONMENT</var>: <var translate="no">REVISION_NUMBER</var>** on the API proxy page. | Apigee hybrid only |
| Ingressgateway | [API calls fail with timeout errors](https://docs.cloud.google.com/apigee/docs/api-platform/troubleshoot/playbooks/ingressgateway/api-calls-fail-timeout-errors.md.txt) | ``` curl: (7) Failed to connect to example.apis.com port 443: Operation timed out ``` > [!NOTE] > **Note:** Sample error message as observed with `curl` command. | Apigee hybrid only |
| Ingressgateway | [API Calls failing with TLS errors](https://docs.cloud.google.com/apigee/docs/api-platform/troubleshoot/playbooks/ingressgateway/api-calls-failing-tls-errors.md.txt) | ``` curl: (35) LibreSSL SSL_connect: SSL_ERROR_SYSCALL in connection to example.apis.com:443 ``` > [!NOTE] > **Note:** Sample error message as observed with `curl` command. | Apigee hybrid only |
| Logging | [Troubleshooting Apigee logs missing from Cloud Logging](https://docs.cloud.google.com/apigee/docs/api-platform/troubleshoot/playbooks/cloud-logging-issues.md.txt) | No error messages are known to be shown in this scenario. | Apigee and Apigee hybrid |
| Management/UI | [Inconsistent/no data observed for entities in hybrid UI or through Management APIs](https://docs.cloud.google.com/apigee/docs/api-platform/troubleshoot/playbooks/management-ui/inconsistent-data-hybrid-ui-mgmt-api.md.txt) | No error messages are known to be shown in this scenario. | Apigee hybrid only |
| Network configuration | [Access routing issues with Apigee](https://docs.cloud.google.com/apigee/docs/api-platform/troubleshoot/playbooks/network-configuration/access-routing-issues.md.txt) | External clients are not able to access/connect to Apigee in a desired manner. These include either network connectivity failures (TLS handshake fails) or `4xx/5xx` responses from Apigee. | Apigee and Apigee hybrid |
| Network configuration | [Apigee connectivity issue with northbound Private Service Connect](https://docs.cloud.google.com/apigee/docs/api-platform/troubleshoot/playbooks/network-configuration/psc.md.txt) | The northbound Private Service Connect setup is not working as expected, and the network endpoint group status remains in `Pending`. | Apigee only |
| Network configuration | [Apigee connectivity issues with southbound Private Service Connect targets](https://docs.cloud.google.com/apigee/docs/api-platform/troubleshoot/playbooks/network-configuration/connectivity-with-sb-psc-targets.md.txt) | A network connection issue or a TCP timeout between Apigee and the target service would show up as a `503` error response and would show an error similar to below if you create a debug session. ``` {"fault":{"faultstring":"The Service is temporarily unavailable","detail":{"errorcode":"messaging.adaptors.http.flow.ServiceUnavailable","reason":"TARGET_CONNECT_TIMEOUT"}}} ``` | Apigee and Apigee hybrid |
| Other | [Expanding Istio property replica counts when draining nodes](https://docs.cloud.google.com/apigee/docs/api-platform/troubleshoot/playbooks/expand-replica-count-to-drain-istio.md.txt) | When draining Istio pods some nodes may not drain because they have a replica count of 1, while 3 or more replicas are required. In order to avoid this, you should set the minimum replica count for each property to at least 3. | Apigee hybrid only |
| Other | [Message processor troubleshooting guide](https://docs.cloud.google.com/apigee/docs/api-platform/troubleshoot/playbooks/ts-message-processor.md.txt) | One or more apigee-runtime pods are not in the `Ready` state. When you use `kubectl` to describe a failed `apigee-runtime` pod, you see the error: ``` Readiness probe failed: HTTP probe failed with statuscode: 500 ``` | Apigee hybrid only |
| Other | [Print build info](https://docs.cloud.google.com/apigee/docs/api-platform/troubleshoot/playbooks/ts-buildinfo.md.txt) | The `buildinfo` API returns information about the current build for a runtime component. This information may be useful if you need to contact support. | Apigee hybrid only |
| Other | [StreamingPull errors 100%](https://docs.cloud.google.com/apigee/docs/api-platform/troubleshoot/playbooks/streaming-pull-error.md.txt) | If you see in your metrics dashboard that the method `google.pubsub.vl.Subscriber.StreamingPull` is failing with 100% errors, you can safely ignore the issue. This is expected behavior. | Apigee hybrid only |
| Deployment | [Instance is not reporting status for environment group](https://docs.cloud.google.com/apigee/docs/api-platform/troubleshoot/playbooks/deployment/apigee-instance-not-reporting-status.md.txt) | Deployments of API proxies fail with Instance <var translate="no">INSTANCE_NAME</var> is not reporting status for environment group <var translate="no">ENV_GROUP_NAME</var> error in the Apigee hybrid UI. | Apigee hybrid only |
| Deployment | [API proxy deployments fail with apigee-serving-cert is not found or expired](https://docs.cloud.google.com/apigee/docs/api-platform/troubleshoot/playbooks/deployment/apigee-serving-cert-not-found-or-expired.md.txt) | API proxy deployments fail with error messages in the `apigee-watcher` logs. | Apigee hybrid only |
| Ingressgateway | [Expand Istio property replica counts to avoid problems when draining Istio nodes](https://docs.cloud.google.com/apigee/docs/api-platform/troubleshoot/playbooks/expand-replica-count-to-drain-istio.md.txt) | When draining Istio pods some nodes may not drain because they have a replica count of `1`, while `3` or more replicas are required. In order to avoid this, you should set the minimum replica count for each property to at least `3`. | Apigee hybrid only |
| Network configuration | [No free IP address space troubleshooting](https://docs.cloud.google.com/apigee/docs/api-platform/troubleshoot/playbooks/network-configuration/no-free-ip-address-space.md.txt) | During Apigee provisioning, if you select a network CIDR range that is not completely free, you may see an error message. | Apigee and Apigee hybrid |
| Network configuration | [503 Service Unavailable error with TARGET_CONNECT_TIMEOUT (Internet and VPC peering targets)](https://docs.cloud.google.com/apigee/docs/api-platform/troubleshoot/playbooks/runtime/vpc-503-target-connect-timeout.md.txt) | This document describes how to diagnose and correct "503 Service Unavailable" errors with TARGET_CONNECT_TIMEOUT when using internet or VPC peering targets. | Apigee |
| Network configuration | [504 Gateway timeout - Target read timeout](https://docs.cloud.google.com/apigee/docs/api-platform/troubleshoot/playbooks/runtime/504-target-read-timeout.md.txt) | This document describes how to diagnose and correct "504 Gateway Timeout" errors with a TARGET_READ_TIMEOUT reason. | Apigee and Apigee hybrid |
| Runtime | [API requests fail with TARGET CONNECT HOST NOT REACHABLE error](https://docs.cloud.google.com/apigee/docs/api-platform/troubleshoot/playbooks/runtime/target-connect-host-not-reachable.md.txt) | This document describes how to diagnose and resolve HTTP 503 response status code errors with a "TARGET_CONNECT_HOST_NOT_REACHABLE" reason. | Apigee and Apigee hybrid |
| Runtime | [API requests fail with 502 Bad Gateway "Unexpected EOF at target" error](https://docs.cloud.google.com/apigee/docs/api-platform/troubleshoot/playbooks/runtime/unexpected-eof-at-target.md.txt) | This playbook describes how to diagnose and resolve 502 Bad Gateway Unexpected EOF errors. | Apigee and Apigee hybrid |
| Other | [Troubleshooting Apigee hybrid stuck in creating or releasing state](https://docs.cloud.google.com/apigee/docs/api-platform/troubleshoot/playbooks/troubleshooting-apigee-hybrid.md.txt) | This document describes how to reset Apigee hybrid components when they are stuck in a `creating` or `releasing` state. | Apigee hybrid only |
| Other | [Immediately delete an Apigee or Apigee hybrid organization](https://docs.cloud.google.com/apigee/docs/api-platform/troubleshoot/playbooks/delete-org.md.txt) | This document outlines how to immediately delete an Apigee or Apigee hybrid organization without retention using the `NO_RETENTION` option. | Apigee and Apigee hybrid |
| Apigee hybrid installation and upgrade | [Cassandra Pods in CrashLoopBackOff Status](https://docs.cloud.google.com/apigee/docs/api-platform/troubleshoot/playbooks/hybrid/cassandra-pod-in-crashloopbackoff-status.md.txt) | Cassandra pods are stuck in a CrashLoopBackOff state after installation or upgrade. | Apigee hybrid only |
| Apigee hybrid installation and upgrade | [Apigee Metrics pods in CrashLoopBackOff status](https://docs.cloud.google.com/apigee/docs/api-platform/troubleshoot/playbooks/hybrid/apigee-metrics-pods-in-crashloopbackoff.md.txt) | Apigee Metrics pods remain in a CrashLoopBackOff state after installation or upgrade. | Apigee hybrid only |
| Apigee hybrid installation and upgrade | [Apigee ingress gateway pods show 1 of 2 containers running](https://docs.cloud.google.com/apigee/docs/api-platform/troubleshoot/playbooks/hybrid/apigee-ingressgateway.md.txt) | Your `apigee-ingressgateway` pods show only 1 of 2 containers running when you get the pod listing. | Apigee hybrid only |
| Apigee hybrid installation and upgrade | [UDCA pod error](https://docs.cloud.google.com/apigee/docs/api-platform/troubleshoot/playbooks/hybrid/universal-data-collection-agent-pod-error.md.txt) | UDCA and Connect Agent pods throw a permission issue error. | Apigee hybrid only |
| Apigee hybrid installation and upgrade | [Connection resets during TLS handshake for non-SNI clients](https://docs.cloud.google.com/apigee/docs/api-platform/troubleshoot/playbooks/hybrid/connection-resets-during-tls-handshake.md.txt) | Connection resets are observed during TLS handshake for non-SNI clients. | Apigee hybrid only |
| Apigee hybrid installation and upgrade | [Insufficient Disk Space in Cassandra](https://docs.cloud.google.com/apigee/docs/api-platform/troubleshoot/playbooks/hybrid/insufficient-disk-space-cassandra.md.txt) | The Apigee hybrid installation process fails due to insufficient disk space for the Cassandra database. | Apigee hybrid only |
| Apigee hybrid installation and upgrade | [Insufficient CPU](https://docs.cloud.google.com/apigee/docs/api-platform/troubleshoot/playbooks/hybrid/insufficient-cpu.md.txt) | When you use `kubectl` to view the pod states, you will see one or more pods in the **CrashLoopBackoff** or **FailedScheduling** state. | Apigee hybrid only |