> [!CAUTION]
> You are currently viewing version 1.7 of the Apigee hybrid documentation. **This version is end of life.** You should upgrade to a newer version. For more information, see [Supported versions](https://docs.cloud.google.com/apigee/docs/hybrid/supported-platforms.md.txt#supported-versions).

<button class="button button-with-icon cloud-select-dropdown__button button-primary" aria-haspopup="true">Version 1.7 (unsupported)</button>

- Supported versions:
- [v1.17 (latest)](https://docs.cloud.google.com/apigee/docs/hybrid/v1.17/ts-message-processor.html)
- [v1.16](https://docs.cloud.google.com/apigee/docs/hybrid/v1.16/ts-message-processor.html)
- [v1.15](https://docs.cloud.google.com/apigee/docs/hybrid/v1.15/ts-message-processor.html)
- [v1.14](https://docs.cloud.google.com/apigee/docs/hybrid/v1.14/ts-message-processor.html)
- [List of supported versions](https://docs.cloud.google.com/apigee/docs/hybrid/supported-platforms.md.txt#supported-versions)
- Unsupported versions:
- [v1.13](https://docs.cloud.google.com/apigee/docs/hybrid/v1.13/ts-message-processor.html)
- [v1.12](https://docs.cloud.google.com/apigee/docs/hybrid/v1.12/ts-message-processor.html)
- [v1.11](https://docs.cloud.google.com/apigee/docs/hybrid/v1.11/ts-message-processor.html)
- [v1.10](https://docs.cloud.google.com/apigee/docs/hybrid/v1.10/ts-message-processor.html)
- [v1.9](https://docs.cloud.google.com/apigee/docs/hybrid/v1.9/ts-message-processor.html)
- [v1.8](https://docs.cloud.google.com/apigee/docs/hybrid/v1.8/ts-message-processor.md.txt)
- [v1.7](https://docs.cloud.google.com/apigee/docs/hybrid/v1.7/ts-message-processor.md.txt)
- [v1.6](https://docs.cloud.google.com/apigee/docs/hybrid/v1.6/ts-message-processor.md.txt)
- [v1.5](https://docs.cloud.google.com/apigee/docs/hybrid/v1.5/ts-message-processor.md.txt)
- [v1.4](https://docs.cloud.google.com/apigee/docs/hybrid/v1.4/ts-message-processor.md.txt)
- [v1.3](https://docs.cloud.google.com/apigee/docs/hybrid/v1.3/ts-message-processor.md.txt)
- [v1.2](https://docs.cloud.google.com/apigee/docs/hybrid/v1.2/ts-message-processor.md.txt)
- [v1.1](https://docs.cloud.google.com/apigee/docs/hybrid/v1.1/ts-message-processor.md.txt)


This topic discusses steps you can take to troubleshoot and fix problems with the
[message processor](https://docs.cloud.google.com/apigee/docs/hybrid/v1.7/what-is-hybrid.md.txt#message-processor). The message
processor is part of the [`apigee-runtime`](https://docs.cloud.google.com/apigee/docs/hybrid/v1.7/what-is-hybrid.md.txt#about-the-runtime-plane) component. See also
[Runtime service configuration overview](https://docs.cloud.google.com/apigee/docs/hybrid/v1.7/service-config.md.txt).

## Readiness probe fails with HTTP status
code 500

### Symptom


One or more `apigee-runtime` pods are not in the Ready state.

### Error message


When you use `kubectl` to describe a failed `apigee-runtime` pod, you see
the error:
`Readiness probe failed: HTTP probe failed with statuscode: 500`

For example:

```
kubectl describe pod -n hybrid \
apigee-runtime-apigee-gcp-prod1-test-blue-67db4f457b-9c7c7

...
apigee-runtime-apigee-gcp-prod1-test-blue-67db4f457b-9c7c7  Readiness probe failed: HTTP probe
failed with statuscode: 500
...
```

### Possible causes


The error means that no active [contract](https://docs.cloud.google.com/apigee/docs/hybrid/v1.7/what-is-hybrid.md.txt#synchronizer) is available for the message processor to serve the
traffic. In this state, the message processor cannot call itself "ready".

| Cause | Description |
|---|---|
| [Synchronizer to management plane connection problem](https://docs.cloud.google.com/apigee/docs/hybrid/v1.7/ts-message-processor.md.txt#diagnosis-synchronizer-to-management-plane-problem) | The synchronizer is unable to connect to the management plane. This problem is usually caused in cases where you override the `contractProvider` URL and associate the wrong service account with it. For example, if you configure a service account for a staging deployment with a `contractProvider` URL that points to the production server. |
| [Message processor to synchronizer connection problem](https://docs.cloud.google.com/apigee/docs/hybrid/v1.7/ts-message-processor.md.txt#diagnosis-message-processor-to-synchronizer-problem) | If the new MP comes up as part of an autoscale or pod restart, you might see this error. Generally, this problem occurs when the synchronizer is down and the MP was unable to load its [contract](https://docs.cloud.google.com/apigee/docs/hybrid/v1.7/what-is-hybrid.md.txt#synchronizer). |

### Diagnosis: Synchronizer to management
plane connection problem


To diagnose a **synchronizer to management plane connection problem**, do the following:

1. List the pods in the cluster:

   ```
   kubectl get pods -n namespace
   ```
2. Open a shell in an `apigee-synchronizer` pod:

   ```
   kubectl exec -it -n namespace synchronizer-pod-name bash
   ```

   For example:

   ```
   kubectl exec -it -n apigee apigee-synchronizer-apigee-gcp-prod1-test-blue-cnj5x bash
   ```
3. Go to the following directory:

       cd /application/opt/apigee/var/log/apigee-synchronizer
       ls

         dr-xr-sr-x 4 apigee apigee  4096 Sep 12 16:52 750
         drwxr-sr-x 2 apigee apigee  4096 Sep 12 16:52 cachedFiles
         -rw-r--r-- 1 apigee apigee 22295 Sep 12 16:52 config.log
         -rw-r--r-- 1 apigee apigee    76 Sep 12 16:52 versions.properties

4. Check the active version in `version.properties`. For example:

   ```
   cat versions.properties

     #active repository version
     #Sat Dec 14 19:45:00 GMT 2019
     active.version=749
   ```
5. Be sure the value of `active.version` matches the name of the contract folder number. In the above example (also shown below), the folder name is ` 750`; therefore, they do not match:

   ```
   dr-xr-sr-x 4 apigee apigee  4096 Sep 12 16:52 750
   ```
6. Exit the pod shell.

### Resolution

If `version.properties` is missing, or if there is a version mis-match as explained
above, check the [synchronizer logs](https://docs.cloud.google.com/apigee/docs/hybrid/v1.7/sync-logs.md.txt) to try to determine why the
most current contracts are not being downloaded. For example:

```
kubectl logs -f -n namespace synchronizer-pod-name
```


For information on interpreting the synchronizer logs, see [Synchronizer logs](https://docs.cloud.google.com/apigee/docs/hybrid/v1.7/sync-logs.md.txt).
If the synchronizer is down, try restarting it using `apigeectl`. For example:

```
$APIGEECTL_HOME/apigeectl apply -f overrides/overrides.yaml --org --env env-name
```

### Diagnosis: Message processor to synchronizer connection problem


To diagnose a **message processor to synchronizer connection problem**, do the following:

1. List the pods in the cluster:

   ```
   kubectl get pods -n namespace
   ```
2. Check the [runtime logs](https://docs.cloud.google.com/apigee/docs/hybrid/v1.7/runtime-logs.md.txt) to try to figure out why the contracts are not being downloaded:

   ```
   kubectl logs -f -n namespace pod-name
   ```

   For example:

   ```
   kubectl logs -f -n apigee apigee-runtime-apigee-gcp-prod1-test-blue-67db4f457b-9c7c7
   ```


   It is possible if the new MP comes up as part of an autoscale or pod restart, that the
   MP might not load the contracts. Generally, the problem occurs when the synchronizer
   is down, preventing the MP from loading the contracts. For example:

   ```
   2019-09-13 16:59:24,331 podName:N/A ipAddress:N/A dpColor:N/A
   HttpClient@331886186-301 DEBUG o.e.j.c.AbstractConnectionPool - AbstractConnectionPool$1.failed() :
   Connection 1/64 creation failed
   java.net.UnknownHostException: apigee-synchronizer-apigee-gcp-prod1-test.hybrid.svc.cluster.local
   at java.net.InetAddress.getAllByName0(InetAddress.java:1281)
   at java.net.InetAddress.getAllByName(InetAddress.java:1193)
   at java.net.InetAddress.getAllByName(InetAddress.java:1127)
   at org.eclipse.jetty.util.SocketAddressResolver$Async.lambda$resolve$1(SocketAddressResolver.java:167)
   at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:672)
   at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:590)
   at java.lang.Thread.run(Thread.java:748)
   	at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:590)
   	at java.lang.Thread.run(Thread.java:748)
   ```

### Resolution


Try restarting the synchronizer. For example:

```
$APIGEECTL_HOME/apigeectl apply -f overrides/overrides.yaml --org --env env-name
```

## Readiness probe fails for an invalid encryption key

### Symptom

The `apigee-runtime` pods are not in the Ready state.

### Diagnosis


When you use `kubectl` to describe a failed `apigee-runtime` pod, you see
this error: `Readiness probe failed: Probe hybrid-encryption-key-validation-probe failed`.
For example:

```
$ kubectl describe pod -n namespace apigee-runtime-pod-name

...
Readiness probe failed: Probe hybrid-encryption-key-validation-probe failed due to
com.apigee.probe.model.ProbeFailedException{ code = hybrid.encryption.key.InvalidEncryptionKey,
message = Invalid encryption key. Please check the length of the encryption key, associated
contexts = []}
...
```

### Resolution

Supported encryption key lengths are 16 or 24 or 32 bytes and the key's value has to be
base64 encoded. For more information on creating a properly formatted key, see
[Data encryption](https://docs.cloud.google.com/apigee/docs/hybrid/v1.7/key-encryption.md.txt).

## Viewing message processor logs


For details about viewing and interpreting message processor logs,
see [Runtime logs](https://docs.cloud.google.com/apigee/docs/hybrid/v1.7/runtime-logs.md.txt).

## Call an API proxy from the runtime pod


In some situations to help isolate a problem, you might want to check if you can make an API proxy call directly from
inside the `apigee-runtime` pod, and thereby bypassing the ingress gateway.

1. Execute the following command to forward to port 8443. This allows you to call the API in the pod:

   ```
   kubectl port-forward -n namespace apigee-runtime-pod-name 8443:8443
   ```
2. Call a deployed API proxy. For example, where the proxy basepath is `ilove-apis`:

   ```
   curl -k -v https://0:8443//ilove-apis

       < HTTP/1.1 200 OK
       < X-Powered-By: Apigee
       < Access-Control-Allow-Origin: *
       < X-Frame-Options: ALLOW-FROM RESOURCE-URL
       < X-XSS-Protection: 1
       < X-Content-Type-Options: nosniff
       < Content-Type: text/html; charset=utf-8
       < Content-Length: 18
       < ETag: W/"12-Jb9QP1bUxNSmZkxQGt5KLQ"
       < Date: Fri, 13 Sep 2019 18:33:46 GMT
       < Via: 1.1 google
       < X-Apigee.Message-ID: 016f5f7f-c59e-404c-86e8-7b0737833f982
       < X-Apigee.dp.color: blue
       < X-Apigee.proxy: /organizations/my-org/environments/test/apiproxies/i-loveapis/revisions/1
       < X-Apigee.proxy.basepath: /ilove-apis
       < X-Apigee.target-latency: 9
       <
       * Connection #0 to host 0 left intact
       <H2>I <3 APIs
   ```

## Check the management API

You can use the API described below to check if the management API is working properly.

1. Get the names of the pods in your cluster:

   ```
   kubectl get pods -n namespace
   ```
2. Use [port-forwarding](https://kubernetes.io/docs/tasks/access-application-cluster/port-forward-access-application-cluster/) to gain access the `apigee-runtime` pod. The syntax for port forwarding is as follows:

   ```
   kubectl port-forward -n namespace podname 8843:8843
   ```

   For example:

   ```
   kubectl port-forward -n apigee \
       apigee-runtime-my-organization-test-blue-57965b7789-6j4bp 8843:8843
   ```
3. Then, in another terminal window, use a utility such as `curl` to send a request to the `classification/tree` API, as the following example shows:

   ```
   curl -k https://0:8843/v1/classification/tree
   ```

   Here is an example response, which lists information about the deployed proxies in the
   "test" environment:

   ```
   [ {
     "condition" : "(always matches)",
     "virtualHost" : {
       "env" : "test",
       "name" : "default",
       "org" : "my-organization",
       "tree" : {
         "elements" : [ {
           "application" : "myproxy",
           "basePath" : "/myproxy",
           "name" : "default",
           "revision" : "1"
         } ],
         "name" : "IdentificationTree"
       }
     }
   } ]
   ```

## Use logging to debug runtime pod problems


To help with troubleshooting, you can create a log session to produce detailed log output
for the `apigee-runtime` pods.

### Create a log session

To create a log session for a runtime pod:

1. List the runtime pods. By default, they are in the `apigee` namespace. If you chose a different namespace, use that instead:

   ```
   kubectl get pods -n apigee
   ```
2. Pick one of the listed `apigee-runtime` pods to debug.
3. Create a log session in the runtime pod you wish to troubleshoot using the `/logsessions` API. For details on the API, see [About the logsessions API](https://docs.cloud.google.com/apigee/docs/hybrid/v1.7/ts-message-processor.md.txt#aboutapi):

   ```
   kubectl exec -it RUNTIME_POD_NAME -n apigee \
     -- curl "https://0:8843/v1/logsessions?loggerNames=ALL&level=FINE&timeout=60000" -k -X POST -v
   ```

   For example:

   ```
   kubectl exec -it apigee-runtime-hybrid-18-01232-dev-d27ca57-190rc6-3klyg-lc7h5 -n apigee \
     -- curl "https://0:8843/v1/logsessions?loggerNames=ALL&level=FINE&timeout=60000" -k -X POST -v
   ```

   The response is a log session ID. For example: `9f831a7d-e533-4ead-8e58-12ec1059a622`
4. Print the logs:

   ```
   kubectl logs -f -n apigee RUNTIME_POD_NAME
   ```

### List log sessions

To get a list of all active log sessions for a runtime pod:

```
kubectl exec -it RUNTIME_POD_NAME -n apigee \
  -- curl "https://0:8843/v1/logsessions" -k -v
```

### Get a log session

To get log session details:

```
kubectl exec -it RUNTIME_POD_NAME -n apigee \
  -- curl "https://0:8843/v1/logsessions/LOG_SESSION_ID" -k -v
```

### End a log session

To end a log session:

```
kubectl exec -it RUNTIME_POD_NAME -n apigee \
    -- curl "https://0:8843/v1/logsessions/LOG_SESSION_ID" -k -X DELETE -v
```

### About the logsessions API

This section describes the logsessions API query parameters:

- `loggerNames`: Specifies the type of log output to show. Possible values include `CONTRACT_REPLICATION`, `DEBUG.SESSION`, or `ALL`
- `level`: Specifies the level of log output. Possible values include `FINEST`, `FINER`, `FINE`, `INFO`, `SEVERE`, `ALL`.
- `timeout`: Specifies the amount of time the log session will operate for the specified log level. After timeout, the log level reverts to `INFO`.

On success, the API returns a session ID for the log session.