Select a documentation version: <button class="button button-with-icon cloud-select-dropdown__button button-primary" aria-haspopup="true">15.13.0</button>

- [Current (18.4.0)](https://docs.cloud.google.com/alloydb/omni/kubernetes/current/docs/access-crash-looping-database-pod.md.txt)
- [18.4.0](https://docs.cloud.google.com/alloydb/omni/kubernetes/18.4.0/docs/access-crash-looping-database-pod.md.txt)
- [18.3.0](https://docs.cloud.google.com/alloydb/omni/kubernetes/18.3.0/docs/access-crash-looping-database-pod.md.txt)
- [18.1.0](https://docs.cloud.google.com/alloydb/omni/kubernetes/18.1.0/docs/access-crash-looping-database-pod.md.txt)
- [17.10.0](https://docs.cloud.google.com/alloydb/omni/kubernetes/17.10.0/docs/access-crash-looping-database-pod.md.txt)
- [17.9.0](https://docs.cloud.google.com/alloydb/omni/kubernetes/17.9.0/docs/access-crash-looping-database-pod.md.txt)
- [17.7.0](https://docs.cloud.google.com/alloydb/omni/kubernetes/17.7.0/docs/access-crash-looping-database-pod.md.txt)
- [17.5.0](https://docs.cloud.google.com/alloydb/omni/kubernetes/17.5.0/docs/access-crash-looping-database-pod.md.txt)
- [16.14.0](https://docs.cloud.google.com/alloydb/omni/kubernetes/16.14.0/docs/access-crash-looping-database-pod.md.txt)
- [16.13.0](https://docs.cloud.google.com/alloydb/omni/kubernetes/16.13.0/docs/access-crash-looping-database-pod.md.txt)
- [16.11.0](https://docs.cloud.google.com/alloydb/omni/kubernetes/16.11.0/docs/access-crash-looping-database-pod.md.txt)
- [16.9.0](https://docs.cloud.google.com/alloydb/omni/kubernetes/16.9.0/docs/access-crash-looping-database-pod.md.txt)
- [15.18.0](https://docs.cloud.google.com/alloydb/omni/kubernetes/15.18.0/docs/access-crash-looping-database-pod.md.txt)
- [15.17.0](https://docs.cloud.google.com/alloydb/omni/kubernetes/15.17.0/docs/access-crash-looping-database-pod.md.txt)
- [15.15.0](https://docs.cloud.google.com/alloydb/omni/kubernetes/15.15.0/docs/access-crash-looping-database-pod.md.txt)
- [15.13.0](https://docs.cloud.google.com/alloydb/omni/kubernetes/15.13.0/docs/access-crash-looping-database-pod.md.txt)

<br />

This page describes how to resolve issues with AlloyDB Omni for
Kubernetes when you encounter the following situations:

- You have a crashlooping database cluster in Kubernetes, and you know the name and namespace of the database, its internal instance, and the pod.
- You need to access a file on the database pod, but the crash loop prevents access.

A *crashlooping* database pod is a pod that repeatedly starts, crashes, and is
then automatically restarted by Kubernetes. This cycle continues indefinitely
until the underlying issue is resolved. The database becomes unavailable and
unstable during this process.

## Before you begin

Before you follow the troubleshooting guidance in this document, try these
solutions:

- If high availability (HA) is available, trigger a failover. With HA, a synchronous standby instance is ready to replace the primary instance.
- Recover from a backup, if one is available. This approach is automated and less error prone.

## Cannot access files on a crashlooping database pod

**Description:** A database pod in your Kubernetes cluster enters a
`CrashLoopBackOff` state. This state prevents you from using `kubectl exec` to
access the database container's file system for debugging.

**Recommended fix:** To resolve this issue, temporarily stop the crash loop by
patching the pod's `StatefulSet` to override the container's command. Before
you proceed, consider simpler alternatives, such as triggering a failover or
recovering from a backup, if available.

To access the pod, follow these steps:

1. Pause all related user-facing AlloyDB Omni resources to
   prevent them from interfering with the recovery process.

   1. List all AlloyDB Omni Kubernetes custom resource definitions:

      ```bash
      kubectl get crd | grep alloydbomni.dbadmin
      ```
   2. Review the list of resources for your database cluster. Keep a list of
      the resources that you pause so that you can unpause them later. At a
      minimum, you must pause the `dbclusters.alloydbomni.dbadmin.goog` resource.

   3. Pause each resource separately by adding the `ctrlfwk.internal.gdc.goog/pause`
      annotation.

      ```bash
      kubectl annotate RESOURCE_TYPE -n NAMESPACE RESOURCE_NAME ctrlfwk.internal.gdc.goog/pause=true
      ```

      Replace the following:
      - `RESOURCE_TYPE`: the type of the resource to pause, for example, `dbclusters.alloydbomni.dbadmin.goog`.
      - `NAMESPACE`: the namespace of the database cluster.
      - `RESOURCE_NAME`: the name of the resource to pause.
2. Pause the internal instance that manages the crashlooping pod.

   1. Find the name of the internal instance.

      ```bash
      kubectl get instances.alloydbomni.internal.dbadmin.goog -n NAMESPACE
      ```

      Replace the following:
      - `NAMESPACE`: the name of the namespace that you're interested in.
   2. Pause the instance. The `ctrlfwk.internal.gdc.goog/pause=true` annotation
      tells the AlloyDB Omni controller to pause its management of the resource. This prevents the controller from automatically reconciling
      or changing the resource while you're manually performing troubleshooting steps, such as patching the underlying `StatefulSet`.

      ```bash
      kubectl annotate instances.alloydbomni.internal.dbadmin.goog -n NAMESPACE INSTANCE_NAME ctrlfwk.internal.gdc.goog/pause=true
      ```

      Replace the following:
      - `NAMESPACE`: the name of the namespace that you're interested in.
      - `INSTANCE_NAME`: the name of the instance that you found.
3. Prevent the pod from crashlooping.

   Patch the pod's `StatefulSet` to override the main container's command.
   This change stops the application from starting and crashing, which lets you
   access the pod.
   1. Confirm the name of the crashlooping pod.

      ```bash
      kubectl get pod -n NAMESPACE
      ```

      Replace `NAMESPACE`, which is the name of the namespace that you're interested in.
   2. Find the name of the `StatefulSet` that owns the pod.

      ```bash
      kubectl get pod -n NAMESPACE POD_NAME -ojsonpath='{.metadata.ownerReferences[0].name}'
      ```

      Replace the following:
      - `NAMESPACE`: the name of the namespace that you're interested in.
      - `POD_NAME`: the name of the crashlooping pod.
   3. Patch the `StatefulSet` to replace the database container's command with `sleep infinity`:

      ```bash
      kubectl patch statefulset STATEFULSET_NAME -n NAMESPACE --type='strategic' -p '
      {
        "spec": {
          "template": {
            "spec": {
              "containers": [
                {
                  "name": "database",
                  "command": ["sleep"],
                  "args": ["infinity"],
                  "livenessProbe": null,
                  "startupProbe": null
                }
              ]
            }
          }
        }
      }
      '
      ```

      Replace the following:
      - `NAMESPACE`: the name of the namespace that you're interested in.
      - `STATEFULSET_NAME`: the name of the `StatefulSet`.
   4. Delete the pod to apply the patch.

      ```bash
      kubectl delete pod -n NAMESPACE POD_NAME
      ```

      Replace the following:
      - `NAMESPACE`: the name of the namespace that you're interested in.
      - `POD_NAME`: the name of the crashlooping pod.
4. Access the database in the pod.

   ```bash
   kubectl exec -ti -n NAMESPACE POD_NAME -c database -- bash
   ```

   Replace the following:
   - `NAMESPACE`: the name of the namespace that you're interested in.
   - `POD_NAME`: the name of the crashlooping pod.

   After the new pod starts, it runs the `sleep` command instead of the
   database application. You can now access the pod's shell.
5. Unpause all components.

   After investigating, restore the original state by removing the `pause`
   annotations. You unpause the resources in the reverse order of pausing them;
   for example, unpause the instance before you unpause the DBCluster.
   This action reconciles the resources and restarts the database pod with its
   original command.
   1. For each resource that you paused, remove the annotation by appending a
      hyphen to the annotation name. The following example removes the
      annotation from an instance:

      ```bash
      kubectl annotate instances.alloydbomni.internal.dbadmin.goog -n NAMESPACE INSTANCE_NAME ctrlfwk.internal.gdc.goog/pause-
      ```

      Replace the following:
      - `NAMESPACE`: the name of the namespace that you're interested in.
      - `INSTANCE_NAME`: the name of the instance that you found in the previous step.
   2. Repeat this process for all other resources that you paused in the first
      step.