Select a documentation version: <button class="button button-with-icon cloud-select-dropdown__button button-primary" aria-haspopup="true">16.3.0</button>

- [Current (18.4.0)](https://docs.cloud.google.com/alloydb/omni/kubernetes/current/docs/troubleshoot-kubernetes-operator.md.txt)
- [18.4.0](https://docs.cloud.google.com/alloydb/omni/kubernetes/18.4.0/docs/troubleshoot-kubernetes-operator.md.txt)
- [18.3.0](https://docs.cloud.google.com/alloydb/omni/kubernetes/18.3.0/docs/troubleshoot-kubernetes-operator.md.txt)
- [18.1.0](https://docs.cloud.google.com/alloydb/omni/kubernetes/18.1.0/docs/troubleshoot-kubernetes-operator.md.txt)
- [17.10.0](https://docs.cloud.google.com/alloydb/omni/kubernetes/17.10.0/docs/troubleshoot-kubernetes-operator.md.txt)
- [17.9.0](https://docs.cloud.google.com/alloydb/omni/kubernetes/17.9.0/docs/troubleshoot-kubernetes-operator.md.txt)
- [17.7.0](https://docs.cloud.google.com/alloydb/omni/kubernetes/17.7.0/docs/troubleshoot-kubernetes-operator.md.txt)
- [17.5.0](https://docs.cloud.google.com/alloydb/omni/kubernetes/17.5.0/docs/troubleshoot-kubernetes-operator.md.txt)
- [16.14.0](https://docs.cloud.google.com/alloydb/omni/kubernetes/16.14.0/docs/troubleshoot-kubernetes-operator.md.txt)
- [16.13.0](https://docs.cloud.google.com/alloydb/omni/kubernetes/16.13.0/docs/troubleshoot-kubernetes-operator.md.txt)
- [16.11.0](https://docs.cloud.google.com/alloydb/omni/kubernetes/16.11.0/docs/troubleshoot-kubernetes-operator.md.txt)
- [16.9.0](https://docs.cloud.google.com/alloydb/omni/kubernetes/16.9.0/docs/troubleshoot-kubernetes-operator.md.txt)
- [16.8.0](https://docs.cloud.google.com/alloydb/omni/kubernetes/16.8.0/docs/troubleshoot-kubernetes-operator.md.txt)
- [16.3.0](https://docs.cloud.google.com/alloydb/omni/kubernetes/16.3.0/docs/troubleshoot-kubernetes-operator.md.txt)
- [15.18.0](https://docs.cloud.google.com/alloydb/omni/kubernetes/15.18.0/docs/troubleshoot-kubernetes-operator.md.txt)
- [15.17.0](https://docs.cloud.google.com/alloydb/omni/kubernetes/15.17.0/docs/troubleshoot-kubernetes-operator.md.txt)
- [15.15.0](https://docs.cloud.google.com/alloydb/omni/kubernetes/15.15.0/docs/troubleshoot-kubernetes-operator.md.txt)
- [15.13.0](https://docs.cloud.google.com/alloydb/omni/kubernetes/15.13.0/docs/troubleshoot-kubernetes-operator.md.txt)
- [15.12.0](https://docs.cloud.google.com/alloydb/omni/kubernetes/15.12.0/docs/troubleshoot-kubernetes-operator.md.txt)
- [15.7.1](https://docs.cloud.google.com/alloydb/omni/kubernetes/15.7.1/docs/troubleshoot-kubernetes-operator.md.txt)

<br />

This page shows you how to resolve issues with the AlloyDB Omni Kubernetes operator, including workarounds for [known issues](https://docs.cloud.google.com/alloydb/omni/kubernetes/16.3.0/docs/troubleshoot-kubernetes-operator.md.txt#known-issues).

<br />

## Collect debugging information

These sections describe how to collect logs and configurations for debugging.

### Fetch logs from operator pods

To fetch logs from the operator pods, run the following commands:

```bash
kubectl logs deployments/fleet-controller-manager -c fleet-manager -n alloydb-omni-system > alloydb-omni-system-fleet-controller-manager.out
kubectl logs deployments/local-controller-manager -c local-manager -n alloydb-omni-system > alloydb-omni-system-local-controller-manager.out
```

### Fetch database pod logs

To fetch database pod logs, run the following commands:

```bash
DB_POD=$(kubectl get pod -l alloydbomni.internal.dbadmin.goog/dbcluster=DB_CLUSTER_NAME,alloydbomni.internal.dbadmin.goog/task-type=database -n DB_CLUSTER_NAMESPACE -o jsonpath='{.items[0].metadata.name}')
kubectl logs -c database ${DB_POD} -n DB_CLUSTER_NAMESPACE > ${DB_POD}.log
kubectl logs -l alloydbomni.internal.dbadmin.goog/dbcluster=DB_CLUSTER_NAME -c database -n DB_CLUSTER_NAMESPACE > dbcluster_DB_CLUSTER_NAME.out
```

The following logs are examples of successful database health checks:

    I0813 11:01:49.210051      27 gateway.go:184] "DatabaseHealthCheck: request handled successfully" log_name="agent" project_ns="default" dbcluster="adb"
    I0813 11:01:59.196796      27 gateway.go:166] "DatabaseHealthCheck: handling request" log_name="agent" project_ns="default" dbcluster="adb"
    I0813 11:01:59.196853      27 database.go:702] "dbdaemon/isRestoreInProgress: starting" log_name="agent" project_ns="default" dbcluster="adb"
    I0813 11:01:59.209824      27 gateway.go:184] "DatabaseHealthCheck: request handled successfully" log_name="agent" project_ns="default" dbcluster="adb"
    I0813 11:02:09.197013      27 gateway.go:166] "DatabaseHealthCheck: handling request" log_name="agent" project_ns="default" dbcluster="adb"
    I0813 11:02:09.197093      27 database.go:702] "dbdaemon/isRestoreInProgress: starting" log_name="agent" project_ns="default" dbcluster="adb"
    I0813 11:02:09.210010      27 gateway.go:184] "DatabaseHealthCheck: request handled successfully" log_name="agent" project_ns="default" dbcluster="adb"
    I0813 11:02:19.197368      27 gateway.go:166] "DatabaseHealthCheck: handling request" log_name="agent" project_ns="default" dbcluster="adb"
    I0813 11:02:19.197425      27 database.go:702] "dbdaemon/isRestoreInProgress: starting" log_name="agent" project_ns="default" dbcluster="adb"
    I0813 11:02:19.210416      27 gateway.go:184] "DatabaseHealthCheck: request handled successfully" log_name="agent" project_ns="default" dbcluster="adb"

### Fetch the postgresql.log

To fetch the `postgresql.log`, run the following command:

```bash
DB_POD=$(kubectl get pod -l alloydbomni.internal.dbadmin.goog/dbcluster=DB_CLUSTER_NAME,alloydbomni.internal.dbadmin.goog/task-type=database -n DB_CLUSTER_NAMESPACE -o jsonpath='{.items[0].metadata.name}')
kubectl exec -c database -n DB_CLUSTER_NAMESPACE -it ${DB_POD} -- cat /obs/diagnostic/postgresql.log > dbcluster_DB_CLUSTER_NAME_postgresql.log
```

### Fetch the DBInstance YAML file

To fetch the DBInstance YAML file, run the following command:

```bash
kubectl get dbclusters.alloydbomni.dbadmin.goog DB_CLUSTER_NAME -n DB_CLUSTER_NAMESPACE -o yaml > dbcluster_DB_CLUSTER_NAME.yaml
```

### Fetch configurations and logs for HA scenarios

To fetch configurations and logs specific to high availability (HA) scenarios,
run the following commands:

```bash
kubectl get replicationconfig.alloydbomni.internal.dbadmin.goog -l alloydbomni.internal.dbadmin.goog/dbcluster=DB_CLUSTER_NAME -n DB_CLUSTER_NAMESPACE -o yaml > replicationconfig_DB_CLUSTER_NAME.yaml
kubectl get deletestandbyjobs.alloydbomni.internal.dbadmin.goog -l alloydbomni.internal.dbadmin.goog/dbcluster=DB_CLUSTER_NAME -n DB_CLUSTER_NAMESPACE -o yaml > deletestandbyjobs_DB_CLUSTER_NAME.yaml
kubectl get createstandbyjobs.alloydbomni.internal.dbadmin.goog -l alloydbomni.internal.dbadmin.goog/dbcluster=DB_CLUSTER_NAME -n DB_CLUSTER_NAMESPACE -o yaml > createstandbyjobs_DB_CLUSTER_NAME.yaml
kubectl get failovers.alloydbomni.dbadmin.goog -l alloydbomni.internal.dbadmin.goog/dbcluster=DB_CLUSTER_NAME -n DB_CLUSTER_NAMESPACE -o yaml > failovers_DB_CLUSTER_NAME.yaml
```

### Fetch pod and STS statuses

To fetch pod and StatefulSet (STS) statuses, run the following commands:

```bash
DB_POD=$(kubectl get pod -n DB_CLUSTER_NAMESPACE -l alloydbomni.internal.dbadmin.goog/dbcluster=DB_CLUSTER_NAME,alloydbomni.internal.dbadmin.goog/task-type=database -o jsonpath='{.items[0].metadata.name}')
kubectl describe pod ${DB_POD} -n DB_CLUSTER_NAMESPACE > pod_${DB_POD}.out
kubectl describe statefulset -l alloydbomni.internal.dbadmin.goog/dbcluster=DB_CLUSTER_NAME -n DB_CLUSTER_NAMESPACE > statefulset_DB_CLUSTER_NAME.out
```

## Identify errors

These sections describe how to identify errors.

### Look for error status and error codes

To identify the error code, check the DBCluster YAML file under the status.
Refer to the error codes documentation for more information.

To fetch the DBCluster YAML file, run the following command:

```bash
kubectl get dbclusters.alloydbomni.dbadmin.goog DB_CLUSTER_NAME -n DB_CLUSTER_NAMESPACE -o yaml > dbcluster_DB_CLUSTER_NAME.yaml
```

Look for `criticalIncidents`. This section contains the error code and a stack
trace.

The following are examples of `criticalIncidents`:

```yaml
status:
    certificateReference:
      certificateKey: ca.crt
      secretRef:
        name: dbs-al-cert-dr-mce
        namespace: dr
    conditions:
    -   lastTransitionTime: "2024-10-07T22:46:03Z"
    ...
    criticalIncidents:
    -   code: DBSE0304
      createTime: "2024-10-03T11:50:54Z"
      message: 'Healthcheck: Health check invalid result.'
      resource:
        component: healthcheck
        location:
          group: alloydbomni.internal.dbadmin.goog
          kind: Instance
          name: bc0f-dr-mce
          namespace: dr
          version: v1
      stackTrace:
      -   component: healthcheck
        message: 'DBSE0304: Healthcheck: Health check invalid result. rpc error: code
          = Code(10304) desc = DBSE0304: Healthcheck: Health check invalid result.
          dbdaemon/healthCheck: invalid timestamp read back from the healthcheck table.
          Lag is 384837.296269 seconds, wanted 35 seconds'
```

You can also retrieve the status by extracting specific fields in JSON format:

```bash
kubectl get dbclusters.alloydbomni.dbadmin.goog DB_CLUSTER_NAME -n DB_CLUSTER_NAMESPACE -o jsonpath='{.status.criticalIncidents}' | jq
```

The output is similar to the following:

```json
[
  {
    "code": "DBSE0085",
    "createTime": "2024-03-14T05:41:37Z",
    "message": "Platform: Pod is unschedulable.",
    "resource": {
      "component": "provisioning",
      "location": {
        "group": "alloydb.internal.dbadmin.goog",
        "kind": "Instance",
        "name": "b55f-testdbcluster",
        "namespace": "dbs-system",
        "version": "v1"
      }
    },
    "stackTrace": [
      {
        "component": "provisioning",
        "message": "DBSE0085: Platform: Pod is unschedulable. 0/16 nodes are available: pod has unbound immediate PersistentVolumeClaims. preemption: 0/16 nodes are available: 16 No preemption victims found for incoming pod..: Pod is unschedulable"
      }
    ]
  }
]
```

If the error message refers to the database pod, check the instances and pod
resources in the same namespace:

```bash
kubectl get instances.alloydbomni.internal.dbadmin.goog -l alloydbomni.internal.dbadmin.goog/dbcluster=DB_CLUSTER_NAME -n DB_CLUSTER_NAMESPACE -o yaml > instance_DB_CLUSTER_NAME.yaml
kubectl get pods -l alloydbomni.internal.dbadmin.goog/dbcluster=DB_CLUSTER_NAME,alloydbomni.internal.dbadmin.goog/task-type=database -n DB_CLUSTER_NAMESPACE
```

## Known issues

This section describes known issues and workarounds for the
AlloyDB Omni operator. For a complete list of release announcements, bug
fixes, and security updates, see the [Release notes](https://docs.cloud.google.com/alloydb/omni/kubernetes/16.3.0/docs/release-notes.md.txt).

### Certificate issuance fails in cert-manager

The AlloyDB Omni operator uses `cert-manager` to provision Transport
Layer Security (TLS) certificates for the operator and database clusters. In
`cert-manager` version 1.21.2 and earlier, a known race condition
([cert-manager issue #6331](https://github.com/cert-manager/cert-manager/issues/6331))
might cause certificate issuance to fail and enter a one-hour retry backoff.

#### Affected versions

AlloyDB Omni operator 1.0.0 and later with `cert-manager` 1.21.2 and
earlier.

#### Symptoms

When this issue occurs, you might observe the following symptoms:

- A `DBCluster` resource stays in the `DBClusterWaitingForCertificate` phase, and database cluster creation or upgrade times out.
- A Helm installation of the AlloyDB Omni operator fails because a `Certificate` resource, such as `local-serving-cert` in the `alloydb-omni-system` namespace, isn't ready.
- The status conditions of a `Certificate` resource include the `CSR not signed by referenced private key` message with the `Failed` or `ErrorKeyMatch` reason.

To find the `Certificate` resources that this issue affects, run the following
command:

```bash
kubectl get certificates.cert-manager.io --all-namespaces \
    -o jsonpath='{range .items[*]}{.metadata.namespace}{"\t"}{.metadata.name}{"\t"}{.status.conditions[*].message}{"\n"}{end}' \
    | grep "CSR not signed by referenced private key"
```

The output lists the namespace, the name, and the status condition messages of
each affected `Certificate` resource. If you view an affected `Certificate`
resource in YAML format, its `status` field is similar to the following:

```yaml
status:
  conditions:
  - message: 'The certificate request has failed to complete and will be retried: Error generating certificate template: CSR not signed by referenced private key'
    reason: Failed
    status: "False"
    type: Issuing
  failedIssuanceAttempts: 1
```

#### Workaround

To prevent controller update conflicts and reduce the retry delay for failed
certificate requests from one hour to 15 seconds, upgrade `cert-manager` with
the `ServerSideApply` feature gate and reduced backoff durations:

```bash
helm upgrade cert-manager oci://quay.io/jetstack/charts/cert-manager \
    --version CERT_MANAGER_VERSION \
    --namespace cert-manager \
    --reuse-values \
    --set featureGates="ServerSideApply=true" \
    --set "extraArgs={--certificate-request-minimum-backoff-duration=15s,--certificate-request-maximum-backoff-duration=5m}"
```

Replace the following:

- `CERT_MANAGER_VERSION`: the version of `cert-manager` installed in your Kubernetes cluster, such as `v1.21.2`. To find the installed version, run the `helm list --namespace cert-manager` command.

> [!NOTE]
> **Note:** Passing the `--set` flags for the `featureGates` and `extraArgs` keys replaces their existing values on the Helm release, even when you include the `--reuse-values` flag. If your `cert-manager` release already configures other feature gates or extra arguments, include them in the comma-separated lists. For GKE Autopilot clusters, also include the `--set global.leaderElection.namespace=cert-manager` flag.

After the `cert-manager` controller restarts, verify that all certificates show
`True` in the `READY` column:

```bash
kubectl get certificates.cert-manager.io --all-namespaces
```

If an installation of the AlloyDB Omni operator failed because of this
issue, run the installation again.

### Low-downtime maintenance minor version upgrade stops progressing on a secondary database cluster

#### Affected versions

AlloyDB Omni operator 1.8.0 and later.

#### Symptoms

During a minor version upgrade or downgrade of a cross-data-center replication
secondary database cluster that has high availability (HA) standbys, the
operation stops progressing after the first standby upgrades. The `DBCluster`
resource remains in the `DBCUpgradeInProgress=True` state with the
`DBCUpgradeFlowBegins` reason.

#### Workaround

1. Disable low-downtime maintenance (LDTM) on the secondary database cluster so
   that the AlloyDB Omni operator completes the upgrade by using the
   standard rolling upgrade workflow:

   ```bash
   kubectl annotate dbclusters.alloydbomni.dbadmin.goog DB_CLUSTER_NAME \
       dbcluster.dbadmin.goog/enableLDTM=false \
       --namespace DB_CLUSTER_NAMESPACE \
       --overwrite
   ```

   Replace the following:
   - `DB_CLUSTER_NAME`: the name of your secondary `DBCluster` resource.
   - `DB_CLUSTER_NAMESPACE`: the namespace of your secondary `DBCluster` resource.
2. Verify that the upgrade completes and the `DBCluster` resource reaches the
   `DBClusterReady` phase:

   ```bash
   kubectl get dbclusters.alloydbomni.dbadmin.goog DB_CLUSTER_NAME \
       --namespace DB_CLUSTER_NAMESPACE
   ```
3. Re-enable low-downtime maintenance on the secondary database cluster so that
   the `DBCluster` resource retains low-downtime maintenance capability if it
   is promoted to a primary database cluster:

   ```bash
   kubectl annotate dbclusters.alloydbomni.dbadmin.goog DB_CLUSTER_NAME \
       dbcluster.dbadmin.goog/enableLDTM=true \
       --namespace DB_CLUSTER_NAMESPACE \
       --overwrite
   ```

### Monitoring agent encounters cgroups permission errors after enabling cross-data-center replication

#### Affected versions

AlloyDB Omni operator 1.9.0.

#### Symptoms

After you enable cross-data-center replication, the monitoring agent encounters
permission errors in the container logs when attempting to gather `cgroups`
information.

#### Workaround

1. List the database Pods in your secondary database cluster to find the name
   of the downstream database Pod:

   ```bash
   kubectl get pods \
       --selector alloydbomni.internal.dbadmin.goog/dbcluster=DB_CLUSTER_NAME,alloydbomni.internal.dbadmin.goog/task-type=database \
       --namespace DB_CLUSTER_NAMESPACE
   ```

   Replace the following:
   - `DB_CLUSTER_NAME`: the name of your secondary `DBCluster` resource.
   - `DB_CLUSTER_NAMESPACE`: the namespace of your secondary `DBCluster` resource.
2. Delete the downstream database Pod so that the AlloyDB Omni operator
   recreates and restarts the Pod:

   ```bash
   kubectl delete pod DB_POD_NAME \
       --namespace DB_CLUSTER_NAMESPACE
   ```

   Replace `DB_POD_NAME` with the name of the downstream
   database Pod from the previous step.

### Database cluster continually triggers automatic failovers

If your `DBCluster` resource creates multiple automatic failovers one after
another, complete the following checks:

1. List the `Failover` resources for your `DBCluster` and compare their
   creation timestamps:

   ```bash
   kubectl get failover.alloydbomni.dbadmin.goog \
       --selector failover.dbadmin.goog/dbcluster=DB_CLUSTER_NAME \
       --namespace DB_CLUSTER_NAMESPACE
   ```

   Replace the following:
   - `DB_CLUSTER_NAME`: the name of your `DBCluster` resource.
   - `DB_CLUSTER_NAMESPACE`: the namespace of your `DBCluster` resource.
2. Connect to the primary database Pod on the `alloydbadmin` database and
   verify the database health:

   ```sql
   SELECT * FROM healthcheck;
   ```

   Verify that the last health check timestamp is recent and that health
   checks run at expected intervals. If the primary instance is unhealthy,
   inspect the `postgresql.log` file to determine why the instance is failing.
   For more information about fetching logs, see
   [Troubleshoot the AlloyDB Omni Kubernetes operator](https://docs.cloud.google.com/alloydb/omni/kubernetes/16.3.0/docs/troubleshoot-kubernetes-operator.md.txt#fetch-postgresql-log).

If you use a large number of database clusters in a single Kubernetes cluster,
or if you have database clusters that are underprovisioned, automatic failover
health checks might time out or fail to connect. If you receive an error that
starts with `HealthCheckProber: health check for instance failed`, see the
recommended fixes in
[Troubleshoot instance healing](https://docs.cloud.google.com/alloydb/omni/kubernetes/16.3.0/docs/kubernetes-ha.md.txt#troubleshoot-instance-healing).
To reduce or prevent automatic failovers, you can also perform the following
actions:

- Increase the `autoFailoverTriggerThreshold` value so that automatic
  failover occurs less frequently. For more information, see
  [Adjust automatic failover trigger settings](https://docs.cloud.google.com/alloydb/omni/kubernetes/16.3.0/docs/kubernetes-ha.md.txt#adjust-failover-trigger-settings).

- Disable automatic failover on database clusters. For more information, see
  [Disable automatic failover](https://docs.cloud.google.com/alloydb/omni/kubernetes/16.3.0/docs/kubernetes-ha.md.txt#disable-failover).

### Database cluster remains stuck in a failover or switchover state

During a high availability (HA) failover or switchover, the
AlloyDB Omni operator fences the former primary instance during the
`StopPrimary` phase by renaming the `PG_VERSION` files to `PG_VERSION.disabled`
in both the root data directory at `/pgdata/db/PG_VERSION.disabled` and every
database subdirectory at
`/pgdata/db/base/OID/PG_VERSION.disabled`. If the
failover or switchover fails before the standby instance is promoted and
automatic rollback doesn't occur or fails, the former primary instance remains
fenced and the `DBCluster` resource remains stuck in an in-progress or failed
rollback phase.

#### Affected versions

AlloyDB Omni operator 1.0.0 and later.

#### Symptoms

When this issue occurs, you might observe the following symptoms:

- The `status.phase` field of the `DBCluster` resource remains in `DBClusterFailoverInProgress`, `DBClusterSwitchoverInProgress`, `DBClusterFailoverRollbackFailed`, or `DBClusterSwitchoverRollbackFailed`.
- The `HAReady` condition on the `DBCluster` resource has `status: "False"` with the `FailoverInProgress` or `SwitchoverInProgress` reason.
- The `Failover` (`failovers.alloydbomni.dbadmin.goog`) or `Switchover` (`switchovers.alloydbomni.dbadmin.goog`) resource is in the `Failed` or `Failed_RollbackFailed` state.
- The `status.internal.phase` field of the `Failover` or `Switchover` resource remains in a phase before `PromoteStandby`, such as `StopPrimary`, or remains in the `PromoteStandby` phase for an hour or more without promoting the standby instance.
- The primary database Pod fails to start PostgreSQL and logs an error indicating that the `PG_VERSION` file is missing because `PG_VERSION` was renamed to `PG_VERSION.disabled` in `/pgdata/db/PG_VERSION.disabled` and `/pgdata/db/base/OID/PG_VERSION.disabled`.
- If a previous manual recovery restored only the top-level
  `/pgdata/db/PG_VERSION` file while leaving `PG_VERSION.disabled` files in
  the `/pgdata/db/base/OID/` subdirectories, subsequent
  `pg_basebackup` operations on standby instances fail with the following
  error in the PostgreSQL logs on the primary instance:

      XX000:ERROR: [basebackup.c:1549] invalid segment number 0 in file "PG_VERSION.disabled"

#### Workaround

> [!WARNING]
> **Warning:** This procedure rolls back the database cluster to the former primary instance. Only perform this procedure if the standby instance hasn't been promoted and is either stopped or still in read-only recovery mode. If the failover or switchover has already progressed past the `PromoteStandby` phase and promoted the standby instance, rolling back to the former primary instance can cause data loss. Instead, restore the database cluster from a backup.

To manually unfence the former primary instance and recover the `DBCluster`
resource, complete the following steps:

1. Inspect the external and internal `Failover` or `Switchover` resources in
   your database cluster namespace to check their phase and identify the
   former primary (`oldPrimary`) and target standby (`newPrimary`) instances:

   ```bash
   kubectl get failovers.alloydbomni.dbadmin.goog,switchovers.alloydbomni.dbadmin.goog,failovers.alloydbomni.internal.dbadmin.goog,instanceswitchovers.alloydbomni.internal.dbadmin.goog \
       --namespace DB_CLUSTER_NAMESPACE \
       -o yaml
   ```

   Replace `DB_CLUSTER_NAMESPACE` with the namespace of your
   `DBCluster` resource.

   Confirm that `status.internal.phase` hasn't progressed past
   `PromoteStandby`. If the standby database Pod is running, verify that
   PostgreSQL on the standby instance is still in read-only recovery mode:

   ```bash
   kubectl exec -c database STANDBY_POD_NAME \
       --namespace DB_CLUSTER_NAMESPACE -- \
       psql -U alloydbadmin -d alloydbadmin -c "SELECT pg_is_in_recovery();"
   ```

   Replace `STANDBY_POD_NAME` with the name of the standby
   database Pod, formatted as
   `al-STANDBY_INSTANCE_NAME-0`, where
   `STANDBY_INSTANCE_NAME` is the value of
   `status.internal.newPrimary`.

   Confirm that the query returns `t` (true), or that the standby database Pod
   is stopped. After you confirm that the standby instance hasn't been
   promoted, delete the standby `Instance` resource to prevent it from being
   promoted during recovery:

   ```bash
   kubectl delete instances.alloydbomni.internal.dbadmin.goog STANDBY_INSTANCE_NAME \
       --namespace DB_CLUSTER_NAMESPACE
   ```

   Replace `STANDBY_INSTANCE_NAME` with the name of the
   standby `Instance` resource from the `status.internal.newPrimary` field.
2. Check the `/pgdata/db` data directory on the primary database Pod for
   `PG_VERSION` and `PG_VERSION.disabled` files:

   ```bash
   kubectl exec -c database PRIMARY_POD_NAME \
       --namespace DB_CLUSTER_NAMESPACE -- \
       find /pgdata/db -type f -name "PG_VERSION*"
   ```

   Replace `PRIMARY_POD_NAME` with the name of the primary
   database Pod, formatted as
   `al-PRIMARY_INSTANCE_NAME-0`, where
   `PRIMARY_INSTANCE_NAME` is the value of
   `status.internal.oldPrimary`.

   If neither `PG_VERSION` nor `PG_VERSION.disabled` exists in `/pgdata/db`,
   the database data directory isn't recoverable and must be restored from a
   backup.

   If `PG_VERSION.disabled` files exist, restore `PG_VERSION` across both the
   root data directory at `/pgdata/db/PG_VERSION.disabled` and every database
   subdirectory at
   `/pgdata/db/base/OID/PG_VERSION.disabled`.
   The following command renames each `PG_VERSION.disabled` file back to
   `PG_VERSION`, or removes the `PG_VERSION.disabled` file if the corresponding
   `PG_VERSION` file was already restored:

   ```bash
   kubectl exec -c database PRIMARY_POD_NAME \
       --namespace DB_CLUSTER_NAMESPACE -- \
       sh -c 'find /pgdata/db -type f -name "PG_VERSION.disabled" | while read -r f; do
         target="${f%.disabled}"
         if [ -f "$target" ]; then rm -f "$f"; else mv "$f" "$target"; fi
       done'
   ```
3. Restart the PostgreSQL service inside the primary database Pod:

   ```bash
   kubectl exec -c database PRIMARY_POD_NAME \
       --namespace DB_CLUSTER_NAMESPACE -- \
       /scripts/supervisor/bin/supervisorctl.par \
       -c /scripts/supervisor/etc/supervisord.conf restart postgres
   ```
4. Clear the failed failover or switchover state by using one of the following
   options:

   - **Option 1: Delete the `Failover` or `Switchover` resources and restart
     the operator controllers.** Delete both the external and internal
     `Failover` resources, and restart the `fleet-controller-manager` and
     `local-controller-manager` deployments in the `alloydb-omni-system`
     namespace to release in-memory locks:

     ```bash
     kubectl delete failovers.alloydbomni.dbadmin.goog FAILOVER_NAME \
         --namespace DB_CLUSTER_NAMESPACE
     kubectl delete failovers.alloydbomni.internal.dbadmin.goog INTERNAL_FAILOVER_NAME \
         --namespace DB_CLUSTER_NAMESPACE
     kubectl rollout restart deployment/fleet-controller-manager deployment/local-controller-manager \
         --namespace alloydb-omni-system
     ```

     Replace the following:
     - `FAILOVER_NAME`: the name of the `failovers.alloydbomni.dbadmin.goog` resource.
     - `INTERNAL_FAILOVER_NAME`: the name of the `failovers.alloydbomni.internal.dbadmin.goog` resource.

     If a switchover is stuck instead of a failover, delete the
     `switchovers.alloydbomni.dbadmin.goog` and
     `instanceswitchovers.alloydbomni.internal.dbadmin.goog` resources
     instead.
   - **Option 2: Mark the `Failover` resources as rolled back.** This
     option applies only to stuck `Failover` resources. If a `Switchover`
     resource is stuck, use Option 1 instead. Edit the `status`
     subresource on both the `failovers.alloydbomni.dbadmin.goog` and
     `failovers.alloydbomni.internal.dbadmin.goog` resources:

     ```bash
     kubectl edit failovers.alloydbomni.dbadmin.goog FAILOVER_NAME \
         --namespace DB_CLUSTER_NAMESPACE \
         --subresource=status
     kubectl edit failovers.alloydbomni.internal.dbadmin.goog INTERNAL_FAILOVER_NAME \
         --namespace DB_CLUSTER_NAMESPACE \
         --subresource=status
     ```

     In the `status` section of each `Failover` resource, set `endTime` to a
     timestamp after `createTime`, set `internal.phase` to
     `RollbackSucceeded`, and set `state` to `Failed`, as shown in the
     following example:

     ```yaml
     apiVersion: alloydbomni.dbadmin.goog/v1
     kind: Failover
     status:
       createTime: "2026-09-01T22:07:42Z"
       endTime: "2026-09-01T22:07:50Z"
       internal:
         attempt: 0
         lastPhaseChangeTime: "2026-09-01T22:07:39Z"
         newPrimary: 4274-dbcluster-sample
         newPrimaryZone: ""
         oldPrimary: dafd-dbcluster-sample
         phase: RollbackSucceeded
       observedGeneration: 2
       reconciled: true
       startTime: "2026-09-01T22:07:39Z"
       state: Failed
     ```
5. Update the `status` subresource of the `DBCluster` resource to clear
   `latestFailoverStatus` and set `phase` to `DBClusterReconciling`:

   ```bash
   kubectl patch dbclusters.alloydbomni.dbadmin.goog DB_CLUSTER_NAME \
       --namespace DB_CLUSTER_NAMESPACE \
       --subresource=status \
       --type=merge \
       --patch '{"status":{"latestFailoverStatus":null,"phase":"DBClusterReconciling"}}'
   ```

   Replace `DB_CLUSTER_NAME` with the name of your
   `DBCluster` resource.

   If the `HAReady` condition in `status.conditions` still prevents
   reconciliation because its `reason` field is set to `FailoverInProgress` or
   `SwitchoverInProgress`, edit the `status` subresource of the `DBCluster`
   resource and either remove the `HAReady` condition entry from
   `status.conditions` so the AlloyDB Omni operator recreates it during
   reconciliation, or set its `reason` field to a valid non-empty `CamelCase`
   value such as `ClusterReady`:

   ```bash
   kubectl edit dbclusters.alloydbomni.dbadmin.goog DB_CLUSTER_NAME \
       --namespace DB_CLUSTER_NAMESPACE \
       --subresource=status
   ```
6. Verify that the AlloyDB Omni operator reconciles the standby instance
   and returns the `DBCluster` resource to the `DBClusterReady` phase:

   ```bash
   kubectl get dbclusters.alloydbomni.dbadmin.goog DB_CLUSTER_NAME \
       --namespace DB_CLUSTER_NAMESPACE
   ```

## Debug memory issues

These sections describe how to debug memory issues.

### Run and take a heapdump

Only turn on this feature for troubleshooting purposes. Remember to turn it
off afterward.

To take a heapdump, complete the following steps:

1. Modify the operator deployment under the `alloydb-omni-system` namespace with name `fleet-controller-manager` and `local-controller-manager`.
2. Add the following argument to the pod `--pprof-address=:8642` or to any other available port.
3. Wait for the controller pod to restart.
4. Port-forward the preceding port. For example:

   ```bash
   kubectl port-forward FLEET_CONTROLLER_MANAGER_POD_NAME -n alloydb-omni-system 8642:8642
   ```
5. In another terminal, run `go tool pprof http://localhost:8642/debug/pprof/heap`.
   Change the port to match the preceding port if you don't use `8642`.

6. Connect to the address and run troubleshooting commands. For example: `top`.

7. After finishing troubleshooting, undo step 1 by removing the argument and
   waiting for the pod to restart.

### Determine the number of resources the operator is watching

To understand the resources that in use, run the following
commands:

```bash
kubectl get backuprepositories -A  | wc -l
kubectl get failovers -A  | wc -l
kubectl get instancebackupplans -A  | wc -l
kubectl get instancebackups -A  | wc -l
kubectl get instancerestores -A  | wc -l
kubectl get instances -A  | wc -l
kubectl get instanceswitchovers -A  | wc -l
kubectl get lrojobs -A  | wc -l
kubectl get replicationconfigs -A  | wc -l
kubectl get sidecars -A  | wc -l
kubectl get deployments -A  | wc -l
kubectl get statefulsets -A  | wc -l
kubectl get certificates.cert-manager.io -A  | wc -l
kubectl get issuers.cert-manager.io -A  | wc -l
kubectl get configmaps -A  | wc -l
kubectl get persistentvolumeclaims -A  | wc -l
kubectl get persistentvolumes -A  | wc -l
kubectl get pods -A  | wc -l
kubectl get secrets -A  | wc -l
kubectl get services -A  | wc -l
kubectl get storageclasses.storage.k8s.io -A  | wc -l
```

For example, if the number of secrets is high, it can lead to an Out Of Memory
(OOM) error.

```bash
kubectl get secrets -A | wc -l
```

## Advanced HA debugging

This section references resources that are internal implementations. These are
subject to change at any point in time and have no backward compatibility
commitments. Only apply manual fixes to issues on
non-production databases. These steps may make the database unrecoverable.

The AlloyDB Omni HA setup has three phases:

1. Set up the primary to receive a connection from the standby.
2. Initialize the standby and connect it to the primary.
3. Set the primary settings to make the connection synchronous.

Step 2 is generally the slowest. Depending on the size of the database, it might
take several hours.

Each instance replicating instance should have a `replicationconfig` attached to it. For example:

```bash
kubectl get replicationconfigs.alloydbomni.internal.dbadmin.goog -n DB_CLUSTER_NAMESPACE
```

Example output:

    NAME                 PARENT     TYPE       ROLE         READY   HEALTHY   SYNC_U   SYNC_D   SLOT_LOG   SLOT_REPLAY
    cd58-adb--58ea-adb   cd58-adb   Physical   Upstream     True    True      true
    ds-58ea-adb          58ea-adb   Physical   Downstream   True    True               true

The spec of the Replication Config indicates the intended settings, while the
status reflects the actual state as read from the database. If there is a
mismatch between the spec and the status, the controller is still attempting to
apply the change, or there is some error that is preventing the change from
being applied. This would be reflected in the status fields.

### Standby jobs

There should be two sets of internal jobs which track the workflow of a standby:

- `createstandbyjobs.alloydbomni.internal.dbadmin.goog`
- `deletestandbyjobs.alloydbomni.internal.dbadmin.goog`

If the setup appears to be stuck, view the jobs relating to the database cluster
(DBC). The job might have error messages which explain which state the setup is
in. Jobs are automatically cleaned up some time after they have completed, so
you may not see any jobs if none are in progress.

```bash
kubectl get createstandbyjobs.alloydbomni.internal.dbadmin.goog -n DB_CLUSTER_NAMESPACE
```

The output is similar to the following:

```yaml
apiVersion: alloydbomni.dbadmin.gdc.goog/v1
  kind: CreateStandbyJob
  metadata:
    creationTimestamp: "2024-11-05T03:34:26Z"
    finalizers:
    -   createstandbyjob.dbadmin.goog/finalizer
    generation: 1804
    labels:
      dbs.internal.dbadmin.goog/dbc: foo-ha-alloydb1-clone1
    name: foo-ha-alloydb1-clone1--ac00-foo-ha-alloydb1-clone1--6036-foo-ha-alloydb1-clone1-1730777666
    namespace: db
    resourceVersion: "11819071"
    uid: 1f24cedf-b326-422f-9405-c96c8720cd90
  spec:
    attempt: 3
    cleanup: false
    currentStep: SetupSynchronous
    currentStepTime: "2024-11-05T03:45:31Z"
    metadata:
      dbc: foo-ha-alloydb1-clone1
      primaryInstance: ac00-foo-ha-alloydb1-clone1
      retryError: 'etcdserver: leader changed'
      standbyInstance: 6036-foo-ha-alloydb1-clone1
    requeueTime: "2024-11-05T18:33:03Z"
    startTime: "2024-11-05T03:36:56Z"
```

### Primary verification

The first thing to verify is that the primary is set up correctly. There should
be a Replication Profile for each standby. If `isSynchronous` is true on the
spec and status, then setup should be complete. If `isSynchronous` is false on
the spec and status, then it has not yet reached step 3. View the standby jobs
to see if there are any running jobs, and if they have any error
messages.

```yaml
  replication:
    profiles:
    -   isActive: true
      isSynchronous: true
      name: ha:4c82-dbcluster-sample::d85d-dbcluster-sample
      password:
        name: ha-rep-pw-dbcluster-sample
        namespace: default
      passwordResourceVersion: "896080"
      role: Upstream
      type: Physical
      username: alloydbreplica
```

Verify that the `disableHealthcheck` annotation is false. It is meant to be
disabled only during a failover or switchover.

```yaml
apiVersion: alloydbomni.internal.dbadmin.goog/v1
kind: Instance
metadata:
  annotations:
    dbs.internal.dbadmin.goog/consecutiveHealthcheckFailures: "0"
    dbs.internal.dbadmin.goog/disableHealthcheck: "false"
    dr-secondary: "false"
    forceReconcile: "1730414498"
```

### Queries

To verify that the resources on the DB pod are set up properly, sign in to the
database as the administrator user `alloydbadmin`. Then issue the following queries:

#### Replication slot

```sql
\x on
select * from pg_replication_slots;
```

    -[ RECORD 1 ]---+---
    slot_name           | d85d_dbcluster_sample
    plugin              |
    slot_type           | physical
    datoid              |
    database            |
    temporary           | f
    active              | t
    active_pid          | 250
    xmin                | 16318
    catalog_xmin        |
    restart_lsn         | 0/CA657F0
    confirmed_flush_lsn |
    wal_status          | reserved
    safe_wal_size       |
    two_phase           | f

A good state is the presence of a replication slot bearing the same name as
the standby instance. The absence of a replication slot indicates that the first setup step has not completed successfully.

If `active` is not `t` (true), then that means the standby is not connecting for
some reason (networking, standby not finishing setup, etc.) and debugging will
likely need to continue on the standby side.

#### Replication stats

```sql
\x on
select * from pg_stat_replication;
```

    -[ RECORD 1 ]---+---
    pid              | 250
    usesysid         | 16385
    usename          | alloydbreplica
    application_name | d85d_dbcluster_sample
    client_addr      | 10.54.79.196
    client_hostname  | gke-samwise-default-pool-afaf152d-8197.us-central1-a.c.foo
    client_port      | 24914
    backend_start    | 2024-10-30 21:44:26.408261+00
    backend_xmin     |
    state            | streaming
    sent_lsn         | 0/CA64DA8
    write_lsn        | 0/CA64DA8
    flush_lsn        | 0/CA64DA8
    replay_lsn       | 0/CA64DA8
    write_lag        |
    flush_lag        |
    replay_lag       |
    sync_priority    | 2
    sync_state       | sync
    reply_time       | 2024-11-04 22:08:04.370838+00

If this doesn't exist, then that means there is no active connection. The
`sync_state` should be `sync`. If it is not `sync`, that means the final step of
setup did not complete. Looking at the logs / jobs should provide more details.

### Standby verification

The standby should have a replication profile that matches the same profile to
the primary:

```yaml
  replication:
    profiles:
    -   host: 10.54.79.210
      isActive: true
      isSynchronous: true
      name: ha:4c82-dbcluster-sample::d85d-dbcluster-sample
      passwordResourceVersion: "896080"
      port: 5432
      role: Downstream
      type: Physical
      username: alloydbreplica
```

If there is no connection from the standby to the primary, there are two common
possibilities:

1. The standby is still setting up.
2. The standby is getting an error while setting up or trying to connect.

To check if option 1 is happening, get the database pod logs and look for log
statements named `dbdaemon/setupPhysicalReplicationDownstream`. The following
are examples of successful setup logs:

    I1104 22:42:42.604871     103 replication.go:107] "dbdaemon/setupPhysicalReplicationDownstream: begin setup" log_name="agent" project_ns="default" dbcluster="dbcluster-sample"
    2024-11-04 22:42:42,605 INFO waiting for postgres to stop
    2024-11-04 22:42:43,566 INFO stopped: postgres (exit status 0)
    I1104 22:42:43.567590     103 replication.go:131] "dbdaemon/setupPhysicalReplicationDownstream: about to call pg_basebackup" log_name="agent" project_ns="default" dbcluster="dbcluster-sample" cmd=["-h","10.54.79.210","-D","/mnt/disks/pgsql/pg_basebackup_data","-U","alloydbreplica","-v","-P","-p","5432","-w","-c","fast"]
    I1104 22:42:44.206403     103 replication.go:139] "dbdaemon/setupPhysicalReplicationDownstream: pg_basebackup finished" log_name="agent" project_ns="default" dbcluster="dbcluster-sample"
    I1104 22:42:44.206440     103 replication.go:141] "dbdaemon/setupPhysicalReplicationDownstream: replacing data directory with pg_basebackup data directory" log_name="agent" project_ns="default" dbcluster="dbcluster-sample"
    I1104 22:42:44.244749     103 replication.go:148] "dbdaemon/setupPhysicalReplicationDownstream: replaced data directory with pg_basebackup data directory" log_name="agent" project_ns="default" dbcluster="dbcluster-sample"
    I1104 22:42:44.244783     103 replication.go:150] "dbdaemon/setupPhysicalReplicationDownstream: Creating config files" log_name="agent" project_ns="default" dbcluster="dbcluster-sample"
    I1104 22:42:44.251565     103 replication.go:155] "dbdaemon/setupPhysicalReplicationDownstream: removing postgresql config file for log archiving" log_name="agent" project_ns="default" dbcluster="dbcluster-sample"
    I1104 22:42:44.251621     103 replication.go:160] "dbdaemon/setupPhysicalReplicationDownstream: removing postgresql auto config file" log_name="agent" project_ns="default" dbcluster="dbcluster-sample"
    I1104 22:42:44.251689     103 replication.go:165] "dbdaemon/setupPhysicalReplicationDownstream: Successfully wrote to config file" log_name="agent" project_ns="default" dbcluster="dbcluster-sample"
    2024-11-04 22:42:44,256 INFO spawned: 'postgres' with pid 271
    2024-11-04 22:42:45,469 INFO success: postgres entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
    I1104 22:42:45.469838     103 replication.go:174] "dbdaemon/setupPhysicalReplicationDownstream: backup replication configuration after changing replication config" log_name="agent" project_ns="default" dbcluster="dbcluster-sample"
    I1104 22:42:45.476732     103 replication.go:179] "dbdaemon/setupPhysicalReplicationDownstream: finished standby setup" log_name="agent" project_ns="default" dbcluster="dbcluster-sample"

If there is a connection error, check the db pod logs as well as the log file on
the database at `/obs/diagnostic/postgresql.log` and see what the error is when
trying to connect. One common error is there is no networking connectivity
between the standby and the primary.

### Manual fixes

The easiest way to fix HA issues is to disable and then re-enable HA by setting
`numberOfStandbys` to 0 and then resetting it to the number you want. If standbys are
stuck being disabled, then follow these steps to manually reset the
HA setup to be empty:

1. Manually delete the standby instances.
2. Connect to the primary database. Query the current replication slots and
   delete any replication slots of standbys you want to delete:

   ```sql
   select pg_drop_replication_slot('REPLICATION_SLOT_NAME');
   ```
3. Delete any replication profiles from the primary instance that you want to
   delete.

If an instance has not reconciled recently, you can edit the `forceReconcile`
annotation value. Set that to any numerical value, which is the timestamp of the
last time that annotation was updated. The only purpose of that annotation is
to provide a field we can update to force a new reconciliation.

```yaml
apiVersion: alloydbomni.internal.dbadmin.goog/v1
kind: Instance
metadata:
  annotations:
    dbs.internal.dbadmin.goog/consecutiveHealthcheckFailures: "0"
    dbs.internal.dbadmin.goog/disableHealthcheck: "false"
    dr-secondary: "false"
    forceReconcile: "1730414498"
```

If a `DBCluster` resource remains stuck in `DBClusterFailoverInProgress`,
`DBClusterSwitchoverInProgress`, `DBClusterFailoverRollbackFailed`, or
`DBClusterSwitchoverRollbackFailed` because the primary instance is fenced with
`PG_VERSION.disabled` files, see [Database cluster remains stuck in a failover
or switchover state](https://docs.cloud.google.com/alloydb/omni/kubernetes/16.3.0/docs/troubleshoot-kubernetes-operator#dbcluster-stuck-failover-switchover).

## Collect database engine and audit logs

The database engine logs and audit logs are available as files inside the
database pod (requiring root access):

- `obs/diagnostic/postgresql.log`
- `obs/diagnostic/postgresql.audit`

```bash
DB_POD=$(kubectl get pod -l alloydbomni.internal.dbadmin.goog/dbcluster=DB_CLUSTER_NAME,alloydbomni.internal.dbadmin.goog/task-type=database -n DB_CLUSTER_NAMESPACE -o jsonpath='{.items[0].metadata.name}')
kubectl exec -c database -n DB_CLUSTER_NAMESPACE ${DB_POD} -it -- /bin/bash
```

When connected to the database container:

```bash
ls -l /obs/diagnostic/
```

Example output:

    drwx--S--- 2 postgres postgres    4096 Aug 13 10:22 archive
    -rw--- 1 postgres postgres  256050 Aug 13 13:25 postgresql.internal
    -rw--- 1 postgres postgres 1594799 Aug 13 13:25 postgresql.log

## Collect database and database pod metrics

The AlloyDB Omni operator provides a set of basic metrics for the
AlloyDB Omni engine and the pod hosting it. The metrics are
available as Prometheus endpoints available at port 9187. To access the
endpoints you need to identify the pod name for the database pod using the
DBCluster label and start port forwarding as follows:

```bash
DB_POD=$(kubectl get pod -l alloydbomni.internal.dbadmin.goog/dbcluster=DB_CLUSTER_NAME -n DB_CLUSTER_NAMESPACE -o jsonpath='{.items[0].metadata.name}')
kubectl port-forward -n DB_CLUSTER_NAMESPACE ${DB_POD} 9187:9187
```

### Access database pod metrics

In another terminal:

```bash
curl http://localhost:9187/metrics | grep HELP
```

For more information about monitoring, see [Monitor AlloyDB Omni](https://docs.cloud.google.com/alloydb/omni/kubernetes/16.3.0/docs/manage.md.txt#monitor).

You can also configure Prometheus to scrape the metrics in your Kubernetes
cluster. Refer to the [Prometheus Kubernetes service discovery config](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#kubernetes_sd_config) for details.