Select a documentation version: <button class="button button-with-icon cloud-select-dropdown__button button-primary" aria-haspopup="true">15.5.5</button>

- [16.3.0](https://docs.cloud.google.com/alloydb/omni/containers/16.3.0/docs/cross-data-center-replication/work-with-cross-data-center-replication.md.txt)
- [15.7.1](https://docs.cloud.google.com/alloydb/omni/containers/15.7.1/docs/cross-data-center-replication/work-with-cross-data-center-replication.md.txt)
- [15.7.0](https://docs.cloud.google.com/alloydb/omni/containers/15.7.0/docs/cross-data-center-replication/work-with-cross-data-center-replication.md.txt)
- [15.5.5](https://docs.cloud.google.com/alloydb/omni/containers/15.5.5/docs/cross-data-center-replication/work-with-cross-data-center-replication.md.txt)

<br />

This page describes how to use cross-data-center replication by creating and working with secondary database clusters in Kubernetes.

<br />

For a conceptual overview of cross-data-center replication, see [About cross-data-center replication](https://docs.cloud.google.com/alloydb/omni/containers/15.5.5/docs/cross-data-center-replication/about-cross-data-center-replication.md.txt).

## Before you begin

- [Install](https://docs.cloud.google.com/alloydb/omni/containers/15.5.5/docs/deploy-kubernetes.md.txt) AlloyDB Omni Operator version 1.1.0 or higher to deploy AlloyDB Omni on a Kubernetes cluster in the primary data center and a Kubernetes cluster in the secondary data center.
- [Create an AlloyDB Omni database cluster](https://docs.cloud.google.com/alloydb/omni/containers/15.5.5/docs/deploy-kubernetes.md.txt#create) on the Kubernetes cluster in the primary data center.

## Create a secondary database cluster

To create an AlloyDB Omni secondary database cluster and enable replication from your primary database cluster, follow these steps:

### Kubernetes

1. Ensure external connectivity is enabled on your AlloyDB Omni primary database cluster. If external connectivity is not enabled, add the following to the spec section of the database cluster manifest:

   ```yaml
   kind: DBCluster
   spec:
     allowExternalIncomingTraffic: true
   ```
2. To enable replication on your primary database cluster, apply a manifest similar to the following to your Kubernetes cluster on the primary data center:

   ```yaml
   apiVersion: v1
   kind: Secret
   metadata:
     name: ha-rep-pw-DB_CLUSTER_NAME
   type: Opaque
   data:
     rep-user-pw: "ENCODED_PASSWORD"
   ---
   apiVersion: alloydbomni.dbadmin.goog/v1
   kind: Replication
   metadata:
     name: REPLICATION_NAME
   spec:
     dbcluster:
       name: DB_CLUSTER_NAME
     upstream:
       password:
         name: ha-rep-pw-DB_CLUSTER_NAME
   ```

   Replace the following:
   - `DB_CLUSTER_NAME`: The name of the database cluster---for example, `dbc-1`.
   - `ENCODED_PASSWORD`: The password for the database user to be used for replication from secondary databases, encoded as a base64 string---for example, `Q2hhbmdlTWUxMjM= for ChangeMe123`. The default value is `alloydbreplica`.
   - `REPLICATION_NAME`: the name of the replication---for example, `replication-1`.

   Wait for the replication status to be ready.
3. To get the upstream connection information that is used to configure replication on the secondary database cluster, run the following command:

   ```bash
     kubectl get replication REPLICATION_NAME
     kubectl get replication REPLICATION_NAME -o json | jq .status.upstream
   ```

   Sample output looks similar to the following:

         {
          "host": "35.230.32.36",
          "password": {
            "name": "ha-rep-pw-dbc-1"
          },
          "port": 5432,
          "replicationSlotName": "dbc_1_replication_1",
          "username": "alloydbreplica"
         }
        ```

4. Take a note of the output since you need it to enable replication on the secondary database cluster in the following step.

5. [Create an AlloyDB Omni cluster](https://docs.cloud.google.com/alloydb/omni/containers/15.5.5/docs/deploy-kubernetes.md.txt#create) on your Kubernetes cluster in the secondary data center using a configuration that's identical to your primary database cluster.

6. Ensure external connectivity is enabled on your AlloyDB Omni secondary database cluster.

7. If external connectivity is not enabled, add the following to the spec section of its manifest:

   ```yaml
    allowExternalIncomingTraffic: true
   ```
8. To enable replication on your secondary database cluster, apply a manifest similar to the following to your Kubernetes cluster on the secondary data center:

   ```yaml
   apiVersion: v1
   kind: Secret
   metadata:
     name: ha-rep-pw-SECONDARY_DB_CLUSTER_NAME
   type: Opaque
   data:
     rep-user-pw: "ENCODED_PASSWORD"
   ---
   apiVersion: alloydbomni.dbadmin.goog/v1
   kind: Replication
   metadata:
     name: SECONDARY_REPLICATION_NAME
   spec:
     dbcluster:
       name: SECONDARY_DB_CLUSTER_NAME
     downstream:
       host: PRIMARY_HOST
       port: PRIMARY_PORT
       username: alloydbreplica
       password:
         name: ha-rep-pw-SECONDARY_DB_CLUSTER_NAME
       replicationSlotName: PRIMARY_REPLICATION_SLOT
       control: setup
   ```

   Replace the following:
   - `SECONDARY_DB_CLUSTER_NAME`: The name of the secondary database cluster---for example, `dbc-2`.
   - `ENCODED_PASSWORD`: The password for the database user to be used for replication the primary database cluster, encoded as a base64 string---for example, `Q2hhbmdlTWUxMjM= for ChangeMe123`. The default value is `alloydbreplica`.
   - `SECONDARY_REPLICATION_NAME`: the name of the replication---for example, `replication-2`.
   - `PRIMARY_HOST`: the connection endpoint of the primary database cluster from the output in step 3 that the secondary database can access for replication.
   - `PRIMARY_PORT`: the connection port of the primary database cluster from the output in step 3 that the secondary database can access for replication.
   - `PRIMARY_REPLICATION_SLOT`: the name of the replication slot on the primary database cluster from the output in step 3 that the secondary database can use for replication.

## View replication on the secondary database cluster

To view detailed information about an AlloyDB Omni secondary database cluster and its replication status,
run the following commands:

### Kubernetes

```bash
kubectl get dbcluster SECONDARY_DB_CLUSTER_NAME
kubectl get replication SECONDARY_REPLICATION_NAME
```

When the secondary database cluster is successfully set up and has streaming replication from the primary database cluster, the replication status is both ready and healthy.

## Promote a secondary database cluster

Before you promote a secondary database cluster, perform the following steps to verify that the secondary database cluster has applied all transactions received from the primary database cluster:

### Kubernetes

- Check the replication status of the secondary database cluster, to ensure that it's both ready and healthy.

  ```bash
  kubectl get replication SECONDARY_REPLICATION_NAME
  ```
- Stop all writes to the primary database cluster. Run the following query on your primary database cluster to check the replication lag of the secondary database. Confirm that the result shows a minimal lag.

  A lag value of 0 is ideal. If the lag is greater than 0, you can still promote the secondary database cluster, at the risk of losing some recent transactions already committed on the primary database cluster.

  ```bash
   psql -h PRIMARY_HOST -U postgres -d postgres -c 'SELECT application_name, pg_wal_lsn_diff(pg_current_wal_lsn(), replay_lsn) AS replay_lag FROM pg_stat_replication;'
  ```

To promote a secondary database cluster to a primary database cluster, update the **control** field of your secondary database cluster's replication manifest to `promote`, and apply that on your Kubernetes cluster on the secondary data center.

### Kubernetes

```yaml
  apiVersion: v1
  kind: Secret
  metadata:
    name: ha-rep-pw-SECONDARY_DB_CLUSTER_NAME
  type: Opaque
  data:
    rep-user-pw: "ENCODED_PASSWORD"
  ---
  apiVersion: alloydbomni.dbadmin.goog/v1
  kind: Replication
  metadata:
    name: SECONDARY_REPLICATION_NAME
  spec:
    dbcluster:
      name: SECONDARY_DB_CLUSTER_NAME
    downstream:
      host: PRIMARY_HOST
      port: PRIMARY_PORT
      username: alloydbreplica
      password:
        name: ha-rep-pw-SECONDARY_DB_CLUSTER_NAME
      replicationSlotName: PRIMARY_REPLICATION_SLOT
      control: promote
```

## Perform a switchover

Before you perform a switchover, verify that the primary and the secondary database clusters that belong to both data centers are online and that the database clusters are in a healthy state.

To ensure data consistency of your primary and secondary database clusters during switchover, perform the following steps to verify that the secondary database cluster has applied all transactions received from the primary database cluster:

### Kubernetes

- Check the replication status of the secondary database cluster, to ensure that it's both ready and healthy.

  ```bash
  kubectl get replication SECONDARY_REPLICATION_NAME
  ```
- Stop all writes to the primary database cluster. Run the following query on your primary database cluster to check the replication lag of the secondary database. Confirm that the result shows a lag value of `0`.

  ```bash
   psql -h PRIMARY_HOST -U postgres -d postgres -c 'SELECT application_name, pg_wal_lsn_diff(pg_current_wal_lsn(), replay_lsn) AS replay_lag FROM pg_stat_replication;'
  ```

To perform a switchover, complete the following steps:

### Kubernetes

1. To convert your AlloyDB Omni secondary database cluster into a primary database cluster, update its replication manifest on your Kubernetes cluster in the secondary data center as follows:

   ```yaml
   apiVersion: v1
   kind: Secret
   metadata:
     name: ha-rep-pw-SECONDARY_DB_CLUSTER_NAME
   type: Opaque
   data:
     rep-user-pw: "ENCODED_PASSWORD"
   ---
   apiVersion: alloydbomni.dbadmin.goog/v1
   kind: Replication
   metadata:
     name: SECONDARY_REPLICATION_NAME
   spec:
     dbcluster:
       name: SECONDARY_DB_CLUSTER_NAME
     upstream:
       password:
         name: ha-rep-pw-SECONDARY_DB_CLUSTER_NAME
   ```

   Wait for the replication status to be ready.
2. To get the upstream connection information for replication, run the following command:

   ```bash
   kubectl get replication SECONDARY_REPLICATION_NAME
   kubectl get replication SECONDARY_REPLICATION_NAME -o json | jq .status.upstream
   ```

   Sample output looks similar to the following:

        {
          "host": "34.23.207.137",
          "password": {
            "name": "ha-rep-pw-dbc-2"
          },
          "port": 5432,
          "replicationSlotName": "dbc_2_replication_2",
          "username": "alloydbreplica"
        }

3. To convert your AlloyDB Omni primary database cluster into a secondary database cluster, update its replication manifest on your Kubernetes cluster on the primary data center similar to the following:

   ```yaml
   apiVersion: v1
   kind: Secret
   metadata:
     name: ha-rep-pw-DB_CLUSTER_NAME
   type: Opaque
   data:
     rep-user-pw: "ENCODED_PASSWORD"
   ---
   apiVersion: alloydbomni.dbadmin.goog/v1
   kind: Replication
   metadata:
     name: REPLICATION_NAME
   spec:
     dbcluster:
       name: DB_CLUSTER_NAME
     downstream:
       host: SECONDARY_HOST
       port: SECONDARY_PORT
       username: alloydbreplica
       password:
         name: ha-rep-pw-DB_CLUSTER_NAME
       replicationSlotName: SECONDARY_REPLICATION_SLOT
       control: rewind
   ```

   Wait for the replication status to become ready and healthy.
4. To verify the replication status use:

   ```bash
   kubectl get replication REPLICATION_NAME
   ```