> [!CAUTION]
> You are currently viewing version 1.3 of the Apigee hybrid documentation. **This version is end of life.** You should upgrade to a newer version. For more information, see [Supported versions](https://docs.cloud.google.com/apigee/docs/hybrid/supported-platforms.md.txt#supported-versions).

<button class="button button-with-icon cloud-select-dropdown__button button-primary" aria-haspopup="true">Version 1.3 (unsupported)</button>

- Supported versions:
- [v1.17 (latest)](https://docs.cloud.google.com/apigee/docs/hybrid/v1.17/multi-region.md.txt)
- [v1.16](https://docs.cloud.google.com/apigee/docs/hybrid/v1.16/multi-region.md.txt)
- [v1.15](https://docs.cloud.google.com/apigee/docs/hybrid/v1.15/multi-region.md.txt)
- [v1.14](https://docs.cloud.google.com/apigee/docs/hybrid/v1.14/multi-region.md.txt)
- [List of supported versions](https://docs.cloud.google.com/apigee/docs/hybrid/supported-platforms.md.txt#supported-versions)
- Unsupported versions:
- [v1.13](https://docs.cloud.google.com/apigee/docs/hybrid/v1.13/multi-region.md.txt)
- [v1.12](https://docs.cloud.google.com/apigee/docs/hybrid/v1.12/multi-region.md.txt)
- [v1.11](https://docs.cloud.google.com/apigee/docs/hybrid/v1.11/multi-region.md.txt)
- [v1.10](https://docs.cloud.google.com/apigee/docs/hybrid/v1.10/multi-region.md.txt)
- [v1.9](https://docs.cloud.google.com/apigee/docs/hybrid/v1.9/multi-region.md.txt)
- [v1.8](https://docs.cloud.google.com/apigee/docs/hybrid/v1.8/multi-region.md.txt)
- [v1.7](https://docs.cloud.google.com/apigee/docs/hybrid/v1.7/multi-region.md.txt)
- [v1.6](https://docs.cloud.google.com/apigee/docs/hybrid/v1.6/multi-region.md.txt)
- [v1.5](https://docs.cloud.google.com/apigee/docs/hybrid/v1.5/multi-region.md.txt)
- [v1.4](https://docs.cloud.google.com/apigee/docs/hybrid/v1.4/multi-region.md.txt)
- [v1.3](https://docs.cloud.google.com/apigee/docs/hybrid/v1.3/multi-region.md.txt)
- [v1.2](https://docs.cloud.google.com/apigee/docs/hybrid/v1.2/multi-region.md.txt)
- [v1.1](https://docs.cloud.google.com/apigee/docs/hybrid/v1.1/multi-region.md.txt)

This topic describes a multi-region deployment for Apigee hybrid on GKE and
Anthos GKE deployed on-prem.

Topologies for multi-region deployment include the following:

- **Active-Active**: When you have applications deployed in multiple geographic locations and you require low latency API response for your deployments. You have the option to deploy hybrid in multiple geographic locations nearest to your clients. For example: US West Coast, US East Coast, Europe, APAC.
- **Active-Passive**: When you have a primary region and a failover or disaster recovery region.

The regions in a multi-region hybrid deployment communicate via Cassandra, as the following image
shows:

![](https://docs.cloud.google.com/static/apigee/docs/hybrid/v1.3/images/arch-multi-region.png)

## Load balancing the MART connection


Each regional cluster must have its own MART IP and hostname; however, you only need
to connect the management plane to one of them. Cassandra propagates information to all
of the clusters. The best option for high availability for MART is to load balance
the individual MART IP addresses and configure your organization to talk to the load balanced
MART URL.

## Prerequisites

Before configuring hybrid for multiple regions, you must complete the following prerequisites:

- Set up Kubernetes clusters in multiple regions with different blocks
- Set up cross-region communication
- Open Cassandra ports 7000 and 7001 between Kubernetes clusters across all regions (7000 may be used as a backup option during troubleshooting). See also [Configure ports](https://docs.cloud.google.com/apigee/docs/hybrid/v1.3/ports.md.txt).

For detailed information, see [Kubernetes](https://kubernetes.io) documentation.

> [!NOTE]
> **NOTE:** Apigee recommends that you ensure that your servers' times are synchronized.
>
> Several features such as expiration and token revocation
> rely on accurate system times. If you host the runtime components in different datacenters,
> then be sure that the system times synchronized.
>
> You can use a tool such as `ntpdate` to verify that server times are
> synchronized.

## Configure the multi-region seed host

This section describes how to expand the existing Cassandra cluster to a new region.
This setup allows the new region to bootstrap the cluster and join the existing
data center. Without this configuration, the multi-region Kubernetes clusters
would not know about each other.

1. Run the following [`kubectl`](https://kubernetes.io/docs/reference/kubectl/overview/) command to identify a seed host address for Cassandra in
   the current region.

   A *seed host address* allows a new regional instance to find the
   original cluster on the very first startup to learn the topology of the cluster.
   The seed host address is designated as the contact point in the cluster.

   ```
   kubectl get pods -o wide -n apigee

   NAME                      READY   STATUS      RESTARTS   AGE   IP          NODE                                          NOMINATED NODE
   apigee-cassandra-default-0        1/1     Running     0          5d    10.0.0.11   gke-k8s-dc-2-default-pool-a2206492-p55d
   apigee-cassandra-default-1        1/1     Running     0          5d    10.0.2.4    gke-k8s-dc-2-default-pool-e9daaab3-tjmz
   apigee-cassandra-default-2        1/1     Running     0          5d    10.0.3.5    gke-k8s-dc-2-default-pool-e589awq3-kjch
   ```
2. Decide which of the IPs returned from the previous command will be the multi-region seed host.
3. The configuration in this step depends on whether you are on GKE or GKE on-prem:

   **GKE Only:** In data center 2, configure `cassandra.multiRegionSeedHost` and
   `cassandra.datacenter` in [Manage runtime plane components](https://docs.cloud.google.com/apigee/docs/hybrid/v1.3/customize-services.md.txt), where `multiRegionSeedHost`
   is one of the IPs returned by the previous command:

   ```
   cassandra:
     multiRegionSeedHost: seed_host_IP
     datacenter: data_center_name
     rack: rack_name
   ```

   For example:

   ```
   cassandra:
     multiRegionSeedHost: 10.0.0.11
     datacenter: "dc-2"
     rack: "ra-1"
   ```

   <br />

   **GKE on-prem Only:** In data center 2, configure
   `cassandra.multiRegionSeedHost` in your overrides file, where `multiRegionSeedHost`
   is one of the IPs returned by the previous command:

   ```
   cassandra:
     hostNetwork: true
     multiRegionSeedHost: seed_host_IP
   ```

   For example:

   ```
   cassandra:
     hostNetwork: true
     dnsPolicy: ClusterFirstWithHostNet
     multiRegionSeedHost: 10.0.0.11
   ```

   <br />

4. In the new data center/region, before you install hybrid, set the same TLS certificates and credentials in `overrides.yaml` as you set in the first region.

   > [!NOTE]
   > NOTE: Be sure to use the same Cassandra TLS certificates and credentials in the second data center as you provided in the original data center. The credentials you set in `overrides.yaml` in the first data center must match the ones you specify in `overrides.yaml` in the second data center. For details see [Configuring TLS for Cassandra](https://docs.cloud.google.com/apigee/docs/hybrid/v1.3/cassandra-tls.md.txt).

## Set up the new region

After you configure the seed host, you can set up the new region.

**To set up the new region:**

1. Copy your certificate from the existing cluster to the new cluster. The new CA root is used by Cassandra and other hybrid components for mTLS. Therefore, it is essential to have consistent certificates across the cluster.
   1. Set the context to the original namespace:

      ```
      kubectl config use-context original-cluster-name
      ```
   2. Export the current namespace configuration to a file:

      ```
      $ kubectl get namespace  -o yaml > apigee-namespace.yaml
      ```
   3. Export the `apigee-ca` secret to a file:

      ```
      kubectl -n cert-manager get secret apigee-ca -o yaml > apigee-ca.yaml
      ```
   4. Set the context to the new region's cluster name:

      ```
      kubectl config use-context new-cluster-name
      ```
   5. Import the namespace configuration to the new cluster. Be sure to update the "namespace" in the file if you're using a different namespace in the new region:

      ```
      kubectl apply -f apigee-namespace.yaml
      ```
   6. Import the secret to the new cluster:

      ```
      kubectl -n cert-manager apply -f apigee-ca.yaml
      ```
2. Install hybrid in the new region. Be sure that the `overrides-DC_name.yaml` file includes the same TLS certificates that are configured in the first region, as explained in the previous section.

   Execute the following two commands to install hybrid in the new region:

   ```
   apigeectl init -f overrides/overrides-DC_name.yaml
   ```

   ```
   apigeectl apply -f overrides/overrides-DC_name.yaml
   ```
3. Run `nodetool rebuild` sequentially on all the pods in new data center. This may take a few minutes to a few hours depending on the data size.

   ```
   kubectl exec apigee-cassandra-default-0 -n apigee  -- nodetool rebuild -- dc-1
   ```
4. Verify the rebuild processes from the logs. Also, verify the data size using the `nodetool status` command:

   ```
   kubectl logs apigee-cassandra-default-0 -f -n apigee
   ```

   The following example shows example log entries:

   ```
   INFO  01:42:24 rebuild from dc: dc-1, (All keyspaces), (All tokens)
   INFO  01:42:24 [Stream #3a04e810-580d-11e9-a5aa-67071bf82889] Executing streaming plan for Rebuild
   INFO  01:42:24 [Stream #3a04e810-580d-11e9-a5aa-67071bf82889] Starting streaming to /10.12.1.45
   INFO  01:42:25 [Stream #3a04e810-580d-11e9-a5aa-67071bf82889, ID#0] Beginning stream session with /10.12.1.45
   INFO  01:42:25 [Stream #3a04e810-580d-11e9-a5aa-67071bf82889] Starting streaming to /10.12.4.36
   INFO  01:42:25 [Stream #3a04e810-580d-11e9-a5aa-67071bf82889 ID#0] Prepare completed. Receiving 1 files(0.432KiB), sending 0 files(0.000KiB)
   INFO  01:42:25 [Stream #3a04e810-580d-11e9-a5aa-67071bf82889] Session with /10.12.1.45 is complete
   INFO  01:42:25 [Stream #3a04e810-580d-11e9-a5aa-67071bf82889, ID#0] Beginning stream session with /10.12.4.36
   INFO  01:42:25 [Stream #3a04e810-580d-11e9-a5aa-67071bf82889] Starting streaming to /10.12.5.22
   INFO  01:42:26 [Stream #3a04e810-580d-11e9-a5aa-67071bf82889 ID#0] Prepare completed. Receiving 1 files(0.693KiB), sending 0 files(0.000KiB)
   INFO  01:42:26 [Stream #3a04e810-580d-11e9-a5aa-67071bf82889] Session with /10.12.4.36 is complete
   INFO  01:42:26 [Stream #3a04e810-580d-11e9-a5aa-67071bf82889, ID#0] Beginning stream session with /10.12.5.22
   INFO  01:42:26 [Stream #3a04e810-580d-11e9-a5aa-67071bf82889 ID#0] Prepare completed. Receiving 3 files(0.720KiB), sending 0 files(0.000KiB)
   INFO  01:42:26 [Stream #3a04e810-580d-11e9-a5aa-67071bf82889] Session with /10.12.5.22 is complete
   INFO  01:42:26 [Stream #3a04e810-580d-11e9-a5aa-67071bf82889] All sessions completed
   ```
5. Update the seed hosts. Remove `multiRegionSeedHost: 10.0.0.11` from `overrides-DC_name.yaml` and reapply.

   > [!NOTE]
   > Seed hosts are local cluster members. To boot up a new region an external seed host is required. Once a region boots up you need to change the seed hosts back to their local clusters in `overrides.yaml` and then reapply the configuration.

   ```
   apigeectl apply -f overrides/overrides-DC_name.yaml
   ```

## Check the Cassandra cluster status


The following command is useful to see if the cluster setup is successful in two
data centers. The command checks the nodetool status for the two regions.

```
kubectl exec apigee-cassandra-default-0 -n apigee -- nodetool status


Datacenter: us-central1
=======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address     Load       Tokens       Owns (effective)  Host ID                               Rack
UN  10.12.1.45  112.09 KiB  256          100.0%            3c98c816-3f4d-48f0-9717-03d0c998637f  ra-1
UN  10.12.4.36  95.27 KiB  256          100.0%            0a36383d-1d9e-41e2-924c-7b62be12d6cc  ra-1
UN  10.12.5.22  88.7 KiB   256          100.0%            3561f4fa-af3d-4ea4-93b2-79ac7e938201  ra-1
Datacenter: us-west1
====================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address     Load       Tokens       Owns (effective)  Host ID                               Rack
UN  10.0.4.33   78.69 KiB  256          0.0%              a200217d-260b-45cd-b83c-182b27ff4c99  ra-1
UN  10.0.0.21   78.68 KiB  256          0.0%              9f3364b9-a7a1-409c-9356-b7d1d312e52b  ra-1
UN  10.0.1.26   15.46 KiB  256          0.0%              1666df0f-702e-4c5b-8b6e-086d0f2e47fa  ra-1
```

*[CIDR]: Classless Inter-Domain Routing