> [!CAUTION]
> You are currently viewing version 1.13 of the Apigee hybrid documentation. **This version is end of life.** You should upgrade to a newer version. For more information, see [Supported versions](https://docs.cloud.google.com/apigee/docs/hybrid/supported-platforms.md.txt#supported-versions).

<button class="button button-with-icon cloud-select-dropdown__button button-primary" aria-haspopup="true">Version 1.13 </button>

- Supported versions:
- [v1.17 (latest)](https://docs.cloud.google.com/apigee/docs/hybrid/v1.17/rotating-cassandra-credentials-in-vault.md.txt)
- [v1.16](https://docs.cloud.google.com/apigee/docs/hybrid/v1.16/rotating-cassandra-credentials-in-vault.md.txt)
- [v1.15](https://docs.cloud.google.com/apigee/docs/hybrid/v1.15/rotating-cassandra-credentials-in-vault.md.txt)
- [v1.14](https://docs.cloud.google.com/apigee/docs/hybrid/v1.14/rotating-cassandra-credentials-in-vault.md.txt)
- [List of supported versions](https://docs.cloud.google.com/apigee/docs/hybrid/supported-platforms.md.txt#supported-versions)
- Unsupported versions:
- [v1.13](https://docs.cloud.google.com/apigee/docs/hybrid/v1.13/rotating-cassandra-credentials-in-vault.md.txt)
- [v1.12](https://docs.cloud.google.com/apigee/docs/hybrid/v1.12/rotating-cassandra-credentials-in-vault.html)
- [v1.11](https://docs.cloud.google.com/apigee/docs/hybrid/v1.11/rotating-cassandra-credentials-in-vault.html)
- [v1.10](https://docs.cloud.google.com/apigee/docs/hybrid/v1.10/rotating-cassandra-credentials-in-vault.html)
- [v1.9](https://docs.cloud.google.com/apigee/docs/hybrid/v1.9/rotating-cassandra-credentials-in-vault.html)
- [v1.8](https://docs.cloud.google.com/apigee/docs/hybrid/v1.8/rotating-cassandra-credentials-in-vault.html)
- [v1.7](https://docs.cloud.google.com/apigee/docs/hybrid/v1.7/rotating-cassandra-credentials-in-vault.html)
- [v1.6](https://docs.cloud.google.com/apigee/docs/hybrid/v1.6/rotating-cassandra-credentials-in-vault.html)
- [v1.5](https://docs.cloud.google.com/apigee/docs/hybrid/v1.5/rotating-cassandra-credentials-in-vault.html)
- [v1.4](https://docs.cloud.google.com/apigee/docs/hybrid/v1.4/rotating-cassandra-credentials-in-vault.html)
- [v1.3](https://docs.cloud.google.com/apigee/docs/hybrid/v1.3/rotating-cassandra-credentials-in-vault.html)
- [v1.2](https://docs.cloud.google.com/apigee/docs/hybrid/v1.2/rotating-cassandra-credentials-in-vault.html)
- [v1.1](https://docs.cloud.google.com/apigee/docs/hybrid/v1.1/rotating-cassandra-credentials-in-vault.html)

## Overview

This feature allows platform administrators to:

- Rotate Cassandra credentials in Hashicorp Vault.
- Roll back to the previous Cassandra credentials in Vault in case of any issues during password rotation.
- Rotate the Cassandra password for one region at a time, so that you can ensure minimal impact on service availability and maintain control over the rotation process.
- Track the start, progress, and completion of the rotation for a single region.


This feature is available in Apigee Hybrid 1.13.1 and later.

> [!NOTE]
> **Note:** Before using this feature, you must have Vault storage of Cassandra credentials set up. Follow the instructions in [Storing Cassandra secrets in Hashicorp Vault](https://docs.cloud.google.com/apigee/docs/hybrid/v1.13/storing-cassandra-secrets-in-vault.md.txt)

## Before you begin


Before setting up credential rotation:

- Backup your Cassandra database. This backup is to ensure recovery is possible to pre-rotated credentials.
- Ensure the cluster is in a healthy state (i.e. all Apigee resources are running, no state changes are pending).

> [!NOTE]
> **Note:** You must have permission to run `kubectl apply -f` within the cluster to set up credential rotation.

## Single region setup

1. Create a new `SecretProviderClass` Kubernetes resource in your Apigee namespace for the new Cassandra credentials. See [Storing Cassandra secrets in Hashicorp Vault](https://docs.cloud.google.com/apigee/docs/hybrid/v1.13/storing-cassandra-secrets-in-vault.md.txt#create-secretproviderclass-object) for a template to use. This allows a Vault role to access secrets within the Kubernetes namespaces.
2. Create a new `SecretRotation` custom resource using the following template:

   ```
   # rotation.yaml

   apiVersion: apigee.cloud.google.com/v1alpha1
   kind: SecretRotation
   metadata:
     name: ROTATION_PROCESS_NAME
     namespace: APIGEE_NAMESPACE
   spec:
     organizationId: ORG_NAME
     rotationId: ROTATION_ID
     timeoutMinutes: 480 # optional. overrides the default (480m == 8hr).
                         # less than or equal to 0 means infinite timeout.
     precheck: true
     cassandra:
       oldSecretProviderClass: OLD_SPC_NAME
       newSecretProviderClass: NEW_SPC_NAME
       jobType: ROTATE
   ```
   - <var translate="no">ROTATION_PROCESS_NAME</var>: A unique name for the rotation job. You will need to set `metadata.name` to a unique value for the rotation precheck job and again for the rotation job. For example `sr-1-precheck` followed by `sr-1`.
   - <var translate="no">ROTATION_ID</var>: Set `spec.rotationId` to a custom identifier, for example `rotation-1-precheck`.
   - <var translate="no">NEW_SPC_NAME</var>: Set `spec.cassandra.newSecretProviderClass` to the new secret provider class name you created in the previous step.
   - <var translate="no">OLD_SPC_NAME</var>: Set `spec.cassandra.oldSecretProviderClass` to the SPC name currently being used by the `ApigeeDatastore`.
3. Trigger the rotation precheck job by applying the `rotation.yaml` file.

   ```
   kubectl -n APIGEE_NAMESPACE apply -f rotation.yaml
   ```
4. Check the job status to verify when the precheck job is complete.

   ```
   kubectl -n APIGEE_NAMESPACE get job sr-(rotationId)-(rotate|rollback|cleanup)-job
   ```

   > [!TIP]
   > **Tip:** You can also view the logs while the job is running.
   >
   > ```
   > kubectl -n APIGEE_NAMESPACE logs sr-(rotationId)-(rotate|rollback|cleanup)-job-(hash)
   > ```

5. Once the rotation precheck job completes, change the value of `metadata.name` and set `spec.precheck` to `false`. Apply the file again to perform the rotation.

   ```
   kubectl -n APIGEE_NAMESPACE apply -f rotation.yaml
   ```
6. After the rotation job completes and you have validated traffic is still flowing correctly, clean up the process with the following two steps:
   1. Update the value of `metadata.name` and set `spec.cassandra.jobType` to `CLEANUP`.
   2. Trigger the cleanup job by applying the file.

      ```
      kubectl -n APIGEE_NAMESPACE apply -f rotation.yaml
      ```

      > [!NOTE]
      > **Note:** Up until this step, you can trigger a [rollback](https://docs.cloud.google.com/apigee/docs/hybrid/v1.13/rotating-cassandra-credentials-in-vault.md.txt#rollback-of-a-rotation) job to revert the rotation process and leave the cluster in its original state.

   When the cleanup job is completed, the rotation process is complete.
7. Backup your Cassandra database. This backup is to ensure recovery is possible to post-rotated credentials.
8. Delete the old Cassandra credentials, role, and policy from Vault.

## Multi-region setup

The multi-region setup procedures are divided into two sections: setup for the first region and setup for the remaining regions.

1. Complete the following steps in the first region before starting the subsequent regions.
   1. Create a new `SecretProviderClass` Kubernetes resource in the `APIGEE_NAMESPACE` namespace for the new Cassandra credentials. See [Storing Cassandra secrets in Hashicorp Vault](https://docs.cloud.google.com/apigee/docs/hybrid/v1.13/storing-cassandra-secrets-in-vault.md.txt#create-secretproviderclass-object) for a template to use. This allows a Vault role to access secrets within the Kubernetes namespaces.
   2. Create a new `SecretRotation` custom resource using the following template:

      ```
      # rotation.yaml

      apiVersion: apigee.cloud.google.com/v1alpha1
      kind: SecretRotation
      metadata:
        name: ROTATION_PROCESS_NAME
        namespace: APIGEE_NAMESPACE
      spec:
        organizationId: ORG_NAME
        rotationId: ROTATION_ID
        timeoutMinutes: -1 # this value is required and should not be changed.
        precheck: true
        cassandra:
          oldSecretProviderClass: OLD_SPC_NAME
          newSecretProviderClass: NEW_SPC_NAME
          jobType: ROTATE
      ```
      - <var translate="no">ROTATION_PROCESS_NAME</var>: A unique name for the rotation job. You will need to set `metadata.name` to a unique value for the rotation precheck job and again for the rotation job. For example `sr-1-precheck` followed by `sr-1`.
      - <var translate="no">ROTATION_ID</var>: Set `spec.rotationId` to a custom identifier, for example `rotation-1-precheck`.
      - <var translate="no">NEW_SPC_NAME</var>: Set `spec.cassandra.newSecretProviderClass` to the new secret provider class name you created in the previous step.
      - <var translate="no">OLD_SPC_NAME</var>: Set `spec.cassandra.oldSecretProviderClass` to the SPC name currently being used by the `ApigeeDatastore`.
   3. Trigger the rotation precheck job by applying the `rotation.yaml` file.

      ```
      kubectl -n APIGEE_NAMESPACE apply -f rotation.yaml
      ```
   4. Check the job status to verify when the precheck job is complete.

      ```
      kubectl -n APIGEE_NAMESPACE get job sr-(rotationId)-(rotate|rollback|cleanup)-job
      ```

      > [!TIP]
      > **Tip:** You can also view the logs while the job is running.
      >
      > ```
      > kubectl -n APIGEE_NAMESPACE logs sr-(rotationId)-(rotate|rollback|cleanup)-job-(hash)
      > ```

   5. Once the rotation precheck job completes:
      - Change the value of `metadata.name`, for example from `sr-1-precheck` to `sr-1`.
      - Set `spec.precheck` to `false` to turn off the precheck and perform the rotation.
      - Set `spec.rotationId` to a new identifier, for example `rotation-1`.
   6. Apply the file again to perform the rotation.

      ```
      kubectl -n APIGEE_NAMESPACE apply -f rotation.yaml
      ```
   7. Check the state of the `SecretRotation` and wait until it is `complete`.

      ```
      kubectl -n APIGEE_NAMESPACE get sr SR_NAME
      ```
2. In each subsequent region, complete the following steps:
   1. Create a new `SecretProviderClass` Kubernetes resource in your Apigee namespace for the new Cassandra credentials. See [Storing Cassandra secrets in Hashicorp Vault](https://docs.cloud.google.com/apigee/docs/hybrid/v1.13/storing-cassandra-secrets-in-vault.md.txt#create-secretproviderclass-object) for a template to use. This should be the same definition as step 1a.
   2. Update your `overrides.yaml` and set `cassandra.auth.secretProviderClass` to the match the value of `spec.cassandra.newSecretProviderClass` in the `rotation.yaml` file.

      ```
      cassandra:
        auth:
          secretProviderClass: NEW_SPC_NAME
      ```
   3. Apply the operator chart:

      ```
      helm upgrade operator apigee-operator/ \
        --namespace APIGEE_NAMESPACE \
        --atomic \
        -f OVERRIDES_FILE
      ```
   4. A new `ReplicaSet` will be created. Check that the new controller-manager pods are using the new SPC:

          export POD=NEW_CONTROLLER_MANAGER_POD_NAME
          kubectl -n APIGEE_NAMESPACE get pods $POD -o jsonpath='{.spec.volumes[?(@.name=="apigee-external-secrets")].csi.volumeAttributes.secretProviderClass}'


      The result should match the value you set for `spec.cassandra.newSecretProviderClass` in `rotation.yaml`, for example:

          kubectl -n apigee get pods $POD -o jsonpath='{.spec.volumes[?(@.name=="apigee-external-secrets")].csi.volumeAttributes.secretProviderClass}'

          my-new-spc

   5. Apply the datastore chart:

      ```
      helm upgrade datastore apigee-datastore/ \
        --namespace APIGEE_NAMESPACE \
        --atomic \
        -f OVERRIDES_FILE
      ```

      > [!NOTE]
      > **Note:** Applying the datastore chart will also trigger releases for the organization and environments.

   6. The datastore will go into a releasing state. Wait until the datastore has finished releasing and is in the running state.

      ```
      kubectl -n APIGEE_NAMESPACE get apigeedatastore DATASTORE_NAME
      ```

      <var translate="no">DATASTORE_NAME</var> is `default` in most installations.
   7. Check that the new datastore pods are using the new SPC:

          export POD=NEW_DATASTORE_POD_NAME
          kubectl -n APIGEE_NAMESPACE get pods $POD -o jsonpath='{.spec.volumes[?(@.name=="apigee-external-secrets")].csi.volumeAttributes.secretProviderClass}'


      The result should match the value you set for `spec.cassandra.newSecretProviderClass` in `rotation.yaml`, for example:

          kubectl -n apigee get pods $POD -o jsonpath='{.spec.volumes[?(@.name=="apigee-external-secrets")].csi.volumeAttributes.secretProviderClass}'

          my-new-spc

   8. Wait until the organization and environments are done releasing and have returned to the running state.

          kubectl -n APIGEE_NAMESPACE get apigeeorg ORG_NAME
          kubectl -n APIGEE_NAMESPACE get apigeeenv ENV_NAME

   9. Check that the new MART, runtime, and synchronizer pods are using the new SPC:

          export POD=NEW_MART_POD_NAME
          kubectl -n APIGEE_NAMESPACE get pods $POD -o jsonpath='{.spec.volumes[?(@.name=="apigee-external-secrets")].csi.volumeAttributes.secretProviderClass}'
          export POD=NEW_RUNTIME_POD_NAME
          kubectl -n APIGEE_NAMESPACE get pods $POD -o jsonpath='{.spec.volumes[?(@.name=="apigee-external-secrets")].csi.volumeAttributes.secretProviderClass}'
          export POD=NEW_SYNCHRONIZER_POD_NAME
          kubectl -n APIGEE_NAMESPACE get pods $POD -o jsonpath='{.spec.volumes[?(@.name=="apigee-external-secrets")].csi.volumeAttributes.secretProviderClass}'


      The result should match the value you set for `spec.cassandra.newSecretProviderClass` in `rotation.yaml`, for example:

          kubectl -n apigee get pods $POD -o jsonpath='{.spec.volumes[?(@.name=="apigee-external-secrets")].csi.volumeAttributes.secretProviderClass}'

          my-new-spc

3. After completing the steps in every region and validate traffic is still flowing correctly, clean up the process in the first region with the following two steps:
   1. In the first region, update the value of `metadata.name` and set `spec.cassandra.jobType` to `CLEANUP`.
   2. Trigger the cleanup job by applying the file.

      ```
      kubectl -n APIGEE_NAMESPACE apply -f rotation.yaml
      ```
   3. Check the job status and watch the job logs to verify when the cleanup job is complete.

   When the cleanup job is completed, the rotation process is complete.
4. Backup your Cassandra database. This backup is to ensure recovery is possible to post-rotated credentials.
5. Delete the old Cassandra credentials, role, and policy from Vault.