Always-on secrets encryption lets you encrypt secrets in Google Distributed Cloud
clusters before they're stored in etcd, without an external Key Management
Service (KMS) or other dependencies. You can enable the feature, find where the
encryption keys are stored so that you can back them up, rotate keys, and
disable encryption. Always-on secrets encryption is available for
[advanced clusters](https://docs.cloud.google.com/kubernetes-engine/distributed-cloud/vmware/docs/how-to/user-cluster-configuration-file-latest.md.txt#enable-advanced-cluster-field) in
versions 1.32 and later.

## Enable always-on secrets encryption

Always-on secrets encryption works by automatically generating an encryption key that is used to encrypt secrets before they are stored on the etcd database for that cluster. The secrets are encrypted using a Data Encryption Key (DEK) generated for every Secret write. This DEK is used in AES-GCM mode to encrypt the Secret object. The DEK is in turn encrypted with a Key Encrypting Key (KEK). The encryption algorithm is A256GCM (AES GCM using 256-bit key).

The key version is a version number to indicate the key currently in use.

You can enable secrets encryption after a cluster has already been created.

- For the admin cluster:

  1. Edit the [admin cluster configuration file](https://docs.cloud.google.com/kubernetes-engine/distributed-cloud/vmware/docs/how-to/admin-cluster-configuration-file-latest.md.txt#secretsencryption-section) to add the `secretsEncryption` section.

  2. Run the `gkectl update` command.

     ```
     gkectl update admin --config ADMIN_CLUSTER_CONFIG_FILE --kubeconfig ADMIN_CLUSTER_KUBECONFIG
     ```
- For a user cluster:

  1. Edit the [user cluster configuration file](https://docs.cloud.google.com/kubernetes-engine/distributed-cloud/vmware/docs/how-to/user-cluster-configuration-file-latest.md.txt#secretsencryption-section) to add the `secretsEncryption` section.

  2. Run the `gkectl update` command.

     ```
     gkectl update cluster --config USER_CONFIG_FILE --kubeconfig ADMIN_CLUSTER_KUBECONFIG
     ```

Replace the following:

- `ADMIN_CLUSTER_KUBECONFIG` with the path of your admin cluster kubeconfig file.
- `ADMIN_CLUSTER_CONFIG_FILE` with the path of your admin cluster configuration file.
- `USER_CONFIG_FILE` with the path of your user cluster configuration file.

The `gkectl update` commands provided in this section can also be used for any other updates to the corresponding cluster.

## Key storage

The encryption keys for the admin cluster are stored on the admin cluster data disk. This disk is mounted on the admin master machine at /opt/data, and the encryption keys can be found at /opt/data/gke-k8s-kms-plugin/generatedkeys/. These keys must be backed up to retain access to the encrypted secrets used by that key. You should enable [VM/storage encryption in the hypervisor](https://techdocs.broadcom.com/us/en/vmware-cis/vsphere/vsphere/9-1/vsphere-security.html) or similar measures to ensure that the control plane VM disks are protected.

## Key rotation

To rotate an existing encryption key for a cluster, increment the `keyVersion` in the corresponding [admin cluster configuration file](https://docs.cloud.google.com/kubernetes-engine/distributed-cloud/vmware/docs/how-to/admin-cluster-configuration-file-latest.md.txt#secretsencryption-generatedkey-version-field) or [user cluster configuration file](https://docs.cloud.google.com/kubernetes-engine/distributed-cloud/vmware/docs/how-to/user-cluster-configuration-file-latest.md.txt#secretsencryption-generatedkey-version-field), and run the appropriate `gkectl update` command. This creates a new key matching the new version number, re-encrypts each secret, and securely erases the old one. All subsequent new secrets are encrypted using the new encryption key.

## Disable always-on secrets encryption

To disable secrets encryption on an existing cluster, add a `disabled: true` field. Next, run the corresponding `gkectl update` command. This update decrypts each existing secret and stores each secret in plain text. All subsequent new secrets are stored in plain text.

```
secretsEncryption:
  mode: GeneratedKey
  generatedKey:
    keyVersion: KEY_VERSION
    disabled: true
```