About vSAN encryption

Encryption of vSAN data at rest requires a key management system (KMS). Google Cloud VMware Engine offers several options for key management for vSAN data encryption:

  • Google-owned and Google-managed encryption keys (GMEK): By default, vSAN data encryption uses Google-owned and Google-managed encryption keys in Cloud Key Management Service for private clouds, at no additional cost.
  • Customer-managed encryption keys (CMEK): You can use CMEK in Cloud Key Management Service to manage encryption keys for individual private clouds.
  • External KMS: You can deploy an external KMS for encryption of vSAN data at rest from one of the supported vendors that follows.

Choose a key management solution

To help you choose a key management solution, review the following comparison:

Feature GMEK (Default) CMEK External KMS
Key management responsibility Google Customer Customer
Key storage Cloud Key Management Service (Google project) Cloud Key Management Service (Customer project) External KMS
Isolation level Regional service Individual private cloud Individual private cloud
Rotation automation Automated by Google Automated by Google upon KMS rotation Managed by customer
Additional license cost None Google Cloud KMS charges apply Third-party KMS licensing

This page explains vSAN encryption behavior and summarizes how to use an external KMS to encrypt virtual machine data at rest in VMware Engine.

vSAN data encryption

By default, VMware Engine enables vSAN encryption for data in the primary cluster and in clusters subsequently added to the private cloud. Encryption of vSAN data at rest uses a data encryption key (DEK) that's stored on the local physical disk of the cluster after encryption. The DEK is a FIPS 140-2 Level 1 compliant AES-256 bit encryption key auto-generated by ESXi hosts. The system uses a key encryption key (KEK) supplied by Cloud Key Management Service or an external KMS to encrypt the DEK.

We strongly recommend against disabling vSAN encryption of data at rest, as it can put you in violation of the service specific terms for Google Cloud VMware Engine. When you disable vSAN encryption of data at rest on a cluster, the VMware Engine monitoring logic raises an alert. To help prevent you from being in violation of service terms, this alert triggers a Cloud Customer Care-driven action to re-enable vSAN encryption on the affected cluster.

Similarly, if you configure an external KMS, we strongly recommend against deleting the key provider configuration of Cloud Key Management Service in vCenter Server.

Google-owned and Google-managed encryption keys (GMEK)

By default, private clouds use Google-owned and Google-managed encryption keys (GMEK). With GMEK, VMware Engine configures vCenter Server to connect to a GMEK provider. VMware Engine creates one instance of the key provider per region, and the key provider uses Cloud KMS for encryption of the KEK. VMware Engine fully manages the key provider and configures it to be highly available in all regions.

The GMEK provider complements the built-in key provider in vCenter Server (in vSphere 7.0 Update 2 and later) and is the recommended approach for production environments. The built-in key provider runs as a process within the vCenter Server, which runs on a vSphere cluster in VMware Engine. VMware recommends against using the built-in key provider for encrypting the cluster that hosts vCenter Server. Instead, use GMEK, CMEK, or an external KMS.

GMEK auto key rotation

Google Cloud VMware Engine periodically triggers an automatic key rotation of GMEK for all clusters. This is a background maintenance task that requires no user intervention and does not impact cluster availability.

Customer-managed encryption keys (CMEK)

Google Cloud VMware Engine supports Customer-Managed Encryption Keys (CMEK) for individual private clouds using Cloud Key Management Service. You can configure each private cloud with a distinct CMEK key, rather than sharing a single key across the entire project. This provides the following benefits:

  • Enhanced isolation: Compromise or rotation of a key for one private cloud doesn't affect others.
  • Granular compliance: Assign keys based on the specific data residency or compliance requirements of the workloads within each private cloud.

Managing CMEK encryption

You can configure and manage customer-managed encryption keys (CMEK) for your private cloud using the Google Cloud console and the VMware Engine API. To use CMEK, you must grant the roles/cloudkms.cryptoKeyEncrypterDecrypter IAM role to your VMware Engine service account (service-<var>PROJECT_NUMBER</var>@gcp-sa-vmwareengine.iam.gserviceaccount.com) for the specific Cloud KMS key you intend to use. This permission allows VMware Engine to encrypt and decrypt the Key Encryption Key (KEK) that protects your vSAN data using the selected Cloud KMS key in your project.

Assign key permissions

Before you can enable CMEK, you must assign the necessary permissions:

  1. Identify your KMS resource: Go to the Key Management page in the Google Cloud console and find the specific Cloud KMS key to use for data protection.

    Go to Key Management

  2. Set up IAM roles: Grant the roles/cloudkms.cryptoKeyEncrypterDecrypter role to the VMware Engine service account (service-<var>PROJECT_NUMBER</var>@gcp-sa-vmwareengine.iam.gserviceaccount.com) directly on the Cloud KMS key dashboard.

Google Cloud console and VMware Engine API instructions for managing CMEK:

Google Cloud console

To configure customer-managed encryption keys (CMEK) using the Google Cloud console, do the following:

Create a private cloud with CMEK

To create a private cloud with CMEK using the Google Cloud console:

  1. In the Google Cloud console, go to the Private clouds page.

    Go to Private clouds

  2. Click Create.

  3. In the Encryption section, choose Customer-managed encryption keys (CMEK).

  4. Enter the full resource name of your Cloud KMS key.

  5. Click Create or Save.

Update the CMEK key for an existing private cloud

To update the CMEK key for an existing private cloud using the Google Cloud console:

  1. In the Google Cloud console, go to the Private clouds page.

    Go to Private clouds

  2. Select the existing private cloud to update.

  3. In the Encryption section, choose Customer-managed encryption keys (CMEK).

  4. Enter the full resource name of your Cloud KMS key.

  5. Click Create or Save.

Update encryption to GMEK for an existing private cloud

To update the encryption type to GMEK for an existing private cloud using the Google Cloud console:

  1. In the Google Cloud console, go to the Private clouds page.

    Go to Private clouds

  2. Select the existing private cloud to update.

  3. In the Encryption section, choose Google-owned and Google-managed encryption keys (GMEK).

  4. Click Create or Save.

VMware Engine API

To use the API commands, first replace the following variables with your environment's values:

  • PROJECT_ID: Your Google Cloud project ID.
  • LOCATION: The VMware Engine region, such as us-east4.
  • PC_ID: The ID for your new private cloud.
  • NETWORK_ID: The ID of your VPC network.
  • CIDR_RANGE: The management CIDR range for the private cloud.
  • CLUSTER_ID: The ID for the management cluster.
  • NODE_TYPE: The node type, for example, standard-72.
  • COUNT: The number of nodes in the cluster.
  • RING: The name of the Cloud KMS key ring.
  • KEY: The name of the Cloud KMS crypto key.
  • PC_NAME: The name of the existing private cloud.

To create a private cloud with CMEK, use the privateClouds.create method and specify the encryptionConfig parameter:

curl -X POST \
  -H "Authorization: Bearer $(gcloud auth print-access-token)" \
  -H "Content-Type: application/json; charset=utf-8" \
  "https://vmwareengine.googleapis.com/v1/projects//locations//privateClouds?private_cloud_id=" \
  -d '{
  "networkConfig": {
    "vmwareEngineNetwork": "projects//locations/global/vmwareEngineNetworks/",
    "managementCidr": ""
  },
  "managementCluster": {
    "clusterId": "",
    "nodeTypeConfigs": {
      "": {
        "nodeCount": 
      }
    }
  },
  "encryptionConfig": {
    "cryptoKeyName": "projects//locations//keyRings//cryptoKeys/"
  }
}'

To update the CMEK key for an existing private cloud, use the privateClouds.patch method:

curl -X PATCH \
  -H "Authorization: Bearer $(gcloud auth print-access-token)" \
  -H "Content-Type: application/json; charset=utf-8" \
  "https://vmwareengine.googleapis.com/v1/projects//locations//privateClouds/?updateMask=encryptionConfig" \
  -d '{
  "encryptionConfig": {
    "cryptoKeyName": "projects//locations//keyRings//cryptoKeys/"
  }
}'

To switch an existing private cloud back to GMEK (Google-owned and Google-managed encryption keys), use the privateClouds.patch method with an empty encryptionConfig object:

curl -X PATCH \
  -H "Authorization: Bearer $(gcloud auth print-access-token)" \
  -H "Content-Type: application/json; charset=utf-8" \
  "https://vmwareengine.googleapis.com/v1/projects//locations//privateClouds/?updateMask=encryptionConfig" \
  -d '\''{
  "encryptionConfig": {}
}'\''

Understanding CMEK constraints and limitations

  • Regional requirements: Both the private cloud and your selected Cloud KMS key must reside in the same region. Note that Google Cloud KMS global keys are unsupported for vSAN encryption.
  • Key availability: If the encryption key becomes inaccessible due to being disabled or revoked, all dependent vSAN operations will fail. Vital procedures, including host reboots and shallow re-keying, cannot proceed until key access is successfully re-established.

Operational efficiency with CMEK automatic KEK rotation

For organizations choosing CMEK, the automatic KEK rotation feature for vSAN data-at-rest encryption automates manual security tasks:

  • Automated synchronization: The system monitors Cloud KMS and automatically detects key rotations.
  • Seamless background KEK rotation: Upon rotation, the system initiates a shallow KEK rotation, ensuring continuous protection without manual intervention in vCenter. This operation updates the key encryption key for your Data Encryption Keys (DEKs) with a new Key Encryption Key (KEK). Because DEKs encrypt your vSAN data directly, this process doesn't require a full re-encryption of your vSAN data.
  • Zero-downtime management: The system generates and applies KEKs in the background, maintaining performance and availability for workloads. This process typically completes within 48 hours following a Cloud KMS key rotation.

Requirement for encrypted virtual machines

You can manage encryption keys for VMs by using the default Google-owned and managed key provider or Cloud Key Management Service.

If you enable VM Encryption (or vTPM) for any VMs in your private cloud and use a KMS to manage encryption keys, you must re-encrypt (shallow re-key) each VM after you rotate your KMS key.

A shallow re-key replaces only the key encryption key (KEK) and doesn't change the data encryption key (DEK) of the VMs. You typically trigger a shallow re-key by using the Re-encrypt action in the vSphere Client.

During this operation, the system re-wraps (re-encrypts) the existing DEK by using a new KEK. This process is fast because it doesn't rewrite actual data on the disk; instead, it only updates the small key bundle that contains the encrypted DEK. For more information, see the following VMware documentation:

Risks of failing to re-key encrypted VMs

Failing to re-key encrypted VMs before you delete the rotated (old) KMS key version can cause the following issues:

  • Failed vMotions: ESXi hosts can't decrypt VM DEKs during vMotion if you reboot the destination hosts or add them to the cluster after the KMS key rotation but before you perform the VM re-key.
  • Power-on failures: If a host reboots or clears its local key cache, it can't reacquire keys from the KMS. If you deleted the required keys from the KMS, the host can't decrypt the DEK, which prevents encrypted VMs from powering on.

Steps to perform a re-key operation on workload VMs

  1. In the vSphere Client, right-click the VM.
  2. Select VM Policies > Re-encrypt.
  3. Confirm the re-encryption request in the dialog that appears.
  4. Wait for the task to complete.
  5. Verify the re-key by migrating the VM to a host that you rebooted or added to the cluster after the KMS key rotation.

External KMS

Optionally, you can select a third-party KMS solution that's KMIP 1.1 compliant and certified by VMware for vSAN. The following vendors have validated their KMS solution with VMware Engine and published deployment guides and support statements:

For configuration instructions, see the following documents:

Use a supported vendor

Each deployment of an external KMS requires the same basic steps:

  • Create a Google Cloud project or use an existing one.
  • Create a new Virtual Private Cloud (VPC) network or choose an existing VPC network.
  • Connect your selected VPC network to the VMware Engine network.

Then, deploy the KMS in a Compute Engine VM instance:

  1. Set up required IAM permissions to deploy Compute Engine VM instances.
  2. Deploy the KMS in Compute Engine.
  3. Establish trust between vCenter and the KMS.
  4. Enable vSAN data encryption.

The following sections briefly describe this process of using one of the supported vendors.

Set up IAM Permissions

You need sufficient permissions to deploy Compute Engine VM instances in a given Google Cloud project and VPC network, connect your VPC network to VMware Engine, and configure firewall rules for the VPC network.

Project owners and IAM principals with the Network Admin role can create allocated IP ranges and manage private connections. For more information on roles, see Compute Engine IAM roles.

Deploy key management system in Compute Engine

Some KMS solutions are available in an appliance form-factor in Google Cloud Marketplace. You can deploy such appliances by importing the OVA directly in your VPC network or Google Cloud project.

For software-based KMS, deploy a Compute Engine VM instance using the configuration (vCPU count, vMem, and disks) recommended by the KMS vendor. Install the KMS software in the guest operating system. Create the Compute Engine VM instance in a VPC network that is connected to VMware Engine network.

Establish trust between vCenter and the KMS

After deploying the KMS in Compute Engine, configure your VMware Engine vCenter to retrieve encryption keys from the KMS.

First add KMS connection details to vCenter. Then, establish trust between vCenter and your KMS. To establish trust between vCenter and your KMS, do the following:

  1. Generate a certificate in vCenter.
  2. Sign it using a token or key generated by your KMS.
  3. Provide or upload that certificate to vCenter.
  4. Confirm the connectivity status by checking the KMS setting and status in the vCenter server configuration page.

Enable vSAN data encryption

In vCenter, the default CloudOwner user has sufficient privileges to enable and manage vSAN data encryption.

To switch from an external KMS back to the default Google-owned and managed key provider, follow the steps for changing the key provider provided in the VMware documentation Configuring and Managing a Standard Key Provider.

What's next