Customer Managed Encryption Keys (CMEK) in Agent Retrieval

Agent Retrieval (formerly Vector Search 2.0) lets you protect your data at rest using cryptographic keys managed through Google Cloud Key Management Service (KMS). This integration provides control over the keys used to encrypt core customer content, including the ability to rotate and revoke those keys.

By default, data is encrypted using Google-managed platform defaults. CMEK provides you with direct control over the encryption keys.

For more information, go to the Customer-managed encryption keys (CMEK) documentation.

Prerequisites and constraints

The following prerequisites and constraints must be met in order to use CMEK with a Collections:

The Cloud KMS key must reside in the same Google Cloud region as the Collection it protects. For example, a Collection in the us-central1 region must use a KMS key located in us-central1.
You must grant your Per-Product-Per-Project (P4) Service Account the cryptoKeyEncrypterDecrypter role on your KMS key. The P4 Service Account follows this format: service-${CONSUMER_PROJECT_NUMBER}@gcp-sa-vectorsearch.iam.gserviceaccount.com

Creating a CMEK-protected Collection

After you meet all pre-requistites, you can use the gcloud CLI, REST API, or Python SDK to create a new Collection with CMEK enabled.

Using the gcloud CLI

Specify the fully qualified resource ID of your KMS key using the --kms-key argument:

gcloud vector-search collections create [COLLECTION_ID] \
    --kms-key projects/[KMS_PROJECT_ID]/locations/[REGION]/keyRings/[KEY_RING]/cryptoKeys/[KEY]

Using the REST API

Include the encryption_spec in the body of your POST request:

{
  "encryption_spec": {
    "crypto_key_name": "projects/[PROJECT_ID]/locations/[REGION]/keyRings/[KEY_RING]/cryptoKeys/[KEY]"
  }
}

Using the Python SDK

Using the Python SDK:

   
request = vectorsearch_v1beta.CreateCollectionRequest(
    parent=f"projects/{PROJECT_ID}/locations/{LOCATION}",
    collection_id=collection_id,
    collection={
        "data_schema": {...},
        "vector_schema": {...},
        # Specifies the customer-managed encryption key spec for a Collection
        "encryption_spec": {
            "crypto_key_name": f"projects/{PROJECT_ID}/locations/{LOCATION}/keyRings/{KEY_RING}/cryptoKeys/{KEY}"
        }
    },
)

operation = vector_search_service_client.create_collection(request=request)
operation.result()

Migrating Data Objects from a non-CMEK Collection into a CMEK-enabled Collection

CMEK cannot be enabled for existing Collections. You must instead migrate Data Objects into a new Collection with CMEK enabled.

To do so:

Export your Data Objects from an existing Collection into a Cloud Storage bucket.
Create a new Collection with CMEK enabled.
Import your Data Objects into the Collection.

Key rotation

When a key is rotated in KMS, new data is encrypted using the primary key version, while existing data remains encrypted with the previous version. For more details, see Key rotation.

Key revocation

Revoking or disabling a key immediately disables the Collection. Associated ANN indexes are automatically undeployed and in-memory data is purged. Read, write, and search operations on the Collection will fail.

Collections aren't automatically re-enabled if the key is restored.

CMEK-supported resources

The current Vertex AI resources covered by CMEK are as follows. CMEK support for Preview features is in Preview status as well.

Resource	Material encrypted	Documentation links
Dataset	All user imported data (for example, text content) for `DataItems` and `Annotations`. User created content such as `AnnotationSpecs, ColumnSpecs`.	Create a dataset for training classification and regression models Create a dataset for training forecast models
Index	All data files used for Vector Search indexes stored in Cloud Storage, Pub/Sub, and internal storage. Index and IndexEndpoint must be created with the same key. All indexes within a Collection must be created with the same key.	Manage indexes in Vector Search : Manage indexes in Vector Search 2
Collections	Collections are used to store related Data Objects.	Manage collections in Vector Search 2
DataObjects	Collections store data as individual JSON objects called Data Objects.	Manage Data Objects in Vector Search 2