Troubleshoot transparent data encryption

Select a documentation version:

This page describes how to troubleshoot transparent data encryption (TDE) issues in AlloyDB Omni.

Retrieve troubleshooting logs

To troubleshoot TDE issues, you can collect logs depending on your deployment type.

Collect the systemd logs for the AlloyDB Omni service and filter for TDE-related messages:

sudo journalctl -u alloydbomni18 | grep "TDE"

The logs might contain entries like the following:

  • TDE: TdeContext initialization failed
  • TDE: Vault server unreachable
  • TDE: CipherTrust request failed
  • TDE: Certificate expired

Error scenarios

This section describes common TDE error scenarios and provides suggested fixes.

JWT errors (HashiCorp Vault)

If you see HashiCorp Vault authentication or initialization failures in the collected logs, they are often related to JSON Web Token (JWT) issues.

Description

You might see errors like the following:

Error Description Reason
TDE: Failed to register Key Store with URL: VAULT_URL, error code: 3, error message: JWT cannot be empty JWT not found or empty The JWT token is missing or empty.
TDE: Failed to register Key Store with URL: VAULT_URL, error code: 3, error message: Invalid JWT format. Expected JWS compact serialization (three dot-separated segments: header, payload, and signature) with base64url encoded values. Malformed JWT format The JWT token isn't in the correct JWS compact serialization format.
TDE: Failed to register Key Store with URL: VAULT_URL, error code: 3, error message: Vault request failed with HTTP code: 400, Vault errors: ["error validating token: invalid expiration time (exp) claim: token is expired"] Expired JWT The JWT token has expired.
TDE: Failed to register Key Store with URL: VAULT_URL, error code: 3, error message: Vault request failed with HTTP code: 400, Vault errors: ["error validating token: error verifying token signature: failed to verify id token signature"] Invalid JWT signature The Vault server couldn't verify the signature of the JWT token.
TDE: Failed to register Key Store with URL: VAULT_URL, error code: 9, error message: Failed to get Vault configuration parameter 'VAULT_JWT' Permission issue AlloyDB Omni lacks permission to read the JWT token file, or the VAULT_JWT environment variable isn't set.

To resolve JWT issues:

  1. Retrieve the active JWT:

    Verify that the JWT token path is defined in your environment or service configuration, for example, in override.conf:

    cat PATH_TO_JWT_FILE

    Replace PATH_TO_JWT_FILE with the path to your JWT token on the host.

  2. Examine JWT structure: Make sure that the token is valid and conforms to the JWT compact serialization format.

  3. Verify token state: Make sure that the token is active and has not expired.

CipherTrust credential errors (Thales CipherTrust Manager)

If you see Thales CipherTrust Manager authentication or initialization failures in the collected logs, they are typically caused by missing or invalid user credentials (username and password).

Description

You might see errors like the following:

Error Description Reason
TDE: Failed to register Key Store with URL: CIPHERTRUST_URL, error code: 3, error message: CipherTrust user cannot be empty Missing username The CIPHERTRUST_USER environment variable or the username key in the Kubernetes Secret is empty.
TDE: Failed to register Key Store with URL: CIPHERTRUST_URL, error code: 3, error message: CipherTrust password cannot be empty Missing password The CIPHERTRUST_PASSWORD environment variable or the password key in the Kubernetes Secret is empty.
TDE: Failed to register Key Store with URL: CIPHERTRUST_URL, error code: 9, error message: Failed to get CipherTrust configuration parameter 'CIPHERTRUST_USER'
Or
TDE: Failed to register Key Store with URL: CIPHERTRUST_URL, error code: 9, error message: Failed to get CipherTrust configuration parameter 'CIPHERTRUST_PASSWORD'		 Environment variable not set The CIPHERTRUST_USER or CIPHERTRUST_PASSWORD environment variable is missing.
TDE: Failed to register Key Store with URL: CIPHERTRUST_URL, error code: 16, error message: CipherTrust request failed with HTTP code: 401 Invalid credentials The username or password was rejected by the Thales CipherTrust Manager authentication endpoint (/api/v1/auth/tokens).

Verify that CIPHERTRUST_USER and CIPHERTRUST_PASSWORD are configured in /etc/systemd/system/alloydbomni18.service.d/override.conf:

systemctl show alloydbomni18 -p Environment

KMS URL errors

If the database fails to initialize or boot, it might be due to a misconfigured Key Management Service (KMS) Key Encryption Key (KEK) URL.

Description

You might see errors like the following:

Error Description Reason
TDE: Failed to register Key Store with URL: KMS_URL, error code: 12, error message: Unsupported key store type Unsupported key store type Occurs when the KEK URL protocol isn't vault or ciphertrust. For example, using http:// or https:// instead of vault:// or ciphertrust://, or omitting the scheme entirely.
TDE: Failed to register Key Store with URL: KMS_URL, error code: 3, error message: Invalid Vault address format. Expected hostname or hostname:port
Or
TDE: Failed to register Key Store with URL: KMS_URL, error code: 3, error message: Invalid CipherTrust address format. Expected hostname or hostname:port		 Invalid KMS address format Occurs when the KMS address part of the URL is incorrectly formatted. For example, if it includes the protocol scheme again, such as vault://https://vault.example.com:8200/... or ciphertrust://https://ciphertrust.example.com:443/....
TDE: Failed to read KEK, error code: 3, error message: Invalid key path format. Must start with '/', not end with '/', and contain only valid characters Invalid key path format Occurs when the path part of the KMS URL is malformed. The path must start with a forward slash / and must not end with a trailing slash.
TDE: Failed to register Key Store with URL: KMS_URL, error code: 13, error message: curl_easy_perform() failed: Could not resolve hostname Hostname resolution failure Occurs when the KMS hostname in the KEK URL can't be resolved. This can happen if the URL is malformed or if there is a DNS configuration issue.
  • Verify KEK URL scheme: Make sure that the URL starts with vault:// (for HashiCorp Vault) or ciphertrust:// (for Thales CipherTrust Manager). Don't use https:// as the prefix.
    • Correct (Vault): vault://vault.example.com:8200/v1/secret/data/key
    • Correct (CipherTrust): ciphertrust://ciphertrust.example.com:443/api/v1/vault/secrets/key
    • Incorrect: https://vault.example.com:8200/v1/secret/data/key
  • Check address format: Make sure that the host and port are correctly specified immediately after the vault:// or ciphertrust:// prefix.
  • Verify path formatting: Make sure that the path starts with / and doesn't have a trailing slash (for example, use /v1/secret/data/key for HashiCorp Vault or /api/v1/vault/secrets/key for Thales CipherTrust Manager, without a trailing /).
  • Verify host resolution: Make sure that the KMS hostname specified in the URL is correct and that it can be resolved from within the database server.

KEK not base64-encoded

If the KEK stored in your external KMS isn't correctly base64-encoded, AlloyDB Omni fails to decode it upon retrieval. This prevents it from unwrapping the local Data Encryption Keys (DEKs) and causes a boot failure.

Description

You might see errors like the following in the database diagnostic logs (postgresql.log):

Error Description Reason
TDE: Failed to read KEK, error code: 13, error message: Error decoding base64 key string Error decoding base64 key string Occurs when the key manager attempts to decode the retrieved secret from the KMS using Base64, but the string isn't valid Base64.
  • Retrieve KEK value: Retrieve and inspect the KEK value directly from your KMS (HashiCorp Vault or Thales CipherTrust Manager).
  • Update value in KMS: Make sure that the KEK stored in your KMS is base64-encoded. If it is stored as raw binary or plain text, update the value in the KMS with its base64-encoded equivalent.
    1. Convert the raw key to Base64: 
      echo -n "RAW_KEY" | base64
       Replace <var>RAW_KEY</var> with your unencoded key value.
    2. Update the secret in your KMS to the generated base64-encoded string.

KEK version missing in KMS

The KEK version missing error occurs when AlloyDB Omni attempts to unwrap local DEKs using a specific historical version of the KEK, but that version is deleted, destroyed, or missing in the external KMS. This is most common during backup restore operations or standby replica initialization.

Description

You might see errors like the following in the database diagnostic logs (postgresql.log):

Error Description Reason
TDE: Failed to read KEK, error code: 5, error message: Vault request failed with HTTP code: 404, Response body: {... "destroyed":true, "version":4 ...}
Or
TDE: Failed to read KEK, error code: 5, error message: CipherTrust request failed with HTTP code: 404, CipherTrust error: "NCERRResourceNotFound: ..."		 Key version missing or destroyed The requested KEK version is missing, deleted, or permanently destroyed in the KMS.
  • Verify KEK version in KMS: Make sure that the requested KEK version is present and active (not deleted or destroyed) in your KMS.
  • Restore missing KEK version: Restore or undelete the requested KEK version in your KMS if it has been deleted but not destroyed.
  • Handle destroyed keys: If the requested key version is permanently destroyed in your KMS, the data or backup encrypted under that version is permanently unrecoverable.

Bad or missing TLS or SSL certificates

The bad or missing TLS or SSL certificate error occurs when AlloyDB Omni can't establish a secure TLS connection to your external KMS (HashiCorp Vault or Thales CipherTrust Manager) because certificates are missing, expired, untrusted, or misconfigured.

Description

You might see errors like the following in the database diagnostic logs (postgresql.log):

Error Description Reason
TDE: Failed to read KEK, error code: 13, error message: curl_easy_perform() failed: SSL peer certificate or SSH remote key was not OK
Or
TDE: Failed to read KEK, error code: 13, error message: curl_easy_perform() failed: Peer certificate cannot be authenticated with given CA certificates		 KMS server certificate verification failed The KMS server's certificate can't be verified using the provided CA certificate.
TDE: Failed to register Key Store with URL: KMS_URL, error code: 13, error message: curl_easy_perform() failed: Problem with the local SSL certificate Local SSL certificate problem Occurs if a client certificate is required and has issues.

To resolve TLS verification issues, make sure that the correct certificates are supplied, mounted, and readable by the postgres user.

Linux expects certificates to be supplied at the path specified in the VAULT_CERT_PATH (for HashiCorp Vault) or CIPHERTRUST_CERT_PATH (for Thales CipherTrust Manager) environment variable.

  1. Confirm that VAULT_CERT_PATH or CIPHERTRUST_CERT_PATH is set correctly in your environment or service override:

    systemctl show alloydbomni18 -p Environment

  2. Check that the certificate files exist and are readable by the postgres user:

    ls -la CERT_PATH

    Make sure that the ca.crt, tls.crt, and tls.key filenames are present and that they have read permissions for the postgres user.

Replace CERT_PATH with the path to the TLS certificates directory on your host (for example, /tde-tls).