Troubleshoot observability buckets

This document describes how to resolve failures that might be related to observability buckets. For example, this page describes how to resolve issues related to configuring default settings for observability buckets. You configure these default settings for a resource, which can be an organization, folder, or project, and they let you do the following:

Configure a default storage location for your observability buckets.
For each location where you choose to store data, you can configure use of customer managed encryption keys (CMEK).

Descendants in the resource hierarchy automatically use these settings, except for those descendants where you've configured default settings.

You don't see any trace data

You expect to see data in the Trace Explorer page, but there isn't any data.

Trace data is stored in an observability buckets named _Trace. There are several reasons why you might not see trace data. To learn how to resolve this failure, see No data in the Trace Explorer page.

Issues related to the storage location

This section lists common issues related to setting or clearing the default storage location, which is a field in the default settings for observability buckets.

Setting the default storage location for a resource fails

You try to set the default storage location for a resource, but the command fails:

An attempt to set a location that isn't supported by observability buckets fails with a 400 (INVALID_ARGUMENT) error code. The failure message is similar to the following:
```
Unsupported default storage location: my-region
```

To resolve these failures, do the following:

Make sure that the location you specify in the command is a supported observability bucket location.

A command to clear the default storage location fails

You issue an updateSettings command to a resource with the value of the default storage location set to an empty string. The command fails with a 403 (INVALID_REQUEST) error code. The error message is similar to one of the following:

Before you can clear the default storage location for this resource, you must set a default storage location for an ancestor in the resource hierarchy.

The default storage location may not be removed from an organization's Settings.

This is expected behavior.

After a default storage location for a resource is set, you can change the value but you can't clear or unset the value unless the default storage location is specified for an ancestor in the resource hierarchy. You can't clear or unset the default storage location for an organization because they don't have ancestors.

The location of an observability bucket conflicts with an organization policy

You notice that the locations of some observability buckets conflict with the allowed locations an organization policy specifies.

This is not an error.

Organization policies that restrict location aren't honored by observability buckets.

Issues related to the CMEK

This section lists common issues related to using CMEK.

How can I find the CMEK settings for an observability bucket?

To determine whether an observability bucket uses CMEK, you can list your observability buckets.

The response data includes information about the Cloud KMS key, its version, and the service account used to access the key.

How do I resolve CMEK configuration errors

After you configure CMEK for a resource and location you either receive an email notification from Google Cloud Observability about CMEK access issues or you notice read or write errors from observability buckets with CMEK enabled. The email, which is sent to essential Contacts, contains information like the following:

Cloud KMS key name.
Cloud KMS key version.
Name of the service account used for encryption and decryption.
The error code seen by Google Cloud Observability when encrypting or decrypting data.
The error message seen when encrypting or decrypting data

The notification provides information about the failure and it lists actions that you can take to mitigate the issue:

Error	Recommendation
Cryptographic key permission denied	The resource's service account doesn't have sufficient IAM permissions to operate on the specified Cloud KMS key. To resolve this failure, follow the instructions in the error. The following information might be helpful to you: Verify service account has required role Grant service account access to a key
Cryptographic key is disabled	The specified Cloud KMS key was disabled. Follow the instructions in the error to re-enable the key. The following information might be helpful to you: Verify a Cloud KMS key is usable Grant service account access to a key
Cryptographic key was destroyed	The specified Cloud KMS key was destroyed. Follow the instructions or see Manage your Cloud KMS key for a resource.

Error

Recommendation

Cryptographic key permission denied

The resource's service account doesn't have sufficient IAM permissions to operate on the specified Cloud KMS key. To resolve this failure, follow the instructions in the error. The following information might be helpful to you:

Cryptographic key is disabled

The specified Cloud KMS key was disabled. Follow the instructions in the error to re-enable the key. The following information might be helpful to you:

Cryptographic key was destroyed

The specified Cloud KMS key was destroyed. Follow the instructions or see Manage your Cloud KMS key for a resource.

Provisioning fails for an observability bucket

You are an essential contact or owner of a resource, and you receive an email notification from Google Cloud Observability about a provisioning failure.

Provisioning an observability bucket might fail due to a configuration error or to an internal error:

When provisioning fails due to an internal error, you don't need to take action. The system automatically recovers from these failures by retrying the create command until it is successful.
When provisioning fails due to a configuration error, you need to correct the error. Provided data continues to arrive, the system automatically issues at least one create observability bucket command per day. Configuration errors include the following:
- A Cloud KMS key isn't usable.
- A service account is missing a required role.

The remainder of this section describes how to investigate common configuration issues.

Review the default storage locations for your resources

Review, and if necessary, update the default settings for observability buckets for organization, folder, or project with a default storage location.

To learn more, see View default settings for observability buckets.

Verify a Cloud KMS key is usable

To determine whether a Cloud KMS key is usable, run gcloud kms keys list:

gcloud kms keys list \
--location=LOCATION_ID \
--keyring=KMS_KEY_RING

Before running the previous command, make the following replacements:

LOCATION_ID: The location of your Cloud KMS key.
KMS_KEY_RING: The Cloud KMS key ring's name.

The previous command returns information about each key in the specified location. For your Cloud KMS key, make sure the following is true:

The Cloud KMS key is listed in the table.
The STATUS column lists ENABLED.
The PURPOSE column lists ENCRYPT_DECRYPT. This setting indicates that the purpose of the key is symmetric encryption:

If necessary, create a new Cloud KMS key and update your default settings for observability buckets for the associated resource and location.

Verify service account has required role

Verify that the resource's service account has been granted the Cloud KMS CryptoKey Encrypter/Decrypter role for the Cloud KMS key listed as part of the default settings for observability buckets for the resource.

To determine the bindings for a Cloud KMS key, run the following command:

gcloud kms keys get-iam-policy KMS_KEY_NAME

If necessary, grant the resource's service account the Cloud KMS CryptoKey Encrypter/Decrypter role for the key. To learn more, see Grant service account access to a key.