Google Cloud Observability storage overview

Google Cloud Observability stores and organizes telemetry data from Cloud Logging, Cloud Monitoring, and Cloud Trace across structured storage containers. Understanding the storage model helps you configure data residency, enforce encryption policies, and connect telemetry to analytics services like BigQuery.

Log data

Log data resides in log buckets, which are the containers that Logging uses to store your log data. Every Google Cloud project, billing account, folder, and organization contains log buckets named _Required and _Default.

By default, log data resides in the Google Cloud project, billing account, folder, or organization where the data originates. However, you can configure Logging to route log data from the resource where it originates to another location, like another project or a centralized log bucket. For more information, see Store log entries and Route log entries.

Cloud Logging lets you regionalize your log data:

Organization policies can restrict the locations of new log buckets and require that log buckets use customer-managed encryption keys (CMEKs).
For organizations and folders, default resource settings for Cloud Logging let you configure the following:
- The location of new _Required and _Default log buckets.
- The KMS key to encrypt your log data.
- The configuration of the default sink.
Descendants in the resource hierarchy automatically inherit these settings, unless they also configure default resource settings. For example, if you configure default resource settings for Cloud Logging for an organization, then all folders and projects in the organization's resource hierarchy automatically inherit those settings. However, if you set the default resource settings for Cloud Logging for a folder in that organization, then the folder-level settings are used.

The default resource settings for Cloud Logging apply only to new resources, not existing resources. For more information, see Configure default resource settings for Cloud Logging.

Metric data

Metric data resides in the Google Cloud project where the data originates.

For information about the storage policies, see Data residency for Monitoring.

Trace data

Trace data is stored in a dataset that is managed by the observability bucket named _Trace. To hold the dataset, the _Trace bucket must exist. The _Trace bucket can be created automatically or manually:

Automatic creation: The system automatically creates the bucket in response to receiving trace data from an application or a Google Cloud service. The system uses the applicable default settings for observability buckets to determine the bucket's location and encryption key. If you haven't defined default settings, then the system selects a supported location and the bucket uses Google-default encryption.

Trace data generated by Cloud Run functions, Cloud Run, and App Engine doesn't cause the system to create the observability bucket. Spans from these services are stored only when the observability bucket exists.
Manual creation: You can use the Observability API to create the _Trace bucket before your Google Cloud project receives trace data. You must provide the location of the bucket. You can provide a Cloud Key Management Service key:
- If you provide a key, then the system uses that key to encrypt the stored data.
- If you don't provide a Cloud KMS key, then the default settings that apply to the bucket's parent resource determine the encryption key. If the default settings specify a Cloud KMS key, then that key encrypts the stored data. Otherwise, Google-default encryption is used.

When the _Trace bucket is created, the system also creates a dataset named Spans for the bucket, and a view named _AllSpans on the dataset. That view includes all data in the dataset.

For more information, see the following:

Observability storage model

The Observability API storage model relies on the following architecture:

Observability buckets

An observability bucket is the management entity for datasets, which store data. An observability bucket is in a specific location and has a data retention policy. When a Google Cloud service uses the Observability API to store their data, the system creates an observability bucket based on the name of the service, a dataset to store data, and a view to provide read access to the stored data. For example, for the Cloud Trace service, the system names the system-created bucket _Trace, the dataset Spans, and the view _AllSpans. To learn about the structure of an observability bucket, see Bucket.

Datasets

A dataset stores data. The system automatically creates one dataset when it creates an observability bucket, which manages the dataset. For example, when the system creates the _Trace bucket, it also creates a dataset named Spans, which stores your trace data. To learn about the structure of a dataset, see Dataset.

Views on datasets

Each dataset hosts one or more views. A view provides read access to a subset of entries in the dataset. The system creates one view when it creates a dataset. That view includes all data in the dataset. The name of the view depends on the service. For example, for the Cloud Trace service, the system creates a view named _AllSpans on the Spans dataset. To learn about the structure of a view, see View.

Links on datasets

Each dataset can contain at most one link. When you create a link for a dataset, the system creates a linked BigQuery dataset. You can then query the data in your dataset using BigQuery or by other services that use the BigQuery API. To learn about the structure of a link, see Link.

The system doesn't automatically create links on datasets.

For example, your trace data resides in a 'Spans' dataset inside the '_Trace' observability bucket. The system creates an '_AllSpans' view on this dataset that includes all stored spans.

Data residency for observability buckets

If you have compliance or regulatory requirements to store your data in specific locations or to use customer-managed encryption keys (CMEKs), then we recommend that you configure both organization policies and default settings for observability buckets.

For organizations, folders, and projects, default settings for observability buckets let you configure the following:

A default storage location.
For each location, a default Cloud Key Management Service key.

There are two different ways the system uses default settings:

When the system automatically creates an observability bucket, it uses the default settings to determine the bucket's location and encryption key. If you haven't defined default settings, then the system selects the location and the bucket uses Google-default encryption.
When you initiate creation of an observability bucket by issuing an API request, you provide the location. However, the system automatically uses the Cloud KMS key defined by the default settings to encrypt the data, unless the arguments to the API request specify the key.

Because the parent of an observability bucket is always a project, when you create a bucket, the system first searches for project-level default settings. If those don't exist, then the system searches the parent's ancestors for default settings. For example, if you define default settings for a folder, then those settings apply to descendants of the folder, except those descendants that have default settings configured.

You can also use organization policies to restrict the locations of new observability buckets, to require the use of CMEKs, or to restrict which Cloud KMS keys can be used for encryption. If you configure organization policies that require use of CMEKs, then you must configure default settings for observability buckets. If you don't, then provisioning for system-created observability buckets fails.

For more information, see Set defaults for observability buckets.

Limitations

You can't do the following:

Modify or delete observability buckets.
Create, delete, or modify datasets.
Create, delete, or modify views.
Use the Google Cloud console to list buckets, datasets, views, or links.