Google Cloud Observability storage overview

This document describes how Google Cloud Observability stores your telemetry data. It includes information about how Cloud Logging, Cloud Monitoring, and Cloud Trace store data. This document also provides a conceptual overview of observability buckets, which Cloud Trace uses as its storage model.

Log data

Log data resides in log buckets, which are the containers that Logging uses to store your log data. Every Google Cloud project, billing account, folder, and organization contains log buckets named _Required and _Default.

By default, log data resides in the Google Cloud project, billing account, folder, or organization where the data originates. However, you can configure Logging to route log data from the resource where it originates to another location, like another project or a centralized log bucket.

To learn more, see Store log entries and Route log entries.

Metric data

Metric data resides in the Google Cloud project where the data originates.

Trace data

Trace data resides in an observability bucket named in the same Google Cloud project where the data originates. The bucket name is _Trace. The next section describes observability buckets.

To learn about the storage format of individual spans, see Trace schema.

Observability storage model

The Observability API storage model relies on the following architecture:

Observability buckets
An observability bucket is the management entity for datasets, which store data. An observability bucket is in a specific location and has a data retention policy. When a Google Cloud service uses the Observability API to store their data, the system creates an observability bucket based on the name of the service. For example, for the Cloud Trace service, the system-created bucket is named _Trace. To learn about the structure of an observability bucket, see Bucket.
Datasets
A dataset is a storage entity. Each dataset is a child of an observability bucket. When the system creates an observability bucket for a Google Cloud service, it also creates one dataset. For example, after the system creates the _Trace bucket, it creates the dataset named Spans. That dataset stores your trace data. To learn about the structure of a dataset, see Dataset.
Views on datasets
Each dataset hosts one or more view. A view provides read access to a subset of entries in the dataset. When a dataset is created, the system automatically creates one view. That view includes all data in the dataset. The name of the view depends on the service. For example, for the Cloud Trace service, the system creates a view named _AllSpans on the Spans dataset. To learn about the structure of a view, see View.
Links on datasets

Each dataset can contain at most one link. When you create a link for a dataset, the system creates a linked BigQuery dataset. You can then query the data in your dataset by using BigQuery or by other services that use the BigQuery API. To learn about the structure of a link, see Link.

The system doesn't automatically create links on datasets.

For example, your trace data resides in a dataset named Spans. This dataset is a child of the observability bucket named _Trace. On the Spans dataset, the system creates the view named _AllSpans. This view includes all data in the dataset.

Limitations

You can't do the following:

  • Modify or delete observability buckets.
  • Create, delete, or modify datasets.
  • Create, delete, or modify views.
  • Use the Google Cloud console to list buckets, datasets, views, or links.
  • Create observability buckets in the locations other than the us; an observability bucket must be in the us location.

What's next