Manage observability buckets

Preview

This product or feature is subject to the "Pre-GA Offerings Terms" in the General Service Terms section of the Service Specific Terms. Pre-GA products and features are available "as is" and might have limited support. For more information, see the launch stage descriptions.

This document describes how to use the Observability API to get information about your observability buckets. It also includes information about how to list datasets, links, and views. To learn more about how Google Cloud Observability stores data, see Storage overview.

Your trace data is stored in observability buckets. This document describes how to manage the storage of your trace data, but it doesn't describe the format of the stored data. For information about that topic, see Trace schema.

This document doesn't apply to the storage of log or metric data. Log and metric data isn't stored in observability buckets.

Before you begin

Sign in to your Google Cloud account. If you're new to Google Cloud, create an account to evaluate how our products perform in real-world scenarios. New customers also get $300 in free credits to run, test, and deploy workloads.
Install the Google Cloud CLI.
If you're using an external identity provider (IdP), you must first sign in to the gcloud CLI with your federated identity.
To initialize the gcloud CLI, run the following command:
gcloud init
Create or select a Google Cloud project.
Roles required to select or create a project
• Select a project: Selecting a project doesn't require a specific IAM role—you can select any project that you've been granted a role on.
• Create a project: To create a project, you need the Project Creator role (roles/resourcemanager.projectCreator), which contains the resourcemanager.projects.create permission. Learn how to grant roles.
Note: If you don't plan to keep the resources that you create in this procedure, create a project instead of selecting an existing project. After you finish these steps, you can delete the project, removing all resources associated with the project.
• Create a Google Cloud project:

  gcloud projects create PROJECT_ID

  Replace PROJECT_ID with a name for the Google Cloud project you are creating.

• Select the Google Cloud project that you created:

  gcloud config set project PROJECT_ID

  Replace PROJECT_ID with your Google Cloud project name.
Verify that billing is enabled for your Google Cloud project.
Enable the Observability API:
Roles required to enable APIs
To enable APIs, you need the Service Usage Admin IAM role (roles/serviceusage.serviceUsageAdmin), which contains the serviceusage.services.enable permission. Learn how to grant roles.
gcloud services enable observability.googleapis.com
Grant roles to your user account. Run the following command once for each of the following IAM roles: roles/observability.viewer
gcloud projects add-iam-policy-binding PROJECT_ID --member="user:USER_IDENTIFIER" --role=ROLE
Replace the following:
• PROJECT_ID: Your project ID.
• USER_IDENTIFIER: The identifier for your user account. For example, [e-mail].
• ROLE: The IAM role that you grant to your user account.

Install the Google Cloud CLI.

If you're using an external identity provider (IdP), you must first sign in to the gcloud CLI with your federated identity.

To initialize the gcloud CLI, run the following command:

gcloud init

Create or select a Google Cloud project.

Roles required to select or create a project

• Select a project: Selecting a project doesn't require a specific IAM role—you can select any project that you've been granted a role on.
• Create a project: To create a project, you need the Project Creator role (roles/resourcemanager.projectCreator), which contains the resourcemanager.projects.create permission. Learn how to grant roles.

Note: If you don't plan to keep the resources that you create in this procedure, create a project instead of selecting an existing project. After you finish these steps, you can delete the project, removing all resources associated with the project.

• Create a Google Cloud project:

  gcloud projects create PROJECT_ID

  Replace PROJECT_ID with a name for the Google Cloud project you are creating.

• Select the Google Cloud project that you created:

  gcloud config set project PROJECT_ID

  Replace PROJECT_ID with your Google Cloud project name.

Verify that billing is enabled for your Google Cloud project.

Enable the Observability API:

Roles required to enable APIs

To enable APIs, you need the Service Usage Admin IAM role (roles/serviceusage.serviceUsageAdmin), which contains the serviceusage.services.enable permission. Learn how to grant roles.

gcloud services enable observability.googleapis.com

Grant roles to your user account. Run the following command once for each of the following IAM roles: roles/observability.viewer

gcloud projects add-iam-policy-binding PROJECT_ID --member="user:USER_IDENTIFIER" --role=ROLE

Replace the following:

• PROJECT_ID: Your project ID.
• USER_IDENTIFIER: The identifier for your user account. For example, [e-mail].
• ROLE: The IAM role that you grant to your user account.

List observability buckets

REST

To list the observability buckets that are in your project and in a specific location, send a request to the projects.locations.buckets.list endpoint.

You must specify the parent parameter, which has the following form:

projects/PROJECT_ID/locations/LOCATION

The fields in the previous expression have the following meanings:

• PROJECT_ID: The identifier of the project.
• LOCATION: The location of the observability bucket. You must use the us location.

The response is an array of Bucket objects. For each object, the value of the name field has the following format:

projects/PROJECT_ID/locations/LOCATION/buckets/BUCKET_ID

For example, when a command was issued to the buckets.list endpoint with the parent parameter set to projects/my-project/locations/us, the response was:

{
  "buckets": [
    {
      "name": "projects/my-project/locations/us/buckets/_Trace",
      "description": "Trace Bucket",
      "createTime": "2025-01-01T15:42:30.988919645Z",
      "updateTime": "2025-02-04T15:42:30.988919645Z",
      "retentionDays": 30
    }
  ]
}

You can issue commands to other Observability API endpoints to get more information about the bucket whose ID is BUCKET_ID. For example, you can list the datasets on that bucket, and the views and links on each dataset. For a complete list of Observability API endpoints, see the Observability API reference documentation.

List datasets on a observability bucket

REST

To list the datasets for an observability bucket, send a request to the projects.locations.buckets.datasets.list endpoint.

You must specify the parent parameter, which has the following form:

projects/PROJECT_ID/locations/LOCATION/buckets/BUCKET_ID

The fields in the previous expression have the following meanings:

• PROJECT_ID: The identifier of the project.
• LOCATION: The location of the observability bucket. You must use the us location.
• BUCKET_ID: The ID of the observability bucket. For example, this ID might be _Trace.

The response is an array of Dataset objects. For each object, the value of the name field has the following format:

projects/PROJECT_ID/locations/LOCATION/buckets/BUCKET_ID/dataset/DATASET_ID

For example, when a command was issued to the buckets.datasets.list endpoint with the parent parameter set to projects/my-project/locations/us/buckets/_Trace, the response was:

{
  "datasets": [
    {
      "name": "projects/my-project/locations/us/buckets/_Trace/datasets/Spans",
      "description": "Trace Spans",
      "createTime": "2025-01-01T15:42:30.988919645Z",
      "updateTime": "2025-02-04T15:42:30.988919645Z",
    }
  ]
}

You can issue commands to other Observability API endpoints to get information about the dataset whose ID is DATASET_ID. For example, you can list the views and links on each dataset. For a complete list of Observability API endpoints, see the Observability API reference documentation.

List views on a dataset

REST

To list the views on a dataset, send a request to the projects.locations.buckets.datasets.views.list endpoint.

You must specify the parent parameter, which has the following form:

projects/PROJECT_ID/locations/LOCATION/buckets/BUCKET_ID/datasets/DATASET_ID/views

The fields in the previous expression have the following meanings:

• PROJECT_ID: The identifier of the project.
• LOCATION: The location of the observability bucket. You must use the us location.
• BUCKET_ID: The ID of the observability bucket. For example, this ID might be _Trace.
• DATASET_ID: The ID of the dataset being queried. For example, this ID might be Spans.

The response is an array of View objects. For each object, the value of the name field has the following format:

projects/PROJECT_ID/locations/LOCATION/buckets/BUCKET_ID/views/OBS_VIEW_ID

In the previous express, the ID of a view is represented by OBS_VIEW_ID. For example, this field might have a value of _AllSpans.

For example, when a command was issued to the buckets.datasets.views.list endpoint with the parent parameter set to projects/my-project/locations/us/buckets/_Trace/datasets/Spans/views, the response was:

{
  "views": [
    {
      "name": "projects/my-project/locations/us/buckets/_Trace/datasets/Spans/views/_AllSpans",
      "filter": "",
      "createTime": "2025-01-01T15:42:30.988919645Z",
      "updateTime": "2025-02-04T15:42:30.988919645Z",
    }
  ]
}

For a complete list of Observability API endpoints, see the Observability API reference documentation.

List links on a dataset

REST

To list the links on a dataset, send a request to the projects.locations.buckets.datasets.links.list endpoint.

You must specify the parent parameter, which has the following form:

projects/PROJECT_ID/locations/LOCATION/buckets/BUCKET_ID/datasets/DATASET_ID

The fields in the previous expression have the following meanings:

• PROJECT_ID: The identifier of the project.
• LOCATION: The location of the observability bucket. You must use the us location.
• BUCKET_ID: The ID of the observability bucket. For example, this ID might be _Trace.
• DATASET_ID: The ID of the dataset being queried. For example, this ID might be Spans.

The response is an array of Link objects. For each object, the value of the name field has the following format:

projects/PROJECT_ID/locations/LOCATION/buckets/BUCKET_ID/links/LINK_ID

The LINK_ID is the name of the BigQuery dataset. This field is globally unique for your Google Cloud project.

For example, when a command was issued to the buckets.datasets.links.list endpoint with the parent parameter set to projects/my-project/locations/us/buckets/_Trace/datasets/Spans/links, the response was:

{
  "links": [
    {
      "name": "projects/my-project/locations/us/buckets/_Trace/datasets/Spans/links/my_link",
      "description": "My link for traces to BigQuery",
      "createTime": "2025-01-12T15:42:30.988919645Z"
    }
  ]
}

For a complete list of Observability API endpoints, see the Observability API reference documentation.

Create a link on a dataset

You can create a link on a dataset, which lets your trace data be queried from BigQuery. You can also delete the Link object which is attached to a dataset.

Before you begin

To get the permissions that you need to create a link on a dataset, ask your administrator to grant you the following IAM roles on your project:

• Observability Editor (roles/observability.editor)
• BigQuery User (roles/bigquery.user)

For more information about granting roles, see Manage access to projects, folders, and organizations.

You might also be able to get the required permissions through custom roles or other predefined roles.

Create a linked BigQuery dataset

REST

To create a link to a BigQuery dataset, send a request to the projects.locations.buckets.datasets.links.create endpoint.

You must specify the parent parameter, which has the following form:

projects/PROJECT_ID/locations/LOCATION/buckets/BUCKET_ID/datasets/DATASET_ID

The fields in the previous expression have the following meaning:

• PROJECT_ID: The identifier of the project.
• LOCATION: The location of the observability bucket. You must use the us location.
• BUCKET_ID: The ID of the observability bucket. For example, this ID might be _Trace.
• DATASET_ID: The ID of the dataset being queried. For example, this ID might be Spans.

This command requires a query parameter and a request body:

• The query parameter, linkId, must be specified and set to the name of the BigQuery dataset. For example, linkId="my_link". The BigQuery dataset name must be unique for your Google Cloud project, and must be limited to 100 characters and can include only letters, digits, and underscores.

• The request body is a Link object. The value of the name field has the following format:

  projects/PROJECT_ID/locations/LOCATION/buckets/BUCKET_ID/dataset/DATASET_ID/links/LINK_ID
  

  The value you provide for the name field must match the linked BigQuery dataset referenced by the query parameter.

  The LINK_ID field is the name of the BigQuery dataset.

The response is an Operation object. This object contains information about the progress of the method. When the method completes, the Operation object contains status data.

For a complete list of Observability API endpoints, see the Observability API reference documentation.

What's next

To learn about querying your telemetry, see View and analyze telemetry.