Create a linked BigQuery dataset

This document describes how to create a linked BigQuery dataset on your observability dataset. You can then use BigQuery services such BigQuery Studio and the BigQuery API to query the linked dataset.

To get started with Observability Analytics, see Query and analyze traces.

When you need a linked dataset

You don't need a linked BigQuery dataset when you want to query your trace data, or query your trace and log data. For those scenarios, you can use the Observability Analytics page. For information about querying log data, see Query and analyze logs with Observability Analytics.

You do need a linked BigQuery dataset when you want to do any of the following:

Create a linked BigQuery dataset

This section describes how to create a linked BigQuery dataset, which lets your trace data be queried by BigQuery services. You can create one linked BigQuery dataset on an observability dataset.

When you create a linked dataset, you initiate a long-running operation. Several actions occur as part of that operation:

  • Audit logs record the request to create a link and the completion of the operation.

  • When the Monitoring Service Account doesn't exist, Google Cloud Observability creates it.

    The Cloud Monitoring API must be enabled for this service account to be created.

  • Google Cloud Observability requires a service account before it can create a linked dataset. When necessary, a service account is created:

  • An audit log records the request from the service agent manager to grant the Monitoring Service Account the IAM role of Monitoring Service Agent.

Before you begin

  1. Sign in to your Google Cloud account. If you're new to Google Cloud, create an account to evaluate how our products perform in real-world scenarios. New customers also get $300 in free credits to run, test, and deploy workloads.

  2. In the Google Cloud console, on the project selector page, select or create a Google Cloud project.

    Roles required to select or create a project

    • Select a project: Selecting a project doesn't require a specific IAM role—you can select any project that you've been granted a role on.
    • Create a project: To create a project, you need the Project Creator role (roles/resourcemanager.projectCreator), which contains the resourcemanager.projects.create permission. Learn how to grant roles.

    Go to project selector

  3. Verify that billing is enabled for your Google Cloud project.

  4. Enable the Cloud Monitoring and Observability APIs.

    Roles required to enable APIs

    To enable APIs, you need the Service Usage Admin IAM role (roles/serviceusage.serviceUsageAdmin), which contains the serviceusage.services.enable permission. Learn how to grant roles.

    Enable the APIs

  5. In the Google Cloud console, on the project selector page, select or create a Google Cloud project.

    Roles required to select or create a project

    • Select a project: Selecting a project doesn't require a specific IAM role—you can select any project that you've been granted a role on.
    • Create a project: To create a project, you need the Project Creator role (roles/resourcemanager.projectCreator), which contains the resourcemanager.projects.create permission. Learn how to grant roles.

    Go to project selector

  6. Verify that billing is enabled for your Google Cloud project.

  7. Enable the Cloud Monitoring and Observability APIs.

    Roles required to enable APIs

    To enable APIs, you need the Service Usage Admin IAM role (roles/serviceusage.serviceUsageAdmin), which contains the serviceusage.services.enable permission. Learn how to grant roles.

    Enable the APIs

    8.

  8. To get the permissions that you need to create a link on an observability dataset, ask your administrator to grant you the Observability Editor (roles/observability.editor) IAM role on your project. For more information about granting roles, see Manage access to projects, folders, and organizations.

    You might also be able to get the required permissions through custom roles or other predefined roles.

Create dataset

gcloud

Before using any of the command data below, make the following replacements:

  • LINK_ID: The name of the BigQuery dataset.
  • DATASET_ID: The ID of the dataset. Your trace data is stored in a dataset named Spans.
  • BUCKET_ID: The ID of the observability bucket. For example, this ID might be _Trace.
  • LOCATION: The location of the observability buckets.
  • PROJECT_ID: The identifier of the project..

Execute the gcloud beta observability buckets datasets links create command:

Linux, macOS, or Cloud Shell

gcloud beta observability buckets datasets links create \
  projects/PROJECT_ID/locations/LOCATION/buckets/BUCKET_ID/datasets/DATASET_ID/links/LINK_ID \
 --dataset=DATASET_ID\
 --bucket=BUCKET_ID \
 --location=LOCATION \
 --project=PROJECT_ID

Windows (PowerShell)

gcloud beta observability buckets datasets links create `
  projects/PROJECT_ID/locations/LOCATION/buckets/BUCKET_ID/datasets/DATASET_ID/links/LINK_ID `
 --dataset=DATASET_ID`
 --bucket=BUCKET_ID `
 --location=LOCATION `
 --project=PROJECT_ID

Windows (cmd.exe)

gcloud beta observability buckets datasets links create ^
  projects/PROJECT_ID/locations/LOCATION/buckets/BUCKET_ID/datasets/DATASET_ID/links/LINK_ID ^
 --dataset=DATASET_ID^
 --bucket=BUCKET_ID ^
 --location=LOCATION ^
 --project=PROJECT_ID

The create command initiates a long-running operation. The following is an example of a response when the command is successful: 

Create request issued for: [mydataset]
Waiting for operation [projects/my-project/locations/us/operations/operation-1775164903749-64e80c9817833-9ff804b6-c3e9cbe7] to complete...done.
Created link [mydataset].

REST

To create a link to a BigQuery dataset, send a request to the projects.locations.buckets.datasets.links.create endpoint.

You must specify the parent parameter, which has the following form:

projects/PROJECT_ID/locations/LOCATION/buckets/BUCKET_ID/datasets/DATASET_ID

The fields in the previous expression have the following meaning:

  • PROJECT_ID: The identifier of the project.
  • LOCATION: The location of the observability bucket.
  • BUCKET_ID: The ID of the observability bucket. For example, this ID might be _Trace.
  • DATASET_ID: The ID of the dataset being queried. For example, this ID might be Spans.

This command requires a query parameter and a request body:

  • The query parameter, linkId, must be specified and set to the name of the BigQuery dataset. For example, linkId="my_link". The BigQuery dataset name must be unique for your Google Cloud project, and must be limited to 100 characters and can include only letters, digits, and underscores.

  • The request body is a Link object. The value of the name field has the following format:

    projects/PROJECT_ID/locations/LOCATION/buckets/BUCKET_ID/dataset/DATASET_ID/links/LINK_ID

    The value you provide for the name field must match the linked BigQuery dataset referenced by the query parameter.

    The LINK_ID field is the name of the BigQuery dataset.

The response is an Operation object. This object contains information about the progress of the method. When the method completes, the Operation object contains status data.

For a complete list of Observability API endpoints, see the Observability API reference documentation.

If you encounter permission errors when creating a linked dataset, then see Troubleshoot permission errors.

What's next