Create a linked BigQuery dataset on a log bucket

This document describes how to create a linked BigQuery dataset on a log bucket. You can then use BigQuery services such as BigQuery Studio, Data Studio, and the BigQuery API to query the linked dataset. A linked BigQuery dataset is a read-only dataset in BigQuery that serves as a pointer to your log bucket. It lets you use BigQuery services to query your log data without duplicating the data or paying for separate storage in BigQuery.

You can't create a linked BigQuery dataset for an analytics view.

If you aren't familiar with Observability Analytics, then see Observability Analytics overview.

When you need a linked dataset

You don't need a linked BigQuery dataset when you want to query your log data, or query your log and trace data. For those scenarios, you can use the Observability Analytics page. You can also save and share your queries, and you can save the query to a custom dashboard. For information about querying your trace data, see Query and analyze traces.

You do need a linked BigQuery dataset when you want to do any of the following:

• Join log entry data with other BigQuery datasets.
• Query log data from another service like BigQuery Studio or Data Studio.
• Create an alerting policy that monitors the result of a SQL query. For more information, see Monitor your SQL query results with an alerting policy.

A linked BigQuery dataset for a log bucket lets BigQuery read the data in the log bucket.

How log buckets and log views map to BigQuery objects

When you create a linked BigQuery dataset for a log bucket, a new dataset becomes available to BigQuery. You specify the name of the linked dataset during the linking process. Any log views within the log bucket are automatically mapped to virtual views in the dataset. The name of a virtual view is the same as the corresponding log view.

For example, suppose the project my_project has a log bucket named example, and that this log bucket has a log view named _AllLogs. If you create a linked dataset for that log bucket with the name my_dataset, then the Explorer page of BigQuery displays a my_project entry. A child of the my_project entry is a dataset with the name my_dataset, and that dataset lists a virtual view named _AllLogs.

Access control implications when using linked datasets

When a principal queries a view on a linked BigQuery dataset, that query runs through the BigQuery permission layer, not the Cloud Logging permission layer. Therefore, the BigQuery roles and permissions granted to the principal on the linked dataset determine whether they can query the dataset.

You can't restrict a principal to a specific virtual view within a linked dataset. When a principal has been granted access to the linked dataset, they can query every view in that dataset.

System actions that occur when you create a linked dataset

When you create a linked dataset, you initiate a long-running operation. Several actions occur as part of that operation:

• Audit logs record the request to create a link and the completion of the operation.

• When the Monitoring Service Account doesn't exist, Google Cloud Observability creates it.

  The Cloud Monitoring API must be enabled for this service account to be created.

• Google Cloud Observability requires a service account before it can create a linked dataset. When necessary, a service account is created:

  • Cloud Observability Service Account: Required for a link on an observability dataset.

  • Cloud Logging Service Account: Required for a link on a log bucket.

• An audit log records the request from the service agent manager to grant the Monitoring Service Account the IAM role of Monitoring Service Agent.

Before you begin

Sign in to your Google Cloud account. If you're new to Google Cloud, create an account to evaluate how our products perform in real-world scenarios. New customers also get $300 in free credits to run, test, and deploy workloads.

In the Google Cloud console, on the project selector page, select or create a Google Cloud project.

Roles required to select or create a project

• Select a project: Selecting a project doesn't require a specific IAM role—you can select any project that you've been granted a role on.
• Create a project: To create a project, you need the Project Creator role (roles/resourcemanager.projectCreator), which contains the resourcemanager.projects.create permission. Learn how to grant roles.

Note: If you don't plan to keep the resources that you create in this procedure, create a project instead of selecting an existing project. After you finish these steps, you can delete the project, removing all resources associated with the project.

Go to project selector

Verify that billing is enabled for your Google Cloud project.

In the Google Cloud console, on the project selector page, select or create a Google Cloud project.

Roles required to select or create a project

• Select a project: Selecting a project doesn't require a specific IAM role—you can select any project that you've been granted a role on.
• Create a project: To create a project, you need the Project Creator role (roles/resourcemanager.projectCreator), which contains the resourcemanager.projects.create permission. Learn how to grant roles.

Note: If you don't plan to keep the resources that you create in this procedure, create a project instead of selecting an existing project. After you finish these steps, you can delete the project, removing all resources associated with the project.

Go to project selector

Verify that billing is enabled for your Google Cloud project.

1. To get the permissions that you need to create a linked dataset, ask your administrator to grant you the Logs Configuration Writer (roles/logging.configWriter) IAM role on your project. For more information about granting roles, see Manage access to projects, folders, and organizations.

   You might also be able to get the required permissions through custom roles or other predefined roles.

2. Make sure that your log bucket has been upgraded to use Observability Analytics:

   1. In the Google Cloud console, go to the Logs Storage page:

      Go to Logs Storage

      If you use the search bar to find this page, then select the result whose subheading is Logging.

   2. If the log bucket displays Upgrade, then click Upgrade and complete the dialog.

Create a linked BigQuery dataset

When you want to use the capabilities of BigQuery to analyze your log data, upgrade a log bucket to use Observability Analytics, and then create a linked dataset. With this configuration, Logging stores your log data but BigQuery can read the log data.

Note: A log bucket can be linked to at most one BigQuery dataset.

Google Cloud console

To create a link to a BigQuery dataset for an existing log bucket, do the following:

1. In the Google Cloud console, go to the Logs Storage page:

   Go to Logs Storage

   If you use the search bar to find this page, then select the result whose subheading is Logging.

2. Locate the log bucket and verify that the Observability Analytics available column displays Open.

   If this column displays Upgrade, then the log bucket hasn't been upgraded to use Observability Analytics. Configure Observability Analytics:

   1. Click Upgrade.
   2. Click Confirm in the dialog.

   After the upgrade completes, proceed to the next step.

3. On the log bucket, click More, and then click Edit bucket.

   The Edit log bucket dialog opens.

4. Select Create a new BigQuery dataset that links to this bucket and enter the name for the new dataset.

   The dataset name must be unique for each Google Cloud project. If you enter the name of an existing dataset, then you receive the following error: Dataset name must be unique in the selected region.

5. Click Done and then click Update bucket.

   After Logging displays the linked dataset name on the Logs Storage page, it might take several minutes before BigQuery recognizes the dataset.

gcloud

To create a linked dataset for a log bucket that is upgraded to use Observability Analytics, run the gcloud logging links create command:

gcloud logging links create LINK_ID --bucket=BUCKET_ID --location=LOCATION

The LINK_ID that you provide is used as the name of the BigQuery dataset, and the value of this field must be unique for your Google Cloud project.

The links create command is asynchronous. The return of an asynchronous method is an Operation object, and it contains information about the progress of the method. When the method completes, the Operation object contains the status. For more information, see Asynchronous API methods.

The links create command takes several minutes to complete.

For example, the following command creates a linked dataset named mylink for the log bucket named my-bucket:

gcloud logging links create mylink --bucket=my-bucket --location=global

The dataset name must be unique for each Google Cloud project. If you attempt to create a dataset with the same name as an existing dataset, then you receive the following error:

BigQuery dataset with name "LINK_ID" already exists.

If you attempt to create a linked dataset for a log bucket that isn't upgraded to use Observability Analytics, then the following error is reported:

A link can only be created for an analytics-enabled bucket.

REST

To create a linked a BigQuery dataset for an existing log bucket that is upgraded use Observability Analytics, call the asynchronous projects.locations.buckets.links.create method of the Cloud Logging API.

Prepare the arguments to the method as follows:

1. Construct the request body for the create command. The request body is formatted as a Link object.
2. For the query parameter of the command, use linkId=LINK_ID. The LINK_ID that you provide is used as the name of the BigQuery dataset, and the value of this field must be unique for your Google Cloud project..

The response to the asynchronous methods is an Operation object. This object contains information about the progress of the method. When the method completes, the Operation object contains the status. For more information, see Asynchronous API methods.

The links.create method takes several minutes to complete.

The dataset name must be unique for each Google Cloud project. If you attempt to create a dataset with the same name as an existing dataset, then you receive the following error:

BigQuery dataset with name "LINK_ID" already exists.

If you attempt to create a linked dataset for a log bucket that isn't upgraded to use Observability Analytics, then the following error is reported:

A link can only be created for an analytics-enabled bucket.

What's next

• Query log data with BigQuery Studio
• Run log queries on BigQuery engine