Run queries on BigQuery engine

This document describes how you can configure the Observability Analytics page to run queries on the BigQuery engine. You must run queries on the BigQuery engine when you have SQL-based alerting policies. You also need to use this query engine when you save SQL-based charts to a dashboard, and the query joins observability data and other business data.

By default, the Observability Analytics page runs queries on the default query engine, which uses the default slots to run your queries. The BigQuery engine also runs queries on the default slots, unless you reserve slots. Reserving slots can remove delays that might occur because default slots are occupied when you submit your query.

Whether you use the default query engine or the BigQuery engine, you can query observability views, views on log buckets, and analytics views.

Before you begin

Sign in to your Google Cloud account. If you're new to Google Cloud, create an account to evaluate how our products perform in real-world scenarios. New customers also get $300 in free credits to run, test, and deploy workloads.

In the Google Cloud console, on the project selector page, select or create a Google Cloud project.

Roles required to select or create a project

• Select a project: Selecting a project doesn't require a specific IAM role—you can select any project that you've been granted a role on.
• Create a project: To create a project, you need the Project Creator role (roles/resourcemanager.projectCreator), which contains the resourcemanager.projects.create permission. Learn how to grant roles.

Note: If you don't plan to keep the resources that you create in this procedure, create a project instead of selecting an existing project. After you finish these steps, you can delete the project, removing all resources associated with the project.

Go to project selector

Verify that billing is enabled for your Google Cloud project.

Enable the Cloud Monitoring and Observability APIs.

Roles required to enable APIs

To enable APIs, you need the Service Usage Admin IAM role (roles/serviceusage.serviceUsageAdmin), which contains the serviceusage.services.enable permission. Learn how to grant roles.

Enable the APIs

In the Google Cloud console, on the project selector page, select or create a Google Cloud project.

Roles required to select or create a project

• Select a project: Selecting a project doesn't require a specific IAM role—you can select any project that you've been granted a role on.
• Create a project: To create a project, you need the Project Creator role (roles/resourcemanager.projectCreator), which contains the resourcemanager.projects.create permission. Learn how to grant roles.

Note: If you don't plan to keep the resources that you create in this procedure, create a project instead of selecting an existing project. After you finish these steps, you can delete the project, removing all resources associated with the project.

Go to project selector

Verify that billing is enabled for your Google Cloud project.

Enable the Cloud Monitoring and Observability APIs.

Roles required to enable APIs

To enable APIs, you need the Service Usage Admin IAM role (roles/serviceusage.serviceUsageAdmin), which contains the serviceusage.services.enable permission. Learn how to grant roles.

Enable the APIs

1. To get the permissions that you need to run queries from Observability Analytics using BigQuery slots, ask your administrator to grant you the following IAM roles:

   • Observability View Accessor (roles/observability.viewAccessor) on the observability views that you want to query. This role supports IAM conditions, which let you restrict the grant to a specific view. If you don't attach a condition to the role grant, then the principal can access all observability views.
   • BigQuery User (roles/bigquery.user) on your project
   • BigQuery Job User (roles/bigquery.jobUser) on your project

   For more information about granting roles, see Manage access to projects, folders, and organizations.

   You might also be able to get the required permissions through custom roles or other predefined roles.

2. Verify that the linked BigQuery dataset for the observability bucket named _Trace exists. If the linked BigQuery dataset doesn't exist, then create it.

If you plan to configure the Observability Analytics page to run queries on your BigQuery slots and if you are using projects with Virtual Private Cloud (VPC) Service Controls, then verify you are using the Enterprise Edition of BigQuery. For more information, see Understand BigQuery editions.

Run queries on the BigQuery engine

To run a query on the BigQuery engine when using the Observability Analytics page, do the following:

1. In the Google Cloud console, go to the Observability Analytics page:

   Go to Observability Analytics

   If you use the search bar to find this page, then select the result whose subheading is Logging.

2. Enter or load a query. For more information, see Query and analyze traces.

   You can save a chart to a dashboard where the chart displays the result of a join between observability data and other business data. For this case, the FROM clause must specify either a view on an observability dataset or a view on a log bucket, and there must also be a linked dataset for the data queried. The JOIN statement can specify a BigQuery table. However, you must have the required IAM permissions to query the BigQuery table.

3. Go to the toolbar and verify that a button labeled Run on BigQuery is displayed.

   If the toolbar displays Run Query, then click Settings and select BigQuery.

   A disabled Run on BigQuery button indicates that there isn't a linked dataset. To resolve this failure, create the linked BigQuery dataset.

4. Click Run on BigQuery.

Pricing

To learn about the costs associated with using BigQuery, see BigQuery pricing.

What's next

• Monitor your SQL query results with an alerting policy.
• Save and share a SQL query.
• Chart SQL query results.
• Sample SQL queries.