Run log queries on BigQuery engine

This document describes how you can configure the Observability Analytics page to run queries on the BigQuery engine. You must run queries on the BigQuery engine when you have SQL-based alerting policies. You also need to use this query engine when you save SQL-based charts to a dashboard, and the chart's query joins observability data and other business data.

The Observability Analytics page runs queries on the default query engine. However, you can configure this page to run queries on the BigQuery engine. Both query engines run queries on your free slots. However, if you use the BigQuery engine and reserve slots, then that engine runs its queries on your reserved slots.

Whether you use the default query engine or the BigQuery engine, you can query observability views, views on log buckets, and analytics views. If you run queries on the default query engine, then your Logging permissions are checked. However, if you run your queries on the BigQuery engine, then BigQuery permissions are checked.

To learn more, see Choose your Observability Analytics query engine.

Before you begin

Sign in to your Google Cloud account. If you're new to Google Cloud, create an account to evaluate how our products perform in real-world scenarios. New customers also get $300 in free credits to run, test, and deploy workloads.

In the Google Cloud console, on the project selector page, select or create a Google Cloud project.

Roles required to select or create a project

• Select a project: Selecting a project doesn't require a specific IAM role—you can select any project that you've been granted a role on.
• Create a project: To create a project, you need the Project Creator role (roles/resourcemanager.projectCreator), which contains the resourcemanager.projects.create permission. Learn how to grant roles.

Note: If you don't plan to keep the resources that you create in this procedure, create a project instead of selecting an existing project. After you finish these steps, you can delete the project, removing all resources associated with the project.

Go to project selector

Verify that billing is enabled for your Google Cloud project.

In the Google Cloud console, on the project selector page, select or create a Google Cloud project.

Roles required to select or create a project

• Select a project: Selecting a project doesn't require a specific IAM role—you can select any project that you've been granted a role on.
• Create a project: To create a project, you need the Project Creator role (roles/resourcemanager.projectCreator), which contains the resourcemanager.projects.create permission. Learn how to grant roles.

Note: If you don't plan to keep the resources that you create in this procedure, create a project instead of selecting an existing project. After you finish these steps, you can delete the project, removing all resources associated with the project.

Go to project selector

Verify that billing is enabled for your Google Cloud project.

1. To get the permissions that you need to query a linked dataset from BigQuery Studio, ask your administrator to grant you the following IAM roles:

   • Logs View Accessor (roles/logging.viewAccessor) on the project or on the view that you want to query.
   • BigQuery User (roles/bigquery.user) on your project.
   • BigQuery Job User (roles/bigquery.jobUser) on your project.

   For more information about granting roles, see Manage access to projects, folders, and organizations.

   You might also be able to get the required permissions through custom roles or other predefined roles.

2. Verify that you have a linked BigQuery dataset on your log bucket. If the dataset doesn't exist, then create it.

If you plan to configure the Observability Analytics page to run queries on the BigQuery engine and if you are using projects with Virtual Private Cloud (VPC) Service Controls, then verify you are using the Enterprise Edition of BigQuery. For more information, see Understand BigQuery editions.

Run Observability Analytics queries on the BigQuery engine

When you want to monitor the results of your SQL query with an alerting policy, run your queries on BigQuery engine by doing the following:

1. Configure Observability Analytics to run your queries on BigQuery engine:

   1. In the Google Cloud console, go to the Observability Analytics page:

      Go to Observability Analytics

      If you use the search bar to find this page, then select the result whose subheading is Logging.

   2. In the Log views list, find the view, and then select Query or enter a query.

      If the query pane displays an error message that references the FROM statement, then the table can't be resolved to a specific log view. For information about how to resolve this failure, see Error FROM clause must contain exactly one log view.

   3. Go to the toolbar and verify that a button labeled Run on BigQuery is displayed.

      If the toolbar displays Run Query, then click Settings and select BigQuery.

      If the Run on BigQuery button is disabled, then you need to create a linked dataset.

   4. Run your query.

      You can use the toolbar options to format your query, clear the query, and open the BigQuery SQL reference documentation.

Pricing

To learn about the costs associated with using BigQuery, see BigQuery pricing.

What's next

• Monitor your SQL query results with an alerting policy
• Save and share a SQL query
• Sample SQL queries
• Chart SQL query results