Run log queries on BigQuery engine

Configuring the Observability Analytics page to run your log queries on the BigQuery engine lets you use reserved slots instead of free slots. You must also use the BigQuery engine when you create SQL-based alerting policies or save dashboard charts whose queries join observability data with other business data.

By default, the Observability Analytics page runs queries on the default query engine, which checks your Logging permissions. When you switch to the BigQuery engine, BigQuery permissions are checked instead. Both query engines let you query observability views, views on log buckets, and analytics views.

To learn more, see Choose your Observability Analytics query engine.

Before you begin

Sign in to your Google Cloud account. If you're new to Google Cloud, create an account to evaluate how our products perform in real-world scenarios. New customers also get $300 in free credits to run, test, and deploy workloads.

In the Google Cloud console, on the project selector page, select or create a Google Cloud project.

Roles required to select or create a project

• Select a project: Selecting a project doesn't require a specific IAM role—you can select any project that you've been granted a role on.
• Create a project: To create a project, you need the Project Creator role (roles/resourcemanager.projectCreator), which contains the resourcemanager.projects.create permission. Learn how to grant roles.

Note: If you don't plan to keep the resources that you create in this procedure, create a project instead of selecting an existing project. After you finish these steps, you can delete the project, removing all resources associated with the project.

Go to project selector

Verify that billing is enabled for your Google Cloud project.

In the Google Cloud console, on the project selector page, select or create a Google Cloud project.

Roles required to select or create a project

• Select a project: Selecting a project doesn't require a specific IAM role—you can select any project that you've been granted a role on.
• Create a project: To create a project, you need the Project Creator role (roles/resourcemanager.projectCreator), which contains the resourcemanager.projects.create permission. Learn how to grant roles.

Note: If you don't plan to keep the resources that you create in this procedure, create a project instead of selecting an existing project. After you finish these steps, you can delete the project, removing all resources associated with the project.

Go to project selector

Verify that billing is enabled for your Google Cloud project.

1. To get the permissions that you need to run queries from Observability Analytics using the BigQuery engine, ask your administrator to grant you the following IAM roles:

   • Logs Viewer (roles/logging.viewer) on the project
   • Logs View Accessor (roles/logging.viewAccessor) on the project or on the view that you want to query
   • BigQuery Data Viewer (roles/bigquery.dataViewer) on your project or the linked dataset
   • BigQuery Job User (roles/bigquery.jobUser) on your project

   For more information about granting roles, see Manage access to projects, folders, and organizations.

   You might also be able to get the required permissions through custom roles or other predefined roles.

2. Verify that you have a linked BigQuery dataset on your log bucket. If the dataset doesn't exist, then create it.

If you plan to configure the Observability Analytics page to run queries on the BigQuery engine in projects that use Virtual Private Cloud (VPC) Service Controls, then verify that you're using the Enterprise Edition of BigQuery. For more information, see Understand BigQuery editions.

Run Observability Analytics queries on the BigQuery engine

When you want to monitor the results of your SQL query with an alerting policy, run your queries on the BigQuery engine by doing the following:

1. In the Google Cloud console, go to the Observability Analytics page:

   Go to Observability Analytics

   If you use the search bar to find this page, then select the result whose subheading is Logging.

2. In the Log views list, find the view, and then select Query or enter a query.

   If the query pane displays an error message that references the FROM statement, then the table can't be resolved to a specific log view. For information about how to resolve this failure, see Error FROM clause must contain exactly one log view.

3. Go to the toolbar and verify that a button labeled Run on BigQuery is displayed.

   If the toolbar displays Run Query, then click Settings and select BigQuery.

   If the Run on BigQuery button is disabled, then you need to create a linked dataset.

4. Run your query.

   You can use the toolbar options to format your query, clear the query, and open the BigQuery SQL reference documentation.

Pricing

To learn about the costs associated with using BigQuery, see BigQuery pricing.

What's next

• Monitor your SQL query results with an alerting policy
• Save and share a SQL query
• Sample SQL queries
• Chart SQL query results