Analyze logs using Logs Explorer and Log Analytics

This document describes how you query, view, and analyze log entries by using the Google Cloud console. There are two interfaces available to you, the Logs Explorer and Log Analytics. You can query, view, and analyze logs with both interfaces; however, they use different query languages and they have different capabilities:

  • For troubleshooting and exploration of log data, use the Logs Explorer.

  • To join your log and trace data, or to generate insights and trends, use Log Analytics.

You can query your logs and save your queries by issuing Logging API commands. You can also query your logs by using Google Cloud CLI.

Use Logs Explorer

The Logs Explorer is designed to help you troubleshoot and analyze the performance of your services and applications. For example, a histogram displays the rate of errors. If you see a spike in errors or something that is interesting, you can locate and view the corresponding log entries. When a log entry is associated with an error group, the log entry is annotated with a menu of options that let you access more information about the error group.

The same query language is supported by the Cloud Logging API, the Google Cloud CLI, and the Logs Explorer. To simplify query construction when you are using the Logs Explorer, you can build queries by using menus, by entering text, and, in some cases, by using options included with the display of an individual log entry.

The Logs Explorer doesn't support aggregate operations, like counting the number of log entries that contain a specific pattern. To perform aggregate operations, enable analytics on the log bucket and then use Log Analytics.

For details about searching and viewing logs with the Logs Explorer, see View logs by using the Logs Explorer.

Explore Log Analytics

Using Log Analytics, you can generate insights by running queries that group and aggregate your log data. These insights can help reduce the time you spend troubleshooting issues. To view your query results, use a table, a chart, or both. Charts can help you identify patterns and trends in your log data. For example, the following screenshot shows a query result being displayed as a table and a chart:

User interface for Log Analytics.

Log Analytics supports the following:

  • Grouping and aggregating log data.

    For example, you can run a SQL query that groups log entries by the hour, and then computes for each group, the average latency for HTTP requests issued to a specific URL.

  • SQL queries that use pipe syntax.

  • Queries of log views and analytics views.

    Log views have a system-defined schema. You define the schema for analytics views.

  • Joins of log and trace data.

    For information about querying your trace data, see Query and analyze traces.

Cloud Logging also lets you query your log data from BigQuery, without exporting that data to BigQuery. After you upgrade your log bucket to use Log Analytics, create a linked dataset. You can query the linked dataset by using BigQuery services.

Upgrading a log bucket doesn't affect your usage of the Logs Explorer. The Logs Explorer only requires that your log data be stored in a log bucket.

Restrictions

  • To upgrade an existing log bucket to use Log Analytics, the following restrictions apply:

    • The log bucket was created at the Google Cloud project level.
    • The log bucket is unlocked unless it is the _Required bucket.
    • There aren't pending updates to the bucket.

  • Log entries written before a bucket is upgraded aren't immediately available. However, when the backfill operation completes, you can analyze these log entries. The backfill process might take several days.

  • You can't use the Log Analytics page to query log views when the log bucket has field-level access controls configured. However, you can issue queries through the Logs Explorer page, and you can query a linked BigQuery dataset. Because BigQuery doesn't honor field-level access controls, if you query a linked dataset, then you can query all fields in the log entries.

  • If you query multiple log views, then they must be stored in the same location. For example, if two views are located in the us-east1 location, then one query can query both views. You can also query two views that are located in the us multi-region. However, if a view's location is global, then that view can reside in any physical location. Therefore, joins between two views that have the location of global might fail.

  • If you query multiple log views and their underlying log buckets are configured with different Cloud KMS keys, then the query fails unless the following constraints are met:

    • A folder or organization that is a parent resource of the log buckets is configured with a default key.
    • The default key is in the same location as the log buckets.

    When the previous constraints are satisfied, the parent's Cloud KMS key encrypts any temporary data generated by a Log Analytics query.

  • Duplicate log entries aren't removed before a query is run. This behavior is different than when you query log entries by using the Logs Explorer, which removes duplicate entries by comparing the log names, timestamps, and insert ID fields. For more information, see Troubleshoot: There are duplicate log entries in my Log Analytics results.

Pricing

For pricing information, see Google Cloud Observability pricing page. If you route log data to other Google Cloud services, then see the following documents:

There are no BigQuery ingestion or storage costs when you upgrade a bucket to use Log Analytics and then create a linked dataset. When you create a linked dataset for a log bucket, you don't ingest your log data into BigQuery. Instead, you get read access to the log data stored in your log bucket through the linked dataset.

BigQuery analysis charges apply when you run SQL queries on BigQuery linked datasets, which includes using the BigQuery Studio page, the BigQuery API, and the BigQuery command-line tool.

Blogs

For more information about Log Analytics, see the following blog posts:

What's next