v1.logs overview

This feature is subject to the "Pre-GA Offerings Terms" in the General Service Terms section of the Service Specific Terms. Pre-GA features are available "as is" and might have limited support. For more information, see the launch stage descriptions.

This document describes how log data that is sent your Google Cloud project by using the Telemetry (OTLP) API is mapped to a Cloud Logging structure. This API implements the OpenTelemetry OTLP Protocol. You can send data to this API when you instrument your applications with an otlphttp exporter and an OpenTelemetry Collector, or when you use the OpenTelemetry SDKs.

OpenTelemetry is a Google Cloud-supported open source project with Google Cloud engineers staffed to ensure support for ingesting and visualizing your telemetry.

General structure of OTLP-formatted log data

When log data is sent to Google Cloud by using the Telemetry API, this data must be in a format that is consistent with OTLP. The general structure for this data is as shown:

"resourceLogs": [
    {
      "resource": {
        "attributes": [...]
      },
      "scopeLogs": [
        {
          "logRecords": [...]
        }
      ]
    }
]

Notice that OpenTelemetry batches individual logs, each of which is represented by a logRecord structure, with information about the source of those logs, which is represented by the resource structure.

When Google Cloud Observability receives a resourceLogs object, it constructs one LogEntry for each logRecord. Unlike OTLP which batches source information with a collection of individual logs, each LogEntry structure contains information about the source of the log and the log itself.

To learn more about the structure of OTLP-formatted log data, see the OpenTelemetry logs.proto.

How OTLP-formatted log data is processed

When you send an OTLP resourceLogs structure to the Telemetry API, Google Cloud Observability performs the following actions:

1. For each logRecord, the system creates a LogEntry.

   The following documents describe how the system populates the LogEntry from the OTLP log data:

   • OTLP log record to log entry mapping.
   • OTLP attributes to Cloud Logging resource type mapping.

   Because each LogEntry contains information about the corresponding Cloud Logging resource type, each LogEntry might be larger than the corresponding OTLP logRecord.

   The conversion from the OTLP resourceLogs structure into a series of LogEntry structure might be lossy. That is, you might not be able to convert from the LogEntry structure to the original resource and logRecord fields.

2. The LogEntry is routed to by aggregated sinks in the project hierarchy and by sinks in the project, in exactly the same manner as if the log entry was sent to your project by using the Cloud Logging API.

Best practices

When instrumenting your applications to send trace data to your Google Cloud project, we recommend that you use an exporter that writes OTLP-formatted data to a Collector, which then sends your trace data to the Telemetry API. In your collector, specify only the root URL:

exporters:
  otlphttp:
    encoding: proto
    endpoint: https://telemetry.googleapis.com

OpenTelemetry detects the data type and automatically appends /v1/traces, /v1/metrics, or /v1/logs as appropriate. For more information, see OTLP/HTTP Request.

For examples that export trace or metric data to the Telemetry API, see the following documents:

• Overview of collector-based instrumentation samples.
• OTLP metric ingestion overview.

When you can't use a collector, you can use an OpenTelemetry library that contains an in-process OTLP exporter to send telemetry to the Telemetry API. To learn how to directly export trace data, see Cloud Trace exporter to the OTLP endpoint.

Authentication

You must configure your exporters with the credentials necessary to send data to your Google Cloud project. For example, when you use collectors, typically you also use the googleclientauth extension to authenticate with Google credentials.

For an example of authentication when using direct export of trace data, see Configure authentication. This example illustrates how to configure the exporter with your Google Cloud Application Default Credentials (ADC) and add a language-specific Google Auth Library to your application.

Cloud Logging and data residency

By default, Cloud Logging routes log entries that originate in a project to system-created log buckets. Log buckets are regional resources. You can update the default log sink to send log entries to a custom log bucket that is in a location of your choice. To learn more, see Regionalize your logs.

Cloud Logging provides settings you can configure for organizations and folders. These settings specify the location of new system-created log buckets, whether these log buckets use customer managed encryption keys (CMEK), and the configuration of the default log sink. To learn more, see Support for organizations and folders.

Where to view ingested data

You can view your log data by using the Logs Explorer and Observability Analytics pages. For information, see the following:

• View and analyze logs.
• Query and analyze logs with SQL.

Limitations

This section describes limits. It also describes how Google Cloud Observability handles certain types of data.

Limits

Description	Value	Notes
Maximum number of logs per OTLP request	8192	Refers to the maximum number of logRecords in an OTLP resourceLogs structure. Limit.
Maximum size of each request	5 MiB	Limit.
Maximum size of a LogEntry that is created from an OTLP log record	256 KiB	Cloud Logging truncates or discards data from an OTLP log record when necessary. Limit.
Maximum length of an attribute key	512 B	Oversized label keys are truncated when the OTLP log record is converted into a LogEntry. Limit.
Maximum length of an attribute value	64 KiB	Oversized label values when the OTLP log record is converted into a LogEntry. Limit.
Maximum depth of attribute nesting	5	Attributes that exceed this limit are truncated when the OTLP log record is converted into a LogEntry.
Maximum number of log-ingestion bytes per minute	2.4 GB for the following regions: asia-east1, asia-northeast1, asia-southeast1, asia-south1, europe-west1, europe-west2, europe-west3, europe-west4, us-central1, us-east4, us-west1. 300 MB for all other regions.	Quota.

Behavior

• When both the OpenTelemetry severity number and severity text are set, the system uses the severity number to determine the Cloud Logging severity level. If the OTLP record doesn't contain severity information, then the Cloud Logging severity level is set to DEFAULT.

• When an OTLP record contains duplicate attribute keys, the system retains the first key and discards the attributes with duplicate keys.

• The system converts attributes attached to a log record into a string. For an example, see Labels field.

• Log names must be URL-safe or they are URL-encoded during ingestion. For information about how log names are set, see How LogEntry fields are set.