OTLP log record to log entry mapping

Preview

This product or feature is subject to the "Pre-GA Offerings Terms" in the General Service Terms section of the Service Specific Terms. Pre-GA products and features are available "as is" and might have limited support. For more information, see the launch stage descriptions.

This document describes how Google Cloud Observability determines the fields of a LogEntry from the OTLP log record, when that record is sent to Google Cloud by using the Telemetry API.

General structure of OTLP-formatted log data

When log data is sent to Google Cloud by using the Telemetry API, this data must be in a format that is consistent with OTLP. The general structure for this data is as shown:

"resourceLogs": [
    {
      "resource": {
        "attributes": [...]
      },
      "scopeLogs": [
        {
          "logRecords": [...]
        }
      ]
    }
]

Notice that OpenTelemetry batches individual logs, each of which is represented by a logRecord structure, with information about the source of those logs, which is represented by the resource structure.

When Google Cloud Observability receives a resourceLogs object, it constructs one LogEntry for each logRecord. Unlike OTLP which batches source information with a collection of individual logs, each LogEntry structure contains information about the source of the log and the log itself.

To learn more about the structure of OTLP-formatted log data, see the OpenTelemetry logs.proto.

How LogEntry fields are set

Google Cloud Observability uses the following rules to determine the values for LogEntry fields:

LogEntry field (Names from HTTP reference)	How the system determines the field value
logName	The system uses the following prioritized list of OpenTelemetry log record attributes to determine the log name: gcp.log_name log_name event_name Log names must be URL-safe or they are URL-encoded during ingestion.
resource	The system uses the information set in the resource field or it infers the resource. For details, see OTLP attributes to resource type mapping.
timestamp	The system uses the following prioritized list of OpenTelemetry log record fields to determine the timestamp: time_unix_nano when non-zero. observed_time_unix_nano when non-zero. The time the log entry was ingested. The retention period for the log bucket that stores the data determines the oldest timestamp that can be ingested. For more information, see Cloud Logging quotas.
receiveTimestamp	Set to the time the LogEntry is ingested.
severity	The system maps the OpenTelemetry severity in the log record to a Cloud Logging severity. For details, see Severity field.
httpRequest	The system maps the gcp.http_request attributes in the log record to equivalent LogEntry fields. For more information, see HttpRequest field. FIXME. There are other fields that get mapped to this field.
labels	The system sets this field by using the log record attributes values. For more information, see Labels field.
trace	The system sets this field to the value from the log record's traceId field. The value must be a valid 32 char hex string, or the entry is rejected.
spanId	The system sets this field to the value from the log record's spanId field. The value must be a valid 16 char hex string, or the entry is rejected.
traceSampled	Not set.
sourceLocation	The system sets this field to the values from the log record's Code attributes. For more information, see SourceLocation field.
split	Not set.
apphub apphubDestination apphubSource	Not set.
otel	For fields in OTLP log data that don't have an equivalent field in a LogEntry, the system converts data types and then adds the converted data to the otel field. For more information, see Otel field.
Payload	The system sets the payload by converting the body of a log record to an appropriate payload type. For more information, see Payload field.

HttpRequest field

Google Cloud Observability maps OTLP attributes that apply to HTTP requests to LogEntry fields. The following sections describe how the system maps flat and nested attributes.

Flat attributes

The following table describes how Google Cloud Observability maps flat attributes that apply to HTTP requests to LogEntry fields. For example, the value from the attribute http.request.method: "GET", is set as the value of the httpRequest.requestMethod field in a log entry:

OpenTelemetry LogRecord.attribute key-value pair.	Value stored in the following LogEntry field (Names from HTTP reference)	Accepted Type
http.request.method	httpRequest.requestMethod	string
url.full http.url	httpRequest.requestUrl	string
http.request.body.size	httpRequest.requestSize	string, int
http.response.status_code	httpRequest.status	string, int
http.response.body.size	httpRequest.responseSize	string, int
user_agent.original http.user_agent	httpRequest.userAgent	string
client.address	remoteIp	string
server.address	serverIp	string
referrer	httpRequest.referer	string
latency	httpRequest.latency	string, int
cacheLookup	httpRequest.cacheLookup	bool
cacheHit	httpRequest.cacheHit	bool
cacheValidatedWithOriginServer	httpRequest.cacheValidatedWithOriginServer	bool
cacheFillBytes	httpRequest.cacheFillBytes	string, int
network.protocol.version protocol	httpRequest.protocol	string

Nested attributes

This section describes how Google Cloud Observability maps nested OTLP attributes that apply to HTTP requests to fields in a LogEntry. The following example illustrates a log record that contains two attributes, each of which contains at least one other attribute:

log_record {
  attributes: {
    gcp.http_request {
      "requestMethod": "GET",
      "requestUrl": "some-URL",
    }
    http_request {
      "requestMethod": "GET",
    }
  }
}

In the table, the nested attributes are shown by using braces. For example, gcp.http_request {requestMethod} means that the attribute gcp.http_request contains the attribute requestMethod. The value from the innermost attribute is assigned to the value of the log entry field:

OpenTelemetry LogRecord.attribute key-value pair.	Value stored in the following LogEntry field (Names from HTTP reference)	Accepted Type
gcp.http_request {requestMethod} http_request {requestMethod}	httpRequest.requestMethod	string
gcp.http_request {requestUrl} http_request {requestUrl}	httpRequest.requestUrl	string
gcp.http_request {requestSize} http_request {requestSize}	httpRequest.requestSize	string, int
gcp.http_request {status} http_request {status}	httpRequest.status	string, int
gcp.http_request {responseSize} http_request {responseSize}	httpRequest.responseSize	string, int
gcp.http_request {userAgent} http_request {userAgent}	httpRequest.userAgent	string
gcp.http_request {remoteIp} http_request {remoteIp}	httpRequest.remoteIp	string
gcp.http_request {serverIp} http_request {serverIp}	httpRequest.serverIp	string
gcp.http_request {referer} http_request {referrer}	httpRequest.referer	string
gcp.http_request {latency} http_request {latency}	httpRequest.latency	string, int
gcp.http_request {cacheLookup} http_request {cacheLookup}	httpRequest.cacheLookup	bool
gcp.http_request {cacheHit} http_request {cacheHit}	httpRequest.cacheHit	bool
gcp.http_request { cacheValidatedWithOriginServer} http_request { cacheValidatedWithOriginServer}	httpRequest.cacheValidatedWithOriginServer	bool
gcp.http_request {cacheFillBytes} http_request {cacheFillBytes}	httpRequest.cacheFillBytes	string, int
gcp.http_request {protocol} http_request {protocol}	httpRequest.protocol	string

Labels field

To determine what labels to attach to the log entry, the system performs the following actions:

1. It removes from the OTLP log record any attributes that have been mapped to specific LogEntry fields.

   For example, suppose there are the attributes attached to a log record:

   attributes: {
       "log_array_attr: ["value1", "value2"],
       "log_json_attr": {"json_key": "json_value"}
       "log-string-attr": "string",
       "code.file.path": "my-file.cc",
       "code.function.name: "my-func",
       "code.line.number": 123,
       "gcp.http_request": {
           "requestMethod": "GET",
           "requestUrl": "my-URL",
     },
   }
   

   After removing fields that are mapped to specific LogEntry fields, the following fields remain:

   attributes: {
       "log_array_attr: ["value1", "value2"],
       "log_json_attr": {"json_key": "json_value"}
       "log-string-attr": "string",
   }
   

2. If an attribute contains an array or JSON elements, then the system converts the value into a string.

   For example, the following illustrates how the LogEntry represents the previous attributes:

   labels: {
       "log_array_attr": "[\"value1\",\"value2\"]",
       "log_json_attr": "{\"json_key\":\"json_value\"}",
       "log-string-attr": "string",
   }
   

   The maximum nesting depth for attributes is five. Any content that has deeper nesting is truncated.

Otel field

For fields in OTLP log data that don't have an equivalent field in a LogEntry, the system converts data types and then adds the converted data to the otel field. For example, the otel field stores attributes from the resource, scope, and entity fields.

The system uses the following rules to convert the OpenTelemetry data types to protobuf Value types:

OpenTelemetry type	protobuf type
string	string
boolean	bool
integer	double
float	double
Array	ListValues
KeyValueList	Struct

To avoid double precision errors, pass integers as strings.

Payload field

Note: The maximum size of a LogEntry is 256 KiB. This limit is approximate and is based on internal data sizes. When necessary, the system truncates the payload field.

The data type of the OTLP logRecord.body field determines the structure of the LogEntry payload:

• string: The system copies the string into the LogEntry.textPayload field.

• Array: The system creates a string of the array elements while preserving new lines. It then copies the string into the LogEntry.textPayload field.

• KeyValueList: The system converts those pairs into JSON and then populates the LogEntry.jsonPayload field, with the following restrictions:

  • When an OTLP record contains duplicate attribute keys, the system retains the first key and discards attributes with duplicate keys.
  • When the nesting depth for a JSON pair is larger than five, the system truncates the content to a depth of five.

Severity field

This section describes how Google Cloud Observability maps the OpenTelemetry severity fields to Cloud Logging severity levels. OpenTelemetry defines a severity number and severity text. The logs.proto defines the severity numbers.

Google Cloud Observability determines the Logging severity from the OpenTelemetry severity number, if set. Otherwise, it uses the severity text. If neither are set, then the Logging severity is set to DEFAULT.

OpenTelemetry severity number Enum (value)	OpenTelemetry severity text (tests are case insensitive)	Cloud Logging severity Enum (value)
SEVERITY_NUMBER_UNSPECIFIED (0)	"default" not set	DEFAULT (0)
SEVERITY_NUMBER_TRACE (1) SEVERITY_NUMBER_TRACE2 (2) SEVERITY_NUMBER_TRACE3 (3) SEVERITY_NUMBER_TRACE4 (4)	"trace" "trace2" "trace3" "trace4"	DEBUG (100)
SEVERITY_NUMBER_DEBUG (5) SEVERITY_NUMBER_DEBUG2 (6) SEVERITY_NUMBER_DEBUG3 (7) SEVERITY_NUMBER_DEBUG4 (8)	"debug" "debug2" "debug3" "debug4"	DEBUG (100)
SEVERITY_NUMBER_INFO (9) SEVERITY_NUMBER_INFO2 (10)	"info" "info2"	INFO (200)
SEVERITY_NUMBER_INFO3 (11) SEVERITY_NUMBER_INFO4 (12)	"notice" "info3" "info4"	NOTICE (300)
SEVERITY_NUMBER_WARN (13) SEVERITY_NUMBER_WARN2 (14) SEVERITY_NUMBER_WARN3 (15) SEVERITY_NUMBER_WARN4 (16)	"warning" "warn" "warn2" "warn3" "warn4"	WARNING (400)
SEVERITY_NUMBER_ERROR (17) SEVERITY_NUMBER_ERROR2 (18) SEVERITY_NUMBER_ERROR3 (19) SEVERITY_NUMBER_ERROR4 (20)	"error" "error2" "error3" "error4"	ERROR (500)
SEVERITY_NUMBER_FATAL (21) SEVERITY_NUMBER_FATAL2 (22)	"critical" "fatal" "fatal2"	CRITICAL (600)
SEVERITY_NUMBER_FATAL3 (23)	"alert" "fatal3"	ALERT (700)
SEVERITY_NUMBER_FATAL4 (24)	"emergency" "fatal4"	EMERGENCY (800)

SourceLocation field

Google Cloud Observability maps the following OTLP Code directly to LogEntry fields. This mapping is possible because these OpenTelemetry attributes are semantically identical to Cloud Logging concepts.

OpenTelemetry LogRecord.attribute key-value pair.	Value stored in the following LogEntry field (Names from HTTP reference)	Accepted Type
code.file.path: Value	sourceLocation.file	string
code.function.name: Value	sourceLocation.function	string
code.function.number: Value	sourceLocation.line	string, int

Limitations

This section describes limits. It also describes how Google Cloud Observability handles certain types of data.

Limits

Description	Value	Notes
Maximum number of logs per OTLP request	8192	Refers to the maximum number of logRecords in an OTLP resourceLogs structure. Limit.
Maximum size of each request	5 MiB	Limit.
Maximum size of a LogEntry that is created from an OTLP log record	256 KiB	Cloud Logging truncates or discards data from an OTLP log record when necessary. Limit.
Maximum length of an attribute key	512 B	Oversized label keys are truncated when the OTLP log record is converted into a LogEntry. Limit.
Maximum length of an attribute value	64 KiB	Oversized label values when the OTLP log record is converted into a LogEntry. Limit.
Maximum depth of attribute nesting	5	Attributes that exceed this limit are truncated when the OTLP log record is converted into a LogEntry.
Maximum number of log-ingestion bytes per minute	2.4 GB for the following regions: asia-east1, asia-northeast1, asia-southeast1, asia-south1, europe-west1, europe-west2, europe-west3, europe-west4, us-central1, us-east4, us-west1. 300 MB for all other regions.	Quota.

Behavior

• When both the OpenTelemetry severity number and severity text are set, the system uses the severity number to determine the Cloud Logging severity level. If the OTLP record doesn't contain severity information, then the Cloud Logging severity level is set to DEFAULT.

• When an OTLP record contains duplicate attribute keys, the system retains the first key and discards the attributes with duplicate keys.

• The system converts attributes attached to a log record into a string. For an example, see Labels field.

• Log names must be URL-safe or they are URL-encoded during ingestion. For information about how log names are set, see How LogEntry fields are set.