Structured logging

In Cloud Logging, a LogEntry object can store a log payload as either structured data or unstructured text. A structured log entry stores its payload as a JSON object in the jsonPayload field, whereas an unstructured log entry stores its payload as a string in the textPayload field.

Formatting your log payload as a JSON object lets you query by JSON paths in the Logs Explorer, query individual fields on the Observability Analytics page, and index specific fields to speed up queries. With unstructured logs, you can search the text string, but you can't query or index individual keys.

How structured logs are written

How structured log entries are written to Logging depends on your environment and how your logs are collected:

• OpenTelemetry and the Telemetry (OTLP) API: When you send OpenTelemetry Protocol (OTLP) log records to Logging, Google Cloud Observability converts structured log data into the jsonPayload field. See OTLP log record to log entry mapping.

• Built-in logging agents: In managed compute environments (such as Google Kubernetes Engine and Cloud Run), built-in agents parse single-line JSON written to stdout or stderr into the jsonPayload field. See Choose a logging framework.
• Ops Agent: On Compute Engine VMs, the agent writes structured logs when it collects JSON logs or parses text logs. See Configure the Ops Agent.

• Logging client libraries: These libraries write structured logs when you pass a JSON object or map as the payload, or when they output single-line JSON to stdout or stderr. See Writing structured logs.

• Google Cloud CLI or the Logging API: You can run gcloud logging write with the --payload-type=json flag (example), or call the entries.write API method with a jsonPayload object.

• BindPlane: This service sends logs with JSON payloads from on-premises or hybrid sources to Logging. See the BindPlane quickstart guide.

Special JSON fields in structured payloads

When logging agents ingest structured logs, they can map specific fields in your log data to top-level fields in the LogEntry object instead of leaving them in jsonPayload:

• Ops Agent: For the JSON fields that the Ops Agent maps automatically when you use the parse_json processor or the tcp and fluent_forward receivers, see Special fields in structured payloads.

• OpenTelemetry (OTLP): For information about how Google Cloud Observability maps OTLP log records to LogEntry fields, see OTLP log record to log entry mapping.

• Built-in logging agents (stdout and stderr): On managed compute platforms such as Google Kubernetes Engine, Cloud Run, Cloud Run functions, and App Engine, the built-in logging agent recognizes the following top-level JSON keys and moves their values into the corresponding LogEntry fields:

  • severity: Sets the LogEntry.severity field.
  • message: Sets the main log message displayed in the Logs Explorer.
  • httpRequest: Sets the LogEntry.httpRequest field.
  • time or timestamp (or timestampSeconds and timestampNanos): Sets the LogEntry.timestamp field.
  • logging.googleapis.com/insertId: Sets the LogEntry.insertId field.
  • logging.googleapis.com/labels: Sets user-defined LogEntry.labels key-value pairs.
  • logging.googleapis.com/operation: Sets the LogEntry.operation field.
  • logging.googleapis.com/sourceLocation: Sets the LogEntry.sourceLocation field.
  • logging.googleapis.com/trace, logging.googleapis.com/spanId, and logging.googleapis.com/trace_sampled: Set the trace, spanId, and traceSampled fields for linking logs with traces.

  Any remaining fields in the JSON record stay in the LogEntry.jsonPayload field.

What's next

• To choose an instrumentation strategy for your compute platform, see Choose an instrumentation approach.

• To query and view log entries, see View logs by using the Logs Explorer.

• To read log entries by using the gcloud CLI, see Reading log entries.

• To read log entries by using the Logging API, see the entries.list method.