OTLP attributes to resource type mapping

Preview

This product or feature is subject to the "Pre-GA Offerings Terms" in the General Service Terms section of the Service Specific Terms. Pre-GA products and features are available "as is" and might have limited support. For more information, see the launch stage descriptions.

This document describes the rules Google Cloud Observability uses to determine the Logging monitored resource type for log data that is ingested by the Telemetry API. The monitored resource describes the source of log data, and in a LogEntry, it is represented by using labels. Google Cloud Observability determines the values of those labels by using the OTLP resource attributes.

General structure of OTLP-formatted log data

When log data is sent to Google Cloud by using the Telemetry API, this data must be in a format that is consistent with OTLP. The general structure for this data is as shown:

"resourceLogs": [
    {
      "resource": {
        "attributes": [...]
      },
      "scopeLogs": [
        {
          "logRecords": [...]
        }
      ]
    }
]

Notice that OpenTelemetry batches individual logs, each of which is represented by a logRecord structure, with information about the source of those logs, which is represented by the resource structure.

When Google Cloud Observability receives a resourceLogs object, it constructs one LogEntry for each logRecord. Unlike OTLP which batches source information with a collection of individual logs, each LogEntry structure contains information about the source of the log and the log itself.

To learn more about the structure of OTLP-formatted log data, see the OpenTelemetry logs.proto.

Set OTLP resource attributes by using the Google Cloud resource detector

To set OTLP resource attributes in your OTLP log data, use a Google Cloud resource detector. These detectors attach to telemetry the OTLP resource attributes that describe the source of the telemetry:

• For collector-based export, configure resource detection as part of the collector's processors. For example, your collector might specify two resource detectors:

  resourcedetection:
      detectors: ["env", "gcp"]
  

  The following documents provide examples of collectors that export metric or trace data:

  • Deploy and use the collector: Describes how to send metric data to the Telemetry API.
  • Deploy Google-Built OpenTelemetry Collector on Google Kubernetes Engine: Describes how to collect log, metric, and trace data when running on Google Kubernetes Engine. The collector sends trace data to the Telemetry API and metric data to a Prometheus exporter.
  Note: We recommend that you use a collector-based export when you use Google Kubernetes Engine. This is the only configuration that lets you configure the resource attributes that Google Cloud Observability requires to associate telemetry with an App Hub application.

• If you are using direct export, then use the appropriate Google Cloud resource detector for your language:

  • Go: Google Cloud resource detector
  • Java: opentelemetry-java-contrib/gcp-resources
  • Node.js: resource-detector-gcp
  • Python: opentelemetry-resourcedetector-gcp

How resource labels are set

Each log entry contains a set of labels that identifies the source of the log entry, which is a also known as a Logging monitored resource. The following illustrates the labels for Google Kubernetes Engine.

resource: {
    labels: {
        cluster_name: "my-cluster"
        container_name: "server"
        location: "us-central1"
        namespace_name: "default"
        pod_name: "my-pod"
        project_id: "my-project"
    }
    type: "k8s_container"
}

As shown, a monitored resource has a type field and one or more labels. At a minimum, the labels for a monitored resource include either project_id or resource_container.

The following table describes the rules that Google Cloud Observability uses to map OTLP resource attributes to a monitored resource. The table is ordered. That is, if the resource attributes include cloud.platform and the value is gcp_compute_engine, then the monitored resource type is set to gce_instance. To determine the value of a monitored resource's label, Google Cloud Observability uses prioritized list of resource attributes. The value of a label is an empty string when none of the resource attributes are available.

OTLP attribute and value attached to a log record	Cloud Logging monitored resource type	Monitored resource label values as determined from OTLP resource attributes, by priority
cloud.platform has value gcp_compute_engine	gce_instance	project_id: See Infer project ID. instance_id: instance_id, host.id zone: zone, cloud.availability_zone
cloud.platform has value gcp_app_engine	gae_app	project_id: See Infer project ID. module_id: gae_module_id, faas.name version_id: gae_version_id, faas.version zone: zone, cloud.availability_zone
cloud.platform has value aws_ec2	aws_ec2_instance	project_id: See Infer project ID. instance_id: instance_id, host.id aws_account: aws_account, cloud.account.id region: region, cloud.availability_zone, cloud.region
cloud.platform has value gcp_bare_metal_solution	baremetal instance	resource_container: See Infer resource_container ID. region: region, cloud.availability_zone, cloud.region instance_id: instance_id, host.id
service.name and service.instance.id are not empty	generic_task	project_id: See Infer project ID. location: location, cloud.availability_zone, cloud.region namespace: namespace, service.namespace job_name: job, service.name, faas.name task_id: task_id, service.instance.id, faas.instance
faas.name and faas.instance are not empty	generic_task	project_id: See Infer project ID. location: location, cloud.availability_zone, cloud.region namespace: namespace, service.namespace job_name: job, service.name, faas.name task_id: task_id, service.instance.id, faas.instance
k8s.cluster.name and k8s.container.name are not empty	k8s_container	project_id: See Infer project ID. location: location, cloud.availability_zone, cloud.region cluster_name: cluster_name, k8s.cluster.name namespace_name: namespace_name, k8s.namespace_name pod_name: pod_name, k8s.pod.name container_name: container_name, k8s.container.name
k8s.cluster.name and k8s.pod.name are not empty	k8s_pod	project_id: See Infer project ID. location: location, cloud.availability_zone, cloud.region cluster_name: cluster_name, k8s.cluster.name namespace_name: namespace_name, k8s.namespace_name pod_name: pod_name, k8s.pod.name
k8s.cluster.name and k8s.node.name are not empty	k8s_node	project_id: See Infer project ID. location: location, cloud.availability_zone, cloud.region cluster_name: cluster_name, k8s.cluster.name node_name: node_name, k8s.node.name
k8s.cluster.name is not empty	k8s_cluster	project_id: See Infer project ID. location: location, cloud.availability_zone, cloud.region cluster_name: cluster_name, k8s.cluster.name
No other label matches	generic_node	project_id: See Infer project ID. location: location, cloud.availability_zone, cloud.region namespace: namespace, service.namespace node_id: node_id, host.id, host_name

Infer project or resource container ID label value

All Logging monitored resource types have either a project_id label or a resource_container label. The system searches the following attributes, in order, to determine the value of those labels:

• project_id
• gcp.project.id
• gcp.project_id
• gcp.dest_project_id
• gcp.resource.container
• resource_container
• cloud.account.id
• cloud.account_id