Telemetry (OTLP) API overview

This document describes the Telemetry (OTLP) API, which implements the OpenTelemetry OTLP Protocol. This API is designed for use with applications that are instrumented by using one of the OpenTelemetry SDKs or that use any OpenTelemetry Collector.

OpenTelemetry is a Google Cloud-supported open source project with Google Cloud engineers staffed to ensure support for ingesting and visualizing your telemetry.

Trace data and the Telemetry API

This section provides information about the Telemetry API and traces.

Why you should use the Telemetry API

When you use the Telemetry API, your data is stored in a format that is generally consistent with the proto files defined by the OpenTelemetry OTLP Protocol. However, fields might be converted from an OpenTelemetry-specific data type to a JSON data type before storage. Further, the limits for the Telemetry API apply. These limits are often more generous than those for the Cloud Trace API. Lastly, your instrumentation doesn't rely on a Google Cloud-specific exporter.

To learn more about the storage format, see Schema for trace data.

When to use the Telemetry API

We recommend that you send your trace data to your Google Cloud project by using the Telemetry API. This API provides compatibility with the open source OpenTelemetry ecosystem and its limits are often more generous than the limits of the Cloud Trace API, which is a proprietary Google Cloud API. Some features, like Application Monitoring, rely on information that is available only when trace data is sent to the Telemetry API.

When instrumenting your applications to send trace data to your Google Cloud project, we recommend that you do one of the following:

• Use an exporter that writes OTLP to a Collector, which then sends your trace data to the Telemetry API.
• Use an in-process OTLP exporter supported by an OpenTelemetry library that sends telemetry to the Telemetry API. There is no collector with this configuration.

For information about how to use the Telemetry API, see Migrate from the Cloud Trace exporter to the OTLP endpoint.

Authentication

Exporters must be authorized to send data to your Google Cloud project. For example, you can configure the exporter with your Google Cloud Application Default Credentials (ADC) by adding a language-specific Google Auth Library to your application. For more information and sample code, see Configure authentication.

Cloud Trace and data residency

If you are using Assured Workloads because you have data-residency or Impact Level 4 (IL4) requirements, then don't use the Telemetry API to send trace spans.

Metric data and the Telemetry API

This section describes how Cloud Monitoring handles metrics ingested by using the otlphttp exporter and an OpenTelemetry Collector or by applications that are instrumented by using one of the OpenTelemetry SDKs.

OTLP metrics in Cloud Monitoring

This feature is subject to the "Pre-GA Offerings Terms" in the General Service Terms section of the Service Specific Terms. Pre-GA features are available "as is" and might have limited support. For more information, see the launch stage descriptions.

When metrics are ingested into Cloud Monitoring by using an OpenTelemetry Collector and the otlphttp exporter or sent directly by using an OpenTelemetry SDK, the OTLP metrics are mapped to Cloud Monitoring metric structures. This section describes the following:

• The mapping between OTLP resources and Cloud Monitoring monitored resources.
• The mapping between OTLP metrics and Cloud Monitoring metrics.

Monitored resource mapping

All metric points are written as they are for Google Cloud Managed Service for Prometheus, by using the Prometheus mapping.

Prometheus mapping

Prometheus metrics require the use of the monitored-resource type prometheus_target.

The following labels on the prometheus_target resource type are used to schematize and efficiently store data within Monarch. The more precisely you specify values for these attributes, the better your queryability and scalability will be.

We strongly recommend being as explicit as possible when setting values for these labels, although we have implemented fallback logic for use in the absence of explicit values.

The following table shows the sources of values for labels, in order of priority:

prometheus-target label	Value used (in priority order)
location (required)	location attribute cloud.availability_zone attribute cloud.region attribute
cluster	cluster attribute k8s.cluster.name attribute __gce__, if cloud.platform attribute is gcp_compute_engine __run__, if cloud.platform attribute is gcp_cloud_run Empty string
namespace	namespace attribute k8s.namespace.name attribute service.namespace attribute Empty string
job	job attribute "service.namespace" + "/" + service.namespace attribute service.name attribute service.name attribute, if not unknown_service:foo faas.name attribute k8s.deployment.name attribute k8s.statefulset.name attribute k8s.job.name attribute k8s.cronjob.name attribute service.name attribute, if unknown_service:foo Empty string
instance Reject the point if empty	instance attribute service.instance.id attribute faas.instance attribute k8s.pod.name:k8s.container.name attribute k8s.pod.name attribute, if no container name present host.id attribute

Metric mapping

Metrics are converted into the Prometheus time-series format. Metric names must have either no domain or the domain prometheus.googleapis.com. After conversion, the metric name will include the prometheus.googleapis.com prefix and an additional suffix, based on the OTLP point kind. The resulting Cloud Monitoring metric has the following structure:

prometheus.googleapis.com/{metric_name}/{suffix}

Additionally, for each unique OpenTelemetry resource, the conversion adds a target_info metric that contains all resource attributes except service.name, service.instance.id, and service.namespace.

All OTLP INT64 metrics are translated to the DOUBLE value type in Cloud Monitoring, even if the collector specifies the value type as INT64. This change is made because once a time series is in Monarch, you can't change the value type. The most common consequence of supporting INT64 values is that you end up with collisions that only can be resolved by deleting a metric.

Note: We expect there to be possible value-type collisions, specifically if you have previously sent INT64 target_info metrics. If you encounter collisions, the best long-term solution for this is for you to delete the metric descriptor for the INT64 target_info metrics. To delete this descriptor from your metric scope, use this Golang script.

Prometheus metric mapping

Metric types are mapped as follows:

• OTLP Gauge maps to Cloud Monitoring gauge.
• OTLP Sum maps as follows:
  • To Cloud Monitoring gauge when is_monotonic is set to false.
  • To Cloud Monitoring cumulative when aggregation_temporality is set to AGGREGATION_TEMPORALITY_CUMULATIVE.
  • To Cloud Monitoring delta when aggregation_temporality is set to AGGREGATION_TEMPORALITY_DELTA.
• OTLP Histogram maps to Cloud Monitoring distribution with a metric kind of cumulative or delta, depending on the value of aggregation_temporality.
• OTLP Summary metrics are expanded into individual time series for each component: count, sum, and each quantile.
  • The names for count and sum metrics are suffixed with _count or _sum respectively, and are written as Cloud Monitoring cumulative metrics of type DOUBLE.
  • Each quantile becomes is its own gauge time series of type DOUBLE, with a quantile label.

The following table summarizes the metric mapping:

OTLP point kind	Monitoring metric kind	Monitoring value type	Suffix	Notes
GAUGE	GAUGE	DOUBLE	/gauge
GAUGE (metric.metadata["prometheus.type"]="unknown")	GAUGE	DOUBLE	/unknown	Prometheus Unknowns are split into a counter and a gauge by the OpenTelemetry Collector.
SUM (monotonic, CUMULATIVE)	CUMULATIVE	DOUBLE	/counter
SUM (monotonic, CUMULATIVE, metric.metadata["prometheus.type"]="unknown")	CUMULATIVE	DOUBLE	/unknown:counter	Prometheus Unknowns are split into a counter and a gauge by the OpenTelemetry Collector.
SUM (monotonic, DELTA)	DELTA	DOUBLE	/delta
SUM (non-monotonic, CUMULATIVE)	GAUGE	DOUBLE	/gauge
HISTOGRAM (CUMULATIVE)	CUMULATIVE	DISTRIBUTION with explicit buckets	/histogram
EXPONENTIAL HISTOGRAM (CUMULATIVE)	CUMULATIVE	DISTRIBUTION with exponential buckets	/histogram
HISTOGRAM (DELTA)	DELTA	DISTRIBUTION with explicit buckets	/histogram:delta
EXPONENTIAL HISTOGRAM (DELTA)	DELTA	DISTRIBUTION with exponential buckets	/histogram:delta
SUMMARY (sum, count, quantile)	CUMULATIVE CUMULATIVE GAUGE	DOUBLE DOUBLE DOUBLE	_sum/summary:counter _count/summary /summary	Summary data points are written as multiple time series, one for count, sum, and each calculated quantile. Quantile metrics are also generated with a quantile label.

Differences between the googlemanagedprometheus exporter and the Telemetry API

The Telemetry API (telemetry.googleapis.com) handles metrics differently from the googlemanagedprometheus exporter:

• The Telemetry API allows the period (.) and slash (/) characters with in metric names. The googlemanagedprometheus exporter converts all instances of these characters to the underscore (_) character. For example, an OTLP metric called prometheus.googleapis.com/foo.bar/gauge is exported verbatim by the OTLP exporter but is exported as prometheus.googleapis.com/foo_bar/gauge by the googlemanagedprometheus exporter.

  When the metrics are ingested, Cloud Monitoring creates metric descriptors based on the names. The difference in how the period (.) and slash (/) characters are handled by the ingestion pathways means that resulting metric descriptors differ between metrics ingested by using the googlemanagedprometheus exporter and those ingested by using the otlphttp exporter. If you use both ingestion paths, then you have two sets of metrics; to get complete results when querying, you have to manually union results from the Prometheus and OTLP versions of the metrics.

• The Telemetry API doesn't append a unit to a metric name when a unit is present, and it doesn't append a _total suffix to counters. So a metric exported as prometheus.googleapis.com/foo/counter when using the Telemetry API is exported as prometheus.googleapis.com/foo_seconds_total/counter by the googlemanagedprometheus exporter. This difference also applies to the _total and _ratio suffixes.

• The API synthesizes the sum_of_squared_deviation value for distribution values derived from exponential histograms. The googlemanagedprometheus exporter does not set this field for exponential histograms.

• The API converts all integer point values to double values for Prometheus metrics.

• The API doesn't set the scope_version or scope_name labels if those labels have empty values.

Where to view ingested data

Trace data that is ingested through the Telemetry API can be viewed using the Trace Explorer page. For information about how to view your trace data, see the following:

• Find and explore traces.
• Query and analyze traces.

Metric data that is ingested through the Telemetry API can be viewed using the Metrics Explorer page. For information about viewing and charting your metric data, see Create charts with Metrics Explorer.

VPC Service Controls support

The Telemetry API service, whose service name is telemetry.googleapis.com, is a VPC Service Controls-supported service. Any VPC Service Controls restrictions that you create for the Telemetry API service apply only to that service. Those restrictions don't apply to any other services, including those like the cloudtrace.googleapis.com service, which can also ingest trace data.

For more information, see the following:

• VPC Service Controls documentation.
• Supported products and limitations.