v1.metrics overview

This document describes how metric data that is sent your Google Cloud project by using the Telemetry (OTLP) API is mapped to the Cloud Monitoring structures. This API implements the OpenTelemetry OTLP Protocol. You can send data to this API when you instrument your applications with an otlphttp exporter and an OpenTelemetry Collector, or when you use the OpenTelemetry SDKs.

OpenTelemetry is a Google Cloud-supported open source project with Google Cloud engineers staffed to ensure support for ingesting and visualizing your telemetry.

OTLP metrics in Cloud Monitoring

This feature is subject to the "Pre-GA Offerings Terms" in the General Service Terms section of the Service Specific Terms. Pre-GA features are available "as is" and might have limited support. For more information, see the launch stage descriptions.

When metrics are ingested into Cloud Monitoring by using an OpenTelemetry Collector and the otlphttp exporter or sent directly by using an OpenTelemetry SDK, the OTLP metrics are mapped to Cloud Monitoring metric structures. This section describes the following:

• The mapping between OTLP resources and Cloud Monitoring monitored resources.
• The mapping between OTLP metrics and Cloud Monitoring metrics.

Monitored resource mapping

All metric points are written as they are for Google Cloud Managed Service for Prometheus, by using the Prometheus mapping.

Prometheus mapping

Prometheus metrics require the use of the monitored-resource type prometheus_target.

The following labels on the prometheus_target resource type are used to schematize and efficiently store data within Monarch. The more precisely you specify values for these attributes, the better your queryability and scalability will be.

We strongly recommend being as explicit as possible when setting values for these labels, although we have implemented fallback logic for use in the absence of explicit values.

The following table shows the sources of values for labels, in order of priority:

prometheus-target label	Value used (in priority order)
location (required)	location attribute cloud.availability_zone attribute cloud.region attribute
cluster	cluster attribute k8s.cluster.name attribute __gce__, if cloud.platform attribute is gcp_compute_engine __run__, if cloud.platform attribute is gcp_cloud_run Empty string
namespace	namespace attribute k8s.namespace.name attribute service.namespace attribute Empty string
job	job attribute "service.namespace" + "/" + service.namespace attribute service.name attribute service.name attribute, if not unknown_service:foo faas.name attribute k8s.deployment.name attribute k8s.statefulset.name attribute k8s.job.name attribute k8s.cronjob.name attribute service.name attribute, if unknown_service:foo Empty string
instance Reject the point if empty	instance attribute service.instance.id attribute faas.instance attribute k8s.pod.name:k8s.container.name attribute k8s.pod.name attribute, if no container name present host.id attribute

Metric mapping

Metrics are converted into the Prometheus time-series format. Metric names must have either no domain or the domain prometheus.googleapis.com. After conversion, the metric name will include the prometheus.googleapis.com prefix and an additional suffix, based on the OTLP point kind. The resulting Cloud Monitoring metric has the following structure:

prometheus.googleapis.com/{metric_name}/{suffix}

Additionally, for each unique OpenTelemetry resource, the conversion adds a target_info metric that contains all resource attributes except service.name, service.instance.id, and service.namespace.

All OTLP INT64 metrics are translated to the DOUBLE value type in Cloud Monitoring, even if the collector specifies the value type as INT64. This change is made because once a time series is in Monarch, you can't change the value type. The most common consequence of supporting INT64 values is that you end up with collisions that only can be resolved by deleting a metric.

Note: We expect there to be possible value-type collisions, specifically if you have previously sent INT64 target_info metrics. If you encounter collisions, the best long-term solution for this is for you to delete the metric descriptor for the INT64 target_info metrics. To delete this descriptor from your metric scope, use this Golang script.

Prometheus metric mapping

Metric types are mapped as follows:

• OTLP Gauge maps to Cloud Monitoring gauge.
• OTLP Sum maps as follows:
  • To Cloud Monitoring gauge when is_monotonic is set to false.
  • To Cloud Monitoring cumulative when aggregation_temporality is set to AGGREGATION_TEMPORALITY_CUMULATIVE.
  • To Cloud Monitoring delta when aggregation_temporality is set to AGGREGATION_TEMPORALITY_DELTA.
• OTLP Histogram maps to Cloud Monitoring distribution with a metric kind of cumulative or delta, depending on the value of aggregation_temporality.
• OTLP Summary metrics are expanded into individual time series for each component: count, sum, and each quantile.
  • The names for count and sum metrics are suffixed with _count or _sum respectively, and are written as Cloud Monitoring cumulative metrics of type DOUBLE.
  • Each quantile becomes is its own gauge time series of type DOUBLE, with a quantile label.

The following table summarizes the metric mapping:

OTLP point kind	Monitoring metric kind	Monitoring value type	Suffix	Notes
GAUGE	GAUGE	DOUBLE	/gauge
GAUGE (metric.metadata["prometheus.type"]="unknown")	GAUGE	DOUBLE	/unknown	Prometheus Unknowns are split into a counter and a gauge by the OpenTelemetry Collector.
SUM (monotonic, CUMULATIVE)	CUMULATIVE	DOUBLE	/counter
SUM (monotonic, CUMULATIVE, metric.metadata["prometheus.type"]="unknown")	CUMULATIVE	DOUBLE	/unknown:counter	Prometheus Unknowns are split into a counter and a gauge by the OpenTelemetry Collector.
SUM (monotonic, DELTA)	DELTA	DOUBLE	/delta
SUM (non-monotonic, CUMULATIVE)	GAUGE	DOUBLE	/gauge
HISTOGRAM (CUMULATIVE)	CUMULATIVE	DISTRIBUTION with explicit buckets	/histogram
EXPONENTIAL HISTOGRAM (CUMULATIVE)	CUMULATIVE	DISTRIBUTION with exponential buckets	/histogram
HISTOGRAM (DELTA)	DELTA	DISTRIBUTION with explicit buckets	/histogram:delta
EXPONENTIAL HISTOGRAM (DELTA)	DELTA	DISTRIBUTION with exponential buckets	/histogram:delta
SUMMARY (sum, count, quantile)	CUMULATIVE CUMULATIVE GAUGE	DOUBLE DOUBLE DOUBLE	_sum/summary:counter _count/summary /summary	Summary data points are written as multiple time series, one for count, sum, and each calculated quantile. Quantile metrics are also generated with a quantile label.

Differences between the googlemanagedprometheus exporter and the Telemetry API

The Telemetry API (telemetry.googleapis.com) handles metrics differently from the googlemanagedprometheus exporter:

• The Telemetry API allows the period (.) and slash (/) characters with in metric names. The googlemanagedprometheus exporter converts all instances of these characters to the underscore (_) character. For example, an OTLP metric called prometheus.googleapis.com/foo.bar/gauge is exported verbatim by the OTLP exporter but is exported as prometheus.googleapis.com/foo_bar/gauge by the googlemanagedprometheus exporter.

  When the metrics are ingested, Cloud Monitoring creates metric descriptors based on the names. The difference in how the period (.) and slash (/) characters are handled by the ingestion pathways means that resulting metric descriptors differ between metrics ingested by using the googlemanagedprometheus exporter and those ingested by using the otlphttp exporter. If you use both ingestion paths, then you have two sets of metrics; to get complete results when querying, you have to manually union results from the Prometheus and OTLP versions of the metrics.

• The Telemetry API doesn't append a unit to a metric name when a unit is present, and it doesn't append a _total suffix to counters. So a metric exported as prometheus.googleapis.com/foo/counter when using the Telemetry API is exported as prometheus.googleapis.com/foo_seconds_total/counter by the googlemanagedprometheus exporter. This difference also applies to the _total and _ratio suffixes.

• The API synthesizes the sum_of_squared_deviation value for distribution values derived from exponential histograms. The googlemanagedprometheus exporter does not set this field for exponential histograms.

• The API converts all integer point values to double values for Prometheus metrics.

• The API doesn't set the scope_version or scope_name labels if those labels have empty values.

Where to view ingested data

Metric data that is ingested through the Telemetry API can be viewed using the Metrics Explorer page. For information about viewing and charting your metric data, see Create charts with Metrics Explorer.