OTLP metric ingestion overview

This document introduces the use of the OpenTelemetry Collector with the otlphttp exporter and the Telemetry (OTLP) API, telemetry.googleapis.com, which implements the OpenTelemetry OTLP Protocol. The combination of the otlphttp exporter and Telemetry API lets you ingest OTLP metrics to Cloud Monitoring.

The OTLP endpoint supports all of the OTLP protocols, including http/proto, http/json, and grpc. For more information, see Protocol support.

You can also send metrics directly to the Telemetry API from applications that use SDKs. For more information, see Use SDKs to send metrics from applications.

If you are using Google Kubernetes Engine, you can follow Managed OpenTelemetry for GKE, instead of manually deploying and configuring an OpenTelemetry Collector that uses the Telemetry API.

OTLP for Prometheus metrics only works when using the OpenTelemetry Collector version 0.140.0 or newer.

Metric and label names in the Telemetry API

This section describes naming conventions and requirements for metrics ingested into Cloud Monitoring by using the Telemetry API:

  • Metric names and label keys don't support full UTF-8.

    • Metric names that don't conform to the regular expression [a-zA-Z][a-zA-Z0-9_:./]* will be rejected.
    • DataPoints containing attributes (that is, label keys) that don't conform to the regular expression [a-zA-Z_][a-zA-Z0-9_.]* will be rejected.

    To prevent rejection of your metrics for these reasons, you can set up a processor to transform your metric names and attributes for you by using the replace_pattern function.

  • Important: Querying metric names and label keys with special characters other than the colon (:) and the underscore (_) requires you to wrap them in braces ({}) and quotes ("), according to PromQL's UTF-8 spec. For example, the following are valid queries:

    • {"my.metric.name"}
    • {"my.metric.name", "label.key.KEY"="value"}

Telemetry API metrics and Cloud Monitoring

This section describes how metrics ingested by using the Telemetry API and the otlphttp exporter interact with Cloud Monitoring:

  • There might be possible value-type collisions between INT64- and DOUBLE-type metrics, specifically if you have previously sent the target_info metric. If you experience this type of conflict, then delete the INT64 metric descriptors. You can use this Golang script to delete all INT64-type target_info metrics from your metrics scope.

  • Retaining the le label when querying Exponential histograms might return unexpected results. The more typical histogram_quantile(.99, sum by (le) (metric)) queries are expected to work.

  • Delta metrics might not query properly in certain circumstances, such as very sparse deltas.

Protocol support

The OTLP endpoint supports all of the OTLP protocols, including http/proto, http/json, and grpc. When exporting from an OpenTelemetry Collector, you can use any of the protocols. However, due to the lack of support for dynamic token refreshing in most SDK exporters, we recommend using only the gRPC OTLP exporter, not the HTTP exporters, when exporting directly from SDKs. For more information, see Use SDKs to send metrics from applications.

Billing

Billing for OTLP metrics is accounted for under the "Prometheus Samples Ingested" SKU, the same one used for metrics from Google Cloud Managed Service for Prometheus. For more information, see the Billing page.

Limits and quotas

Ingestion of metrics by using the Telemetry API is subject to the Telemetry API metric limits.

Additionally, all standard Cloud Monitoring and Google Cloud Managed Service for Prometheus metric quotas and limits apply. For example, metrics can't have more than 200 labels.

The default quota for metrics ingested by the Telemetry API is 60,000 requests per minute. At a maximum batch size of 200 points per request, this is an effective default quota of 200,000 samples per second. You can request a quota increase.

What's next