Migrate to the OTLP exporter

This feature is subject to the "Pre-GA Offerings Terms" in the General Service Terms section of the Service Specific Terms. Pre-GA features are available "as is" and might have limited support. For more information, see the launch stage descriptions.

This document describes how to modify a OpenTelemetry Collector configuration that uses an existing exporter — in this case, the googlemanagedprometheus exporter — to use the otlphttp exporter and the Telemetry (OTLP) API, telemetry.googleapis.com.

OTLP for Prometheus metrics only works when using the OpenTelemetry Collector version 0.140.0 or newer.

Enable the Telemetry API

The otlphttp exporter writes to the Telemetry API, so that API must be enabled in your project. Enable the Telemetry API by running the following command:

gcloud services enable telemetry.googleapis.com

Authorize the Kubernetes service account

The Kubernetes service account must have permission to use the Telemetry API. The following commands grant the necessary Identity and Access Management (IAM) role to the Kubernetes service account. These commands assume you are using Workload Identity Federation for GKE:

export PROJECT_NUMBER=$(gcloud projects describe PROJECT_ID --format="value(projectNumber)")

gcloud projects add-iam-policy-binding projects/PROJECT_ID \
  --role=roles/telemetry.metricsWriter \
  --member=principal://iam.googleapis.com/projects/$PROJECT_NUMBER/locations/global/workloadIdentityPools/PROJECT_ID.svc.id.goog.goog/subject/ns/opentelemetry/sa/opentelemetry-collector \
  --condition=None

Replace the PROJECT_ID variable with the ID of your Google Cloud project.

If your service account has a different format, then you can use the command in the Google Cloud Managed Service for Prometheus documentation to authorize the service account, with the following changes:

• Replace the service-account name gmp-test-sa with your service account.
• Run the command to grant the roles/telemetry.metricsWriter role.

Set the PROJECT_ID environment variable

Set the PROJECT_ID environment variable in your collector deployment. The result looks like the following:

env:
- name: PROJECT_ID
  value: PROJECT_ID

Locate the exporter to be replaced

In your configuration file, locate the exporter and the service that uses the exporter:

exporters:
  googlemanagedprometheus:
service:
  pipelines:
    metrics:
      exporters: [googlemanagedprometheus]

Add configuration for the otlphttp expporter

In this step, you add the otlphttp exporter and the metricstarttime processor to the configuration. This additional exporter causes the collector to double-write the metrics.

• If you are using OTLP-native metrics, then double-writing causes no problem because the structure of the metric descriptors created by the two methods differs. They are separate metrics.
• If you are using Prometheus metrics, then you will see collisions until you remove the googlemanagedpromethus exporter.

The updated config looks like the following:

exporters:
  googlemanagedprometheus:
  otlphttp:
    encoding: json
    endpoint: https://telemetry.googleapis.com
    auth:
      authenticator: googleclientauth
processors:
  # This processor ensures the start time is set for Prometheus metrics. This must be set in the pipeline before the k8sattributes processor, if used.
  metricstarttime:
    strategy: subtract_initial_point
  resource/gcp_project_id:
    attributes:
    - action: insert
      # Make sure you set the PROJECT_ID environment variable.
      value: ${PROJECT_ID}
      key: gcp.project_id
extensions:
  googleclientauth:
service:
  extensions: [googleclientauth]
  pipelines:
    metrics:
      processors: [resource/gcp_project_id, metricstarttime]
      exporters: [googlemanagedprometheus, otlphttp]

After double-writing your metrics, update your dashboards and alerting policies to use the new metric names as generated by the otlphttp exporter. The otlphttp exporter doesn't rename metrics to be "Prometheus-style", using underscore replacements.

Note: Because the Telemetry API sets the value type of all metrics to DOUBLE, we expect there to be possible value-type collisions, specifically if you have previously sent INT64 target_info metrics. If you encounter collisions, the best long-term solution for this is for you to delete the metric descriptor for the INT64 target_info metrics. To delete this descriptor from your metric scope, use this Golang script.

Remove configuration for the previous exporter

After updating your dashboards and alerts to refer to the otlphttp-exported metrics, remove the old exporter, googlemanagedprometheus in this case, from the configuration. The revised configuration looks like the following:

exporters:
  otlphttp:
    encoding: json
    endpoint: https://telemetry.googleapis.com
    auth:
      authenticator: googleclientauth
processors:
  # This processor ensures the start time is set for Prometheus metrics. This must be set in the pipeline before the k8sattributes processor, if used.
  metricstarttime:
    strategy: subtract_initial_point
  resource/gcp_project_id:
    attributes:
    - action: insert
      # Make sure you set the PROJECT_ID environment variable.
      value: ${PROJECT_ID}
      key: gcp.project_id
extensions:
  googleclientauth:
service:
  extensions: [googleclientauth]
  pipelines:
    metrics:
      processors: [resource/gcp_project_id, metricstarttime]
      exporters: [otlphttp]

Migrate dashboards and alerting policies, if necessary

If you used the googlemanagedprometheus exporter only for Prometheus metric data, then when you switch over to the OTLP exporter, your existing dashboards and alert policies will continue to work.

However, if you used the googlemanagedprometheus exporter to collect and send data that was originally emitted using OTLP, or if you scraped Prometheus metrics with UTF-8 characters, then you need to update your dashboards and alerts to use the UTF-8 metric name.

This is because of the following differences between metrics exported by using OTLP and metrics exported by using the googlemanagedprometheus exporter:

• The Telemetry API allows the period (.) and slash (/) characters with in metric names. The googlemanagedprometheus exporter converts all instances of these characters to the underscore (_) character. For example, an OTLP metric called prometheus.googleapis.com/foo.bar/gauge is exported verbatim by the OTLP exporter but is exported as prometheus.googleapis.com/foo_bar/gauge by the googlemanagedprometheus exporter.

  When the metrics are ingested, Cloud Monitoring creates metric descriptors based on the names. The difference in how the period (.) and slash (/) characters are handled by the ingestion pathways means that resulting metric descriptors differ between metrics ingested by using the googlemanagedprometheus exporter and those ingested by using the otlphttp exporter. If you use both ingestion paths, then you have two sets of metrics; to get complete results when querying, you have to manually union results from the Prometheus and OTLP versions of the metrics.

• The Telemetry API doesn't append a unit to a metric name when a unit is present, and it doesn't append a _total suffix to counters. So a metric exported as prometheus.googleapis.com/foo/counter when using the Telemetry API is exported as prometheus.googleapis.com/foo_seconds_total/counter by the googlemanagedprometheus exporter. This difference also applies to the _total and _ratio suffixes.

For more information about differences between metrics, see Differences between the googlemanagedprometheus exporter and the Telemetry API.

Because the transformation rules don't apply to metrics with UTF-8 characters, you need to rewrite your dashboards and alert policies either to use the new metric names or to union the old and new metric names.

We don't recommend writing processor rules to recreate these transformations to continue writing UTF-8 metrics as if they were collected by the googlemanagedprometheus exporter. Doing so retains backward compatibility, but it sacrifices forward compatibility, and you won't be able to easily use open-source assets that reference the UTF-8 metric names.