Choose an instrumentation approach

You can choose the most effective instrumentation approach for your application based on your Google Cloud compute platform and telemetry requirements. This guide provides recommendations for Google Kubernetes Engine (GKE), Compute Engine, and Cloud Run, helping you decide between the Google-Built OpenTelemetry Collector, the Ops Agent, and direct in-process OpenTelemetry export.

This document provides general recommendations, as well as guidance for the following platforms:

For sample applications, see Samples and implementation guides.

The recommendations on this page aren't the only solutions, and other approaches can work. For additional guidance, contact Cloud Customer Care.

General recommendations

This section contains general recommendations about how to instrument your application. For platform-specific guidance, see Platform recommendations.

  • Log data: We recommend that you use a framework that can be configured to output JSON-structured logs for Cloud Logging. For writing log data, we recommend the following:

  • Metric data: We recommend that you use OpenTelemetry or Prometheus client libraries:

    • OpenTelemetry provides a vendor-neutral, open-source framework and client libraries to instrument applications for metrics.
    • Prometheus client libraries (Go, Java, Python, etc.) generate metrics that can be exposed at an HTTP endpoint and scraped by an agent.
  • Trace data: We recommend that you use OpenTelemetry to generate distributed trace data from your application code.

OpenTelemetry collection architectures

When you instrument an application with OpenTelemetry libraries, you can route telemetry through the standalone OpenTelemetry Collector or export directly from your application process. The OpenTelemetry Collector configuration file controls how the collector receives, processes, and exports telemetry. In general, we recommend that collectors receive and export telemetry by using the OpenTelemetry Protocol (OTLP). The Telemetry (OTLP) API implements this protocol.

There are two fundamental approaches for sending telemetry to your Google Cloud project:

  • Collector-based export (vendor-neutral): You instrument your application code with the OpenTelemetry SDK and use the in-process OTLP exporter to send data to an OpenTelemetry collector, such as the Google-Built OpenTelemetry Collector or an Ops Agent receiver. The collector receives the telemetry and forwards it to your Google Cloud project. Your application code remains completely vendor-neutral.

  • Direct export (vendor-specific): You instrument your application code with the OpenTelemetry SDK and configure OpenTelemetry to send telemetry to your project by using the Telemetry API. You don't need to deploy a collector, but your application code includes vendor-specific dependencies.

We recommend that you use an OpenTelemetry collector when your compute environment supports one. For environments where running a separate collector process isn't practical, use direct in-process export.

Platform recommendations

Because Google Cloud compute environments have different lifecycle and sidecar capabilities, the optimal collection pattern varies by platform. The following sections provide specific recommendations for your deployment environment.

GKE

For general information about GKE, see GKE overview.

Type Recommendation
Metrics

We recommend that you use Google Cloud Managed Service for Prometheus.

For instrumentation, do one of the following:

Traces

Do the following:

  1. Deploy the Google-Built OpenTelemetry Collector on Google Kubernetes Engine.

    The collector receives trace data from the SDK's in-process OTLP exporter, processes that data, and then sends the processed data to your Google Cloud project by using the Telemetry (OTLP) API.

  2. Use the OpenTelemetry SDK and the OTLP exporter for your language.
Logs

Configure your app to output JSON-structured logs to stdout and stderr. For a list of frameworks, see Recommended logging frameworks.

GKE collects logs written to stdout and stderr automatically. For more information, see About GKE logs.

Compute Engine

For general information about Compute Engine, see Virtual machine instances.

Type Recommendation
Metrics and Traces

Do the following:

  1. Use the Ops Agent to collect metrics and traces.

    For an example, see Collect OpenTelemetry Protocol (OTLP) metrics and traces. This guide describes how to configure the Ops Agent to receive metric and trace data from the SDK's in-process OTLP exporters, transform that data, and then send the data to your Google Cloud project. Metric data is sent by using the Cloud Monitoring API and trace data is sent by using the Cloud Trace API.

  2. Use the OpenTelemetry SDK and the OTLP exporter for your language.

Alternatively, if you only want to configure collection for Prometheus-format metrics, then use the Ops Agent Prometheus receiver to collect metrics instrumented by using Prometheus client libraries or the OpenTelemetry SDK.

Logs

Do the following:

  1. Configure your app to output JSON-structured logs to a file. For a list of frameworks, see Recommended logging frameworks.
  2. Install the Ops Agent and configure a receiver. For an example, see Logging receivers.

Cloud Run

For general information about Cloud Run, see What is Cloud Run.

Type Recommendation
Metrics and Traces

Do the following:

  1. Use the OpenTelemetry SDK and the OTLP exporter for your language.
  2. Deploy an OpenTelemetry sidecar to collect metrics and traces.

    For examples, see the following documents:

  3. For your Cloud Run service, use instance-based billing. With instance-based billing, the CPU is allocated for the entire instance lifecycle, which is necessary because OpenTelemetry instrumentation does background processing. For more information, see Billing settings for Cloud Run services.

Alternatively, if you only want to configure collection for Prometheus-format metrics, then use the Prometheus sidecar for Cloud Run to collect metrics instrumented by using Prometheus client libraries or the OpenTelemetry SDK.

Logs

Configure your app to output JSON-structured logs to stdout and stderr. For a list of frameworks, see Recommended logging frameworks.

Cloud Run collects logs written to stdout and stderr automatically. For more information, see Write container logs.

Cloud Run functions

For general information about Cloud Run functions, see Cloud Run functions overview.

Type Recommendation
Metrics Writing metric data directly isn't supported in Cloud Run functions. You can generate log-based metrics.
Traces Use the OpenTelemetry SDK and the Cloud Trace exporter for your language.
Logs

Configure your app to output JSON-structured logs to stdout and stderr. For a list of frameworks, see Recommended logging frameworks.

Cloud Run functions collects logs written to stdout and stderr automatically. For more information, see Cloud Run functions: Monitoring and logging overview.

App Engine

For general information about App Engine, see An overview of App Engine.

Type Recommendation
Metrics Use the OpenTelemetry SDK and the Cloud Monitoring exporter for your language.
Traces Use the OpenTelemetry SDK and the Cloud Trace exporter for your language.
Logs

Configure your app to output JSON-structured logs to stdout and stderr. For a list of frameworks, see Recommended logging frameworks.

App Engine collects logs written to stdout and stderr automatically. For more information, see Write and view logs.

Samples and implementation guides

OpenTelemetry documentation

This section provides links to the OpenTelemetry SDK and the exporters for OTLP, Cloud Trace, and Cloud Monitoring.

General references:

Go

Java

JavaScript

Python