Enable telemetry signals in Go client libraries

Google Cloud provides powerful monitoring, logging, and diagnostics for Go applications.

The Go client libraries are instrumented to emit tracing, metrics, and logging data. The instrumentation is opt-in; you need to explicitly enable it. This document describes the available signals and how to enable them.

Available signals

The signals include the following telemetry data, adhering to OpenTelemetry Semantic Conventions:

• Traces: Low-level HTTP/gRPC traces representing the network requests made by the client libraries.
• Metrics: Client request metrics, tracking latency and request rates. The primary metric is gcp.client.request.duration.
• Logs: Actionable error logs at the DEBUG level (and above), providing details for failed requests at the transport layer, even if they are eventually retried successfully.

The signals include standard OpenTelemetry attributes (for example, http.response.status_code and rpc.system.name) and Google Cloud-specific custom attributes, which may include these and similar attributes:

• gcp.client.service: The service name (for example, pubsub or storage).
• gcp.client.repo: The client library repository (for example, googleapis/google-cloud-go).
• gcp.client.version: The client library version.
• gcp.client.artifact: The specific module path (for example, cloud.google.com/go/secretmanager).
• gcp.resource.destination.id: The ID of the resource being acted upon.
• gcp.errors.domain: The error domain for actionable error logs.
• gcp.errors.metadata.<key>: Additional error metadata keys for failed requests (flattened).

For a full list of standard attributes, see the OpenTelemetry HTTP and gRPC semantic conventions.

Enabling telemetry

To protect sensitive data, Golden Signals are disabled by default. You must explicitly opt in to enable them.

Traces, logs, or metrics may contain sensitive data such as resource names, full URLs, and error messages.

Before configuring subscribers or exporters for traces, logs, or metrics, review the contents of the spans and logs and consult the OpenTelemetry and slog documentation to set up filters and formatters to prevent leaking sensitive information, depending on your intended use case.

In Go, use the following environment variables to turn on traces, metrics, and logs globally across your Google Cloud client libraries:

# Enable trace generation (span emission)
export GOOGLE_SDK_GO_TRACING=true

# Enable metrics
export GOOGLE_SDK_GO_METRICS=true

# Enable logging
export GOOGLE_SDK_GO_LOGGING=true

Trace context propagation

The Go client libraries automatically propagate active trace contexts to Google Cloud services, even if trace generation (GOOGLE_SDK_GO_TRACING) is disabled.

If your application has an active OpenTelemetry span in the context.Context passed to a client library method, the library uses it to provide a tracing context for the outgoing requests. This ensures that your application-level traces can be correlated with backend service logs and behaviors without requiring you to emit client-side spans.

Set up a global OpenTelemetry text map propagator in your application to provide a tracing context for the client libraries.

Exporting telemetry

Once telemetry is enabled in the client libraries, your application must be configured to collect and export this data to your observability backend.

Tracing and metrics

To export the traces and metrics generated by the Go client libraries, initialize the OpenTelemetry SDK with your preferred exporter (for example, OTLP) and set up the global text map propagator in your main.go:

package main

import (
    "context"
    "log"

    "go.opentelemetry.io/otel"
    "go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc"
    "go.opentelemetry.io/otel/propagation"
    "go.opentelemetry.io/otel/sdk/trace"
)

func main() {
    ctx := context.Background()

    // Initialize the exporter
    exporter, err := otlptracegrpc.New(ctx)
    if err != nil {
        log.Fatalf("failed to initialize exporter: %v", err)
    }

    // Set up the tracer provider
    tp := trace.NewTracerProvider(trace.WithBatcher(exporter))
    defer func() {
        // Ensure all spans are flushed before exit
        if err := tp.Shutdown(ctx); err != nil {
            log.Fatalf("failed to shutdown TracerProvider: %v", err)
        }
    }()

    otel.SetTracerProvider(tp)

    // Set up the global propagator
    otel.SetTextMapPropagator(propagation.TraceContext{})

    // ... initialize Google Cloud client libraries ...
}

For more details on connecting the OpenTelemetry SDK to Cloud Monitoring or Cloud Trace, see the Google Cloud Observability guides for Go.

Logging

The Go client libraries use structured logging using slog to emit actionable errors at the DEBUG level. The exported logs automatically include trace IDs and span IDs if your logging framework is configured to extract them from the context.Context.

To receive these logs, you must provide a configured *slog.Logger to the client upon initialization (for example, by using option.WithLogger()).

To route these structured logs to Cloud Logging, configure slog to write JSON to standard output (stdout). If you are deploying to an environment like Google Kubernetes Engine or Cloud Run, the built-in agents automatically scrape these logs.

package main

import (
    "context"
    "log/slog"
    "os"

    "cloud.google.com/go/secretmanager/apiv1"
    "google.golang.org/api/option"
)

func main() {
    ctx := context.Background()

    // Configure slog to output JSON to stdout at the DEBUG level
    opts := &slog.HandlerOptions{Level: slog.LevelDebug}
    logger := slog.New(slog.NewJSONHandler(os.Stdout, opts))

    // Provide the logger to the client
    client, err := secretmanager.NewClient(ctx, option.WithLogger(logger))
    if err != nil {
        // handle error
    }
    defer client.Close()
}

For further instructions on mapping OpenTelemetry trace contexts and formatting structured JSON logs to align with Cloud Logging's expected payload fields, see Configure structured logging for Go