Cloud Trace overview

Cloud Trace is a distributed tracing system for Google Cloud that tracks request latency and helps you troubleshoot performance bottlenecks across services and generative AI applications. By collecting latency data from Google Cloud services and instrumented applications, Trace helps you understand how requests are processed in a microservices architecture and identify relevant logs.

Trace helps you answer questions like the following:

  • How long does your application take to handle a given request?
  • Why does a request take a long time to complete?
  • Why do some requests take longer than others?
  • What is the overall latency of requests to your application?
  • Has application latency increased or decreased over time?
  • How can you reduce application latency?
  • What are your application's dependencies?

For information about how to use traces and logs together for root-cause analysis, see the blog post Troubleshooting distributed applications: Using traces and logs together for root-cause analysis.

For information about profiling your application, see Cloud Profiler.

Environment support

Trace runs on Linux in the following environments:

Components

Trace consists of a tracing client, which collects traces and sends them to your Google Cloud project. You can then use the Google Cloud console to view and analyze the data collected by the tracing client. For information about the data model, see Traces and spans.

Tracing client

A tracing client collects latency and span data from your application and exports it to your Google Cloud project. Depending on your environment and requirements, you can collect trace data automatically or manually by instrumenting your application code.

Tracing interface

To view and analyze your span data, you can use the Trace Explorer and Observability Analytics pages in the Google Cloud console:

  • Trace Explorer: Displays aggregate information about your trace data and lets you examine individual traces in detail. The aggregated latency data is shown on a heatmap, which you can explore with your pointer. To restrict which data is displayed, you can add filters. You can also view and explore individual spans and traces:

  • Observability Analytics: Provides a SQL query interface. Your queries can join your trace and log data, and you can view the query results either as a table or a chart. You can use BigQuery to analyze your trace data if you create a linked BigQuery dataset. For more information, see Query and analyze traces.

Configurations with automatic tracing

The following configurations automatically capture trace data:

  • App Engine standard environment

    • First-generation runtimes automatically intercept and send trace spans to Cloud Trace. For more information, see Overview of legacy bundled services.

    • Second-generation runtimes automatically capture overall request latency and add the X-Cloud-Trace-Context HTTP header to requests. For more information, see Runtime support schedule.

  • Cloud Run functions and Cloud Run

    For incoming and outgoing HTTP requests, latency data is automatically sent to Trace.

Instrument your application

Instrument your application to collect specific information that helps you understand its performance and troubleshoot failures. Several open-source instrumentation frameworks collect log, metric, and trace data, and can send that data to any vendor, including Google Cloud. For your agentic applications, some frameworks can collect your prompts and responses or pass context that allows tracing of some remote Google Cloud MCP servers calls.

To instrument your application, we recommend that you use an open-source, vendor-neutral instrumentation framework, such as OpenTelemetry, instead of vendor- and product-specific APIs or client libraries. For information about these frameworks, see Instrumentation and observability and Choose an instrumentation approach.

The instrumentation samples we provide use OpenTelemetry:

Although you can use Cloud Trace client libraries to instrument your application, we recommend that you use OpenTelemetry. OpenTelemetry libraries are preferable to Trace client libraries because they are simpler and export trace data in OTLP format, which OpenTelemetry defines. For more information, see Instrument for Trace and Client libraries for Cloud Trace.

Cloud Trace and agentic applications

To understand the behavior of your agentic applications, configure them to collect prompts and responses or to generate spans when they call remote Google Cloud MCP servers. Prompts and responses help you understand the reasoning your agentic application uses. Spans that record tool calls help you confirm tool invocation, call statuses, and request latencies.

Several instrumentation samples show you how to configure an application to collect prompts and responses. These samples rely on OpenTelemetry. For more information, see How to instrument your generative AI applications.

Google Cloud MCP servers can generate trace spans. For more information, see Investigate MCP calls using Trace.

APIs that ingest trace data

You can send trace data to your project by using either the Telemetry API or the Cloud Trace API. We recommend the Telemetry API for the following reasons:

  • The API provides compatibility with the open source OpenTelemetry ecosystem, and its limits are often more generous than the limits of the Cloud Trace API, which is a proprietary Google Cloud API.

  • Your trace data is stored in a format that is generally consistent with the proto files defined by OTLP. Some fields might be converted from an OpenTelemetry-specific data type to a JSON data type before storage. For information about the storage format, see Schema for trace data.

  • For collector-based export of trace data, your instrumentation doesn't rely on a Google Cloud-specific exporter.

  • Some features, like Application Monitoring, rely on information that is available only when you send trace data to the Telemetry API.

To prevent your Google Cloud project from storing trace data, disable the Cloud Trace API. Disabling the Cloud Trace API has the following effects:

  • Google Cloud services don't send trace data to your project.
  • Google Cloud replies to requests sent to a Cloud Trace API endpoint with an error code.
  • Google Cloud Observability discards trace data sent to the trace-specific Telemetry API endpoint. Don't disable the Telemetry API, as that API can receive log, metric, and trace data.

If you manage an organization and want to prevent usage of Cloud Trace, then create an organization policy constraint.

VPC Service Controls support

Trace is a VPC Service Controls-supported service. The Trace service name is cloudtrace.googleapis.com. Any VPC Service Controls restrictions that you create for the Trace service apply only to that service. Those restrictions don't apply to any other services, including services like the telemetry.googleapis.com service, which can also ingest trace data.

For more information, see the following:

Cloud Trace and data residency

If you are using Assured Workloads because you have data-residency or Impact Level 4 (IL4) requirements, then don't use the Cloud Trace API to send trace spans.

To prevent your Google Cloud project from storing trace data, disable the Cloud Trace API. Don't disable the Telemetry API, as that API can receive log, metric, and trace data.

Trace retention

Category Retention period
Spans stored in the _Trace bucket 30 days

IAM roles

Cloud Trace uses Identity and Access Management (IAM) to control access to resources. For a list of Cloud Trace API and Telemetry API roles, see Control access with IAM.

Because the Telemetry API is a consumer API, sending data to the Telemetry API requires that you specify a quota project and grant the application's service account permission to use that quota. For more information, see Telemetry API: Authentication.

Pricing

To learn about pricing for Cloud Trace, see the Google Cloud Observability pricing page.

What's next