Trace sampling

Trace sampling determines which requests and spans Cloud Trace ingests, helping you control storage costs and stay within quotas while capturing sufficient data to troubleshoot application performance. When every span in an end-to-end request is recorded, the trace is complete. However, to manage high request volumes, components in a distributed tracing system typically sample only a subset of traces or follow the sampling decisions of parent spans.

Sampling is distinct from context propagation. Sampling controls whether a component records span data. In contrast, context propagation passes trace identifiers between components so that sampled spans can be linked together.

Sampling strategies

Sampling decisions can be either head-based or tail-based. In head-based sampling, the sampling decision is made when the request is received by the component processing the span. In tail-based sampling, the sampling decision is delayed until after the entire trace is available.

You might encounter the phrase "100% sampling" in documentation for distributed tracing systems. This phrase might apply to a trace or to a component. When applied to a trace, it means that all spans have been sampled, or equivalently, that the trace is complete. When applied to a component, it means that the component samples every span it processes.

Head-based sampling

Head-based samplers are typically configured to always sample spans or to use a probabilistic sampling strategy:

  • With always sample configurations, components that process and write trace data sample every span. Ideally, all traces are complete, and therefore you have the information necessary to troubleshoot failures. However, an always-sample configuration might cause you to exceed quotas or your storage cost limits.

  • With probabilistic sampling, not all spans are sampled. The actual behavior for this approach depends on the component's implementation. In some implementations, all spans have the same probability of being sampled. In others, the sampling decision of the parent influences whether a span is sampled.

A trace might not contain every span. If you use probabilistic sampling, exceed quota, or use components that process but don't sample spans, then incomplete traces are expected.

Tail-based sampling

Cloud Trace doesn't support tail-based sampling; sampling decisions must be made in the components that send data to Cloud Trace.

If you use tail-based sampling, then you can also use an intermediary server to receive trace data, evaluate sampling decisions, and relay sampled spans to Trace. For example, you can use an OpenTelemetry Collector with the Tail Sampling Processor to make a delayed sampling decision.

If you plan to use tail sampling, consider the following:

  • You must store all spans in a trace before you make a sampling decision. Therefore, you might require a large amount of temporary storage or incur other overhead.
  • In general, all components that can generate spans for a trace need to coordinate. Typically, developers that use OpenTelemetry route all spans for the same trace ID to the same collector.

Components make sampling decisions

Each component makes its own decision about whether to sample the span it is processing. However, the parent's sampling decision, which might be available to the component through the trace context, can influence the component's decision. Applications that use the traceparent header can pass the parent's sampling decision by using the sampled flag.

For example, assume every component has a rule that says "if the parent span is sampled, then sample the current span; otherwise, sample 50% of the spans". In this scenario, the following is true:

  • The root span determines whether all spans in the trace are sampled.
  • When the root span is sampled, all spans in the trace are sampled. Therefore, the trace is complete.

Sampling and Google Cloud services

Each Google Cloud service makes its own sampling decisions, and not all Google Cloud services sample. That is, a service might never send data to Cloud Trace.

When sampling is supported by a Google Cloud service, that service typically implements the following:

  • A default sample rate.
  • A mechanism to use the parent's sampling decision as a hint about whether to sample the span.
  • Maximum sampling rate.

To request that a Google Cloud service add support for sampling, use the Google Issue Tracker.

What's next