Trace inbound traffic with Cloud Trace

This document explains how to configure distributed tracing for Application Load Balancers by using Cloud Trace. You create a telemetry policy that targets the forwarding rule of your load balancer, and then view the resulting traces in Cloud Trace.

The examples in this document use an ecommerce application that has the following architecture:

A global external Application Load Balancer that receives customer traffic for a web storefront.
A regional internal Application Load Balancer that routes checkout requests from the storefront to internal cart, inventory, and payment services.

For information about how distributed tracing works, including sampling modes and trace context header handling, see Distributed tracing overview.

Before you begin

Enable the Cloud Trace API and the Network Services API in your project:

gcloud services enable \
    cloudtrace.googleapis.com \
    networkservices.googleapis.com

Make sure that you have a supported load balancer deployed. For more information, see the following:
Note: Classic Application Load Balancers don't support distributed tracing.
Note the name and scope of the forwarding rule of your load balancer. The scope of the forwarding rule depends on the load balancer type:
- Global forwarding rule: used by global external Application Load Balancers and cross-region internal Application Load Balancers.
- Regional forwarding rule: used by regional external Application Load Balancers and regional internal Application Load Balancers.
To list the forwarding rules in your project, use the gcloud compute forwarding-rules list command:
```
gcloud compute forwarding-rules list
```

Required roles

To get the permissions that you need to configure telemetry policies and view traces, ask your administrator to grant you the following Identity and Access Management (IAM) roles on your project:

To create, update, or delete telemetry policies: Compute Network Admin (roles/compute.networkAdmin) or Compute Admin (roles/compute.admin)
To view traces in the Google Cloud console: Cloud Trace User (roles/cloudtrace.user)

To let the load balancer write trace spans to Cloud Trace, grant the Cloud Trace Agent role (roles/cloudtrace.agent) on your project to the following service accounts:

Compute Engine service agent: service-PROJECT_NUMBER@compute-system.iam.gserviceaccount.com
Network Security service agent: service-PROJECT_NUMBER@gcp-sa-networksecurity.iam.gserviceaccount.com
The service accounts of your backend workloads, if your backends export their own spans to Cloud Trace. Without this role, Cloud Trace shows only the load balancer span, and the trace doesn't contain backend spans.

Replace PROJECT_NUMBER with your Google Cloud project number.

For more information about granting roles, see Manage access to projects, folders, and organizations.

Telemetry policy configuration

You define a telemetry policy in a YAML file that uses the following format:

name: "projects/PROJECT_ID/locations/LOCATION/telemetryPolicies/POLICY_NAME"
displayName: "DISPLAY_NAME"
telemetryTarget:
  resources:
  - "FORWARDING_RULE_URI"
tracingConfiguration:
  samplingRate: SAMPLING_RATE
  parentBasedSampling:
    enabled: ENABLE_PARENT_BASED_SAMPLING
    samplingRate: PARENT_BASED_SAMPLING_RATE

The following table describes the fields in a telemetry policy:

Field	Description
`name`	The resource name of the telemetry policy. Replace the following: `PROJECT_ID`: the ID of your Google Cloud project. `LOCATION`: the location of the telemetry policy, which must match the scope of the forwarding rules that the policy targets. For global forwarding rules, use `global`. For regional forwarding rules, use the region of the forwarding rule, such as `us-central1`. `POLICY_NAME`: the name of the telemetry policy. The name must be 4 to 63 characters long, start with a lowercase letter, contain only lowercase letters, numbers, and hyphens, and end with a lowercase letter or number.
`displayName`	Optional. A human-readable name (`DISPLAY_NAME`) for the telemetry policy.
`telemetryTarget.resources`	A list of the full resource names (`FORWARDING_RULE_URI`) of the forwarding rules that the policy applies to. You can specify up to 50 resources. Use one of the following formats: Global forwarding rule: `//compute.googleapis.com/projects/PROJECT_ID/global/forwardingRules/FORWARDING_RULE_NAME` Regional forwarding rule: `//compute.googleapis.com/projects/PROJECT_ID/regions/REGION/forwardingRules/FORWARDING_RULE_NAME` Replace the following: `FORWARDING_RULE_NAME`: the name of the forwarding rule `REGION`: the region of the forwarding rule, such as `us-central1`
`tracingConfiguration.samplingRate`	The proportion of requests to trace (`SAMPLING_RATE`), specified as a value from `0.0` (0%) to `1.0` (100%). For example, `0.01` samples 1% of requests. The default value is `0.0`.
`tracingConfiguration.parentBasedSampling.enabled`	Optional. Whether the load balancer honors the sampled flag in the incoming W3C `traceparent` header (`ENABLE_PARENT_BASED_SAMPLING`) is `true` or `false`. The default value is `false`. Supported only on internal Application Load Balancers.
`tracingConfiguration.parentBasedSampling.samplingRate`	Optional. The proportion of upstream-sampled requests to trace (`PARENT_BASED_SAMPLING_RATE`), specified as a value from `0.0` to `1.0`. For example, `1.0` traces all requests that an upstream client or service has already sampled.

Configure tracing for an external load balancer

This example configures probabilistic sampling on the forwarding rule of a global external Application Load Balancer for an ecommerce storefront. A sampling rate of 0.01 traces 1% of requests, which captures edge latency and errors while keeping Cloud Trace ingestion costs predictable during high traffic.

Create a YAML file named storefront-tracing-policy.yaml:
```
cat <<EOF > storefront-tracing-policy.yaml
name: "projects/PROJECT_ID/locations/global/telemetryPolicies/storefront-tracing-policy"
displayName: "Tracing policy for the storefront load balancer"
telemetryTarget:
  resources:
    - "//compute.googleapis.com/projects/PROJECT_ID/global/forwardingRules/FORWARDING_RULE_NAME"
tracingConfiguration:
  samplingRate: 0.01
EOF
```
Replace the following:
- PROJECT_ID: the ID of your Google Cloud project
- FORWARDING_RULE_NAME: the name of the global forwarding rule of your load balancer
For a regional external Application Load Balancer, replace global in the policy name with the region of the forwarding rule, and use the regional forwarding rule format.

Import the telemetry policy by using the gcloud beta network-services telemetry-policies import command:

gcloud beta network-services telemetry-policies import storefront-tracing-policy \
    --source=storefront-tracing-policy.yaml \
    --location=global

Configure tracing for an internal load balancer

This example configures a regional internal Application Load Balancer for internal checkout and payment services. The policy uses the following sampling settings:

A baseline samplingRate of 0.01 (1%) for requests that don't carry an upstream sampling decision.
Parent-based sampling with a samplingRate of 1.0 (100%), so that every checkout request that's already sampled at the storefront, or by any upstream service, is traced across the internal load balancer.

To configure the policy, do the following:

Create a YAML file named checkout-tracing-policy.yaml:

cat <<EOF > checkout-tracing-policy.yaml
name: "projects/PROJECT_ID/locations/REGION/telemetryPolicies/checkout-tracing-policy"
displayName: "Tracing policy for the internal checkout load balancer"
telemetryTarget:
  resources:
    - "//compute.googleapis.com/projects/PROJECT_ID/regions/REGION/forwardingRules/FORWARDING_RULE_NAME"
tracingConfiguration:
  samplingRate: 0.01
  parentBasedSampling:
    enabled: true
    samplingRate: 1.0
EOF

Replace the following:

PROJECT_ID: the ID of your Google Cloud project
REGION: the region of the forwarding rule, such as us-central1
FORWARDING_RULE_NAME: the name of the regional forwarding rule of your load balancer

For a cross-region internal Application Load Balancer, use global as the location in the policy name, and use the global forwarding rule format.

Import the telemetry policy:

gcloud beta network-services telemetry-policies import checkout-tracing-policy \
    --source=checkout-tracing-policy.yaml \
    --location=REGION

Verify the telemetry policy

To verify that the telemetry policy exists and targets the correct forwarding rules, use the gcloud beta network-services telemetry-policies describe command:

gcloud beta network-services telemetry-policies describe POLICY_NAME \
    --location=LOCATION

Replace the following:

POLICY_NAME: the name of the telemetry policy
LOCATION: global or the region of the telemetry policy

To list all telemetry policies in a location, use the gcloud beta network-services telemetry-policies list command:

gcloud beta network-services telemetry-policies list \
    --location=LOCATION

Update a telemetry policy

To change the sampling settings, or to apply the same policy to additional forwarding rules, do the following:

Edit the policy YAML file. For example, to apply the storefront policy to multiple global forwarding rules, add each forwarding rule to the telemetryTarget.resources list:

telemetryTarget:
  resources:
    - "//compute.googleapis.com/projects/PROJECT_ID/global/forwardingRules/storefront-https-fr"
    - "//compute.googleapis.com/projects/PROJECT_ID/global/forwardingRules/cart-https-fr"

Import the updated policy:

gcloud beta network-services telemetry-policies import POLICY_NAME \
    --source=POLICY_FILE \
    --location=LOCATION

Replace POLICY_FILE with the path to your policy YAML file.

View traces

After you configure tracing and send traffic through your load balancer, view the traces in the Google Cloud console:

In the Google Cloud console, go to the Trace Explorer page.

Go to Trace Explorer
Select your Google Cloud project.
In the Filter field, enter a filter to find the traces that you want to inspect. For example, attribute ID or latency threshold.
Optionally, to explore a specific trace, go to the toolbar, click Trace ID, and enter the trace ID.
Click a trace to view its spans in the timeline and graph views.
Inspect the load balancer span to see details such as the total duration and the backend that served the request.

For more information about using Trace Explorer, see Find and explore traces.

Correlate traces with logs and metrics

You can correlate load balancer trace spans with request logs and metrics to isolate the source of latency or errors.

Correlate traces with logs

If you enable logging on your backend service, the load balancer adds a trace field to each request log entry for a sampled request. The field uses the following format:

projects/PROJECT_ID/traces/TRACE_ID

View a trace from a load balancer log entry

In the Google Cloud console, go to the Logs Explorer page.

Go to Logs Explorer
Expand a load balancer log entry.
In the log entry summary, click the trace icon.
Select View trace details.

The trace opens on the Trace Explorer page.

View load balancer logs from a trace

In the Google Cloud console, go to the Trace Explorer page.

Go to Trace Explorer
Click a trace to open the Details pane.
Click View Logs.

The load balancer log entries for the request appear.

Correlate traces with metrics

Compare latency spikes in load balancer metrics with trace span breakdowns to determine whether latency originates in the load balancer or in the backend. You can view and query these metrics in Metrics Explorer in Cloud Monitoring. Use the following metrics:

Global external Application Load Balancer: loadbalancing.googleapis.com/https/total_latencies and loadbalancing.googleapis.com/https/backend_latencies
Regional external Application Load Balancer: loadbalancing.googleapis.com/https/external/regional/total_latencies
Internal Application Load Balancer: loadbalancing.googleapis.com/https/internal/total_latencies

Disable tracing

To stop tracing requests for a load balancer, do one of the following:

Remove the forwarding rule from the telemetryTarget.resources list of the policy, and then import the updated policy.
Delete the telemetry policy by using the gcloud beta network-services telemetry-policies delete command:
```
gcloud beta network-services telemetry-policies delete POLICY_NAME \
    --location=LOCATION
```

Best practices

Consider the following best practices when you configure distributed tracing:

Size sampling rates for your traffic volume: for high-traffic production workloads, use a samplingRate of 0.01 (1%) or 0.001 (0.1%) to balance observability with Cloud Trace ingestion costs.
Use parent-based sampling for multi-tier architectures: on internal Application Load Balancers, enable parent-based sampling so that requests sampled upstream are traced across every internal hop.
Instrument your backends: use an OpenTelemetry SDK with the W3C Trace Context propagator in your backend services so that backend spans join the trace that the load balancer starts.

Troubleshoot

This section describes common issues and their resolutions.

Traces don't appear in Cloud Trace

If traces don't appear in Cloud Trace, do the following:

Make sure that the Cloud Trace API is enabled in your project:

gcloud services enable cloudtrace.googleapis.com \
    --project=PROJECT_ID

Verify that the Cloud Trace Agent role (roles/cloudtrace.agent) is granted to the service accounts listed in Required roles.
Verify that the telemetry policy exists in the correct location and that it targets the correct forwarding rule. For more information, see Verify the telemetry policy.
Check the configured samplingRate. If the traffic volume or the sampling rate is low, the load balancer might not capture traces immediately. To confirm that tracing works, temporarily set samplingRate to 1.0, and then restore your production rate when you finish testing.

Trace spans are disconnected or incomplete

If load balancer spans appear as separate root traces instead of as part of your application's traces, do the following:

For internal Application Load Balancers, verify that parent-based sampling is enabled (parentBasedSampling.enabled: true) in the telemetry policy.
Verify that the service accounts of your client and backend workloads have the Cloud Trace Agent role (roles/cloudtrace.agent), so that their spans can be exported to Cloud Trace.
Verify that your applications use an OpenTelemetry SDK with W3C Trace Context propagation enabled, so that the traceparent header is propagated across service boundaries. For more information, see the OpenTelemetry context propagation documentation.