Troubleshoot distributed tracing errors

This document describes how to diagnose and resolve issues with distributed tracing when using Cloud Trace and telemetry policies with Secure Web Proxy.

Traces don't appear in Trace

If trace spans generated by Secure Web Proxy don't appear in the Trace Explorer, check the following items:

  1. Verify that the Cloud Trace API is enabled: make sure that the Trace API (cloudtrace.googleapis.com) is enabled in the Google Cloud project that hosts your Secure Web Proxy gateway.

    <pre class="devsite-click-to-copy">
gcloud services enable cloudtrace.googleapis.com
</pre>

  2. Verify the telemetry policy exists and is active: use the gcloud beta network-services telemetry-policies describe command to check that your telemetry policy exists in the correct region and references the appropriate Secure Web Proxy gateway:

    gcloud beta network-services telemetry-policies describe POLICY_NAME \
    --location=REGION

    Replace the following:

    • POLICY_NAME: name of your telemetry policy, such as my-swp-tracing-policy
    • REGION: region where your telemetry policy is deployed, such as us-central1

  3. Check the sampling rate: if your policy's samplingRate is low (for example, 0.01 for 1% or 0.001 for 0.1%), individual manual test requests might not be sampled. To confirm that tracing is functioning, temporarily update your policy to use 100% sampling (samplingRate: 1.00) and then restore your production rate.

  4. Validate the target gateway resource URI: verify that the telemetryTarget.resources field in the telemetry policy specifies the exact, fully qualified resource URL or short name of the Secure Web Proxy gateway.

    //networkservices.googleapis.com/projects/PROJECT_ID/locations/REGION/gateways/GATEWAY_NAME

    Replace the following:

    • PROJECT_ID: ID of your Google Cloud project
    • REGION: region where your Secure Web Proxy gateway is deployed, such as us-central1
    • GATEWAY_NAME: name of your Secure Web Proxy gateway instance

    Any mismatch in the project ID, region, or gateway name prevents the proxy from receiving the policy configuration.

  5. Check Identity and Access Management (IAM) permissions:

    • Make sure that you've granted your user account or service account the Cloud Trace User role (roles/cloudtrace.user) role to view spans in the Google Cloud console.
    • Make sure that you've granted the Cloud Trace Agent role (roles/cloudtrace.agent) to the following service accounts:
      • service-PROJECT_NUMBER@compute-system.iam.gserviceaccount.com
      • service-PROJECT_NUMBER@gcp-sa-networksecurity.iam.gserviceaccount.com
      • Client virtual machine (VM) service accounts, if client applications generate or propagate spans

Missing child spans or broken trace graphs

If Secure Web Proxy proxy spans appear as independent or disconnected root traces instead of child spans of your application requests, then do the following:

  1. Enable parent-based sampling: in your telemetry policy YAML file, verify that parentBasedSampling.enabled is set to true.

    tracingConfiguration:
  samplingRate: 0.01
  parentBasedSampling:
    enabled: true

    When parent-based sampling is disabled, Secure Web Proxy might drop traces that were sampled upstream if they don't match the local samplingRate.

  2. Check the OpenTelemetry instrumentation: verify that your application uses an OpenTelemetry SDK that has distributed tracing context propagation enabled. For more information, see the OpenTelemetry TraceContext propagator documentation.

Unexpectedly high trace ingestion volume

If you observe higher-than-expected trace ingestion volume or costs in Trace, then do the following:

  1. Reduce the baseline sampling rate: in high-traffic production environments, set samplingRate to a smaller fraction, such as 0.01 (1%) or 0.001 (0.1%).
  2. Rely on parent-based sampling: keep the gateway baseline samplingRate low and enable parentBasedSampling. This combination helps ensure that the proxy samples only the requests that upstream applications specifically select.
  3. Remove temporary debug policies: if you had applied 100% sampling (samplingRate: 1.00) during troubleshooting, then remove the policy or revert to your previous configuration after the debugging process is complete.

What's next