Trace outbound traffic with Cloud Trace

This document explains how to configure distributed request tracing for Secure Web Proxy by using Cloud Trace.

For information about the concepts, key benefits, and architecture details, see Distributed tracing overview.

Before you begin

Complete the initial setup steps.

Enable the Compute Engine API, Network Services API, and Cloud Trace API in your project.

gcloud services enable \
    compute.googleapis.com \
    networkservices.googleapis.com \
    cloudtrace.googleapis.com

Make sure that you have an active Secure Web Proxy gateway instance deployed. For more information, see Deploy a Secure Web Proxy instance.
To generate HTTP-level distributed trace spans and propagate W3C trace context headers for HTTPS requests, you must enable TLS inspection on your Secure Web Proxy gateway.

Roles and permissions

To obtain the permissions that are required to configure telemetry policies and view traces, request your administrator to grant you the following Identity and Access Management (IAM) roles:

To create, update, or delete telemetry policies: Network Services Admin (roles/networkservices.admin) or Compute Network Admin (roles/compute.networkAdmin)
To view and analyze trace spans in the Google Cloud console: Cloud Trace User (roles/cloudtrace.user)
To export trace spans to Trace, make sure that the following service accounts have the Cloud Trace Agent role (roles/cloudtrace.agent):
- Compute Engine service agent: service-PROJECT_NUMBER@compute-system.iam.gserviceaccount.com
- Network Security service agent: service-PROJECT_NUMBER@gcp-sa-networksecurity.iam.gserviceaccount.com
- Client virtual machine (VM) service accounts, if client applications generate or propagate spans

Telemetry policy configuration schema

To define a telemetry policy in a YAML file, use the following schema:

    name: projects/PROJECT_ID/locations/REGION/telemetryPolicies/POLICY_NAME
    telemetryTarget:
      resources:
      - "//networkservices.googleapis.com/projects/PROJECT_ID/locations/REGION/gateways/GATEWAY_NAME"
    description: "OPTIONAL_DESCRIPTION"
    tracingConfiguration:
      samplingRate: SAMPLING_RATE
      parentBasedSampling:
        enabled: true

The following table describes the fields in a telemetry policy:

Field	Type	Description
`name`	String	Unique identifier of the telemetry policy.
`telemetryTarget.resources`	List of strings	Resource URLs of the Secure Web Proxy gateways to which this policy applies. Your gateways must reside in the same region as the policy. Format: `//networkservices.googleapis.com/projects/PROJECT_ID/locations/REGION/gateways/GATEWAY_NAME` Alternatively, use the short name format: `projects/PROJECT_ID/locations/REGION/gateways/GATEWAY_NAME`.
`description`	String	Optional. This field provides a human-readable description of the telemetry policy.
`tracingConfiguration.samplingRate`	Float	Proportion of requests to trace, specified as a decimal between `0.0` (0%) and `1.0` (100%). For example, `0.01` samples 1% of requests.
`tracingConfiguration.parentBasedSampling.enabled`	Boolean	Indicates whether you want to enable upstream trace sampling decisions (`true` or `false`).

Configure distributed tracing

To apply a tracing policy to your Secure Web Proxy gateway by using the Google Cloud CLI, do the following:

Create a YAML policy file named swp-tracing-policy.yaml.

cat <<EOF > swp-tracing-policy.yaml
name: projects/PROJECT_ID/locations/REGION/telemetryPolicies/my-swp-tracing-policy
telemetryTarget:
  resources:
    - "//networkservices.googleapis.com/projects/PROJECT_ID/locations/REGION/gateways/GATEWAY_NAME"
description: "Distributed tracing policy for the Secure Web Proxy gateway"
tracingConfiguration:
  samplingRate: 0.01
  parentBasedSampling:
    enabled: true
EOF

Replace the following:

PROJECT_ID: ID of your Google Cloud project
REGION: region where your Secure Web Proxy gateway is deployed, such as us-central1
GATEWAY_NAME: name of your Secure Web Proxy gateway instance

Import the telemetry policy by using the gcloud beta network-services telemetry-policies import command:

gcloud beta network-services telemetry-policies import my-swp-tracing-policy \
    --source=swp-tracing-policy.yaml \
    --location=REGION

Verify that the policy is applied by using the gcloud beta network-services telemetry-policies describe command:
```
gcloud beta network-services telemetry-policies describe my-swp-tracing-policy \
    --location=REGION
```

Attach or detach target gateways

To update the policy definition or attach multiple regional gateways, do the following:

Update the telemetryTarget.resources list in the swp-tracing-policy.yaml file to include all target gateway resource URIs.

telemetryTarget:
  resources:
    - "//networkservices.googleapis.com/projects/PROJECT_ID/locations/us-central1/gateways/prod-swp-uc1"
    - "//networkservices.googleapis.com/projects/PROJECT_ID/locations/us-central1/gateways/staging-swp-uc1"

Import the updated telemetry policy by using the gcloud beta network-services telemetry-policies import command:
```
gcloud beta network-services telemetry-policies import my-swp-tracing-policy \
    --source=swp-tracing-policy.yaml \
    --location=REGION
```
Replace the following:
- PROJECT_ID: ID of your Google Cloud project
- REGION: region where your Secure Web Proxy gateway is deployed, such as us-central1

View and analyze traces in Trace

In the Google Cloud console, go to the Trace Explorer page.

Go to Trace Explorer
Select your Google Cloud project.
In the Filter field, search for traces by using various attributes.
Click any trace in the scatter plot or timeline view to inspect span details, such as total duration, proxy processing latency, and external server IP details.

Correlate traces with logs and metrics

Secure Web Proxy provides correlation between traces, logs, and metrics:

Log-trace correlation: every proxy transaction log entry written to Cloud Logging contains a trace field with the format: projects/PROJECT_ID/traces/TRACE_ID.
- In the Logs Explorer, click the log entry and then select View trace details to navigate directly to the corresponding span in Trace.
- In Trace, click Show logs on the Trace Details pane to view all transaction logs that are associated with that request.
Metric correlation: compare latency spikes observed in Cloud Monitoring metrics, such as networkservices.googleapis.com/https/total_latencies, with trace span breakdowns to identify whether a latency spike occurred within the proxy or at the external origin server.