Troubleshoot

This page contains troubleshooting information for Trace.

Known issues

This section lists known issues:

Spans written to your Google Cloud project by using the Telemetry API aren't accessible to the Cloud Trace API. For example, if you try to list these traces, then the command fails with a 404 Not Found error.

Troubleshoot Log Analytics

This section describes how to resolve failures you might see when using Log Analytics to query your trace data.

Error message stating a view does not exist

You enter a SQL query in the query pane of the Log Analytics page, but the SQL parser displays the following error:

projects/PROJECT_ID/locations/LOCATION/buckets/BUCKET_ID/datasets/DATASET_ID/views/OBS_VIEW_ID does not exist

The previous error is reported when the view specified in the FROM statement can't be found.

To resolve this error, verify that your view has the proper syntax:

Verify that the fully-qualified name of the view follows the syntax required by the Log Analytics naming scheme. You can find the required syntax for a view by displaying its default query.
If the Google Cloud project ID, location, bucket ID, dataset ID, or view ID contain period characters, (.), then verify that the field is wrapped by single backquotes, (`).

For example, if the ID of your Google Cloud project is example.com:bluebird, then the FROM statement is as follows:
```
FROM `example.com:bluebird`.`us`.`_Trace`.`Spans`.`_AllSpans`
```

No data in the Trace Explorer page

You have an application that is sending trace data to your Google Cloud project. However, when you open the Trace Explorer page, no data is shown.

There are several possible reasons why you can't view trace data:

You aren't granted the permissions required to view the data.
Trace spans weren't sent to your project.
Your application doesn't have the permissions required to write trace data.
Your trace spans aren't being stored.

The following subsections provide information about how to troubleshoot the listed failure scenarios.

Verify that you have permission to view trace data

To view trace data, make sure that you have been granted the Cloud Trace User role (roles/cloudtrace.user).

Verify that trace spans are sent to your project

To verify that spans are being sent to your project, do the following:

Enable the Cloud Trace and Telemetry APIs.
Roles required to enable APIs
To enable APIs, you need the Service Usage Admin IAM role (roles/serviceusage.serviceUsageAdmin), which contains the serviceusage.services.enable permission. Learn how to grant roles.
Enable the APIs

Both APIs can ingest trace spans. However, the Telemetry API is recommended because it is compatible with the OpenTelemetry ecosystem and because it has more generous limits than the Cloud Trace API.

Note: There might be org policies in place that have disabled the Cloud Trace API. These policies prevent you from enabling the API.
Go to the Enabled APIs and Services page, find the rows for the Cloud Trace API and Telemetry API.

If the Requests count for these two APIs is zero, then no trace data is being sent to your project.

Verify that your application has the required permissions to write trace spans

To determine whether your application has permission to write trace data to your project, do the following:

Go to the Enabled APIs and Services page, find the rows for the Cloud Trace API and Telemetry API, and examine the Errors column.
If you see a non-zero value in the Errors column for either API, then there are errors reading or writing trace data through that API. To identify the type of error, select the API, select the Metrics tab, and view the Errors by API method:

If writes are failing, then grant the service account that is providing credentials the following roles:
- Cloud Trace API: Cloud Trace Agent Role (roles/cloudtrace.agent).
- Telemetry API: Cloud Telemetry Trace Writer (roles/telemetry.tracesWriter).

Verify that your trace data is stored

Trace spans are stored in an observability bucket named _Trace. That bucket is provisioned automatically when your Google Cloud project receives trace spans. However, there are several scenarios where provisioning fails.

To determine whether an observability bucket exists for your trace data, you can either [list your observability buckets][trace-storage-list-buckets] or you can open the Trace Explorer page. For example, you can do the following:

In the Google Cloud console, go to the Trace explorer page:
Go to Trace explorer

You can also find this page by using the search bar.
If you see for a banner similar to the following, then that indicates that storage for your trace data isn't provisioned.
```
Trace storage is not initialized for this project. Enable trace storage to begin collecting trace data.
```
To provision a observability bucket for your trace data, go to the banner and click Enable.

When you click Enable, that action causes a span to be sent to your project. When the system receives the span, it issues the command to create an observability bucket named _Trace. This process can take several minutes to complete.

When initialization is successful, a notification banner is shown and Cloud Trace ingests any trace data that was sent in the last hour. This data was stored in a temporary buffer. It might take a few minutes before data appears in the Trace Explorer. If you don't see any data, then refresh your window.
If the enable command fails, then the following message is shown:
```
Initializing trace storage has failed for an unexpected reason. Please file a support ticket for assistance.
```
To resolve the failure, contact Google Cloud support by clicking File a ticket.

Search for a specific trace fails

You enter a trace ID into the Trace Explorer page. The trace isn't found and a message similar to the following is shown:

The select trace with ID abcde does not exist or is older than 30 days and has been deleted per our retention policy.

To resolve this failure, try the following:

Verify that the timestamp associated with the trace ID is within the retention period.
Identify the Google Cloud project that stores the trace, and verify that the resource picker in the Google Cloud console selects this project. By default, the Trace Explorer page only has access to trace data stored in the selected project.

Missing older data in the Trace Explorer page

You are using the Trace Explorer page and you can view recent data, but when you set the time-range selector to 30 days or to a larger value, the older data isn't shown.

The Trace Explorer page doesn't display data for time periods that are larger than the data retention period of Cloud Trace, which is 30 days.

If the time-range selector is 30 days or less, then the missing data indicates that the database queried by the Trace Explorer page queries was created more recently than your time-range setting. For example, if you set this value to 20 days and you can only see the most recent 10 days of data, then the database was created 10 days in the past. Also, this database only contains traces that were sent to your Google Cloud project after the database was created.

Incomplete trace is shown

You open the Trace Explorer page and select a span to view. The Details flyout shows the trace, but the trace isn't complete. Some spans aren't shown.

Spans might be missing for the following reasons:

The Trace Explorer page isn't searching all Google Cloud projects that store span data for the trace.
Your IAM role on a Google Cloud project that stores span data for the trace doesn't contain the permissions necessary to view trace data.
There is an instrumentation issue. For example, only some spans in a trace were sent to your Google Cloud project.

To resolve these problems, do the following:

In the Trace Explorer page, make sure to set the Scope element to a trace scope that lists projects that that store the spans for the selected trace.

If there isn't a trace scope that includes the projects that you identified in the previous step, then create or modify an existing trace scope. For more information, see Create and manage trace scopes.
Verify you have the Cloud Trace User role (roles/cloudtrace.user) on the projects that store the span data.

You don't have the required permissions to view trace data

You are viewing the Trace Explorer page and see the following notification:

You don't have the required permissions to view trace data for one or more projects listed in the trace scope.

To resolve this message, in the toolbar, do the following:

Expand the Scope element and identify the selected trace scope.
In the Refine scope flyout, select Manage scopes.
Locate the trace scope you identified in the first step, and then expand the details to view the list of Google Cloud projects.
For each Google Cloud project in the trace scope, verify that you have the role of Cloud Trace User (roles/cloudtrace.user). If you don't have that role on a project, then ask an administrator or project owner to grant you that role.

Missing span ID message in trace

Your trace contains a "Missing span ID" message.

In distributed tracing systems, incomplete traces are expected. A trace is incomplete when a sampled span contains a reference to another span that hasn't been received. The unresolved reference can occur for the following reasons:

The referenced span wasn't sampled.
The referenced span was sampled but hasn't yet been received by Cloud Trace or the span was received but not stored.

When you are viewing an incomplete trace, Cloud Trace displays the message "Missing span ID" on the trace details pane.

If you are consistently seeing the message "Missing span ID", then try the following:

For components that you manage, verify that they respect and propagate the header's sampled flag flag, when this field is present. This setting is a hint to child components to sample the request. For more information about trace headers, see Protocols for context propagation.

Google Cloud services typically respect this hint. However, they also limit the rate at which they write trace data.
If you are using Cloud Service Mesh, then verify that you follow the guidance for propagating the trace context for those configurations. For Cloud Service Mesh guidance, see Trace context propagation.

Unable to correlate log and trace data

You are doing one of the following:

You are viewing a trace span and you want to view associated log entries. However, either no log data is listed or, when you open the Logs Explorer page, it doesn't show any log entries.
You are viewing a log entry and you want to view associated trace spans. However, when you use the options on the log entry to open the Trace Explorer page, it doesn't show any trace data.

To resolve these failures, configure the observability scope. This scope specifies which of your trace scopes and log scopes to use when the corresponding explorer pages open. To learn more, see Configure observability scopes for multi-project queries.

No trace data after updating Go app to use OpenTelemetry

Your application relies on the client library to capture traces, and after updating your application to use OpenTelemetry, you no longer see Cloud Trace data.

Because some Cloud Client Libraries for Go are integrated with OpenCensus, you must use an OpenCensus Bridge. For more information about the problem solved by the bridge, see OpenCensus Bridge.

For information about the update of the Cloud Client Libraries for Go, see Issue #4237.