This document describes how to access and view traces and spans generated by
your Gemini Enterprise app. Traces and spans provide detailed insights into the performance and execution
flow of requests within your application, such as when using the assistant or
agents.

You can view trace data using either the **Trace Explorer** or directly within individual agents using the agent's **Traces** tab.

## Key concepts

This section introduces key concepts related to observability in
Gemini Enterprise.

| Concept | Description |
| Trace | A trace is a collection of spans that represents a single request or transaction as it flows through different services and components. <br /> For example, a trace represents the entire lifecycle of a request. This includes a user asking a question from the Gemini Enterprise assistant, the Gemini Enterprise assistant responding, and any subsequent actions triggered by the answer, such as sending an email. |
| Span | A span is a single, timed unit of work within a trace. It represents a specific operation, such as a function call, an API request, or a database query. Each span includes details like its start and end times, a unique ID, and its relationship to other spans. These relationships together form a trace. |
| Span logs | Span logs are timestamped, free-form messages or events associated with a specific span. They provide detailed, contextual information about a span's execution, helping users debug issues and understand the flow of a request. |
|---|---|

## Before you begin

Make sure you have the following:

- The [Gemini Enterprise Admin](https://docs.cloud.google.com/gemini/enterprise/docs/access-control.md.txt#discoveryengine.agentspaceAdmin)
  role or Google Cloud console Gemini Enterprise User role.

- An existing Gemini Enterprise web app. For information about creating a new
  app, see [Create an app](https://docs.cloud.google.com/gemini/enterprise/docs/create-app.md.txt).

- To access traces, you must have the
  [Cloud Trace User](https://docs.cloud.google.com/iam/docs/roles-permissions/cloudtrace.md.txt#cloudtrace.user)
  role (`roles/cloudtrace.user`).

- Enable the **Enable instrumentation of OpenTelemetry traces and logs**
  observability setting. Depending on the agent type, you enable this in the
  app-level configurations (for the **Core Assistant** agent) or inside the
  agent's **Configuration** tab (for **Workflow Builder** employee-made
  agents and **Deep Research** agents). For more information, see
  [Manage observability settings](https://docs.cloud.google.com/gemini/enterprise/docs/manage-observability-settings.md.txt).

## Data retention

Traces and spans generated by your Gemini Enterprise app are stored in
Cloud Trace in your Google Cloud project, and their retention is governed by
Cloud Trace. By default, Cloud Trace retains span data for **30 days** ;
data older than the retention period is automatically deleted. For the
authoritative and most current retention values, see
[Cloud Trace retention periods](https://docs.cloud.google.com/trace/docs/quotas.md.txt#trace-retention-periods)
in the [Cloud Trace quotas and limits](https://docs.cloud.google.com/trace/docs/quotas.md.txt) documentation.

## Access traces in Trace Explorer

To access traces and spans using the Trace Explorer, follow these steps:

1. In the Google Cloud console, go to the **Trace Explorer** page.

   [Go to Trace Explorer](https://console.cloud.google.com/traces/explorer)
2. Select the Google Cloud project where your Gemini Enterprise app was created. You
   see all the spans for your project.

3. In the table, click a **Span ID** to
   view the trace details, including the span's duration and other attributes.

   > [!NOTE]
   > **Note:** For more information on using the Trace Explorer, see [Find and explore traces](https://docs.cloud.google.com/trace/docs/finding-traces.md.txt).

   ![Screenshot of the Trace Explorer in the Google Cloud console, showing a list of spans with columns for Span ID, Trace ID, Start time, Duration, and Service.](https://docs.cloud.google.com/static/gemini/enterprise/docs/images/observability/traces-explorer-gemini-enterprise.png)

### Filter and search traces

Within the Trace Explorer for your project, you can isolate and analyze
Gemini Enterprise agent and data connector executions using these filters,
query parameters, and attributes:

- **Faceted filters** :
  - **Service** : Filter by `AssistantService` or `connectorsexecution` to isolate the platform agent and data connector request paths.
  - **Span** : Filter by `/AssistantService.StreamAssist` to locate the root user-turn spans, or search for child spans like `execute_tool
    <tool_name>` and `invoke_connector <connector>`.
- **Direct Trace ID Lookup** : Find traces directly using the standard 32-character W3C trace ID, which can be extracted from the `x-cloud-trace-context` response headers from `StreamAssist`.
- **Custom attributes** : Query by the `gemini_enterprise.assist_token` attribute on `execute_tool` or `invoke_connector` spans to map turn-level legacy assist tokens to the W3C trace waterfall.
- **Resource attributes** : Every exported span carries `cloud.region`, the
- **Resource attributes** : Every exported span carries the `cloud.region` attribute. This attribute represents the Gemini Enterprise serving location that handled the request, such as `global`, `us`, or `eu`. You can filter by this attribute to compare performance across regions. This comparison is useful when a data connector succeeds in one location but fails in another. This value represents the location serving your app, not the region hosting the third-party data source. To apply this filter, see [Apply span filters](https://docs.cloud.google.com/trace/docs/finding-traces.md.txt#apply-span-filters).

## Access traces for agents

You can also view trace spans directly for individual agents on the agent's **Traces** tab, which displays a detailed chronological summary of span trace records and parent-child execution structures.

> [!NOTE]
> **Note:** The **Traces** tab is available for the **Core Assistant** agent, **Workflow Builder** employee-made agents, and **Deep Research** agents.

To access traces for an agent:

1. In the Google Cloud console, navigate to your app, and then click **Agents**.
2. Select the agent you want to inspect, and then click the **Traces** tab.

### Trace summary table

The main table in the **Traces** tab shows the following information for each trace span:

- **Status** : The current status of the trace (currently displays **Unset**).
- **Trace span ID**: The unique identifier for the trace span. Click the span ID link to drill down into its details.
- **Span Name** : A brief description of the work the span is doing. Examples of spans for Gemini Enterprise data connectors include:
  - `execute_tool <tool_name>`: Emitted during the tool-dispatch layer in the agent runtime (for example, `execute_tool federated_search`).
  - `invoke_connector <connector>`: Generated by the connectors execution layer to represent the request logic to the specific connector (for example, `invoke_connector jira` or `invoke_connector servicenow`), which replaces the default service-side RPC spans.
- **Span Type** : The type of span. Options include:
  - `Agent to Tool`: An invocation from the agent to an external tool (reused for `execute_tool` and `invoke_connector` operations).
  - `Invoke Agent`: An invocation of an agent.
  - `Agent to Model`: A request from the agent to a large language model.
- **Duration**: The time taken to complete the trace span (in minutes).
- **Input**: The input data or message sent to the span, if any.
- **Output**: The output result returned by the span, if any.
- **Start Time**: The exact timestamp when the span began execution.

### Detailed trace view

Clicking a **Trace span ID** opens a detailed page featuring:

1. **Summary** : View the baseline metadata including **Start time** , **Duration** , and total **Spans** count. You can click to **View in Cloud Trace** or switch between **Graph** and **Timeline** view buttons.
2. **Visualizations**: Toggle between a graphical hierarchy or a timeline view showing parent-child relationships of all trace spans.
3. **Selected span details** : Select a single span to display detailed sections, which can include:
   - **Summary**: Displays the name of the span, target service, count of total spans with this action, and total duration (in seconds).
   - **Logs** : View relevant event details if configured. Click **View logs** to navigate directly to standard logs.
   - **Evaluation**: Shows any evaluation results, if configured.
   - **Input \& Output**: Complete request and response payloads.
   - **Logs \& Events**: List of events that occurred during this span execution.
   - **Stacktraces**: Error details if a span execution fails.
   - **Metadata \& Links**: Links referencing related cloud explorer entities.
   - **Attributes**: Resource tags and additional key-value observability parameters.

![Screenshot of the Traces tab in the Google Cloud console, showing a timeline of spans and the span summary](https://docs.cloud.google.com/static/gemini/enterprise/docs/images/observability/core-assistant-traces-details.png)

### Parent-child span hierarchy

For data connectors execution, parent-child span nesting within a single W3C
trace helps pinpoint exactly where latency or errors occur:

1. `/AssistantService.StreamAssist`: Root C++ platform span initiating the request.
2. `invoke_agent core_assistant`: The agent execution span.
3. `execute_tool <tool_name>`: The active tool execution span (for example, `execute_tool federated_search`).
4. `invoke_connector <connector_name>` (for example, `invoke_connector jira` or `invoke_connector servicenow`): The connectors execution span, representing the request lifecycle to the third-party endpoint.

## Experimental generative AI semantic convention telemetry

> [!WARNING]
> **Warning:** The generative AI semantic convention telemetry described in this section is experimental. The attributes, schemas, and formats are subject to change in future releases.

In addition to the stable telemetry, Gemini Enterprise trace spans include richer
metadata based on the
[OpenTelemetry generative AI semantic conventions](https://github.com/open-telemetry/semantic-conventions-genai/blob/main/docs/gen-ai/README.md).
OpenTelemetry classifies these conventions as
[Development status](https://opentelemetry.io/docs/specs/otel/document-status).
That status applies to early-stage specifications that can change or be removed
and shouldn't be relied on in production. Because these conventions aren't yet
stable, this Gemini Enterprise telemetry is experimental. These spans include
`gen_ai.*` attributes that describe agent, model, and tool activity, such as
agent and tool execution and prompt-and-response steps.

Prompt and response content in trace spans (such as prompt text and generated
answers) appears only when your observability settings allow logging of prompt
inputs and response outputs. Otherwise, the content is redacted or omitted. To
change these settings, see
[Manage observability settings](https://docs.cloud.google.com/gemini/enterprise/docs/manage-observability-settings.md.txt).