Monitor data insights token usage

This document explains how to monitor token consumption for Knowledge Catalog (formerly Dataplex Universal Catalog) data insights scans using Cloud Logging and Cloud Monitoring. Tracking token usage per scan and per job lets you understand model usage trends and estimate billing costs across your project.

How token usage logging works

When a data insights scan job (jsonPayload.type="DATA_DOCUMENTATION") completes with a SUCCEEDED state, Knowledge Catalog writes a log entry to the dataplex.googleapis.com/data_scan log. This log entry contains a billableTokenUsage block under jsonPayload.dataDocumentation that records the token breakdown for the job. Other log entries for the job (such as CREATED, STARTED, FAILED, or CANCELLED events) don't include a billableTokenUsage block.

From the billableTokenUsage block, you can track three primary token categories to estimate billing:

Metric Description Formula (using billableTokenUsage fields)
Non-cached input tokens Input tokens that weren't served from the context cache, including tool-use prompt tokens. max(promptTokenCount - cachedContentTokenCount, 0) + toolUsePromptTokenCount
Cached input tokens Input tokens served from the context cache. Cached tokens are tracked separately because they aren't priced at the standard non-cached input rate. cachedContentTokenCount
Output tokens Generated response tokens plus reasoning (thinking) tokens. Thinking tokens are billed as output tokens. candidatesTokenCount + thoughtsTokenCount

For token pricing details, see Data Cloud Agent pricing.

Token fields in the log entry

Each dataplex.googleapis.com/data_scan log entry includes standard resource and job identifiers such as resource.labels.datascan_id, resource.labels.location, jsonPayload.jobId, and jsonPayload.state. For succeeded data insights scans, the jsonPayload.dataDocumentation.billableTokenUsage object contains the following fields:

Field Description
promptTokenCount Total prompt tokens sent to the model, including cached tokens.
cachedContentTokenCount Portion of the prompt tokens served from the context cache.
toolUsePromptTokenCount Prompt tokens added by tool calls (such as schema, profile, or query history context).
candidatesTokenCount Generated response tokens, excluding thinking tokens.
thoughtsTokenCount Reasoning (thinking) tokens generated by the model.

The following example shows a dataplex.googleapis.com/data_scan log entry for a succeeded data insights scan job:

{
  "resource": {
    "type": "dataplex.googleapis.com/DataScan",
    "labels": {
      "datascan_id": "my-insights-scan",
      "location": "us-central1",
      "resource_container": "projects/1234567890"
    }
  },
  "jsonPayload": {
    "type": "DATA_DOCUMENTATION",
    "jobId": "8f2b3c4d-1234-5678-9abc-def012345678",
    "state": "SUCCEEDED",
    "dataDocumentation": {
      "billableTokenUsage": {
        "promptTokenCount": "20412",
        "candidatesTokenCount": "7053"
      }
    }
  }
}

In this example, the job didn't use context caching, thinking tokens, or tool calls. As a result, non-cached input tokens equal 20,412, cached input tokens equal 0, and output tokens equal 7,053.

Before you begin

Before you query data insights token logs or create a monitoring dashboard, complete the following prerequisites.

Required roles

To get the permissions that you need to monitor data insights token usage, ask your administrator to grant you the following IAM roles on your project:

  • View data scan logs and run Observability Analytics queries: Logs Viewer (roles/logging.viewer)
  • Create custom dashboards in Monitoring: Monitoring Editor (roles/monitoring.editor)
  • View custom dashboards in Monitoring: Monitoring Viewer (roles/monitoring.viewer)

For more information about granting roles, see Manage access to projects, folders, and organizations.

You might also be able to get the required permissions through custom roles or other predefined roles.

Enable APIs

Enable the Dataplex, Logging, Monitoring APIs, if any are not already enabled.

Roles required to enable APIs

To enable APIs, you need the serviceusage.services.enable permission. If you created the project, then you likely already have this permission through the Owner role (roles/owner). Otherwise, you can get this permission through the Service Usage Admin role (roles/serviceusage.serviceUsageAdmin). Learn how to grant roles.

Enable the APIs

Verify log routing configuration

Data insights scans write logs under the log name dataplex.googleapis.com/data_scan. The dashboard configurations in this document read those logs from your project's _Default log bucket. Verify that your project's log routing meets the following conditions:

  • The _Default sink is enabled and routes logs to the _Default log bucket.

  • The _Default sink has no exclusion filter that drops dataplex.googleapis.com/data_scan or DATA_DOCUMENTATION logs.

Console

  1. In the Google Cloud console, go to the Logging > Log Router page.

    Go to Log Router

  2. Locate the _Default sink and verify that its status is Enabled and its destination is the _Default bucket.

  3. Click the sink's menu () and select View sink details. Verify that no exclusion filter matches dataplex.googleapis.com/data_scan or DATA_DOCUMENTATION.

gcloud

Describe the _Default sink to inspect its status, destination, and exclusion filters:

gcloud logging sinks describe _Default --project=PROJECT_ID

Replace PROJECT_ID with your Google Cloud project ID.

Verify that disabled isn't set to true, destination ends with /buckets/_Default, and no entry under exclusions matches data_scan.

Enable Observability Analytics on the _Default bucket

To run SQL queries on your data insights scan logs and power the custom Observability Analytics dashboard, upgrade your project's _Default log bucket to use Observability Analytics.

Before you enable Observability Analytics on the _Default log bucket, review the following considerations:

  • Upgrading a bucket to Observability Analytics can't be undone, and it incurs no extra charge.
  • You don't need to create a linked BigQuery dataset.

  • Observability Analytics SQL queries only return logs that arrive after the upgrade completes. Older logs remain available in Logs Explorer, but don't appear in SQL query results. After you upgrade the bucket, run at least one data insights scan so that your dashboard has data to display.

Console

  1. In the Google Cloud console, go to the Logging > Logs Storage page.

    Go to Logs Storage

  2. Locate the _Default bucket row and check the Observability Analytics available column:

    • If the column shows Open, Observability Analytics is already enabled.
    • If the column shows Upgrade, click Upgrade and confirm the upgrade.

  3. Wait a few minutes until the column status changes to Open.

gcloud

  1. Upgrade the _Default log bucket to enable Observability Analytics:

    gcloud logging buckets update _Default \
    --location=global --enable-analytics --project=PROJECT_ID

    Replace PROJECT_ID with your Google Cloud project ID.

  2. After a few minutes, verify that the upgrade is complete:

    gcloud logging buckets describe _Default \
    --location=global --project=PROJECT_ID

    Confirm that the output includes analyticsEnabled: true.

View per-job token usage in Logs Explorer

To inspect the token consumption of individual data insights scan jobs, use Logs Explorer or the Google Cloud CLI. For hourly or daily aggregations across your project, you can create a dashboard using Observability Analytics.

Console

  1. In the Google Cloud console, go to the Logging > Logs Explorer page.

    Go to Logs Explorer

  2. If the query editor isn't visible, turn on Show query.

  3. Enter the following query in the query editor:

    log_id("dataplex.googleapis.com/data_scan")
jsonPayload.type="DATA_DOCUMENTATION"
jsonPayload.dataDocumentation.billableTokenUsage.promptTokenCount:*

    Optional: To filter for a specific scan or job execution, append one of the following lines to the query:

    resource.labels.datascan_id="DATA_SCAN_ID"
jsonPayload.jobId="JOB_ID"

    Replace the following:

    • DATA_SCAN_ID: the ID of the data insights scan.
    • JOB_ID: the ID of a specific scan job run.

  4. Select a time range (for example, Last 24 hours), and then click Run query.

  5. Expand a log entry and open jsonPayload > dataDocumentation > billableTokenUsage to inspect the token counts for that job.

gcloud

Read recent data insights scan log entries and format the token fields as a table:

gcloud logging read \
    'log_id("dataplex.googleapis.com/data_scan") AND jsonPayload.type="DATA_DOCUMENTATION" AND jsonPayload.dataDocumentation.billableTokenUsage.promptTokenCount:*' \
    --project=PROJECT_ID --freshness=1d --limit=20 \
    --format='table(timestamp, resource.labels.datascan_id:label=SCAN,
jsonPayload.jobId:label=JOB,
jsonPayload.dataDocumentation.billableTokenUsage.promptTokenCount:label=PROMPT,
jsonPayload.dataDocumentation.billableTokenUsage.cachedContentTokenCount:label=CACHED,
jsonPayload.dataDocumentation.billableTokenUsage.toolUsePromptTokenCount:label=TOOL_USE,
jsonPayload.dataDocumentation.billableTokenUsage.candidatesTokenCount:label=CANDIDATES,
jsonPayload.dataDocumentation.billableTokenUsage.thoughtsTokenCount:label=THOUGHTS)'

Replace PROJECT_ID with your Google Cloud project ID.

An empty column in the output table indicates 0 tokens for that field. To turn these fields into the metrics, use the formulas in How token usage logging works.

Query token usage using Observability Analytics

You can also run or customize the widget SQL queries directly on the Log Analytics page in the Google Cloud console.

The daily and hourly charts share the same chart query, which returns one row per scan job. Each chart widget bins and sums the rows by day (timeBinSize: 86400s) or by hour (timeBinSize: 3600s). If you edit a chart query and paste it back into a dashboard widget, preserve the column aliases timestamp, total_non_cached_input_tokens, total_cached_input_tokens, and total_output_tokens.

To filter a query to a single scan, append the following condition to the WHERE clause:

AND JSON_VALUE(resource.labels, '$.datascan_id') = 'DATA_SCAN_ID'

Replace DATA_SCAN_ID with the ID of your data insights scan.

Sample chart query (daily and hourly charts)

SELECT
  timestamp,
  GREATEST(
    IFNULL(SAFE_CAST(JSON_VALUE(json_payload, '$.dataDocumentation.billableTokenUsage.promptTokenCount') AS INT64), 0)
    - IFNULL(SAFE_CAST(JSON_VALUE(json_payload, '$.dataDocumentation.billableTokenUsage.cachedContentTokenCount') AS INT64), 0),
    0)
  + IFNULL(SAFE_CAST(JSON_VALUE(json_payload, '$.dataDocumentation.billableTokenUsage.toolUsePromptTokenCount') AS INT64), 0)
    AS total_non_cached_input_tokens,
  IFNULL(SAFE_CAST(JSON_VALUE(json_payload, '$.dataDocumentation.billableTokenUsage.cachedContentTokenCount') AS INT64), 0)
    AS total_cached_input_tokens,
  IFNULL(SAFE_CAST(JSON_VALUE(json_payload, '$.dataDocumentation.billableTokenUsage.candidatesTokenCount') AS INT64), 0)
  + IFNULL(SAFE_CAST(JSON_VALUE(json_payload, '$.dataDocumentation.billableTokenUsage.thoughtsTokenCount') AS INT64), 0)
    AS total_output_tokens
FROM `PROJECT_ID.global._Default._Default`
WHERE
  log_id = 'dataplex.googleapis.com/data_scan'
  AND JSON_VALUE(json_payload, '$.type') = 'DATA_DOCUMENTATION'
  AND JSON_VALUE(json_payload, '$.dataDocumentation.billableTokenUsage.promptTokenCount') IS NOT NULL

Replace PROJECT_ID with your Google Cloud project ID.

Sample per-scan-job query

SELECT
  timestamp,
  JSON_VALUE(resource.labels, '$.datascan_id') AS data_scan_id,
  JSON_VALUE(resource.labels, '$.location') AS location,
  JSON_VALUE(json_payload, '$.jobId') AS job_id,
  JSON_VALUE(json_payload, '$.state') AS state,
  GREATEST(
    IFNULL(SAFE_CAST(JSON_VALUE(json_payload, '$.dataDocumentation.billableTokenUsage.promptTokenCount') AS INT64), 0)
    - IFNULL(SAFE_CAST(JSON_VALUE(json_payload, '$.dataDocumentation.billableTokenUsage.cachedContentTokenCount') AS INT64), 0),
    0)
  + IFNULL(SAFE_CAST(JSON_VALUE(json_payload, '$.dataDocumentation.billableTokenUsage.toolUsePromptTokenCount') AS INT64), 0)
    AS non_cached_input_tokens,
  IFNULL(SAFE_CAST(JSON_VALUE(json_payload, '$.dataDocumentation.billableTokenUsage.cachedContentTokenCount') AS INT64), 0)
    AS cached_input_tokens,
  IFNULL(SAFE_CAST(JSON_VALUE(json_payload, '$.dataDocumentation.billableTokenUsage.candidatesTokenCount') AS INT64), 0)
  + IFNULL(SAFE_CAST(JSON_VALUE(json_payload, '$.dataDocumentation.billableTokenUsage.thoughtsTokenCount') AS INT64), 0)
    AS output_tokens
FROM `PROJECT_ID.global._Default._Default`
WHERE
  log_id = 'dataplex.googleapis.com/data_scan'
  AND JSON_VALUE(json_payload, '$.type') = 'DATA_DOCUMENTATION'
  AND JSON_VALUE(json_payload, '$.dataDocumentation.billableTokenUsage.promptTokenCount') IS NOT NULL
ORDER BY timestamp DESC
LIMIT 500

Replace PROJECT_ID with your Google Cloud project ID.

Create a dashboard using Observability Analytics

To visualize and track model token consumption over time, you can create a custom Monitoring dashboard using Observability Analytics.

This approach runs SQL queries directly against your data insights scan logs in the _Default log bucket. It offers several benefits:

  • Accuracy: Queries compute exact token counts directly from log entry JSON payloads rather than aggregated metric estimates.
  • Granular job history: In addition to aggregated charts, you can inspect individual scan jobs in a tabular view.
  • No additional cost: Upgrading the _Default log bucket to Observability Analytics doesn't incur additional charges.

This option creates a Monitoring dashboard with three widgets that run SQL queries on your logs:

  • Daily token usage (stacked bar chart)
  • Hourly token usage (line chart)
  • Token usage per scan job (table showing the latest 500 jobs)

The following dashboard JSON configuration uses the built-in @__project_id variable (@__project_id.global._Default._Default), which automatically queries the project in which you open the dashboard. You don't need to edit any project placeholders in the JSON before importing it.

Console

  1. In the Google Cloud console, go to the Monitoring > Dashboards page.

    Go to Dashboards

  2. Click Create dashboard.

  3. In the toolbar, click Settings, select JSON, and then select JSON Editor.

  4. Replace the existing contents of the editor with the dashboard JSON definition provided in the next section.

  5. Click Apply changes. With autosave enabled, Monitoring saves the dashboard as Data Insights - Token Usage.

  6. The first time you open the dashboard, each widget prompts you to confirm running the SQL query on the BigQuery engine for your project. Verify that the prompt lists your project, and then click Continue execution on each widget.

  7. Select a time range in the dashboard toolbar (for example, Last 7 days).

gcloud

  1. Save the dashboard JSON definition to a local file named dashboard.json.

  2. Create the dashboard in Monitoring:

    gcloud monitoring dashboards create \
    --project=PROJECT_ID \
    --config-from-file=dashboard.json

    Replace PROJECT_ID with your Google Cloud project ID.

  3. Open the Dashboards page in the Google Cloud console, select Insights Scan - Token Usage, and click Continue execution on each widget to authorize the SQL query execution.

Dashboard JSON definition

To create the dashboard, expand the following section and copy the JSON definition:

JSON definition

{
  "displayName": "Data Insights - Token Usage",
  "mosaicLayout": {
    "columns": 48,
    "tiles": [
      {
        "xPos": 0,
        "yPos": 0,
        "width": 24,
        "height": 16,
        "widget": {
          "title": "Daily token usage",
          "xyChart": {
            "chartOptions": {
              "mode": "COLOR",
              "displayHorizontal": false
            },
            "dataSets": [
              {
                "plotType": "STACKED_BAR",
                "targetAxis": "Y1",
                "breakdowns": [],
                "dimensions": [
                  {
                    "column": "timestamp",
                    "columnType": "TIMESTAMP",
                    "sortColumn": "timestamp",
                    "sortOrder": "SORT_ORDER_ASCENDING",
                    "timeBinSize": "86400s"
                  }
                ],
                "measures": [
                  {
                    "column": "total_non_cached_input_tokens",
                    "aggregationFunction": {
                      "type": "sum",
                      "parameters": []
                    }
                  },
                  {
                    "column": "total_cached_input_tokens",
                    "aggregationFunction": {
                      "type": "sum",
                      "parameters": []
                    }
                  },
                  {
                    "column": "total_output_tokens",
                    "aggregationFunction": {
                      "type": "sum",
                      "parameters": []
                    }
                  }
                ],
                "timeSeriesQuery": {
                  "opsAnalyticsQuery": {
                    "sql": "SELECT\n  timestamp,\n  GREATEST(\n    IFNULL(SAFE_CAST(JSON_VALUE(json_payload, '$.dataDocumentation.billableTokenUsage.promptTokenCount') AS INT64), 0)\n    - IFNULL(SAFE_CAST(JSON_VALUE(json_payload, '$.dataDocumentation.billableTokenUsage.cachedContentTokenCount') AS INT64), 0),\n    0)\n  + IFNULL(SAFE_CAST(JSON_VALUE(json_payload, '$.dataDocumentation.billableTokenUsage.toolUsePromptTokenCount') AS INT64), 0)\n    AS total_non_cached_input_tokens,\n  IFNULL(SAFE_CAST(JSON_VALUE(json_payload, '$.dataDocumentation.billableTokenUsage.cachedContentTokenCount') AS INT64), 0)\n    AS total_cached_input_tokens,\n  IFNULL(SAFE_CAST(JSON_VALUE(json_payload, '$.dataDocumentation.billableTokenUsage.candidatesTokenCount') AS INT64), 0)\n  + IFNULL(SAFE_CAST(JSON_VALUE(json_payload, '$.dataDocumentation.billableTokenUsage.thoughtsTokenCount') AS INT64), 0)\n    AS total_output_tokens\nFROM `@__project_id.global._Default._Default`\nWHERE\n  log_id = 'dataplex.googleapis.com/data_scan'\n  AND JSON_VALUE(json_payload, '$.type') = 'DATA_DOCUMENTATION'\n  AND JSON_VALUE(json_payload, '$.dataDocumentation.billableTokenUsage.promptTokenCount') IS NOT NULL"
                  }
                }
              }
            ],
            "yAxis": {
              "label": "tokens",
              "scale": "LINEAR"
            }
          }
        }
      },
      {
        "xPos": 24,
        "yPos": 0,
        "width": 24,
        "height": 16,
        "widget": {
          "title": "Hourly token usage",
          "xyChart": {
            "chartOptions": {
              "mode": "COLOR",
              "displayHorizontal": false
            },
            "dataSets": [
              {
                "plotType": "LINE",
                "targetAxis": "Y1",
                "breakdowns": [],
                "dimensions": [
                  {
                    "column": "timestamp",
                    "columnType": "TIMESTAMP",
                    "sortColumn": "timestamp",
                    "sortOrder": "SORT_ORDER_ASCENDING",
                    "timeBinSize": "3600s"
                  }
                ],
                "measures": [
                  {
                    "column": "total_non_cached_input_tokens",
                    "aggregationFunction": {
                      "type": "sum",
                      "parameters": []
                    }
                  },
                  {
                    "column": "total_cached_input_tokens",
                    "aggregationFunction": {
                      "type": "sum",
                      "parameters": []
                    }
                  },
                  {
                    "column": "total_output_tokens",
                    "aggregationFunction": {
                      "type": "sum",
                      "parameters": []
                    }
                  }
                ],
                "timeSeriesQuery": {
                  "opsAnalyticsQuery": {
                    "sql": "SELECT\n  timestamp,\n  GREATEST(\n    IFNULL(SAFE_CAST(JSON_VALUE(json_payload, '$.dataDocumentation.billableTokenUsage.promptTokenCount') AS INT64), 0)\n    - IFNULL(SAFE_CAST(JSON_VALUE(json_payload, '$.dataDocumentation.billableTokenUsage.cachedContentTokenCount') AS INT64), 0),\n    0)\n  + IFNULL(SAFE_CAST(JSON_VALUE(json_payload, '$.dataDocumentation.billableTokenUsage.toolUsePromptTokenCount') AS INT64), 0)\n    AS total_non_cached_input_tokens,\n  IFNULL(SAFE_CAST(JSON_VALUE(json_payload, '$.dataDocumentation.billableTokenUsage.cachedContentTokenCount') AS INT64), 0)\n    AS total_cached_input_tokens,\n  IFNULL(SAFE_CAST(JSON_VALUE(json_payload, '$.dataDocumentation.billableTokenUsage.candidatesTokenCount') AS INT64), 0)\n  + IFNULL(SAFE_CAST(JSON_VALUE(json_payload, '$.dataDocumentation.billableTokenUsage.thoughtsTokenCount') AS INT64), 0)\n    AS total_output_tokens\nFROM `@__project_id.global._Default._Default`\nWHERE\n  log_id = 'dataplex.googleapis.com/data_scan'\n  AND JSON_VALUE(json_payload, '$.type') = 'DATA_DOCUMENTATION'\n  AND JSON_VALUE(json_payload, '$.dataDocumentation.billableTokenUsage.promptTokenCount') IS NOT NULL"
                  }
                }
              }
            ],
            "yAxis": {
              "label": "tokens",
              "scale": "LINEAR"
            }
          }
        }
      },
      {
        "xPos": 0,
        "yPos": 16,
        "width": 48,
        "height": 20,
        "widget": {
          "title": "Token usage per scan job (latest 500)",
          "timeSeriesTable": {
            "columnSettings": [],
            "dataSets": [
              {
                "tableTemplate": "",
                "timeSeriesQuery": {
                  "opsAnalyticsQuery": {
                    "sql": "SELECT\n  timestamp,\n  JSON_VALUE(resource.labels, '$.datascan_id') AS data_scan_id,\n  JSON_VALUE(resource.labels, '$.location') AS location,\n  JSON_VALUE(json_payload, '$.jobId') AS job_id,\n  JSON_VALUE(json_payload, '$.state') AS state,\n  GREATEST(\n    IFNULL(SAFE_CAST(JSON_VALUE(json_payload, '$.dataDocumentation.billableTokenUsage.promptTokenCount') AS INT64), 0)\n    - IFNULL(SAFE_CAST(JSON_VALUE(json_payload, '$.dataDocumentation.billableTokenUsage.cachedContentTokenCount') AS INT64), 0),\n    0)\n  + IFNULL(SAFE_CAST(JSON_VALUE(json_payload, '$.dataDocumentation.billableTokenUsage.toolUsePromptTokenCount') AS INT64), 0)\n    AS non_cached_input_tokens,\n  IFNULL(SAFE_CAST(JSON_VALUE(json_payload, '$.dataDocumentation.billableTokenUsage.cachedContentTokenCount') AS INT64), 0)\n    AS cached_input_tokens,\n  IFNULL(SAFE_CAST(JSON_VALUE(json_payload, '$.dataDocumentation.billableTokenUsage.candidatesTokenCount') AS INT64), 0)\n  + IFNULL(SAFE_CAST(JSON_VALUE(json_payload, '$.dataDocumentation.billableTokenUsage.thoughtsTokenCount') AS INT64), 0)\n    AS output_tokens\nFROM `@__project_id.global._Default._Default`\nWHERE\n  log_id = 'dataplex.googleapis.com/data_scan'\n  AND JSON_VALUE(json_payload, '$.type') = 'DATA_DOCUMENTATION'\n  AND JSON_VALUE(json_payload, '$.dataDocumentation.billableTokenUsage.promptTokenCount') IS NOT NULL\nORDER BY timestamp DESC\nLIMIT 500"
                  }
                }
              }
            ],
            "metricVisualization": "NUMBER"
          }
        }
      }
    ]
  }
}

Limitations

  • Token metrics are complete only if dataplex.googleapis.com/data_scan logs aren't excluded or routed away from the _Default log bucket.
  • Only succeeded jobs with token metadata are counted. Log entries without a billableTokenUsage block are excluded from the queries and metrics.
  • Use these dashboards to monitor usage trends and estimate costs, not to reconcile an official Cloud Billing invoice.
  • Each Google Cloud project requires its own dashboard setup.
  • SQL queries don't return log entries written before Observability Analytics was enabled on the _Default bucket.
  • The per-job table widget displays at most the 500 most recent jobs within the selected time range. To inspect older jobs, query Logs Explorer.

What's next