Troubleshoot data lineage issues

This document helps you troubleshoot and resolve the most common issues with Knowledge Catalog (formerly Dataplex Universal Catalog) data lineage graphs not showing up. Resolving these issues ensures that you can successfully track data movement, understand data origins, and debug data pipelines.

Project types

Data assets can reside in different projects. The following is a summary of possible projects and their asset names.

BigQuery storage project

This project stores your BigQuery data assets. You can find it in asset details as a part of Table ID, before the first dot.

Figure 1. The name of a BigQuery storage project.

Compute project

This project stores the data lineage metadata. For BigQuery, this is where you run a job. If you run a job using the Google Cloud console, you can find the compute project name in the project selector:

Figure 2. The name of a compute project that runs BigQuery jobs.

When sending requests to the BigQuery API, specify the compute project in the URL, for example:

POST /bigquery/v2/projects/docs-compute/jobs HTTP/1.1
Host: bigquery.googleapis.com
User-Agent: Go-http-client/1.1
Authorization: <REDACTED 1031 BYTES>
Accept-Encoding: gzip
{
  "configuration": {
    "query": {
      "useLegacySql": false,
      "query": "CREATE OR REPLACE TABLE `docs-target.dataset.target-002` AS SELECT * FROM `docs-source.dataset.source-002`;"
    }
  },
  "jobReference": {
    "projectId": "docs-compute",
    "jobId": "docs-compute-job-id",
    "location": "us",
  }
}

Active project

This is the project from which you are viewing the data lineage. The Google Cloud console shows the active project in the project selector. If you're using the API, the active project is the project from which you're making API calls.

Figure 3. The active project in the the Google Cloud console.

BigQuery data lineage not showing

The following issue occurs after running a BigQuery job. In this case, the problem can be caused by three scenarios:

• The Data Lineage API is disabled in the active project or the compute project.
• You don't have the Data Lineage Viewer role (roles/datalineage.viewer) in the active or the compute project.
• The data lineage hasn't arrived yet. Depending on the volume and complexity of the data being processed, it can take from 30 minutes up to 24 hours for the data lineage to display.

If you see the message "Fetching lineage failed due to missing permissions.", you are missing permissions on the active project. Otherwise, you are missing permissions on the compute project.

Figure 4. Example of lineage not showing in the BigQuery UI.

To resolve this issue, check if the Data Lineage API is enabled for the compute project. After you enable the API, you need to run a job to see the data lineage. Depending on the volume and complexity of the data being processed, it can take from 30 minutes up to 24 hours for the data lineage to display.

Next, check if the Data Lineage API is enabled for the active project.

When the Data Lineage API is enabled, grant the Data Lineage Viewer role (roles/datalineage.viewer) in both the active and the compute projects.

BigQuery process metadata not showing

The following issue occurs when you open the table details pane, which doesn't show all the details such as the SQL statement or the Process type property. This happens even though the data lineage displays properly.

This can happen when you don't have permissions to see metadata in the compute project.

Example:

• BigQuery source table: docs-source.dataset.source-001
• BigQuery target table: docs-target.dataset.target-001
• Data lineage between docs-source.dataset.source-001 and docs-target.dataset.target-001 in compute project docs-compute
• You have the Data Lineage Viewer role for the active and compute docs-compute projects.

Clicking the BigQuery process details displays the following message in the Google Cloud console:

You don't have permission to view BigQuery process metadata in project X.

Figure 5. Example of BigQuery process details not showing in the BigQuery UI.

To resolve this issue, grant the user the bigquery.jobs.get permission (for example, included in the BigQuery Resource Viewer role) in the compute project.

BigQuery table details not showing

The following issue occurs when you open the table details pane, which shows only the Fully qualified name property. This happens even though the data lineage displays properly. This can happen when you don't have all the required permissions in the table's storage projects.

Example:

• BigQuery table docs-source.dataset.source-001
• BigQuery table docs-target.dataset.target-001
• Data lineage between docs-source.dataset.source-001 and docs-target.dataset.target-001 with compute project docs-compute
• You have the Data Lineage Viewer role for the active and compute docs-compute projects.

In this case, when you click BigQuery node details, you see the message Entry with this fully qualified name is not available in Knowledge Catalog or you do not have permissions to view it.

Figure 6. Example of BigQuery table details not showing in the BigQuery UI.

To resolve this issue, grant the bigquery.tables.get permission (for example, included in the BigQuery Data Viewer role) in the storage project.

Unexpected Knowledge Catalog Premium Processing charges

You disabled the Dataplex API (dataplex.googleapis.com) to stop charges, but you continue to see daily charges for the "Knowledge Catalog Premium Processing" SKU.

This issue can occur if the Data Lineage API (datalineage.googleapis.com) remains enabled. The Data Lineage API is billed under the "Knowledge Catalog Premium Processing" SKU, but is managed as a separate API in the Google Cloud console. Disabling the Dataplex API doesn't disable the Data Lineage API or stop its charges.

To identify if data lineage is the source of the charges, check your Cloud Billing report for the label goog-dataplex-workload-type with the value LINEAGE.

To stop the charges, turn off data lineage by disabling the Data Lineage API in your projects.