Track data lineage with Knowledge Catalog

This page explains how to track data lineage for your Looker (Google Cloud core) instance using Knowledge Catalog.

Data lineage is the process of tracking how data flows through your systems. By integrating Looker (Google Cloud core) with Knowledge Catalog, you can visualize the end-to-end journey of your data from its source in BigQuery through the Looker semantic layer (LookML views and Explores) to downstream consumption in Looker dashboards, Looks, LookML dashboards, and Conversational Analytics agents.

This visibility provides bidirectional lineage tracking for data governance, impact analysis, and audit workflows:

• Downstream impact analysis: Predict which dashboards, Explores, and downstream consumers are affected before you make schema or model changes. For example, before dropping a column in a BigQuery table, you can check the lineage graph to see which Looker dashboards might break.
• Upstream data audit: Trace the origin, transformations, and chain of custody of the metrics that are displayed in executive dashboards and Looks.

Before you begin

To use data lineage with Looker (Google Cloud core), you must meet the following prerequisites:

1. Looker (Google Cloud core): Data lineage is supported for all edition types of Looker (Google Cloud core) instances. Looker (original) instances don't integrate with Knowledge Catalog.
2. Data source: Data lineage is supported only for BigQuery data sources.

3. Required Permissions: To view lineage graphs, you need the following IAM roles:

   • Looker Schema Viewer (roles/looker.schemaViewer) on the project that's hosting the Looker (Google Cloud core) instance. This role is required to view Looker (Google Cloud core)-specific metadata that populates the side panel of a lineage graph.

     Note: You can add the Looker Schema Viewer role only by using Google Cloud CLI.

     gcloud projects add-iam-policy-binding YOUR_PROJECT_ID
     --member=PRINCIPAL_TYPE:PRINCIPAL_ID
     --role=roles/looker.schemaViewer

     For more information, see the add-iam-policy-binding documentation.

   • Dataplex Catalog Viewer (roles/dataplex.catalogViewer) or equivalent permissions to view Knowledge Catalog assets

   • Data Lineage Viewer (roles/datalineage.viewer) to read lineage data

Caution: SQL table references in LookML must match the exact casing of the corresponding BigQuery tables. For example, if a BigQuery table name is lowercase, the LookML reference must also be lowercase. Case discrepancies prevent downstream lineage from populating in the lineage graph.

Enable data lineage

To enable data lineage, complete the following steps in the Google Cloud console:

1. Enable the Data Lineage API: You must enable the Data Lineage API (datalineage.googleapis.com) on any Google Cloud projects that host your Looker (Google Cloud core) instance and your BigQuery data.

   Enable the Data Lineage API

2. Ensure that service-level lineage ingestion is enabled: Ensure that the Looker (Google Cloud core) service-level lineage integration is enabled. Lineage ingestion is enabled by default for Looker (Google Cloud core) instances in projects where the Data Lineage API is enabled.

Note: Data lineage ingestion operates independently of the Looker (Google Cloud core) Knowledge Catalog metadata integration opt-out setting and requires no configuration within the Looker Admin panel. Any previously configured preview settings in the Looker Admin panel during the preview program are no longer applied to the data lineage integration.

To view the lineage configuration for a Google Cloud project, see the Get current configuration documentation. If the integration with Looker (Google Cloud core) is disabled, the command will return output similar to the following:

    {
    "name": "projects/123456789012/locations/global/config",
    "ingestion": {
      "rules": [
        {
          "integrationSelector": {
            "integration": "LOOKER_CORE"
          },
          "lineageEnablement": {
            "enabled": false
          }
        }
      ]
    },
    "etag": "Wb35wDxTTLd6Z+QAL+Yd4g=="
  }

The project ID in the response will reflect the ID in your request. The etag field is a checksum that is generated by the server and that is based on the current value of the configuration.

View data lineage

Once the integration is enabled and the initial sync is complete (which may take up to 24 hours), you can view lineage in the Knowledge Catalog console.

1. In the Google Cloud console, go to the Knowledge Catalog page.

   Go to Knowledge Catalog

2. Click Search in the left navigation pane.
3. Search for a BigQuery table or a Looker (Google Cloud core) asset (like a dashboard or an Explore).
   • You can use the Filters panel to filter by Systems > Looker.
4. Click the name of the asset to open its details page.
5. Click the Lineage tab.

The lineage graph displays the asset as a central node, with upstream sources to the left and downstream consumers to the right. This bidirectional layout lets you evaluate downstream impacts to the right, or trace upstream provenance and source tables to the left.

Interpret the lineage graph

The lineage graph consists of nodes and links:

• Nodes: Represent data assets. Supported Looker (Google Cloud core) assets include the following:
  • Looker dashboard
  • Looker dashboard element (tile)
  • Looker Look
  • LookML dashboard
  • Looker agent (Conversational Analytics agent)
  • LookML Explore
  • LookML view
• Links: Represent the flow of data. For example, a link from a BigQuery table to a LookML view indicates that the view selects data from that table.

Identify asset owners

To find out who owns a downstream asset that might be impacted by a change, follow these steps:

1. In the lineage graph, click the node for the asset (for example, a Looker dashboard).
2. An information panel opens on the right side of the screen.
3. Look for the Aspects section to find the Owner (email address).

Filter the lineage list

When in the List view for lineage, you can filter entities by property name or value. For example, complex LookML models can generate large lineage graphs with many intermediate entities. To focus on business impact, you could filter by entity type by following these steps:

1. In the Lineage tab, toggle to the List view.
2. Locate the Filter options in the toolbar.
3. In the Entity filter, enter Looker Dashboard, LookML Dashboard, and Looker Look to filter out intermediate types like LookML View and LookML Explore.

The entity list updates to show only the selected asset types, making it easier to identify user-facing content.

Limitations

The Looker (Google Cloud core) lineage integration has the following limitations:

• Instance types: Lineage is supported only for Looker (Google Cloud core) instances. Looker (original) instances don't support lineage.
• Data Sources: Lineage is supported only for BigQuery data sources.
• Granularity: Lineage is provided at the object level (table, view, Explore, Looker dashboard, LookML dashboard, and Conversational Analytics agent). Column-level lineage is not supported.
• Latency: Lineage data is not real-time. The synchronization process typically takes four hours. However, synchronization may take up to eight hours, depending on the timing of Looker metadata exports and lineage data consumption. Changes made in Looker or BigQuery may take some time to appear in the lineage graph. Initial sync after enabling the integration may take up to 24 hours.
• Complex SQL: LookML that's defined with complex custom SQL (for example, Liquid templates, derived tables with complex joins) may not be fully parsed, potentially resulting in disconnected nodes.

Pricing

Data lineage for Looker (Google Cloud core) is generally available. Pricing will begin at a future date with advance notice provided through a Mandatory Service Announcement (MSA).

For more information, see the Knowledge Catalog pricing page.

What's next

• Manage metadata resources with Knowledge Catalog
• Learn more about data lineage