Data lineage in Knowledge Catalog lets you track downstream dependencies for tables and views in your BigQuery project. You can then identify assets that aren't actively used as sources for downstream jobs or workflows and flag them as candidates for deletion to reduce storage costs.
In this tutorial, you manage a BigQuery project with datasets derived from the Quarterly Census of Employment and Wages (QCEW). The project contains numerous tables and views for dashboards and analyses, such as tracking employment trends or wage comparisons, which increases your storage costs. By checking each asset for downstream links, you can systematically flag unused resources for deletion.
Set up Lineage Explorer
To complete the use case, first set up the environment and run the data transformations by connecting a remote repository to Dataform. This repository contains the code necessary to set up the dataset and transform the data. For more information, see Prerequisites and setup.
After you finish the environment setup, use BigQuery and Lineage Explorer to visually identify assets without downstream dependencies.
Identify unused assets and optimize costs
After the dataset is prepared, identify assets that lack downstream dependencies. While you can do this visually for individual tables, you can also use a programmatic approach for larger datasets.
In this example, compare active assets with standalone assets to determine which can be safely removed:
- In Google Cloud console, go to the BigQuery page.
- Use the search field to find a standalone table (for example, an old analysis table from a specific year).
- Click the Lineage tab.
- In the Lineage Explorer pane, do the following:
- In the Direction section, select the Downstream direction.
- Click Apply.
Observe the graph. If there are no downstream nodes, the asset is not being used as a source for other tracked BigQuery jobs. For more information about visualizing data with the data lineage graph, see Lineage graph view.