Analyze causes of personally identifiable information (PII) leak

Data lineage in Knowledge Catalog (formerly Dataplex Universal Catalog) lets you trace how data moves across tables and views to identify where personally identifiable information (PII) crosses security boundaries. In this tutorial, you investigate a scenario where consumer email addresses (user_email) intended for account creation, invoicing, and shipping leak into an organization-wide analytics table. You use column-level lineage to trace the exposed column from the public order_status_stats table through the status_counts_by_user_v view back to the transformation process that joins it with identity data.

Before you begin

Before starting this tutorial, you must enable the required APIs, grant the required Identity and Access Management roles, and run the sample data transformations in Dataform to populate the dataset. For complete setup instructions, see Prerequisites and setup.

Analyze personal information leak with Lineage Explorer

After you prepare the dataset, trace the personal information leak using the BigQuery Lineage tab.

In this example, you trace the user_email column from the public table back to its source:

  1. In Google Cloud console, go to the BigQuery page.
  2. Use the search field to find the order_status_stats table.
  3. Click the Lineage tab.
  4. In the Lineage Explorer pane, do the following:
    1. In the Column Level Lineage section, select the user_email column name from the list.
    2. In the Direction section, select the Upstream direction.
    3. Click Apply.
  5. Follow the graph back one step. The graph shows that the email is pulled from the status_counts_by_user_v intermediate view.
  6. Click the process node between the view and its upstream dependencies. The process node shows that a join operation occurs between anonymized order data and a table containing identity information.

The lineage proves that personal information crosses from a restricted functional table into a broader analytics schema, where unauthorized users can see it.

For more information on visualizing data with data lineage graph, see Lineage graph view.