About multi-region lineage search

Multi-region lineage search in Knowledge Catalog (formerly Dataplex Universal Catalog) lets you trace data assets across Google Cloud projects and regions to build complete lineage graphs. This unified view helps you maintain data governance and analyze dependencies across your organization when pipelines span geographic boundaries. You can discover and connect cross-regional graphs by using recommended server-side automation with the searchLineageStreaming API (Preview) or client-side fan-out with the searchLinks API. For instructions on using the streaming API, see Search multi-region lineage using server-side automation.

By default, data lineage in Knowledge Catalog is a regional service. Whenever your data moves or transforms, the resulting lineage data, such as links, processes, and events, is stored in the specific region where that action took place. However, real-world data pipelines frequently span multiple Google Cloud projects and regions. For example, you might have a BigQuery table in us-central1 that copies data to a storage bucket in europe-west1, requiring a multi-region search to trace your data assets across these boundaries.

To understand multi-region lineage discovery, it helps to understand how Knowledge Catalog handles graph traversal:

  • Root criteria: The starting point of your lineage search, defined by one or more asset names (such as a BigQuery table or a Pub/Sub topic) or fine-grained column fields.

  • Direction: The orientation of the graph traversal relative to the root criteria. You can search upstream (to see where your data came from) or downstream (to see where your data is going).

  • Breadth-first search: The architectural mechanism used to find connected nodes. The search traverses the lineage graph layer by layer, accurately calculating the execution depth of each connected asset across regional boundaries.

How do the multi-region search methods compare?

While both methods let you to piece together a cross-regional view of your data, they handle the heavy lifting differently:

Feature Server-side automation
searchLineageStreaming API		 Client-side fan-out
searchLinks API
Execution model Server-side automation: The Google Cloud routing engine traverses multiple regions natively. Client-side orchestration: Your application script must manually loop and manage requests.
Request overhead Single API request: A single HTTP POST call starts the multi-region search. Multiple API requests: Requires a separate HTTP call for every region and every graph layer.
Response handling Real-time stream: Results are pushed to the client as they are found, preventing timeouts. Static payloads: Individual JSON arrays must be received, collected, and merged manually.
Deep graphs (greater than 2 layers) Handles deep, nested lineage graphs automatically up to 100 levels. Suffers from the N+1 query problem; requires iterative, slow round-trips from the client.

Choose the right multi-region search method

Review the following scenarios to determine which multi-region search method fits your workload.

Choose the streaming API method for the following use cases:

  • Trace deep or complex graphs: Your data moves through multiple intermediate tables, buckets, or pipelines across different regions, requiring multi-level traversal (maxDepth greater than 2).

  • Track column-level lineage: You want to track fields across regions or leverage wildcard (*) searches to pull all column dependencies at once.

  • Maintain lightweight code: You prefer to make a single API call and let Google Cloud handle the routing, deduplication, and graph assembly.

  • Require pipeline metadata: You want to optionally retrieve structural details about the processes running your pipelines in the same request payload.

Choose the client-side fan-out method for the following scenarios:

  • You only trace shallow, single-hop lineage: Your lineage graph isn't complex, and you only need to look up direct parent or child links (maxDepth equals 1) across a small, fixed number of known regions.

  • You are working within strict legacy systems: You have an existing data-governance application built heavily around the standard SearchLinks endpoint and want to maintain structural backward compatibility without implementing streaming response consumers.

What's next