Method: projects.locations.searchLineageStreaming

Retrieves a streaming response of lineage links connected to the requested assets by performing a breadth-first search in the given direction. Links represent the data flow between source (upstream) and target (downstream) assets in transformation pipelines. Links are stored in the same project as the Lineage Events that create them. This method retrieves links from all valid locations provided in the request. This method supports Column-Level Lineage (CLL) along with wildcard support to retrieve all CLL for an Entity FQN.

Following permissions are required to retrieve links: * datalineage.events.get permission for the project where the link is stored for entity-level lineage. * datalineage.events.getFields permission for the project where the link is stored for column-level lineage.

This method also returns processes that created the links if explicitly requested by setting maxProcessPerLink is non-zero and full process details are requested via links.processes.process in the FieldMask.

Permission required to retrieve processes: * datalineage.processes.get permission for the project where the process is stored.

HTTP request


POST https://datalineage.googleapis.com/v1/{parent}:searchLineageStreaming

Path parameters

Parameters
parent

string

Required. The project and location to initiate the search from. It takes the form projects/{project}/locations/{location}.

Request body

The request body contains data with the following structure:

JSON representation 
{
  "locations": [
    string
  ],
  "rootCriteria": {
    object (RootCriteria)
  },
  "direction": enum (SearchDirection),
  "filters": {
    object (SearchFilters)
  },
  "limits": {
    object (SearchLimits)
  }
}
Fields
locations[]

string

Required. The locations to search in. This list should contain the location from the parent field.
rootCriteria

object (RootCriteria)

Required. Criteria for the root of the search.
direction

enum (SearchDirection)

Required. Direction of the search.
filters

object (SearchFilters)

Optional. Filters for the search.
limits

object (SearchLimits)

Optional. Limits for the search.

Response body

Response message for locations.searchLineageStreaming.

If successful, the response body contains data with the following structure:

JSON representation 
{
  "links": [
    {
      object (LineageLink)
    }
  ],
  "unreachable": [
    string
  ]
}
Fields
unreachable[]

string

Unordered list. Unreachable resources. If non-empty, the result set might be incomplete.

Currently, only locations are supported.

Format: projects/[PROJECT_NUMBER]/locations/[LOCATION] Example: projects/123456789/locations/us-east1

Authorization scopes

Requires one of the following OAuth scopes:

  • https://www.googleapis.com/auth/cloud-platform
  • https://www.googleapis.com/auth/datalineage.readonly
  • https://www.googleapis.com/auth/datalineage.read-write

For more information, see the Authentication Overview.

IAM Permissions

Requires the following IAM permission on the parent resource:

  • datalineage.locations.searchLinks

For more information, see the IAM documentation.

RootCriteria

Criteria for the root of the search.

JSON representation 
{

  // Union field criteria can be only one of the following:
  "entities": {
    object (MultipleEntityReference)
  }
  // End of list of possible types for union field criteria.
}
Fields
Union field criteria. Criteria for the root of the search. criteria can be only one of the following:
entities

object (MultipleEntityReference)

Optional. The entities to initiate the search from. Entities can be specified by FQN only, or by FQN and field. To search by FQN and all available fields for that FQN, use the wildcard * as the field value.

SearchDirection

Direction of the search.

Enums
SEARCH_DIRECTION_UNSPECIFIED Direction is unspecified.
DOWNSTREAM Retrieve links that lead from the specified asset to downstream assets.
UPSTREAM Retrieve links that lead from upstream assets to the specified asset.

SearchFilters

Filters for the search.

JSON representation 
{
  "dependencyTypes": [
    enum (DependencyType)
  ],
  "entitySet": enum (EntitySet),
  "timeRange": {
    object (Interval)
  }
}
Fields
dependencyTypes[]

enum (DependencyType)

Optional. Types of dependencies between entities to retrieve. If unspecified, all dependency types are returned.
entitySet

enum (EntitySet)

Optional. Entity set restriction. If unspecified, the method returns all entities.
timeRange

object (Interval)

Optional. Time interval to search for lineage. If unspecified, all lineage is returned. Currently, at most one of startTime and endTime can be set.

EntitySet

Entity set restriction.

Enums
ENTITY_SET_UNSPECIFIED The entity set is unspecified. Returns all the data.
ENTITIES Returns entities with only FQN specified. For example, entities with the field field set are not returned.

Interval

Represents a time interval, encoded as a Timestamp start (inclusive) and a Timestamp end (exclusive).

The start must be less than or equal to the end. When the start equals the end, the interval is empty (matches no time). When both start and end are unspecified, the interval matches any time.

JSON representation 
{
  "startTime": string,
  "endTime": string
}
Fields
startTime

string (Timestamp format)

Optional. Inclusive start of the interval.

If specified, a Timestamp matching this interval will have to be the same or after the start.

Uses RFC 3339, where generated output will always be Z-normalized and use 0, 3, 6 or 9 fractional digits. Offsets other than "Z" are also accepted. Examples: "2014-10-02T15:01:23Z", "2014-10-02T15:01:23.045123456Z" or "2014-10-02T15:01:23+05:30".
endTime

string (Timestamp format)

Optional. Exclusive end of the interval.

If specified, a Timestamp matching this interval will have to be before the end.

Uses RFC 3339, where generated output will always be Z-normalized and use 0, 3, 6 or 9 fractional digits. Offsets other than "Z" are also accepted. Examples: "2014-10-02T15:01:23Z", "2014-10-02T15:01:23.045123456Z" or "2014-10-02T15:01:23+05:30".

SearchLimits

Limits for the search results.

JSON representation 
{
  "maxDepth": integer,
  "maxResults": integer,
  "maxProcessPerLink": integer
}
Fields
maxDepth

integer

Optional. The maximum depth of the search. The default value is 5 and maximum value is 100.
maxResults

integer

Optional. The maximum number of links to return in the response. The default value is 1_000 and the maximum value is 10_000.

LineageProcess

Process metadata for the link.

JSON representation 
{
  "process": {
    object (Process)
  }
}
Fields
process

object (Process)

Process that created the link.

DependencyInfo

Dependency info describes how one entity is dependent on another.

JSON representation 
{
  "dependencyType": enum (DependencyType)
}
Fields
dependencyType

enum (DependencyType)

The type of dependency.