# Manage pipelines

[BigQuery pipelines](https://docs.cloud.google.com/bigquery/docs/pipelines-introduction.md.txt)
let you inspect run history, configure failure alerts, and control code assets
powered by
[Dataform](https://docs.cloud.google.com/dataform/docs/overview.md.txt).
You can also enrich workflow metadata in
[Knowledge Catalog](https://docs.cloud.google.com/knowledge-catalog/docs/introduction.md.txt)
to track data quality and delete obsolete workflows to keep your project
organized. Depending on where your team stores code, you can access pipelines
in **Files and Folders** , including user folders and team folders, or in
**BigQuery Studio Git repositories (Git Folders)**
([Preview](https://cloud.google.com/products#product-launch-stages)).

## Before you begin

1. [Create a BigQuery pipeline](https://docs.cloud.google.com/bigquery/docs/create-pipelines.md.txt).
2. To manage pipeline metadata in Knowledge Catalog, ensure that the [Dataplex API](https://docs.cloud.google.com/knowledge-catalog/docs/enable-api.md.txt) is enabled in your Google Cloud project.

### Required roles


To get the permissions that
you need to manage pipelines,

ask your administrator to grant you the
following IAM roles:

- To view and run pipelines:
  - [Dataform Viewer](https://docs.cloud.google.com/iam/docs/roles-permissions/dataform.md.txt#dataform.Viewer) (`roles/dataform.Viewer`) on the project
  - [BigQuery Job User](https://docs.cloud.google.com/iam/docs/roles-permissions/bigquery.md.txt#bigquery.jobUser) (`roles/bigquery.jobUser`) on the project
- To run pipelines with user credentials for a Google Account: [BigQuery Data Editor](https://docs.cloud.google.com/iam/docs/roles-permissions/bigquery.md.txt#bigquery.dataEditor) (`roles/bigquery.dataEditor`) on the project or specific BigQuery datasets
- To delete pipelines: [Dataform Admin](https://docs.cloud.google.com/iam/docs/roles-permissions/dataform.md.txt#dataform.Admin) (`roles/dataform.Admin`) on the pipeline
- To manage pipelines in user folders:
  - [Code Owner](https://docs.cloud.google.com/iam/docs/roles-permissions/dataform.md.txt#dataform.codeOwner) (`roles/dataform.codeOwner`) on the folder
  - [Code Editor](https://docs.cloud.google.com/iam/docs/roles-permissions/dataform.md.txt#dataform.codeEditor) (`roles/dataform.codeEditor`) on the folder
  - [Code Viewer](https://docs.cloud.google.com/iam/docs/roles-permissions/dataform.md.txt#dataform.codeViewer) (`roles/dataform.codeViewer`) on the folder
- To manage pipelines in team folders:
  - [Team Folder Owner](https://docs.cloud.google.com/iam/docs/roles-permissions/dataform.md.txt#dataform.teamFolderOwner) (`roles/dataform.teamFolderOwner`) on the team folder
  - [Team Folder Contributor](https://docs.cloud.google.com/iam/docs/roles-permissions/dataform.md.txt#dataform.teamFolderContributor) (`roles/dataform.teamFolderContributor`) on the team folder
  - [Team Folder Viewer](https://docs.cloud.google.com/iam/docs/roles-permissions/dataform.md.txt#dataform.teamFolderViewer) (`roles/dataform.teamFolderViewer`) on the team folder
- To manage pipelines in Git repositories: [Developer Connect OAuth User](https://docs.cloud.google.com/iam/docs/roles-permissions/developerconnect.md.txt#developerconnect.oauthUser) (`roles/developerconnect.oauthUser`) on the project
- To view and manage metadata in Knowledge Catalog: [Dataplex Catalog Editor](https://docs.cloud.google.com/iam/docs/roles-permissions/dataplex.md.txt#dataplex.catalogEditor) (`roles/dataplex.catalogEditor`) on the project or `@bigquery` entry group
- To view data lineage for pipelines: [Data Lineage Viewer](https://docs.cloud.google.com/iam/docs/roles-permissions/datalineage.md.txt#datalineage.viewer) (`roles/datalineage.viewer`) on the project
- To open pipelines from the lineage graph: [Dataform Viewer](https://docs.cloud.google.com/iam/docs/roles-permissions/dataform.md.txt#dataform.viewer) (`roles/dataform.viewer`) on the project that contains the pipeline


For more information about granting roles, see [Manage access to projects, folders, and organizations](https://docs.cloud.google.com/iam/docs/granting-changing-revoking-access.md.txt).


You might also be able to get
the required permissions through [custom
roles](https://docs.cloud.google.com/iam/docs/creating-custom-roles) or other [predefined
roles](https://docs.cloud.google.com/iam/docs/roles-overview#predefined).

For more information about Dataform IAM, see
[Control access with IAM](https://docs.cloud.google.com/dataform/docs/access-control.md.txt).
For more information about roles for folders, see
[Create and manage folders](https://docs.cloud.google.com/bigquery/docs/create-manage-folders.md.txt#required_roles).
For more information about roles for Git repositories, see
[Manage code with BigQuery Studio Git repositories](https://docs.cloud.google.com/bigquery/docs/git-repositories.md.txt#required_roles).

> [!NOTE]
> **Note:** When you create a pipeline, BigQuery grants you the [Dataform Admin role](https://docs.cloud.google.com/dataform/docs/access-control.md.txt#dataform.admin) (`roles/dataform.admin`) on that pipeline. All users with the Dataform Admin role granted on the Google Cloud project have owner access to all pipelines created in the project.

If you use a custom service account to run pipelines, you must grant the following roles to that service account:

- [BigQuery Job User](https://docs.cloud.google.com/iam/docs/roles-permissions/bigquery.md.txt#bigquery.jobUser) (`roles/bigquery.jobUser`) on the project
- [BigQuery Data Editor](https://docs.cloud.google.com/iam/docs/roles-permissions/bigquery.md.txt#bigquery.dataEditor) (`roles/bigquery.dataEditor`) on the project or specific BigQuery datasets
- [Dataplex Catalog Editor](https://docs.cloud.google.com/iam/docs/roles-permissions/dataplex.md.txt#dataplex.catalogEditor) (`roles/dataplex.catalogEditor`) on the project or `@bigquery` entry group
- [Data Lineage Events Producer](https://docs.cloud.google.com/iam/docs/roles-permissions/datalineage.md.txt#datalineage.producer) (`roles/datalineage.producer`) on the project that contains the pipeline, to [report data lineage](https://docs.cloud.google.com/bigquery/docs/manage-pipelines.md.txt#data-lineage)

## View pipelines

You can view and inspect pipelines stored in folders or Git folders
in the **Files** pane. You can view standalone pipelines and pipelines
stored in folders in the **Explorer** pane. Pipelines stored in Git folders
are not displayed in the **Explorer** pane.

### View pipelines in the Files pane

Pipelines stored in folders or in Git folders appear directly in the **Files** pane.

To view a pipeline stored in a folder or Git folder, follow these steps:

1. In the Google Cloud console, go to the
   **BigQuery** page.

   [Go to BigQuery](https://console.cloud.google.com/bigquery)
2. In the left pane, click **Files** to open the file browser.

   If you don't see the left pane, click **Expand left pane** to open the pane.
3. Expand your **User** folder, a **Team folder**, or a connected Git repository folder.

   Pipelines are displayed with a **Pipeline** icon instead of a standard folder icon.
4. Click a pipeline folder to open the **Pipeline Viewer**.

   The **Pipeline Viewer** displays the compiled Directed Acyclic Graph (DAG) of your pipeline tasks, execution status, and configuration tabs.
5. Expand the pipeline directory in the file browser to browse nested directories (such as `definitions/`) and individual task files.

> [!NOTE]
> **Note:** If a connected Git repository contains `workflow_settings.yaml` in its root directory, BigQuery Studio recognizes it as a root-level pipeline. The Git repository folder itself displays the **Pipeline** icon. When you click the Git repository folder, the **Pipeline Viewer** opens, displaying the compiled DAG for the entire repository.

### View pipelines in the Explorer pane

To view a list of standalone pipelines and pipelines stored in folders,
do the following:

1. In the Google Cloud console, go to the
   **BigQuery** page.

   [Go to BigQuery](https://console.cloud.google.com/bigquery)
2. In the left pane, click **Explorer**:

   ![Highlighted button for the Explorer pane.](https://docs.cloud.google.com/static/bigquery/images/explorer-tab.png)

   If you don't see the left pane, click **Expand left pane** to open the pane.
3. In the **Explorer** pane, expand your project and click **Pipelines**.

4. Select a pipeline to open it in the **Pipeline Viewer**.

## View past manual runs

To view past manual runs of a selected pipeline, follow these steps:

1. In the Google Cloud console, go to the
   **BigQuery** page.

   [Go to BigQuery](https://console.cloud.google.com/bigquery)
2. In the left pane, click **Explorer**:

   ![Highlighted button for the Explorer pane.](https://docs.cloud.google.com/static/bigquery/images/explorer-tab.png)
3. In the **Explorer** pane, expand your project, click **Pipelines**,
   and then select a pipeline.

4. Click **Executions**.

5. Optional: To refresh the list of past runs, click **Refresh**.

## Configure alerts for failed pipeline runs

Each pipeline has a corresponding Dataform repository ID.
Each BigQuery pipeline run is logged in
[Cloud Logging](https://docs.cloud.google.com/logging/docs.md.txt) using the corresponding
Dataform repository ID. You can use Cloud Monitoring to observe
trends in Cloud Logging logs for
BigQuery pipeline runs and to notify
you when conditions you describe occur.

To receive alerts when a BigQuery pipeline run fails,
you can create a log-based alerting policy for the corresponding
Dataform repository ID. For instructions, see
[Configure alerts for failed workflow invocations](https://docs.cloud.google.com/dataform/docs/monitor-runs.md.txt#configure-alerts-failed-workflow-invocations).

To find the Dataform repository ID of your pipeline, do the following:

1. In the Google Cloud console, go to the
   **BigQuery** page.

   [Go to BigQuery](https://console.cloud.google.com/bigquery)
2. In the left pane, click **Explorer**:

   ![Highlighted button for the Explorer pane.](https://docs.cloud.google.com/static/bigquery/images/explorer-tab.png)
3. In the **Explorer** pane, expand your project, click **Pipelines**,
   and then select a pipeline.

4. Click **Settings**.

   The Dataform repository ID of your pipeline is displayed at
   the bottom of the **Settings** tab.

## Delete a pipeline

To permanently delete a pipeline, follow these steps:

1. In the Google Cloud console, go to the **BigQuery** page. [Go to BigQuery](https://console.cloud.google.com/bigquery)

- To delete a pipeline stored in a folder or Git folder, do the following:

  1. In the left pane, click **Files**.

  2. In the file tree, find the pipeline folder that you want to delete.

  3. Click
     **View actions** next to the pipeline folder, and then click **Delete**.

  4. In the confirmation dialog, click **Delete**.

     The pipeline folder and all tasks and files contained within it are deleted from your workspace.
- To delete a pipeline listed in the **Explorer** pane, do the following:

  1. In the left pane, click **Explorer**:

     ![Highlighted button for the Explorer pane.](https://docs.cloud.google.com/static/bigquery/images/explorer-tab.png)
  2. In the **Explorer** pane, expand your project and click **Pipelines**.

  3. Find the pipeline that you want to delete.

  4. Click
     **View actions** next to the pipeline, and then click **Delete**.

  5. Click **Delete**.

## Manage metadata in Knowledge Catalog

Knowledge Catalog lets you store and manage metadata for
pipelines. Pipelines are available in Knowledge Catalog
by default, without additional configuration.

You can use Knowledge Catalog to manage pipelines
in all [pipeline locations](https://docs.cloud.google.com/bigquery/docs/locations.md.txt).
Managing pipelines in Knowledge Catalog
is subject to [Knowledge Catalog quotas and limits](https://docs.cloud.google.com/knowledge-catalog/docs/quotas.md.txt)
and [Knowledge Catalog pricing](https://cloud.google.com/dataplex/pricing).

Knowledge Catalog automatically retrieves
the following metadata from pipelines:

- Data asset name
- Data asset parent
- Data asset location
- Data asset type
- Corresponding Google Cloud project

Knowledge Catalog logs pipelines as
[entries](https://docs.cloud.google.com/knowledge-catalog/docs/ingest-custom-sources.md.txt#entries) with the following
entry values:

System entry group
:   The [system entry group](https://docs.cloud.google.com/knowledge-catalog/docs/ingest-custom-sources.md.txt#entry-groups)
    for pipelines is `@dataform`. To view details of pipeline entries
    in Knowledge Catalog, you need to view the `dataform` system entry group.
    For instructions about how to view a list of all entries in an entry group, see
    [View details of an entry group](https://docs.cloud.google.com/knowledge-catalog/docs/ingest-custom-sources.md.txt#entry-group-details)
    in the Knowledge Catalog documentation.

System entry type
:   The [system entry type](https://docs.cloud.google.com/knowledge-catalog/docs/ingest-custom-sources.md.txt#entry-types)
    for pipelines is `dataform-code-asset`. To view details of
    pipelines,you need to view the `dataform-code-asset` system entry type,
    filter the results with an aspect-based filter,
    and [set the `type` field inside `dataform-code-asset` aspect to `WORKFLOW`](https://docs.cloud.google.com/knowledge-catalog/docs/search-syntax.md.txt#aspect-search).
    Then, select an entry of the selected pipeline.
    For instructions about how to view details of a selected entry type, see
    [View details of an entry type](https://docs.cloud.google.com/knowledge-catalog/docs/ingest-custom-sources.md.txt#entry-type-details)
    in the Knowledge Catalog documentation.
    For instructions about how to view details of a selected entry, see
    [View details of an entry](https://docs.cloud.google.com/knowledge-catalog/docs/search-assets.md.txt#view-entry-details)
    in the Knowledge Catalog documentation.

System aspect type
:   The [system aspect type](https://docs.cloud.google.com/knowledge-catalog/docs/enrich-entries-metadata.md.txt#aspect-types)
    for pipelines is `dataform-code-asset`. To
    provide additional context to pipelines in Knowledge Catalog
    by annotating data pipeline entries with
    [aspects](https://docs.cloud.google.com/knowledge-catalog/docs/enrich-entries-metadata.md.txt#aspects),
    view the `dataform-code-asset` aspect type,
    filter the results with an aspect-based filter,
    and [set the `type` field inside `dataform-code-asset` aspect to `WORKFLOW`](https://docs.cloud.google.com/knowledge-catalog/docs/search-syntax.md.txt#aspect-search).
    For instructions about how to annotate entries with aspects, see
    [Manage aspects and enrich metadata](https://docs.cloud.google.com/knowledge-catalog/docs/enrich-entries-metadata.md.txt)
    in the Knowledge Catalog documentation.

Type
:   The type for data canvases is `WORKFLOW`.
    This type lets you filter pipelines in the `dataform-code-asset`
    system entry type and the `dataform-code-asset` aspect type by using the
    `aspect:dataplex-types.global.dataform-code-asset.type=WORKFLOW`
    query in an [aspect-based filter](https://docs.cloud.google.com/knowledge-catalog/docs/search-syntax.md.txt#aspect-search).

For instructions about how to search for assets in Knowledge Catalog, see
[Search for data assets in Knowledge Catalog](https://docs.cloud.google.com/knowledge-catalog/docs/search-assets.md.txt)
in the Knowledge Catalog documentation.

### Metadata enrichment and data quality scorecard integration

Dataform can publish the following metadata to
Knowledge Catalog:

- Overview aspect type
- Generic aspect type
- Data quality scorecard aspect type

Dataform assertions are automatically integrated with the
[Knowledge Catalog data quality scorecard](https://docs.cloud.google.com/knowledge-catalog/docs/enrich-entries-metadata.md.txt#data-quality-scorecard).
During pipeline execution, the results of any Dataform assertions
are automatically published to Knowledge Catalog. These results
populate the Knowledge Catalog data quality scorecard with a pass or fail
status.

> [!NOTE]
> **Note:** Each execution overwrites existing data quality scorecards published by prior Dataform runs, but doesn't affect scorecards created by Knowledge Catalog data scans.

To check the status of a metadata update, follow the instructions in
[View past manual runs](https://docs.cloud.google.com/bigquery/docs/manage-pipelines.md.txt#view-manual-runs).

After the metadata has been synchronized, you can search for and view the entry
in Knowledge Catalog. For more information, see
[Search for resources](https://docs.cloud.google.com/knowledge-catalog/docs/search-assets.md.txt#search-data-assets).

## Track data lineage

BigQuery reports table-level
[data lineage](https://docs.cloud.google.com/knowledge-catalog/docs/about-data-lineage.md.txt) to
Knowledge Catalog for the actions that run in your BigQuery
pipeline. For each executed action, the lineage graph in BigQuery
and Knowledge Catalog shows a process between the tables that the action
reads and the table that it writes. You don't need to change your SQLX files to
report lineage.

Tables, views, incremental tables, materialized views, assertions, property
graphs, data preparations, and operations with `hasOutput: true` report
table-to-table lineage. Operations without `hasOutput: true` report the process
and run only. Declarations, notebooks, and SQL unit tests don't report lineage.
Declarations appear only as inputs to other actions.

Reporting lineage never causes a pipeline run to fail. If the
Data Lineage API is disabled, a permission is missing, or a quota is
exceeded, the pipeline run continues without reporting lineage.

Data lineage ingestion is billed under the Knowledge Catalog Premium
Processing SKU, and only events that write table-to-table lineage links are
billed. Data lineage storage is billed under the metadata storage SKU. For more
information, see
[Knowledge Catalog pricing](https://cloud.google.com/dataplex/pricing).

### Data lineage limitations

Data lineage for pipelines is subject to the following limitations:

- Only table-level lineage is reported. Column-level lineage isn't reported.

- Data lineage reporting is subject to the
  [Data Lineage API quotas](https://docs.cloud.google.com/knowledge-catalog/docs/quotas.md.txt) for writes per
  project, identity, and region.

### Enable data lineage

> [!NOTE]
> **Note:** During the initial rollout period, data lineage reporting is opt-in.

To enable data lineage for your pipeline, follow these steps:

1. Enable the Data Lineage API (`datalineage.googleapis.com`) in the project
   that contains the pipeline and in the project where users view lineage.

2. In the project that contains the pipeline, grant the
   [Data Lineage Events Producer role](https://docs.cloud.google.com/iam/docs/roles-permissions/datalineage.md.txt#datalineage.producer)
   (`roles/datalineage.producer`) to the identity that runs the pipeline. This
   identity is the custom service account of the pipeline, or the
   Dataform service agent
   (`service-PROJECT_NUMBER@gcp-sa-dataform.iam.gserviceaccount.com`)
   if no custom service account is set.

3. In the `workflow_settings.yaml` file of your pipeline, add the following
   `lineage` block:

   ```yaml
   lineage:
     enabled: true
   ```

Pipeline runs that use end user credentials for a Google Account don't report
lineage. To report lineage, run the pipeline with a custom service account.

To stop data lineage reporting for a pipeline, set `enabled` to `false` in the
`lineage` block. To stop data lineage reporting for a whole project, disable
the Data Lineage API in the project. Disabling the API also stops data
lineage reporting from other services in the project.

### View data lineage

To view data lineage for a table that a pipeline action produces, follow these
steps:

1. In the Google Cloud console, go to **BigQuery Studio**.

   [Go to BigQuery Studio](https://console.cloud.google.com/bigquery)
2. In the **Explorer** pane, expand your project and dataset, and then select a
   table that a pipeline action produces.

3. Click the **Lineage** tab.

4. To view the action, the pipeline, and the runs, select the process node.

5. To open the pipeline, click **Open in Dataform**.

For more information about navigating lineage graphs, see
[View data lineage in Knowledge Catalog](https://docs.cloud.google.com/knowledge-catalog/docs/use-lineage.md.txt).

### Troubleshoot data lineage

The following table describes common data lineage issues for pipelines and how
to fix them:

| Symptom | Likely cause | Fix |
|---|---|---|
| The **Lineage** tab doesn't show pipeline process nodes. | The Data Lineage API is disabled. | Enable the API in the project that contains the pipeline and in the project where you view lineage. |
| A pipeline doesn't report lineage. | The identity that runs the pipeline doesn't have the Data Lineage Events Producer role. | Grant `roles/datalineage.producer` to the custom service account or to the Dataform service agent. |
| Pipeline runs that use end user credentials don't report lineage. | Lineage reporting doesn't support end user credentials. | Run the pipeline with a custom service account. |
| A pipeline doesn't report lineage although the role is granted. | Dataform can't act as the custom service account. | Grant the Dataform service agent the [Service Account Token Creator role](https://docs.cloud.google.com/iam/docs/roles-permissions/iam.md.txt#iam.serviceAccountTokenCreator) (`roles/iam.serviceAccountTokenCreator`) on the custom service account. |
| One pipeline doesn't report lineage. | The pipeline is opted out. | Remove `lineage.enabled: false` from `workflow_settings.yaml`. |
| Lineage is missing for some runs. | The Data Lineage API quota is exceeded. | Spread out pipeline runs or request a higher quota. |
| Clicking **Open in Dataform** returns a permission error. | The user doesn't have access to the pipeline. | Grant the user the Dataform Viewer role (`roles/dataform.viewer`). |

After the first permission or API error in a pipeline run,
BigQuery stops reporting lineage for the rest of that run. The
next pipeline run tries to report lineage again.

To check whether a pipeline run reported lineage, view the
`dataform.googleapis.com/lineage_emission` log entries in Cloud Logging. To
filter for skipped or failed lineage reporting in Logs Explorer, run the
following query:

    log_id("dataform.googleapis.com/lineage_emission")
    jsonPayload.outcome=("SKIPPED" OR "FAILED")

## What's next

- Learn more about [BigQuery pipelines](https://docs.cloud.google.com/bigquery/docs/pipelines-introduction.md.txt).
- Learn how to [create pipelines](https://docs.cloud.google.com/bigquery/docs/create-pipelines.md.txt).
- Learn how to [schedule pipelines](https://docs.cloud.google.com/bigquery/docs/schedule-pipelines.md.txt).
- Learn how to [manage code with BigQuery Studio Git repositories](https://docs.cloud.google.com/bigquery/docs/git-repositories.md.txt).
- Learn how to [organize code assets with folders](https://docs.cloud.google.com/bigquery/docs/code-asset-folders.md.txt).
- Learn more about [Dataform Deployments](https://docs.cloud.google.com/dataform/docs/deployments.md.txt).