# Schedule pipelines

Scheduling
[BigQuery pipelines](https://docs.cloud.google.com/bigquery/docs/pipelines-introduction)
lets you automate multi-step SQL and notebook workflows powered by
[Dataform](https://docs.cloud.google.com/dataform/docs/overview)
on a recurring interval or in response to table updates. You can authenticate
scheduled runs with your Google Account user credentials or a
[custom service account](https://docs.cloud.google.com/dataform/docs/access-control#about-service-accounts),
and then inspect execution history to monitor workflow health.

BigQuery automatically saves your changes to the pipeline, and
these changes are available to you and to users who have the Dataform Admin
role on the project. To update the schedule with a new version of the pipeline,
you need to
[deploy the pipeline](https://docs.cloud.google.com/bigquery/docs/schedule-pipelines#deploy).
Schedules always run the latest deployed version.

For workflows that include notebooks, scheduled runs use a
[default runtime specification](https://docs.cloud.google.com/colab/docs/runtimes#default_runtime_specifications)
and write output to the
[Cloud Storage bucket](https://docs.cloud.google.com/storage/docs/buckets)
that you select during setup.

## Before you begin

Before you begin, [create a pipeline](https://docs.cloud.google.com/bigquery/docs/create-pipelines).

### Enable pipeline scheduling

To schedule pipelines, you must grant the following role to the custom
service accounts that you plan to use for pipeline schedules:

[Service Account User](https://docs.cloud.google.com/iam/docs/roles-permissions/iam#iam.serviceAccountUser) (`roles/iam.serviceAccountUser`)
:   Follow [Grant a single role on a service account](https://docs.cloud.google.com/iam/docs/manage-access-service-accounts#grant-single-role)
    to add your service account as a principal to itself. In other words, add the
    service account as a principal to the same service account. Then, grant the
    Service Account User role to this principal.

If your pipeline contains SQL queries, you must grant the following roles to
the custom service accounts or Google Accounts that you plan to use for
pipeline schedules:

[BigQuery Job User](https://docs.cloud.google.com/bigquery/docs/access-control#bigquery.jobUser) (`roles/bigquery.jobUser`)
:   Follow
    [Grant a single role on a project](https://docs.cloud.google.com/iam/docs/granting-changing-revoking-access#grant-single-role)
    to grant the BigQuery Job User role to the custom service accounts or Google
    Accounts on projects from which your pipelines read data.

[BigQuery Data Viewer](https://docs.cloud.google.com/bigquery/docs/access-control#bigquery.dataViewer) (`roles/bigquery.dataViewer`)
:   Follow
    [Grant a single role on a project](https://docs.cloud.google.com/iam/docs/granting-changing-revoking-access#grant-single-role)
    to grant the BigQuery Data Viewer role to the custom service accounts or
    Google Accounts on projects from which your pipelines read data.

[BigQuery Data Editor](https://docs.cloud.google.com/bigquery/docs/access-control#bigquery.dataEditor) (`roles/bigquery.dataEditor`)
:   Follow
    [Grant a single role on a project](https://docs.cloud.google.com/iam/docs/granting-changing-revoking-access#grant-single-role)
    to grant the BigQuery Data Editor role to the custom service accounts or
    Google Accounts on projects to which your pipelines write data.

If your pipeline contains notebooks, you must grant the following roles to
the custom service accounts or Google Accounts that you plan to use for
pipeline schedules:

[Notebook Executor User](https://docs.cloud.google.com/iam/docs/roles-permissions/aiplatform#aiplatform.notebookExecutorUser) (`roles/aiplatform.notebookExecutorUser`)
:   Follow
    [Grant a single role on a project](https://docs.cloud.google.com/iam/docs/granting-changing-revoking-access#grant-single-role)
    to grant the Notebook Executor User role to the custom service accounts or
    Google Accounts on the selected project.

[Storage Admin](https://docs.cloud.google.com/iam/docs/roles-permissions/storage#storage.admin) (`roles/storage.admin`)
:   Follow [Add a principal to a bucket-level policy](https://docs.cloud.google.com/storage/docs/access-control/using-iam-permissions#bucket-add)
    to add the custom service accounts or Google Accounts as principals to the
    Cloud Storage bucket that you plan to use for storing the output of
    notebooks executed in scheduled pipeline runs, and then grant the Storage
    Admin role to these principals.

Additionally, you must grant the following roles to the default
Dataform service agent:

[Service Account Token Creator](https://docs.cloud.google.com/iam/docs/service-account-permissions#token-creator-role) (`roles/iam.serviceAccountTokenCreator`)
:   Follow [Grant token creation access to a service account](https://docs.cloud.google.com/dataform/docs/access-control#grant-token-creation-access)
    to add the default Dataform service agent as a principal to your
    service account, and grant the Service Account Token Creator role
    to this principal.

[Service Account User](https://docs.cloud.google.com/iam/docs/roles-permissions/iam#iam.serviceAccountUser) (`roles/iam.serviceAccountUser`)
:   Follow
    [Grant or revoke multiple IAM roles using Google Cloud console](https://docs.cloud.google.com/iam/docs/manage-access-service-accounts#multiple-roles-console)
    to grant the Service Account User role to the default
    Dataform service agent on the custom service account.

To enable metadata enrichment in Knowledge Catalog,
you must enable the
[Dataplex API](https://console.cloud.google.com/flows/enableapi?apiid=dataplex.googleapis.com).
Additionally, you must grant the following role on the project or on the
`@bigquery` entry group to the custom service accounts or Google Accounts
that you plan to use for pipeline schedules:

[Dataplex Catalog Editor role](https://docs.cloud.google.com/iam/docs/roles-permissions/dataplex#dataplex.catalogEditor) (`roles/dataplex.catalogEditor`)
:   Follow
    [Grant a single role on a project](https://docs.cloud.google.com/iam/docs/granting-changing-revoking-access#grant-single-role)
    to grant the Dataplex Catalog Editor role to the custom service accounts or
    Google Accounts on the project or `@bigquery` entry group.

To learn more about service accounts in Dataform, see
[About service accounts in Dataform](https://docs.cloud.google.com/dataform/docs/access-control#about-service-accounts).

### VPC Service Controls requirements

If you use VPC Service Controls to protect your pipelines, you should be aware that
scheduled runs are powered by Dataform. When you configure
VPC Service Controls for scheduled runs, ensure that the following requirements are
met:

- You must set the [`dataform.restrictGitRemotes` Organization Policy Service](https://docs.cloud.google.com/dataform/docs/restrict-git-remotes).
- Dataform and BigQuery must be restricted by the same VPC Service Controls service perimeter.
- To allow users to authenticate with the user credentials for their Google Account when scheduling or manually triggering runs, you must add their user identities to your ingress rules. For more information, see [Updating ingress and egress policies for a service perimeter](https://docs.cloud.google.com/vpc-service-controls/docs/configuring-ingress-egress-policies#updating) and [Ingress rules reference](https://docs.cloud.google.com/vpc-service-controls/docs/ingress-egress-rules#ingress-rules-reference).

For detailed configuration steps and security considerations, see
[Configure VPC Service Controls for Dataform](https://docs.cloud.google.com/dataform/docs/vpc-service-controls).

### Required roles


To get the permissions that
you need to manage pipelines,

ask your administrator to grant you the
following IAM roles:

- View and manage schedules:
  - [Dataform Admin](https://docs.cloud.google.com/iam/docs/roles-permissions/dataform#dataform.admin) (`roles/dataform.admin`) on the pipeline
  - [Dataform Editor](https://docs.cloud.google.com/iam/docs/roles-permissions/dataform#dataform.editor) (`roles/dataform.editor`) on the project
- View and run pipelines:
  - [Dataform Viewer](https://docs.cloud.google.com/iam/docs/roles-permissions/dataform#dataform.viewer) (`roles/dataform.viewer`) on the project
  - [BigQuery Job User](https://docs.cloud.google.com/iam/docs/roles-permissions/bigquery#bigquery.jobUser) (`roles/bigquery.jobUser`) on the project
- Run a pipeline schedule with user credentials for a Google Account: [BigQuery Job User](https://docs.cloud.google.com/iam/docs/roles-permissions/bigquery#bigquery.jobUser) (`roles/bigquery.jobUser`) on the project
- Run a pipeline schedule with a custom service account: [Service Account User](https://docs.cloud.google.com/iam/docs/roles-permissions/iam#iam.serviceAccountUser) (`roles/iam.serviceAccountUser`) on the custom service account
- Manage pipeline schedules in user folders:
  - [Code Owner](https://docs.cloud.google.com/iam/docs/roles-permissions/dataform#dataform.codeOwner) (`roles/dataform.codeOwner`) on the folder
  - [Code Editor](https://docs.cloud.google.com/iam/docs/roles-permissions/dataform#dataform.codeEditor) (`roles/dataform.codeEditor`) on the folder
- Manage pipeline schedules in team folders:
  - [Team Folder Owner](https://docs.cloud.google.com/iam/docs/roles-permissions/dataform#dataform.teamFolderOwner) (`roles/dataform.teamFolderOwner`) on the team folder
  - [Team Folder Contributor](https://docs.cloud.google.com/iam/docs/roles-permissions/dataform#dataform.teamFolderContributor) (`roles/dataform.teamFolderContributor`) on the team folder
- Manage pipelines in Git repositories: [Developer Connect OAuth User](https://docs.cloud.google.com/iam/docs/roles-permissions/developerconnect#developerconnect.oauthUser) (`roles/developerconnect.oauthUser`) on the project
- Enable metadata enrichment: [Dataplex Catalog Editor](https://docs.cloud.google.com/iam/docs/roles-permissions/dataplex#dataplex.catalogEditor) (`roles/dataplex.catalogEditor`) on the project or `@bigquery` entry group


For more information about granting roles, see [Manage access to projects, folders, and organizations](https://docs.cloud.google.com/iam/docs/granting-changing-revoking-access).


You might also be able to get
the required permissions through [custom
roles](https://docs.cloud.google.com/iam/docs/creating-custom-roles) or other [predefined
roles](https://docs.cloud.google.com/iam/docs/roles-overview#predefined).

To enhance security for scheduling, see
[Implement enhanced scheduling permissions](https://docs.cloud.google.com/dataform/docs/access-control#enhanced-scheduling-permissions).

For more information about Dataform IAM, see
[Control access with IAM](https://docs.cloud.google.com/dataform/docs/access-control).

For more information about roles for folders, see
[Create and manage folders](https://docs.cloud.google.com/bigquery/docs/create-manage-folders#required_roles).

For more information about roles for Git repositories, see
[Manage code with BigQuery Studio Git repositories](https://docs.cloud.google.com/bigquery/docs/git-repositories#required_roles).

To use Colab notebook runtime templates when scheduling pipelines,
you need the
[Notebook Runtime User role](https://docs.cloud.google.com/iam/docs/roles-permissions/aiplatform#aiplatform.notebookRuntimeUser)
(`roles/aiplatform.notebookRuntimeUser`).

## Create a pipeline schedule

You can also use the **Pipelines \& Connections** page to schedule a
Dataform pipeline using a [streamlined,
BigQuery-specific workflow](https://docs.cloud.google.com/bigquery/docs/pipeline-connection-page).
This feature is in [preview](https://cloud.google.com/products/#product-launch-stages).

> [!NOTE]
> **Note:** For pipelines stored in BigQuery Studio Git folders, scheduling is managed through [Dataform deployments](https://docs.cloud.google.com/bigquery/docs/schedule-pipelines#deploy-git-pipelines).

To create a pipeline schedule, follow these steps:

### **Files** pane

1. In the Google Cloud console, go to the **BigQuery** page.

   [Go to BigQuery](https://console.cloud.google.com/bigquery)
2. In the left pane, click **Files**
   to open the file browser.

   If you don't see the left pane, click **Expand left pane** to open the pane.
3. In the **Files** pane, expand your **User** folder or a **Team folder**,
   and then select a pipeline.

4. In the **Pipeline Viewer** toolbar, click **Schedule** or **Trigger**.

5. In the **Schedule pipeline** pane, in the **Schedule name** field,
   enter a name for the schedule.

6. In the **Authentication** section, authorize the
   pipeline with your Google Account user credentials or a service
   account.

   - To use your Google Account user credentials
     ([Preview](https://cloud.google.com/products#product-launch-stages)), select
     **Execute with my user credentials**.

     Optional: In the **Extended access options** section, select the
     additional services that your pipeline requires:
     - **Knowledge Catalog**: Allows Google Cloud Knowledge Catalog metadata updates.
     - **Google Drive**: Allows read-only access to Google Drive files.
     - **Bigtable**: Allows read-only access to Google Bigtable data.
   - To use a service account, select
     **Execute with selected service account**, and then select a service account.

7. If your pipeline contains a notebook, in the **Notebook options** section,
   in the **Runtime template** field, select a Colaboratory notebook runtime
   template or the default runtime specifications. For details on creating a
   Colab notebook runtime template, see
   [Create a runtime template](https://docs.cloud.google.com/colab/docs/create-runtime-template).

   > [!NOTE]
   > **Note:** A notebook runtime template must be in the same region as the pipeline.

   > [!NOTE]
   > **Note:** If you don't have the [required role](https://docs.cloud.google.com/bigquery/docs/schedule-pipelines#required_roles) for using Colab notebook runtime templates, you can still run and schedule pipelines with the default runtime specifications.

8. If your pipeline contains a notebook, in the **Notebook options** section, in the **Cloud Storage bucket** field,
   click **Browse** and select or create a Cloud Storage bucket for
   storing the output of notebooks in your pipeline.

   Your selected service account must be granted the
   Storage Admin IAM role on the selected bucket.
   For more information, see [Enable pipeline scheduling](https://docs.cloud.google.com/bigquery/docs/schedule-pipelines#enable-scheduling).
9. Under **Configuration Type** , select **Schedule (time-based recurrence)**.

10. Under **Schedule frequency**, do the following:

    1. In the **Repeats** menu, select the frequency of scheduled pipeline runs.
    2. In the **At time** field, enter the time for scheduled pipeline runs.
    3. In the **Timezone** menu, select the timezone for the schedule.
11. Set the BigQuery query job priority with the
    **Execute as interactive job with high priority (default)** option.
    By default, BigQuery runs queries as
    [interactive query jobs](https://docs.cloud.google.com/bigquery/docs/running-queries#interactive-batch),
    which are intended to start running as quickly as possible.
    Clearing this option runs the queries as
    [batch query jobs](https://docs.cloud.google.com/bigquery/docs/running-queries#interactive-batch),
    which have lower priority.

12. Optional: To limit how many tasks can run concurrently during scheduled
    pipeline runs, select **Limit concurrent actions** , and in the **Maximum
    concurrent actions** field, enter a whole number between `1` and `5000`.

    If the number of executed tasks exceeds the limit,
    BigQuery places the remaining tasks in a
    queue until a concurrency slot becomes available.
13. Click **Create schedule** . If you selected **Execute with my user credentials**
    for your authentication method, you must
    [authorize your Google Account](https://docs.cloud.google.com/bigquery/docs/schedule-pipelines#authorize-google-account)
    ([Preview](https://cloud.google.com/products#product-launch-stages)).

When you create the schedule, the current version of the pipeline is
automatically deployed. To update the schedule with a new version of the
pipeline, [deploy the pipeline](https://docs.cloud.google.com/bigquery/docs/schedule-pipelines#deploy).

The latest deployed version of the pipeline runs at the selected time and
frequency.

> [!NOTE]
> **Note:** If a scheduled pipeline run doesn't finish before the start of the next scheduled run, the next scheduled run is skipped and marked with an error.

### **Explorer** pane

1. In the Google Cloud console, go to the **BigQuery** page.

   [Go to BigQuery](https://console.cloud.google.com/bigquery)
2. In the left pane, click **Explorer**:

   ![Highlighted button for the Explorer pane.](https://docs.cloud.google.com/static/bigquery/images/explorer-tab.png)

   If you don't see the left pane, click **Expand left pane** to open the pane.
3. In the **Explorer** pane, expand your project, click **Pipelines**,
   and then select a pipeline.

4. Click **Schedule**.

5. In the **Schedule pipeline** pane, in the **Schedule name** field,
   enter a name for the schedule.

6. In the **Authentication** section, authorize the
   pipeline with your Google Account user credentials or a service
   account.

   - To use your Google Account user credentials, select
     **Execute with my user credentials**.

     Optional: In the **Extended access options** section, select the
     additional services that your pipeline requires:
     - **Knowledge Catalog**: Allows Google Cloud Knowledge Catalog metadata updates.
     - **Google Drive**: Allows read-only access to Google Drive files.
     - **Bigtable**: Allows read-only access to Google Bigtable data.
   - To use a service account, select
     **Execute with selected service account**, and then select a service account.

7. If your pipeline contains a notebook, in the **Notebook options** section,
   in the **Runtime template** field, select a Colaboratory notebook runtime
   template or the default runtime specifications. For details on creating a
   Colab notebook runtime template, see
   [Create a runtime template](https://docs.cloud.google.com/colab/docs/create-runtime-template).

   > [!NOTE]
   > **Note:** A notebook runtime template must be in the same region as the pipeline.

   > [!NOTE]
   > **Note:** If you don't have the [required role](https://docs.cloud.google.com/bigquery/docs/schedule-pipelines#required_roles) for using Colab notebook runtime templates, you can still run and schedule pipelines with the default runtime specifications.

8. If your pipeline contains a notebook, in the **Notebook options** section, in the **Cloud Storage bucket** field,
   click **Browse** and select or create a Cloud Storage bucket for
   storing the output of notebooks in your pipeline.

   Your selected service account must be granted the
   Storage Admin IAM role on the selected bucket.
   For more information, see [Enable pipeline scheduling](https://docs.cloud.google.com/bigquery/docs/schedule-pipelines#enable-scheduling).
9. Under **Configuration Type** , select **Schedule (time-based recurrence)**.

10. Under **Schedule frequency**, do the following:

    1. In the **Repeats** menu, select the frequency of scheduled pipeline runs.
    2. In the **At time** field, enter the time for scheduled pipeline runs.
    3. In the **Timezone** menu, select the timezone for the schedule.
11. Set the BigQuery query job priority with the
    **Execute as interactive job with high priority (default)** option.
    By default, BigQuery runs queries as
    [interactive query jobs](https://docs.cloud.google.com/bigquery/docs/running-queries#interactive-batch),
    which are intended to start running as quickly as possible.
    Clearing this option runs the queries as
    [batch query jobs](https://docs.cloud.google.com/bigquery/docs/running-queries#interactive-batch),
    which have lower priority.

12. Optional: To limit how many tasks can run concurrently during scheduled
    pipeline runs, select **Limit concurrent actions** , and in the **Maximum
    concurrent actions** field, enter a whole number between `1` and `5000`.

    If the number of executed tasks exceeds the limit,
    BigQuery places the remaining tasks in a
    queue until a concurrency slot becomes available.
13. Click **Create schedule** . If you selected **Execute with my user credentials**
    for your authentication method, you must
    [authorize your Google Account](https://docs.cloud.google.com/bigquery/docs/schedule-pipelines#authorize-google-account).

When you create the schedule, the current version of the pipeline is
automatically deployed. To update the schedule with a new version of the
pipeline, [deploy the pipeline](https://docs.cloud.google.com/bigquery/docs/schedule-pipelines#deploy).

The latest deployed version of the pipeline runs at the selected time and
frequency.

> [!NOTE]
> **Note:** If a scheduled pipeline run doesn't finish before the start of the next scheduled run, the next scheduled run is skipped and marked with an error.

### **Scheduling** page

1. In the Google Cloud console, go to the **Scheduling** page.

   [Go to Scheduling](https://console.cloud.google.com/bigquery/orchestration)
2. Click **Create** , and then select **Pipeline schedule** from the menu.

3. In the **Schedule pipeline** pane, select a pipeline to schedule.

4. In the **Schedule name** field, enter a name for the schedule.

5. In the **Authentication** section, authorize the
   pipeline with your Google Account user credentials or a service
   account.

   - To use your Google Account user credentials, select
     **Execute with my user credentials**.

     Optional: In the **Extended access options** section, select the
     additional services that your pipeline requires:
     - **Knowledge Catalog**: Allows Google Cloud Knowledge Catalog metadata updates.
     - **Google Drive**: Allows read-only access to Google Drive files.
     - **Bigtable**: Allows read-only access to Google Bigtable data.
   - To use a service account, select
     **Execute with selected service account**, and then select a service account.

6. If your pipeline contains a notebook, in the **Notebook options** section,
   in the **Runtime template** field, select a Colab notebook
   runtime template or the default runtime specifications. For details on
   creating a Colab notebook runtime template, see
   [Create a runtime template](https://docs.cloud.google.com/colab/docs/create-runtime-template).

   > [!NOTE]
   > **Note:** A notebook runtime template must be in the same region as the pipeline.

   > [!NOTE]
   > **Note:** If you don't have the [required role](https://docs.cloud.google.com/bigquery/docs/schedule-pipelines#required_roles) for using Colab notebook runtime templates, you can still run and schedule pipelines with the default runtime specifications.

7. If your pipeline contains a notebook, in the **Cloud Storage bucket** field,
   click **Browse** and select or create a Cloud Storage bucket for
   storing the output of notebooks in your pipeline.

   Your selected service account must be granted the
   Storage Admin IAM role on the selected bucket.
   For more information, see [Enable pipeline scheduling](https://docs.cloud.google.com/bigquery/docs/schedule-pipelines#enable-scheduling).
8. Under **Configuration Type** , select **Schedule (time-based recurrence)**.

9. Under **Schedule frequency**, do the following:

   1. In the **Repeats** menu, select the frequency of scheduled pipeline runs.
   2. In the **At time** field, enter the time for scheduled pipeline runs.
   3. In the **Timezone** menu, select the timezone for the schedule.
10. Set the BigQuery query job priority with the
    **Execute as interactive job with high priority (default)** option.
    By default, BigQuery runs queries as
    [interactive query jobs](https://docs.cloud.google.com/bigquery/docs/running-queries#interactive-batch),
    which are intended to start running as quickly as possible.
    Clearing this option runs the queries as
    [batch query jobs](https://docs.cloud.google.com/bigquery/docs/running-queries#interactive-batch),
    which have lower priority.

11. Optional: To limit how many tasks can run concurrently during scheduled
    pipeline runs, select **Limit concurrent actions** , and in the **Maximum
    concurrent actions** field, enter a whole number between `1` and `5000`.

    If the number of executed tasks exceeds the limit,
    BigQuery places the remaining tasks in a
    queue until a concurrency slot becomes available.
12. Click **Create schedule** . If you selected **Execute with my user credentials**
    for your authentication method, you must
    [authorize your Google Account](https://docs.cloud.google.com/bigquery/docs/schedule-pipelines#authorize-google-account).

> [!NOTE]
> **Note:** If a scheduled pipeline run doesn't finish before the start of the next scheduled run, the next scheduled run is skipped and marked with an error.

### Authorize your Google Account

To authenticate the resource with your
[Google Account](https://docs.cloud.google.com/iam/docs/principals-overview#google-account)
user credentials, you must manually grant permission for BigQuery
pipelines to get the access token for your Google Account and access the source
data on your behalf. You can grant manual approval with the OAuth dialog
interface. If you select any **Extended access options**, you must grant access to those services---for example, Google Drive or Knowledge Catalog.

> [!NOTE]
> **Note:** Context-Aware Access (CAA) policies---including IP-based, geolocation-based, and device compliance policies---aren't supported when executing or scheduling BigQuery pipelines with user credentials for a Google Account, because the token requests originate from Google infrastructure. CAA policies block these executions unless the Dataform OAuth client ID is [exempted from the policies](https://docs.cloud.google.com/dataform/docs/troubleshooting#euc-permission-denied).

You only need to give permission to BigQuery pipelines once.

To revoke the permission that you granted, follow these steps:

1. Go to your [Google Account page](https://myaccount.google.com/).
2. Click **BigQuery Pipelines**.
3. Click **Remove access**.

> [!WARNING]
> **Warning:** Revoking access permissions prevents any future pipeline runs that this Google Account owns across all regions.

Changing the pipeline schedule owner by updating credentials also
requires manual approval if the new Google Account owner has never created a
schedule before.

If your pipeline contains a notebook, you must also manually grant
permission for Colab Enterprise to get the access token for your
Google Account and access the source data on your behalf. You only need
to give permission once. You can revoke this permission on the
[Google Account page](https://myaccount.google.com/).

## Trigger-based scheduling

> [!WARNING]
>
> **Preview**
>
>
> This feature is
>
> subject to the "Pre-GA Offerings Terms" in the General Service Terms section of the
> [Service Specific
> Terms](https://docs.cloud.google.com/terms/service-terms#1).
>
> Pre-GA features are available "as is" and might have limited support.
>
> For more information, see the
> [launch stage descriptions](https://cloud.google.com/products/#product-launch-stages).

> [!NOTE]
> **Note:** To get support or provide feedback for this feature, contact [bigquery-event-based-triggers@google.com](mailto:bigquery-event-based-triggers@google.com).

You can configure BigQuery pipelines to automatically trigger
executions based on updates to specified BigQuery tables. You can
create trigger-based schedules to automate pipeline executions in response to
changes to your BigQuery data, rather than on a fixed schedule.

When the pipeline detects changes to the specified table or tables, it triggers
a new execution of the associated workflow. You can define conditions based on
updates to a single table, to all of a set of tables, or to any of a set of
tables.

You can also adjust the optional settings of your trigger-based schedules to
control the minimum interval between pipeline triggers. For example, adjust the
**Min Execution Duration** value to ensure that trigger-based schedules aren't
activated more frequently than intended. You can also adjust the **Max Wait
Duration** value to ensure that the trigger-based schedule is forced to activate
once within that duration, even if no table updates were detected.

### Limitations

Trigger-based schedules are subject to the following limitations:

- Trigger-based schedules aren't instantaneous. When you configure a trigger-based schedule, the pipeline checks the status of the BigQuery table approximately once every 3 minutes. This time period is called the polling interval and can result in a delay between a table modification and the trigger activation.
- Each monitored table results in API calls to BigQuery during every polling interval. Monitoring a very large number of tables can contribute to [BigQuery API quota consumption](https://docs.cloud.google.com/bigquery/docs/troubleshoot-quotas#ts-maximum-api-request-limit).

### Create a trigger

To create a trigger, follow these steps:

1. In the Google Cloud console, go to the **BigQuery** page.

   [Go to BigQuery](https://console.cloud.google.com/bigquery)
2. Open your pipeline in the Pipeline Viewer by doing one of the following:

   - In the left pane, click **Files** , expand your **User** folder or a **Team folder**, and then select a pipeline.
   - In the left pane, click **Explorer** , expand your project, click **Pipelines**, and then select a pipeline.
3. In the Pipeline Viewer toolbar, click **Trigger**.

4. In the **Trigger** field, enter a name for the trigger.

5. In the **Authentication** section, authorize the
   pipeline with your Google Account user credentials or a service
   account.

   - To use your Google Account user credentials, select
     **Execute with my user credentials**.

     Optional: In the **Extended access options** section, select the
     additional services that your pipeline requires:
     - **Knowledge Catalog**: Allows Google Cloud Knowledge Catalog metadata updates.
     - **Google Drive**: Allows read-only access to Google Drive files.
     - **Bigtable**: Allows read-only access to Google Bigtable data.
   - To use a service account, select
     **Execute with selected service account**, and then select a service account.

6. If your pipeline contains a notebook, in the **Notebook options** section,
   in the **Runtime template** field, select a Colaboratory notebook runtime
   template or the default runtime specifications. For details on creating a
   Colab notebook runtime template, see
   [Create a runtime template](https://docs.cloud.google.com/colab/docs/create-runtime-template).

   > [!NOTE]
   > **Note:** A notebook runtime template must be in the same region as the pipeline.

   > [!NOTE]
   > **Note:** If you don't have the [required role](https://docs.cloud.google.com/bigquery/docs/schedule-pipelines#required_roles) for using Colab notebook runtime templates, you can still run and schedule pipelines with the default runtime specifications.

7. If your pipeline contains a notebook, in the **Notebook options** section, in the **Cloud Storage bucket** field,
   click **Browse** and select or create a Cloud Storage bucket for
   storing the output of notebooks in your pipeline.

   Your selected service account must be granted the
   Storage Admin IAM role on the selected bucket.
   For more information, see [Enable pipeline scheduling](https://docs.cloud.google.com/bigquery/docs/schedule-pipelines#enable-scheduling).
8. Under **Configuration Type** , select **Trigger (event-based execution)**.

9. In the **Search tables** field, add a table or tables to be monitored for
   the trigger.

10. Under **Trigger Condition**, select one of the following options:

    - **Wait for ALL tables to update**: trigger the workflow only when all listed tables have been updated since the last check.
    - **Trigger if ANY table updates**: trigger this workflow if any of the listed tables are updated since the last check.
11. (Optional) For **Max Wait Duration**, enter a duration to force the
    activation of a trigger if no table updates are detected within this duration.
    Supports values between 1 second to 7 days. If not specified, then
    the workflow will only run if the monitored table is updated, and the minimum
    execution duration is satisfied.

12. (Optional) For **Min Execution Duration**, select a duration to prevent
    triggers from activating more frequently than this minimum duration. Supports
    values between 3 minutes to 24 hours. If not specified, the default value is 3
    minutes.

13. Optional: To limit how many actions can run concurrently during trigger-based
    executions, select **Limit concurrent actions** , and in **Maximum concurrent
    actions** , enter a whole number between `1` and `5000`.

    If the number of ready actions exceeds the limit, BigQuery
    places the remaining actions in a queue until a concurrency slot becomes
    available.
14. Click **Create schedule** . If you selected **Execute with my user credentials**
    for your authentication method, you must
    [authorize your Google Account](https://docs.cloud.google.com/bigquery/docs/schedule-pipelines#authorize-google-account).

### Troubleshooting trigger-based schedules

This section describes common issues with trigger-based schedules and how to
resolve them.

Issue: The trigger isn't activating
:   **Resolution:** Try one of the following steps:

    - Verify that the user credentials or the service account has all the [required permissions](https://docs.cloud.google.com/bigquery/docs/schedule-pipelines#required_roles).
    - Verify that the specified BigQuery table is being modified.
    - Check that the trigger isn't being affected by the [polling
      interval](https://docs.cloud.google.com/bigquery/docs/schedule-pipelines#limitations).
    - Check if the minimum execution duration, or the **Min Execution Duration** value, is preventing more frequent runs. You can decrease this value to increase the frequency of the trigger activation.
    - Check if the trigger condition option (**ALL** or **ANY**) is affecting the trigger activation.
    - Examine the [audit logs](https://docs.cloud.google.com/bigquery/docs/introduction-audit-workloads) to check for errors when Dataform attempts to call the BigQuery API to check the status of the monitored table.

Issue: The trigger is activating too often
:   **Resolution:** Adjust the minimum execution duration, or the
    **Min Execution Duration** value. You can increase this value to decrease the
    frequency of the trigger activation.

## Deploy a pipeline

Deploying a pipeline updates its schedule or orchestration configuration with the current version of the
pipeline. Scheduled runs always execute the latest deployed version of the pipeline.

### Deploy pipelines stored in folders

To deploy standalone pipelines and pipelines stored in folders,
follow these steps:

1. In the Google Cloud console, go to the **BigQuery** page.

   [Go to BigQuery](https://console.cloud.google.com/bigquery)
2. Open your pipeline in the **Pipeline Viewer** by doing one of the following:

   - In the left pane, click **Files** , expand your **User** folder or a **Team folder**, and then select a pipeline.
   - In the left pane, click **Explorer** , expand your project, click **Pipelines**, and then select a pipeline.
3. In the **Pipeline Viewer** toolbar, click **Deploy**.

The corresponding schedule is updated with the current version of the pipeline.
The latest deployed version of the pipeline runs at the scheduled time.

### Deploy pipelines stored in Git repositories

> [!WARNING]
>
> **Preview**
>
>
> This feature is
>
> subject to the "Pre-GA Offerings Terms" in the General Service Terms section of the
> [Service Specific
> Terms](https://docs.cloud.google.com/terms/service-terms#1).
>
> Pre-GA features are available "as is" and might have limited support.
>
> For more information, see the
> [launch stage descriptions](https://cloud.google.com/products/#product-launch-stages).

Pipelines stored in Git Folders use Dataform [deployments](https://docs.cloud.google.com/dataform/docs/deployments) for orchestration and scheduling. Deployments let you configure release configurations, workflow configurations, compilation overrides, and multi-environment orchestration.

To deploy a pipeline stored in a Git repository:

1. In the Google Cloud console, go to the **BigQuery** page.

   [Go to BigQuery](https://console.cloud.google.com/bigquery)
2. In the left pane, click **Files**.

3. In the file tree, expand your connected Git repository folder, and select a pipeline or click the Git repository folder if it is a root-level pipeline.

4. In the **Pipeline Viewer** toolbar, click **Deploy**.

5. In the deployment dialog, configure the release configurations, workflow configurations, and execution settings.

6. Click **Deploy**.

For more information about configuring deployments, release configurations, and workflow configurations, see [Dataform Deployments](https://docs.cloud.google.com/dataform/docs/deployments).

## Disable a schedule

To pause the scheduled runs of a selected pipeline without deleting the schedule,
you can disable the schedule.

To disable a schedule for a selected pipeline, follow these steps:

### **Files** pane

1. In the Google Cloud console, go to the **BigQuery** page.

   [Go to BigQuery](https://console.cloud.google.com/bigquery)
2. In the left pane, click **Files**.

3. In the **Files** pane, expand your **User** folder or a **Team folder**, and then select a pipeline.

4. In the Pipeline Viewer toolbar, click **View trigger**.

5. In the **Schedule details** table, in the **Schedule state** row,
   click the **Schedule is enabled** toggle.

### **Explorer** pane

1. In the Google Cloud console, go to the **BigQuery** page.

   [Go to BigQuery](https://console.cloud.google.com/bigquery)
2. In the left pane, click **Explorer**:

   ![Highlighted button for the Explorer pane.](https://docs.cloud.google.com/static/bigquery/images/explorer-tab.png)
3. In the **Explorer** pane, expand your project, click **Pipelines**,
   and then select a pipeline.

4. Click **View trigger**.

5. In the **Schedule details** table, in the **Schedule state** row,
   click the **Schedule is enabled** toggle.

### **Scheduling** page

1. In the Google Cloud console, go to the **Scheduling** page.

   [Go to Scheduling](https://console.cloud.google.com/bigquery/orchestration)
2. Click the name of the selected pipeline.

3. On the **Schedule details** page, click **Disable**.

## Enable a schedule

To resume scheduled runs of a disabled pipeline schedule, follow these steps:

### **Files** pane

1. In the Google Cloud console, go to the **BigQuery** page.

   [Go to BigQuery](https://console.cloud.google.com/bigquery)
2. In the left pane, click **Files**.

3. In the **Files** pane, expand your **User** folder or a **Team folder**, and then select a pipeline.

4. In the **Pipeline Viewer** toolbar, click **View trigger**.

5. In the **Schedule details** table, in the **Schedule state** row,
   click the **Schedule is disabled** toggle.

### **Explorer** pane

1. In the Google Cloud console, go to the **BigQuery** page.

   [Go to BigQuery](https://console.cloud.google.com/bigquery)
2. In the left pane, click **Explorer**:

   ![Highlighted button for the Explorer pane.](https://docs.cloud.google.com/static/bigquery/images/explorer-tab.png)
3. In the **Explorer** pane, expand your project, click **Pipelines**,
   and then select a pipeline.

4. Click **View trigger**.

5. In the **Schedule details** table, in the **Schedule state** row,
   click the **Schedule is disabled** toggle.

### **Scheduling** page

1. In the Google Cloud console, go to the **Scheduling** page.

   [Go to Scheduling](https://console.cloud.google.com/bigquery/orchestration)
2. Click the name of the selected pipeline.

3. On the **Schedule details** page, click **Enable**.

## Manually run a deployed pipeline

When you manually run a pipeline deployed in a selected schedule,
BigQuery executes the deployed pipeline once,
independently from the schedule.

To manually run a deployed pipeline, follow these steps:

1. In the Google Cloud console, go to the **Scheduling** page.

   [Go to Scheduling](https://console.cloud.google.com/bigquery/orchestration)
2. Click the name of the selected pipeline schedule.

3. On the **Schedule details** page, click **Run**.

## View all pipeline schedules

To view all pipeline schedules in your Google Cloud project, follow these steps:

1. In the Google Cloud console, go to the **Scheduling** page.

   [Go to Scheduling](https://console.cloud.google.com/bigquery/orchestration)
2. Optional: To display additional columns with pipeline schedule details,
   click **Column display options** ,
   and then select columns and click **OK**.

## View pipeline schedule details

To view details for a selected pipeline schedule, follow these steps:

### **Files** pane

1. In the Google Cloud console, go to the **BigQuery** page.

   [Go to BigQuery](https://console.cloud.google.com/bigquery)
2. In the left pane, click **Files**.

3. In the **Files** pane, expand your **User** folder or a **Team folder**, and then select a pipeline.

4. In the **Pipeline Viewer** toolbar, click **View trigger** (or **View trigger**).

### **Explorer** pane

1. In the Google Cloud console, go to the **BigQuery** page.

   [Go to BigQuery](https://console.cloud.google.com/bigquery)
2. In the left pane, click **Explorer**:

   ![Highlighted button for the Explorer pane.](https://docs.cloud.google.com/static/bigquery/images/explorer-tab.png)
3. In the **Explorer** pane, expand your project, click **Pipelines**,
   and then select a pipeline.

4. Click **View trigger**.

### **Scheduling** page

1. In the Google Cloud console, go to the **Scheduling** page.

   [Go to Scheduling](https://console.cloud.google.com/bigquery/orchestration)
2. Click the name of the selected pipeline schedule.

## View past scheduled runs

To view past runs of a selected pipeline schedule, follow these steps:

### **Files** pane

1. In the Google Cloud console, go to the **BigQuery** page.

   [Go to BigQuery](https://console.cloud.google.com/bigquery)
2. In the left pane, click **Files**.

3. In the **Files** pane, expand your **User** folder, a **Team folder**, or a Git repository, and then select a pipeline.

4. In the **Pipeline Viewer** toolbar, click **Executions**.

5. Optional: To refresh the list of past runs, click **Refresh**.

### **Explorer** pane

1. In the Google Cloud console, go to the **BigQuery** page.

   [Go to BigQuery](https://console.cloud.google.com/bigquery)
2. In the left pane, click **Explorer**:

   ![Highlighted button for the Explorer pane.](https://docs.cloud.google.com/static/bigquery/images/explorer-tab.png)
3. In the **Explorer** pane, expand your project, click **Pipelines**,
   and then select a pipeline.

4. Click **Executions**.

5. Optional: To refresh the list of past runs, click **Refresh**.

### **Scheduling** page

1. In the Google Cloud console, go to the **Scheduling** page.

   [Go to Scheduling](https://console.cloud.google.com/bigquery/orchestration)
2. Click the name of the selected pipeline.

3. On the **Schedule details** page, in the **Past executions** section,
   inspect past runs.

4. Optional: To refresh the list of past runs, click **Refresh**.

## Edit a pipeline schedule

To edit a pipeline schedule, follow these steps:

### **Files** pane

1. In the Google Cloud console, go to the **BigQuery** page.

   [Go to BigQuery](https://console.cloud.google.com/bigquery)
2. In the left pane, click **Files**.

3. In the **Files** pane, expand your **User** folder or a **Team folder**, and then select a pipeline.

4. In the **Pipeline Viewer** toolbar, click **View trigger** , and then click **Edit**.

5. In the **Schedule pipeline** dialog, edit the schedule,
   and then click **Update schedule**.

### **Explorer** pane

1. In the Google Cloud console, go to the **BigQuery** page.

   [Go to BigQuery](https://console.cloud.google.com/bigquery)
2. In the left pane, click **Explorer**:

   ![Highlighted button for the Explorer pane.](https://docs.cloud.google.com/static/bigquery/images/explorer-tab.png)
3. In the **Explorer** pane, expand your project, click **Pipelines**,
   and then select a pipeline.

4. Click **View trigger** , and then click **Edit**.

5. In the **Schedule pipeline** dialog, edit the schedule,
   and then click **Update schedule**.

### **Scheduling** page

1. In the Google Cloud console, go to the **Scheduling** page.

   [Go to Scheduling](https://console.cloud.google.com/bigquery/orchestration)
2. Click the name of the selected pipeline.

3. On the **Schedule details** page, click **Edit**.

4. Click **View trigger** , and then click **Edit**.

5. In the **Schedule pipeline** dialog, edit the schedule,
   and then click **Update schedule**.

## Delete a pipeline schedule

To permanently delete a pipeline schedule, follow these steps:

1. In the Google Cloud console, go to the **Scheduling** page.

   [Go to Scheduling](https://console.cloud.google.com/bigquery/orchestration)
2. Do either of the following:

   - Click the name of the selected pipeline schedule, and then
     on the **Schedule details** page, click **Delete**.

   - In the row that contains the selected pipeline schedule, click
     **View actions** in the **Actions** column, and then click **Delete**.

3. In the dialog that appears, click **Delete**.

## What's next

- Learn more about [pipelines in BigQuery](https://docs.cloud.google.com/bigquery/docs/pipelines-introduction).
- Learn how to [create pipelines](https://docs.cloud.google.com/bigquery/docs/create-pipelines).
- Learn how to [manage pipelines](https://docs.cloud.google.com/bigquery/docs/manage-pipelines).
- Learn how to [manage code with BigQuery Studio Git repositories](https://docs.cloud.google.com/bigquery/docs/git-repositories).
- Learn how to [organize code assets with folders](https://docs.cloud.google.com/bigquery/docs/code-asset-folders).
- Learn more about [Dataform Deployments](https://docs.cloud.google.com/dataform/docs/deployments).