# Create pipelines
You can create [pipelines in BigQuery](https://docs.cloud.google.com/bigquery/docs/pipelines-introduction),
which let you orchestrate and automate workflows such as scheduling SQL queries,
notebooks, and data preparation tasks to run in sequence.
## Before you begin
### VPC Service Controls requirements
If you use VPC Service Controls to protect your pipelines, you should be aware that
pipeline runs are powered by Dataform. When you configure
VPC Service Controls for scheduled runs, ensure that the following requirements are
met:
- You must set the [`dataform.restrictGitRemotes` Organization Policy Service](https://docs.cloud.google.com/dataform/docs/restrict-git-remotes).
- Dataform and BigQuery must be restricted by the same VPC Service Controls service perimeter.
- To allow users to authenticate with the user credentials for their Google Account when scheduling or manually triggering runs, you must add their user identities to your ingress rules. For more information, see [Updating ingress and egress policies for a service perimeter](https://docs.cloud.google.com/vpc-service-controls/docs/configuring-ingress-egress-policies#updating) and [Ingress rules reference](https://docs.cloud.google.com/vpc-service-controls/docs/ingress-egress-rules#ingress-rules-reference).
For detailed configuration steps and security considerations, see
[Configure VPC Service Controls for Dataform](https://docs.cloud.google.com/dataform/docs/vpc-service-controls).
### Required roles for pipelines
To get the permissions that
you need to create pipelines,
ask your administrator to grant you the
following IAM roles:
- To create pipelines: [Code Creator](https://docs.cloud.google.com/iam/docs/roles-permissions/dataform#dataform.codeCreator) (`roles/dataform.codeCreator`) on the project
- To edit and run pipelines: [Dataform Editor](https://docs.cloud.google.com/iam/docs/roles-permissions/dataform#dataform.editor) (`roles/dataform.editor`) on the project
- To create pipelines in user folders:
- [Code Creator](https://docs.cloud.google.com/iam/docs/roles-permissions/dataform#dataform.codeCreator) (`roles/dataform.codeCreator`) on the project
- [Code Owner](https://docs.cloud.google.com/iam/docs/roles-permissions/dataform#dataform.codeOwner) (`roles/dataform.codeOwner`) on the folder
- [Code Editor](https://docs.cloud.google.com/iam/docs/roles-permissions/dataform#dataform.codeEditor) (`roles/dataform.codeEditor`) on the folder
- To create pipelines in team folders:
- [Team Folder Contributor](https://docs.cloud.google.com/iam/docs/roles-permissions/dataform#dataform.teamFolderContributor) (`roles/dataform.teamFolderContributor`) on the team folder
- [Team Folder Owner](https://docs.cloud.google.com/iam/docs/roles-permissions/dataform#dataform.teamFolderOwner) (`roles/dataform.teamFolderOwner`) on the team folder
- To connect a Git repository for Git folders: [Developer Connect OAuth User](https://docs.cloud.google.com/iam/docs/roles-permissions/developerconnect#developerconnect.oauthUser) (`roles/developerconnect.oauthUser`) on the project
For more information about granting roles, see [Manage access to projects, folders, and organizations](https://docs.cloud.google.com/iam/docs/granting-changing-revoking-access).
You might also be able to get
the required permissions through [custom
roles](https://docs.cloud.google.com/iam/docs/creating-custom-roles) or other [predefined
roles](https://docs.cloud.google.com/iam/docs/roles-overview#predefined).
For more information about Dataform IAM, see
[Control access with IAM](https://docs.cloud.google.com/dataform/docs/access-control).
For more information about roles for folders, see
[Create and manage folders](https://docs.cloud.google.com/bigquery/docs/create-manage-folders#required_roles).
For more information about roles for Git repositories, see
[Manage code with BigQuery Studio Git repositories](https://docs.cloud.google.com/bigquery/docs/git-repositories#required_roles).
> [!NOTE]
> **Note:** When you create a pipeline, BigQuery grants you the [Dataform Admin role](https://docs.cloud.google.com/dataform/docs/access-control#dataform.admin) (`roles/dataform.admin`) on that pipeline. All users with the Dataform Admin role granted on the Google Cloud project have owner access to all the pipelines created in the project. To override this behavior, see [Grant a specific role upon resource creation](https://docs.cloud.google.com/dataform/docs/access-control#grant-specific-role).
### Required roles for notebook options
To get the permissions that
you need to select a runtime template in notebook options,
ask your administrator to grant you the
[Notebook Runtime User](https://docs.cloud.google.com/iam/docs/roles-permissions/aiplatform#aiplatform.notebookRuntimeUser) (`roles/aiplatform.notebookRuntimeUser`) IAM role on the project.
For more information about granting roles, see [Manage access to projects, folders, and organizations](https://docs.cloud.google.com/iam/docs/granting-changing-revoking-access).
You might also be able to get
the required permissions through [custom
roles](https://docs.cloud.google.com/iam/docs/creating-custom-roles) or other [predefined
roles](https://docs.cloud.google.com/iam/docs/roles-overview#predefined).
If you don't have this role, you can select the default notebook runtime
specification.
### Security considerations for pipelines
Because code assets in BigQuery are powered by Dataform, you should consider the following security implications for users with access to these assets:
- Visibility for code assets is governed by project-level Dataform permissions. Users with the `dataform.repositories.list` permission---which is included in standard BigQuery roles such as [BigQuery Job User](https://docs.cloud.google.com/bigquery/docs/access-control#bigquery.jobUser), [BigQuery Studio User](https://docs.cloud.google.com/bigquery/docs/access-control#bigquery.studioUser), and [BigQuery User](https://docs.cloud.google.com/bigquery/docs/access-control#bigquery.user)---can see all code assets in the **Explorer** panel of the Google Cloud project, regardless of whether they created these assets or these assets were shared with them. To restrict visibility, you can create [custom roles](https://docs.cloud.google.com/iam/docs/creating-custom-roles) that exclude the `dataform.repositories.list` permission.
- Any secrets shared with the Dataform service agent can potentially be accessed by users who can edit these assets. To secure your credentials, restrict creation and edit access to trusted users, and limit the secrets accessible to the Dataform service agent. For more information, see [Secrets access during package installation](https://docs.cloud.google.com/dataform/docs/access-control#secret-access-risk).
For more information, see [Security considerations for Dataform permissions](https://docs.cloud.google.com/dataform/docs/access-control#security-considerations-permissions).
### Set the default region for code assets
All new code assets in your Google Cloud project use a default region. After the
asset is created, you can't change its region.
> [!IMPORTANT]
> **Important:** If you change the region while creating a code asset, that region becomes the default for all subsequent code assets. Existing code assets are not affected.
To set the default region for new code assets, do the following:
1. Go to the **BigQuery** page.
[Go to BigQuery](https://console.cloud.google.com/bigquery)
2. In the left pane, click **Files**
to open the file browser:
3. Next to the project name, click
**View files panel actions** \> **Switch code region**.
4. Select the code region that you want to use as a default.
5. Click **Save**.
For a list of supported regions, see
[BigQuery Studio locations](https://docs.cloud.google.com/bigquery/docs/locations#bqstudio-loc).
## Create a pipeline
You can create and store pipelines in the following ways:
- **In folders** : organize pipelines inside personal or team folders in the **Files** pane of BigQuery Studio.
- **In BigQuery Studio Git folders** ([Preview](https://cloud.google.com/products#product-launch-stages)): host, version control, and manage multiple independent pipelines or root-level pipelines within a folder with a Git repository connected using Developer Connect.
- **From the editor tab bar** : create a pipeline from the **+** menu in the query editor.
- **From the Pipelines \& Connections page** ([Preview](https://cloud.google.com/products#product-launch-stages)): create a pipeline using a streamlined, BigQuery-specific workflow.
### Create a pipeline in a folder
You can create and organize pipelines directly inside your personal user folders or shared team folders in BigQuery Studio. For more information about folders, see [Organize code assets with folders](https://docs.cloud.google.com/bigquery/docs/code-asset-folders).
To create a pipeline in a folder, follow these steps:
1. In the Google Cloud console, go to the **BigQuery** page.
[Go to BigQuery](https://console.cloud.google.com/bigquery)
2. In the left pane, click **Files**
to open the file browser.
If you don't see the left pane, click **Expand left pane** to open the pane.
3. Do one of the following to create a pipeline:
- Select the user folder or team folder where you want to place the pipeline, click the **+** (Create) button on the toolbar, and select **Pipeline**.
- In the file tree, click **View actions** next to a folder name, select **Create** \> **Pipeline**.
- Open a folder to view its contents in the main panel, and click **Create Pipeline**.
4. In the creation dialog, enter a name for the pipeline.
5. Click **Create** . A new pipeline folder appears in the file tree with the **Pipeline** icon.
6. Click the pipeline folder to open the **Pipeline Viewer** , and then configure [pipeline settings](https://docs.cloud.google.com/bigquery/docs/create-pipelines#configure-pipeline-settings).
### Create a pipeline in a Git repository
> [!WARNING]
>
> **Preview**
>
>
> This product or feature is
>
> subject to the "Pre-GA Offerings Terms" in the General Service Terms section
> of the [Service Specific
> Terms](https://docs.cloud.google.com/terms/service-terms#1).
>
> Pre-GA products and features are available "as is" and might have limited support.
>
> For more information, see the
> [launch stage descriptions](https://cloud.google.com/products/#product-launch-stages).
> [!NOTE]
> **Note:** To provide feedback or request support for this feature, send an email to [cloud-pipelines-ui@google.com](mailto:cloud-pipelines-ui@google.com).
You can create and manage pipelines directly within Git-integrated folders in BigQuery Studio. A Git folder represents a remote GitHub or GitLab repository connected to BigQuery Studio using Developer Connect.
You can host and organize multiple independent pipelines within a single Git repository.
Git repository pipelines support Dataform [deployments](https://docs.cloud.google.com/dataform/docs/deployments) for scheduling, release configurations, and multi-environment orchestration.
To create a pipeline in a Git folder, follow these steps:
1. In the Google Cloud console, go to the **BigQuery** page.
[Go to BigQuery](https://console.cloud.google.com/bigquery)
2. If you have not already connected your Git repository, [create and connect a BigQuery Studio Git repository](https://docs.cloud.google.com/bigquery/docs/git-repositories#create-git-repository).
3. In the left pane, click **Files**
to open the file browser.
4. In the file tree, locate your connected Git repository folder or a nested directory within it.
5. Click **View actions** next to the folder name, and select **Create** \> **Pipeline**.
6. In the dialog, enter the pipeline name and click **Save**.
A new pipeline directory is created, initialized with the default files:
- `workflow_settings.yaml`: Pipeline configuration.
- `definitions/actions.yaml`: Empty task definitions.
7. Click the pipeline directory in the file browser to open the **Pipeline Viewer**.
> [!NOTE]
> **Note:** If your remote Git repository contains the pipeline configuration (`workflow_settings.yaml`) in the root directory of the repository, BigQuery Studio automatically recognizes the repository as a **Root-Level Pipeline** .The Git repository folder itself displays the **Pipeline** icon. When you click the Git repository folder, the **Pipeline Viewer** opens, displaying the compiled DAG for the entire repository. You don't need to create a nested pipeline directory.
### Create a pipeline from the editor tab bar
To create a pipeline from the editor tab bar, follow these steps:
1. In the Google Cloud console, go to the **BigQuery** page.
[Go to BigQuery](https://console.cloud.google.com/bigquery)
2. In the tab bar of the editor pane, click the
arrow next to the **+** sign, and then click
**Pipeline**.
3. Optional: To rename the pipeline, click the pipeline name, and then type a new name.
4. Configure [pipeline settings](https://docs.cloud.google.com/bigquery/docs/create-pipelines#configure-pipeline-settings).
### Create a pipeline from the Pipelines \& Connections page
You can use the BigQuery **Pipelines \& Connections**
page in the Google Cloud console to create a Dataform pipeline that uses a
[streamlined, BigQuery-specific workflow](https://docs.cloud.google.com/bigquery/docs/pipeline-connection-page).
This feature is in [preview](https://cloud.google.com/products/#product-launch-stages).
For instructions on how to create a pipeline from the BigQuery
**Pipelines \& Connections** page, see
[Create a data integration asset](https://docs.cloud.google.com/bigquery/docs/pipeline-connection-page#create_a_data_integration_asset).
## Configure pipeline settings
To configure settings for your pipeline, open the pipeline in the
**Pipeline Viewer** , click the **Settings** tab,
and configure the following sections:
1. In the **Authentication** section, choose to authorize the
pipeline with the user credentials for your Google Account or a service
account.
- To use the user credentials for your Google Account, select
**Run with my user credentials**.
Optional: In the **Extended access options** section, select the
additional services that your pipeline requires:
- **Knowledge Catalog**: Allows Google Cloud Knowledge Catalog metadata updates.
- **Google Drive**: Allows read-only access to Google Drive files.
- **Bigtable**: Allows read-only access to Google
Bigtable data.
> [!NOTE]
> **Note:** Authenticating API-based runs with user credentials isn't supported. To [run all the tasks in a pipeline](https://docs.cloud.google.com/bigquery/docs/create-pipelines#run-pipeline-all-tasks) using the Dataform API, you must configure the pipeline to use a service account.
- To use a service account, select
**Run with selected service account** , and then select a
service account. If you need to create a service account, click
**New service account**.
2. In the **Processing location** section, select a processing location for the
pipeline.
- To enable the automatic selection of a location, select
**Automatic location selection**. This option selects a location
based on the datasets referenced in the request. The selection process is
as follows:
- If your query references datasets from the same location, BigQuery uses that location.
- If your query references datasets from two or more different locations, an error occurs. For details about this limitation, see [Cross-region dataset replication](https://docs.cloud.google.com/bigquery/docs/data-replication).
- If your query doesn't reference any datasets, BigQuery defaults to the `US` multi-region.
- To pick a specific region, select **Region** , then choose a
region in the **Region** menu. Alternatively, you can use the
[`@@location` system variable](https://docs.cloud.google.com/bigquery/docs/reference/system-variables)
in your query. For more information, see
[Specify locations](https://docs.cloud.google.com/bigquery/docs/locations#specify_locations).
- To pick a multi-region, select **Multi-region** , then choose a
multi-region in the **Multi-region** menu.
The pipeline processing location doesn't need to match your default
storage location for code assets.
### SQLX options
To configure the SQLX settings for your pipeline, do the following in the
**SQLX options** section:
1. In the **Default project** field, enter the name of an existing
Google Cloud project. This value is used for `defaultProject` in the
`workflow_settings.yaml` file and for `defaultDatabase` in the
`dataform.json` file. The default project is used by pipeline tasks during their
execution.
> [!NOTE]
> **Note:** The project name isn't validated, so it's possible to enter any non-empty string. However, if the project doesn't exist, the pipeline execution fails.
2. Optional: In the **Default dataset** field, search for and select an existing
dataset. The list of available datasets is filtered based on the selected
project and processing location. This value is used for `defaultDataset` in
the `workflow_settings.yaml` file. The default dataset is used by pipeline tasks
during their execution.
> [!NOTE]
> **Note:** Setting the default dataset and then changing the pipeline's region invalidates the dataset selection. Changing the project can also invalidate the dataset selection. If a given dataset doesn't exist in the selected project, it is created.
### Notebook options
To add a notebook to your pipeline, do the following in the **Notebook options**
section:
1. In the **Runtime template field**, either accept the default notebook
runtime, or search for and select an existing runtime.
- To view specifications for the default runtime, click the adjacent arrow.
- To create a new runtime, see [Create a runtime template](https://docs.cloud.google.com/colab/docs/create-runtime-template).
> [!NOTE]
> **Note:** A notebook runtime template must be located in the same region as the pipeline that specifies it.
> [!NOTE]
> **Note:** If you use the default runtime for a notebook in a BigQuery pipeline, you can't change the network of the Gemini Enterprise Agent Platform runtime instance. The runtime is restricted to the default network, and selecting a different network isn't supported. If you use a custom runtime template, the runtime uses the network and subnetwork configured in the template. If your project enforces the `compute.requireShieldedVm` organization policy, ensure that your custom runtime template has Secure Boot enabled to avoid VM creation failures.
2. In the **Cloud Storage bucket** field, click **Browse**
and select or create a Cloud Storage bucket for storing the output
of notebooks in your pipeline.
3. Follow the steps in
[Add a principal to a bucket-level policy](https://docs.cloud.google.com/storage/docs/access-control/using-iam-permissions#bucket-add)
to add your custom Dataform service account as a principal to the
Cloud Storage bucket that you plan to use for storing output of
scheduled pipeline runs, and grant the
[Storage Admin role](https://docs.cloud.google.com/iam/docs/roles-permissions/storage#storage.admin)
(`roles/storage.admin`) to this principal.
The selected custom Dataform service account must be granted the
Storage Admin IAM role on the selected bucket.
## Add a pipeline task
You can add multiple tasks to a pipeline that executes code assets in a specific
sequence.
Supported code assets inside pipelines include:
- SQL queries
- Notebooks
- Data preparations
- Tables and Views (SQLX tasks)
### Task naming conventions
When you name a pipeline task, follow these naming conventions:
- Task names can only contain letters (a-z, A-Z), numbers (0-9), underscores (`_`), and hyphens (`-`).
- Task names must start with a letter, number, or underscore.
- Periods (`.`) are prohibited in task names. When naming or renaming a task, don't include file extensions such as `.sql` or `.py`. If a task name contains a period, the **Run** button is deactivated.
- Task names can contain characters with a total of up to 1,024 UTF-8 bytes.
- Task names must be unique within the pipeline container.
- Task names are case-sensitive.
### Add a task
To add a task to a pipeline, follow these steps:
1. In the Google Cloud console, go to the **BigQuery** page.
[Go to BigQuery](https://console.cloud.google.com/bigquery)
2. In the left pane, click
**Explorer**:
If you don't see the left pane, click
**Expand left pane**
to open the pane.
3. In the **Explorer** pane, expand your project, click **Pipelines**,
and then select a pipeline.
4. To add a code asset, select one of the following options:
### SQL query
You can either create a new query or import an existing one.
1. Do one of the following:
- Click **Add task** , and then select **Query**.
You can either create a new query or import an existing one.
- When working with pipelines in folders or Git folders,
right-click the pipeline folder in the
**Files** pane, and select **Create Query**.
- When working with pipelines in folders or Git folders,
click the **Add task** button in the **Pipeline Viewer**.
2. Optional: In the **Query task details** pane, in the
**Run after** menu, select a task to precede your query.
**Create a new query**
1. Click the
arrow menu next to **Edit Query** and
select either **In context** or **In new tab**.
2. Search for an existing query.
3. Select a query name and then press **Enter**.
4. Click **Save**.
5. Optional: To rename the query,
click the query name on the pipeline pane,
click **Edit Query**, click the existing query name at the top of the
screen, and then type a new name.
**Import an existing query**
1. Click the
arrow menu next to **Edit Query** and
click **Import a copy**.
2. Search for an existing query to import or select an existing
query from the search pane. When you import a
query, the original remains unchanged because the query's source
file is copied into the pipeline.
3. Click **Edit** to open the imported query.
4. Click **Save**.
### Notebook
You can either create a new notebook or import an existing one.
To change settings for notebook runtime templates, see
[Notebook options](https://docs.cloud.google.com/bigquery/docs/create-pipelines#notebook_options).
1. Do one of the following:
- Click **Add task** , and then select **Notebook**.
- When working with pipelines in folders or Git folders,
right-click the pipeline folder in the
**Files** pane, and select **Create Notebook**.
- When working with pipelines in folders or Git folders,
click the **Add task** button in the **Pipeline Viewer**.
1. Optional: In the **Notebook task details** pane, in the **Run after** menu, select a task to precede your notebook.
**Create a new notebook**
2. Click the
arrow menu next to **Edit Notebook** and
select either **In context** or **In new tab**.
3. Search for an existing notebook.
4. Select a notebook name and then press **Enter**.
5. Click **Save**.
6. Optional: To rename the notebook,
click the notebook name on the pipeline pane,
click **Edit Notebook**, click the existing notebook name at the top of
the screen, and then type a new name.
**Import an existing notebook**
1. Click the
arrow menu next to **Edit Notebook** and
click **Import a copy**.
2. Search for an existing notebook to import or select an existing
notebook from the search pane. When you import a
notebook, the original remains unchanged because the notebook's
source file is copied into the pipeline.
3. To open the imported notebook, click **Edit**.
4. Click **Save**.
### Data preparation
You can either create a new data preparation or import an existing one.
1. Do one of the following:
- Click **Add task** , and then select **Data preparation**.
- When working with pipelines in folders or Git folders,
right-click the pipeline folder in the
**Files** pane, and select **Create Data preparation**.
- When working with pipelines in folders or Git folders,
click the **Add task** button in the **Pipeline Viewer**.
2. Optional: In the **Data preparation task details** pane, in the
**Run after** menu, select a task to precede your data preparation.
**Create a new data preparation**
1. Click the
arrow menu adjacent to **Edit Data preparation** and
select either **In context** or **In new tab**.
2. Search for an existing data preparation.
3. Select a data preparation name and press **Enter**.
4. Click **Save**.
5. Optional: To rename the data preparation, click the data preparation
name on the pipeline pane, click **Edit Data preparation**, click the
name at the top of the screen, and enter a new name.
**Import an existing data preparation**
1. Click the
arrow drop-down menu next to **Edit Data preparation** and
click **Import a copy**.
2. Search for an existing data preparation to import or select an existing
data preparation from the search pane. When you import a data
preparation, the original remains unchanged because the data
preparation's source file is copied into the pipeline.
3. To open the imported data preparation, click **Edit**.
4. Click **Save**.
### Table
1. Do one of the following:
- Click **Add task** , and then select **Table**.
- When working with pipelines in folders or Git folders,
right-click the pipeline folder in the
**Files** pane, and select **Create Table**.
- When working with pipelines in folders or Git folders,
click the **Add task** button in the **Pipeline Viewer**.
1. In the **Create new** pane, select **Table** or **Incremental table**.
2. Verify the default project for the table, or select a new project.
3. Verify the default dataset for the table, or select a new dataset.
4. Enter a name for the table.
5. In the table task details pane, click **Open** to open the task.
6. Configure the task using the settings in
**Details \> Configuration** or in the `config` block of the
code editor for the table.
For metadata changes, use the **Configuration** tab. This tab lets
you edit a specific value in the `config` block from the code editor,
such as a string or an array, that is formatted like a JavaScript
object. Using this tab helps you avoid syntax errors and verify that
your settings are correct.
Optional: In the **Run after** menu, select a task to precede your
table.
You can also define the metadata for your pipeline task in the
`config` block in the editor. For more information, see
[Creating tables](https://docs.cloud.google.com/dataform/docs/reference/sample-scripts#creating_tables).
The editor validates your code and displays the validation status.
> [!NOTE]
> **Note:** When you use JavaScript functions as values in the `config` block, you can't edit the JavaScript functions on the **Configuration** tab.
1. Use the `metadata` key to specify information for Knowledge Catalog. This enrichment process supports the following metadata constructs:
- Overview: documentation and summary text for the entry.
- Generic aspects: semantic details such as table system and type information.
The following example configuration shows you how to add an overview and
generic metadata aspects to a table configuration for
Knowledge Catalog:
```javascript
config {
type: "table",
metadata: {
overview: "This table provides standardized trip data.",
extraProperties: {
generic: {
system: "BigQuery",
type: "fact table"
}
}
}
}
```
1. In **Details \> Compiled queries**, view the SQL compiled
from the SQLX code.
2. Click **Run** to run the SQL in your pipeline.
3. In **Query results**, inspect the data preview.
### View
1. Do one of the following:
- Click **Add task** , and then select **View**.
- When working with pipelines in folders or Git folders,
right-click the pipeline folder in the
**Files** pane, and select **Create View**.
- When working with pipelines in folders or Git folders,
click the **Add task** button in the **Pipeline Viewer**.
2. In the **Create new** pane, select **View** or **Materialized view**.
3. Verify the default project for the view, or select a new project.
4. Verify the default dataset for the view, or select a new dataset.
5. Enter a name for the view.
6. In the view task details pane, click **Open** to open the task.
7. Configure the task using the settings in
**Details \> Configuration** or in the `config` block of the
code editor for the view.
For metadata changes, use the **Configuration** tab. This tab lets
you edit a specific value in the `config` block from the code editor,
such as a string or an array, that is formatted like a JavaScript
object. Using this tab helps you avoid syntax errors and verify that
your settings are correct.
Optional: In the **Run after** menu, select a task to precede your
view.
You can also define the metadata for your pipeline task in the
`config` block in the editor. For more information, see
[Creating a view with Dataform core](https://docs.cloud.google.com/dataform/docs/reference/sample-scripts#create-view).
The editor validates your code and displays the validation status.
> [!NOTE]
> **Note:** When you use JavaScript functions as values in the `config` block, you can't edit the JavaScript functions on the **Configuration** tab.
1. Use the `metadata` key to specify information for Knowledge Catalog. This enrichment process supports the following metadata constructs:
- Overview: documentation and summary text for the entry.
- Generic aspects: semantic details such as table system and type information.
The following example configuration shows you how to add an overview and
generic metadata aspects to a table configuration for
Knowledge Catalog:
```javascript
config {
type: "view",
metadata: {
overview: "This view provides standardized trip data.",
extraProperties: {
generic: {
system: "BigQuery",
type: "view"
}
}
}
}
```
1. In **Details \> Compiled queries**, view the SQL compiled
from the SQLX code.
2. Click **Run** to run the SQL in your pipeline.
3. In **Query results**, inspect the data preview.
### Declare source
1. Click **Add task** , and then select **Declare source**.
2. In the **Declare source** pane, verify the default project for the data
source, or select a new project.
3. Verify the default dataset for the data source, or select a new
dataset.
4. In the **Table / View** field, select the table or view that you want
to use as a data source.
5. Click **Create**.
6. In the declare source task details pane, click **Open** to
open the task.
7. Configure the data source using the settings in
**Details \> Configuration** or in the `config` block of the
code editor.
For more information about configuring a data source, see
[Declare a data source](https://docs.cloud.google.com/dataform/docs/declare-source).
The editor validates your code and displays the validation status.
### Assertion data test
1. Click **Add task** , and then select **Data quality test**.
2. In the **Create new** pane, verify the default project for the data
quality test, or select a new project.
3. Verify the default dataset for the data quality test, or select a new
dataset.
4. In the **Table** field, enter a name for the data quality test.
5. Click **Create**.
6. In the data quality test task details pane, click **Open** to
open the task.
7. Configure assertions for the test using the settings in
**Details \> Configuration** or in the `config` block of the
code editor.
For more information about configuring assertions, see
[Test data quality](https://docs.cloud.google.com/dataform/docs/test-data#assertions).
Optional: In the **Run after** menu, select a task to precede your
data quality test in the pipeline sequence.
The editor validates your code and displays the validation status.
8. Click **Run** to run the data quality test as part of your
pipeline sequence.
### Unit test
1. Click **Add task** , and then select **Unit test**.
2. In the **Action to test** field, select the
table or view that you want to test.
3. In the **BigQuery Unit test** field, enter a name for the unit test.
4. Click **Create**.
5. In the task details pane, click **Open** to
open the unit test.
6. Configure inputs and the expected output rows for the unit test.
For more information about configuring unit tests, see
[Test data quality](https://docs.cloud.google.com/dataform/docs/test-data#unit-tests).
7. Optional: To ensure that the inputs and outputs are what you expect,
click **Run task** and select expected or actual statements.
8. Click **Run** to run the data quality test as part of your
pipeline sequence.
## Edit a pipeline task
To edit a pipeline task, follow these steps:
1. In the Google Cloud console, go to the **BigQuery** page.
[Go to BigQuery](https://console.cloud.google.com/bigquery)
2. In the left pane, click
**Explorer**:
If you don't see the left pane, click
**Expand left pane**
to open the pane.
3. In the **Explorer** pane, expand your project, click **Pipelines**,
and then select a pipeline.
4. Click the selected task.
5. To change the preceding task, in the **Run after** menu, select a task that
will precede your task.
6. To edit the contents of the selected task, click **Edit**.
7. In the new tab that opens, edit the task contents, and then save changes to
the task.
## Delete a pipeline task
To delete a task from a pipeline, follow these steps:
1. In the Google Cloud console, go to the **BigQuery** page.
[Go to BigQuery](https://console.cloud.google.com/bigquery)
2. In the left pane, click
**Explorer**:
If you don't see the left pane, click
**Expand left pane**
to open the pane.
3. In the **Explorer** pane, expand your project, click **Pipelines**,
and then select a pipeline.
4. Click the selected task.
5. In the **Task details** pane, click
Delete **Delete**.
## Share a pipeline
> [!IMPORTANT]
> **Important:** If you enhance security by setting the `enable_private_workspace` field [(Preview)](https://cloud.google.com/products#product-launch-stages) to `true` in the [`projects.locations.updateConfig` Dataform API method](https://docs.cloud.google.com/dataform/reference/rest/v1beta1/projects.locations/updateConfig), only the pipeline creator can read and write code in that pipeline. For more information, see [Enable private workspaces](https://docs.cloud.google.com/dataform/docs/access-control#enable-private-workspaces).
> [!NOTE]
> **Note:** You can share a pipeline but not a task within the pipeline.
To share a pipeline, follow these steps:
1. In the Google Cloud console, go to the **BigQuery** page.
[Go to BigQuery](https://console.cloud.google.com/bigquery)
2. In the left pane, click
**Explorer**:
If you don't see the left pane, click
**Expand left pane**
to open the pane.
3. In the **Explorer** pane, expand your project, click **Pipelines**,
and then select a pipeline.
4. Click **Share** , and then select **Manage permissions**.
5. Click **Add user/group**.
6. In the **New principals** field, enter the name of at least one user or group.
7. For **Assign Roles**, select a role.
8. Click **Save**.
## Share a link to a pipeline
1. In the Google Cloud console, go to the **BigQuery** page.
[Go to BigQuery](https://console.cloud.google.com/bigquery)
2. In the left pane, click
**Explorer**:
If you don't see the left pane, click
**Expand left pane**
to open the pane.
3. In the **Explorer** pane, expand your project, click **Pipelines**,
and then select a pipeline.
4. Click **Share** , and then select **Share link**. The URL for your pipeline
is copied to your computer's clipboard.
## Run a pipeline
When running a pipeline, you can choose to run all the tasks in the pipeline,
manually select specific tasks to run, run tasks with selected tags, or run unit tests.
### Run all the tasks in a pipeline
To manually run the current version of a pipeline, select one of the
following options:
### Console
To run all the tasks in a pipeline, do the following:
1. In the Google Cloud console, go to the **BigQuery** page.
[Go to BigQuery](https://console.cloud.google.com/bigquery)
2. In the left pane, click
**Explorer**:
If you don't see the left pane, click
**Expand left pane**
to open the pane.
3. In the **Explorer** pane, expand your project, click **Pipelines**,
and then select a pipeline.
4. Click **Run**
\> **Run all tasks** . If you selected
**Run with my user credentials** for your
[authentication](https://docs.cloud.google.com/bigquery/docs/create-pipelines#create_a_pipeline), you must
[authorize your Google Account](https://docs.cloud.google.com/bigquery/docs/create-pipelines#authorize-google-account).
5. Optional: To inspect the run,
[view past manual runs](https://docs.cloud.google.com/bigquery/docs/manage-pipelines#view-manual-runs).
### API
> [!NOTE]
> **Note:** The Dataform API doesn't support user credentials for pipeline runs. You must select a service account in the **Authentication** section of your pipeline settings to use the API.
To run a pipeline manually, compile the default workspace and use the
compilation result to create a workflow invocation.
1. To create a compilation result for the default workspace, use the
[`projects.locations.repositories.compilationResults.create` method](https://docs.cloud.google.com/dataform/reference/rest/v1/projects.locations.repositories.compilationResults/create).
Run the API request with the following information:
curl -X POST \
-H "Authorization: Bearer $(gcloud auth print-access-token)" \
-H "Content-Type: application/json" \
-d '{
"workspace": "projects/PROJECT_ID/locations/LOCATION/repositories/REPOSITORY_ID/workspaces/default"
}' \
"https://dataform.googleapis.com/v1/projects/PROJECT_ID/locations/LOCATION/repositories/REPOSITORY_ID/compilationResults"
Replace the following:
- `LOCATION`: the Google Cloud region for your repository, for example, `us-central1`. You can find the repository location in the Google Cloud console by navigating to the **Explorer** pane, selecting the pipeline, opening the **Settings** tab, and clicking **Open pipeline in Dataform** . The location is in the URL in the format of `/locations/LOCATION/`.
- `PROJECT_ID`: the unique identifier of your Google Cloud project.
- `REPOSITORY_ID`: the unique identifier for your Dataform repository, for example, `my-secure-repo`. You can find the repository ID in the Google Cloud console by navigating to the **Explorer** pane, selecting the pipeline, opening the **Settings** tab, and viewing the **Dataform repository ID** field.
2. In the response body, locate the `name` field and copy its value,
for example,
`projects/my-project/locations/us-central1/repositories/my-repo/compilationResults/12345-67890`.
3. Trigger the pipeline run using the
[`projects.locations.repositories.workflowInvocations.create` method](https://docs.cloud.google.com/dataform/reference/rest/v1/projects.locations.repositories.workflowInvocations/create).
Run the API request with the following information:
curl -X POST \
-H "Authorization: Bearer $(gcloud auth print-access-token)" \
-H "Content-Type: application/json" \
-d '{
"compilationResult": "COMPILATION_RESULT"
}' \
"https://dataform.googleapis.com/v1/projects/PROJECT_ID/locations/LOCATION/repositories/REPOSITORY_ID/workflowInvocations"
Replace the following:
- `COMPILATION_RESULT`: the full resource name of the compilation result that you copied in the previous step.
- `LOCATION`: the Google Cloud region for your repository, for example, `us-central1`.
- `PROJECT_ID`: the unique identifier of your Google Cloud project.
- `REPOSITORY_ID`: the unique identifier for your Dataform repository, for example, `my-secure-repo`.
### Run selected tasks in a pipeline
To run selected tasks in a pipeline, do the following:
1. In the Google Cloud console, go to the **BigQuery** page.
[Go to BigQuery](https://console.cloud.google.com/bigquery)
2. In the left pane, click
**Explorer**:
If you don't see the left pane, click
**Expand left pane**
to open the pane.
3. In the **Explorer** pane, expand your project, click **Pipelines**,
and then select a pipeline.
4. Click **Run**
\> **Select tasks to run**.
5. In the **Run** pane, in the **Authentication** section, authorize the
execution with the user credentials for your Google Account or a service
account.
- To use the user credentials for your Google Account,
select **Run with user credentials**.
Optional: In the **Extended access options** section, select the
additional services that your pipeline requires:
- **Knowledge Catalog**: Allows Google Cloud Knowledge Catalog metadata updates.
- **Google Drive**: Allows read-only access to Google Drive files.
- **Bigtable**: Allows read-only access to Google
Bigtable data.
- To use a custom service account, select
**Run with selected service account**, and then select a custom
service account.
> [!NOTE]
> **Note:** To see service accounts in the menu, you must have the `iam.serviceAccounts.list` permission at the project level, which is available in the [View Service Accounts role](https://docs.cloud.google.com/iam/docs/roles-permissions/iam#iam.serviceAccountViewer) (`roles/iam.serviceAccountViewer`). If you don't have this permission, you can select the service account by clicking **Enter manually** and entering the service account ID.
If you need to create a service account, click **New service account**.
6. Ensure **Selection of tasks** is selected.
7. In the **Select tasks to run** menu, search for specific tasks
and select the tasks that you want to run.
The **Tasks** table lists the tasks that you've selected. Click a task name
to open it directly in the SQL editor.
8. Optional: Configure the following execution options:
- **Include dependencies**: select this option to run the selected tasks and their dependencies.
- **Include dependents**: select this option to run the selected tasks and their transitive downstream dependents.
- **Run with full refresh**: select this option to rebuild all tables from scratch.
- **Run as interactive job with high priority (default)** : select this option to set the BigQuery query job priority. By default, BigQuery runs queries as [interactive query jobs](https://docs.cloud.google.com/bigquery/docs/running-queries#interactive-batch), which are intended to start running as quickly as possible. Clearing this option runs the queries as [batch query jobs](https://docs.cloud.google.com/bigquery/docs/running-queries#interactive-batch), which have lower priority.
- **Limit concurrent actions** : to limit how many tasks can run concurrently during the execution, select this option. Then, in the **Maximum concurrent actions** field, enter a whole number between 1 and 5000. Ready tasks exceeding the limit remain queued until a concurrency slot becomes available.
9. Click **Run** . If you selected **Run with user credentials**
for your authentication method, you must
[authorize your Google Account](https://docs.cloud.google.com/bigquery/docs/create-pipelines#authorize-google-account)
([Preview](https://cloud.google.com/products#product-launch-stages)).
10. Optional: To inspect the run,
[view past manual runs](https://docs.cloud.google.com/bigquery/docs/manage-pipelines#view-manual-runs).
### Run tasks with selected tags in a pipeline
To run tasks with selected tags in a pipeline, do the following:
1. In the Google Cloud console, go to the **BigQuery** page.
[Go to BigQuery](https://console.cloud.google.com/bigquery)
2. In the left pane, click
**Explorer**:
If you don't see the left pane, click
**Expand left pane**
to open the pane.
3. In the **Explorer** pane, expand your project, click **Pipelines**,
and then select a pipeline.
4. Click
**Run**
\> **Run by tag**, then do either of the following:
- Click a tag that you want to run.
- Click **Select tags to run**.
5. In the **Run** pane, in the **Authentication** section, authorize the execution
with the user credentials for your Google Account or a service account.
- To use the user credentials for your Google Account,
select **Run with user credentials**.
Optional: In the **Extended access options** section, select the
additional services that your pipeline requires:
- **Knowledge Catalog**: Allows Google Cloud Knowledge Catalog metadata updates.
- **Google Drive**: Allows read-only access to Google Drive files.
- **Bigtable**: Allows read-only access to Google
Bigtable data.
- To use a custom service account, select
**Run with selected service account**, and then select a custom
service account.
> [!NOTE]
> **Note:** To see service accounts in the menu, you must have the `iam.serviceAccounts.list` permission at the project level, which is available in the [View Service Accounts role](https://docs.cloud.google.com/iam/docs/roles-permissions/iam#iam.serviceAccountViewer) (`roles/iam.serviceAccountViewer`). If you don't have this permission, you can select the service account by clicking **Enter manually** and entering the service account ID.
If you need to create a service account, click **New service account**.
6. Ensure **Selection of tags** is selected.
7. In the **Select tags to run** menu, search for specific tags and select the
tags that you want to run.
The **Tasks** table lists the tasks that you've selected. Click a task name
to open it directly in the SQL editor.
8. Optional: Configure the following execution options:
- **Include dependencies**: select this option to run the selected tasks and their dependencies.
- **Include dependents**: select this option to run the selected tasks and their transitive downstream dependents.
- **Run with full refresh**: select this option to rebuild all tables from scratch.
- **Run as interactive job with high priority (default)** : select this option to set the BigQuery query job priority. By default, BigQuery runs queries as [interactive query jobs](https://docs.cloud.google.com/bigquery/docs/running-queries#interactive-batch), which are intended to start running as quickly as possible. Clearing this option runs the queries as [batch query jobs](https://docs.cloud.google.com/bigquery/docs/running-queries#interactive-batch), which have lower priority.
- **Limit concurrent actions** : to limit how many tasks can run concurrently during the execution, select this option. Then, in the **Maximum concurrent actions** field, enter a whole number between 1 and 5000. Ready tasks exceeding the limit remain queued until a concurrency slot becomes available.
9. Click **Run** . If you selected **Run with user credentials**
for your authentication method, you must
[authorize your Google Account](https://docs.cloud.google.com/bigquery/docs/create-pipelines#authorize-google-account)
([Preview](https://cloud.google.com/products#product-launch-stages)).
10. Optional: To inspect the run,
[view past manual runs](https://docs.cloud.google.com/bigquery/docs/manage-pipelines#view-manual-runs).
### Run unit tests in a pipeline
You [define unit tests in Dataform](https://docs.cloud.google.com/dataform/docs/test-data#unit-tests)
by creating `.sqlx` test files in the `definitions/` directory of your
repository. After you create unit tests in Dataform, you can run
them directly in your BigQuery pipeline to validate your SQL
transformation logic against mock datasets.
To run unit tests in a pipeline, do the following:
1. In the Google Cloud console, go to the **BigQuery** page.
[Go to BigQuery](https://console.cloud.google.com/bigquery)
2. In the left pane, click
**Explorer**:
If you don't see the left pane, click
**Expand left pane**
to open the pane.
3. In the **Explorer** pane, expand your project, click **Pipelines**,
and then select a pipeline.
4. Click **Run**.
5. In the **Run** pane, in the **Authentication** section, authorize the
execution with the user credentials for your Google Account or a service
account:
- To use the user credentials for your Google Account
([Preview](https://cloud.google.com/products#product-launch-stages)),
select **Run with user credentials**.
Optional: In the **Extended access options** section, select the
additional services that your pipeline requires:
- **Knowledge Catalog**: Allows Google Cloud Knowledge Catalog metadata updates.
- **Google Drive**: Allows read-only access to Google Drive files.
- **Bigtable**: Allows read-only access to Google
Bigtable data.
- To use a custom service account, select
**Run with selected service account**, and then select a custom
service account.
To see service accounts in the menu, you must have the
`iam.serviceAccounts.list` permission at the project level,
which is available in the
[View Service Accounts role](https://docs.cloud.google.com/iam/docs/roles-permissions/iam#iam.serviceAccountViewer)
(`roles/iam.serviceAccountViewer`). If you don't have this permission,
you can select the service account by clicking **Enter manually** and
entering the service account ID.
If you need to create a service account, click **New service account**.
6. In the **Execution mode** section, select **Unit tests**.
7. Select one of the following execution scope options:
- **Select unit tests**: search for and select specific unit tests from your pipeline.
- **Select unit test tags**: filter and execute unit tests assigned specific tags.
- **All Unit Tests**: execute all unit tests defined in the pipeline.
8. Optional: In the **Execution options** sections, select the
**Execute as interactive job with high priority** checkbox to run unit tests
immediately, prioritizing execution speed.
If you don't select the **Execute as interactive job with high priority**
checkbox, Dataform runs unit tests using batch
resources by default, prioritizing compute costs savings.
9. Click **Run** . If you selected **Run with user credentials**
for your authentication method, you must
[authorize your Google Account](https://docs.cloud.google.com/bigquery/docs/create-pipelines#authorize-google-account)
([Preview](https://cloud.google.com/products#product-launch-stages)).
10. Optional: To inspect the run,
[view past manual runs](https://docs.cloud.google.com/bigquery/docs/manage-pipelines#view-manual-runs).
### Authorize your Google Account
To authenticate the resource with your
[Google Account](https://docs.cloud.google.com/iam/docs/principals-overview#google-account)
user credentials, you must manually grant permission for BigQuery
pipelines to get the access token for your Google Account and access the source
data on your behalf. You can grant manual approval with the OAuth dialog
interface. If you select any **Extended access options**, you must grant access to those services---for example, Google Drive or Knowledge Catalog.
> [!NOTE]
> **Note:** Context-Aware Access (CAA) policies---including IP-based, geolocation-based, and device compliance policies---aren't supported when executing or scheduling BigQuery pipelines with user credentials for a Google Account, because the token requests originate from Google infrastructure. CAA policies block these executions unless the Dataform OAuth client ID is [exempted from the policies](https://docs.cloud.google.com/dataform/docs/troubleshooting#euc-permission-denied).
You only need to give permission to BigQuery pipelines once.
To revoke the permission that you granted, follow these steps:
1. Go to your [Google Account page](https://myaccount.google.com/).
2. Click **BigQuery Pipelines**.
3. Click **Remove access**.
> [!WARNING]
> **Warning:** Revoking access permissions prevents any future pipeline runs that this Google Account owns across all regions.
If your pipeline contains a notebook, you must also manually grant
permission for Colab Enterprise to get the access token for your
Google Account and access the source data on your behalf. You only need
to give permission once. You can revoke this permission on the
[Google Account page](https://myaccount.google.com/).
## What's next
- Learn more about [BigQuery pipelines](https://docs.cloud.google.com/bigquery/docs/pipelines-introduction).
- Learn how to [manage pipelines](https://docs.cloud.google.com/bigquery/docs/manage-pipelines).
- Learn how to [schedule pipelines](https://docs.cloud.google.com/bigquery/docs/schedule-pipelines).
- Learn how to [manage code with BigQuery Studio Git repositories](https://docs.cloud.google.com/bigquery/docs/git-repositories).
- Learn how to [organize code assets with folders](https://docs.cloud.google.com/bigquery/docs/code-asset-folders).
- Learn how to [test data quality with unit tests](https://docs.cloud.google.com/dataform/docs/test-data#unit-tests).