Create and manage deployments

Preview

This product or feature is subject to the "Pre-GA Offerings Terms" in the General Service Terms section of the Service Specific Terms. Pre-GA products and features are available "as is" and might have limited support. For more information, see the launch stage descriptions.

Note: To provide feedback or ask questions that are related to this Preview feature, contact [e-mail].

Deployments let you schedule code to run—for example, pipelines—that's stored in remote Git repositories. You can choose the directory where your code is located and perform the following actions:

• Create and manage deployments connected to remote repositories.
• Configure triggers to pull code from remote repositories and run code on a regular basis.
• Run pipelines on a schedule or with other triggers.
• View logs of your executions directly within the Google Cloud console. For alerting, Dataform deployments integrate with Cloud Logging and Cloud Monitoring.

Limitations

• Deployments don't have built-in alerting. You must use Logging and Monitoring to configure alerts.
• Authentication with Developer Connect only supports connections and not account connectors.

Before you begin

Before you create a deployment, complete the following tasks:

• Opt in to using the Pipelines & Connections page in the Google Cloud console.

• Enable the BigQuery API, Dataform API, and Developer Connect API:

  Enable the APIs

• Connect a remote repository using Developer Connect.

Required roles

To get the permissions that you need to complete the tasks in this document, ask your administrator to grant you the following IAM roles:

• Dataform Admin (roles/dataform.admin) on the project
• BigQuery Job User (roles/bigquery.jobUser) on the project
• Developer Connect Viewer (roles/developerconnect.viewer) on the project
• Use a custom service account to run a deployment: Service Account User (roles/iam.serviceAccountUser) on the custom service account

For more information about granting roles, see Manage access to projects, folders, and organizations.

You might also be able to get the required permissions through custom roles or other predefined roles.

To use a custom service account to run a deployment, you must also grant the BigQuery Data Editor role (roles/bigquery.dataEditor) to the custom service account.

Create a deployment

To create a Dataform deployment, do the following:

1. In the Google Cloud console, go to the Pipelines & Connections page.

   Go to Pipelines & Connections

2. Click Create > Create deployment.

3. Select a repository authentication type, and then select one of the following options:

   Developer Connect

   1. Select a repository and branch.
   2. Enter the path for the directory where the code is located.
   3. Enter a name for the deployment.
   4. Select a region for the deployment.

   5. In the Schedule frequency section, do the following:

      1. In the Repeats menu, select the frequency of scheduled pipeline runs.
      2. In the At time field, enter the time for scheduled pipeline runs.
      3. In the Timezone menu, select the time zone for the schedule.

      Alternatively, select On-demand to run your deployment on demand.

   6. Optional: Enter the Google Cloud project ID.

   7. Optional: Enter a schema suffix.

   8. Optional: Enter a table prefix.

   9. Optional: Enter the default schema.

   10. Optional: Enter compilation variables.

   11. Click Deploy.

   HTTPS

   1. Enter the URL of your remote Git repository.
   2. Select a branch.
   3. Enter the path for the directory where the code is located.
   4. Enter the secret containing the personal access token for HTTPS remotes. For more information, see Connect a remote repository through HTTPS.
   5. Enter a name for the deployment.
   6. Select a region for the deployment.

   7. In the Schedule frequency section, do the following:

      1. In the Repeats menu, select the frequency of scheduled pipeline runs.
      2. In the At time field, enter the time for scheduled pipeline runs.
      3. In the Timezone menu, select the time zone for the schedule.

      Alternatively, select On-demand to run your deployment on demand.

   8. Optional: Enter the Google Cloud project ID.

   9. Optional: Enter a schema suffix.

   10. Optional: Enter a table prefix.

   11. Optional: Enter the default schema.

   12. Optional: Enter compilation variables.

   13. Click Deploy.

   SSH

   1. Enter the URL of your remote Git repository.
   2. Select a branch.
   3. Enter the path for the directory where the code is located.
   4. Enter the secret containing a user private key and the SSH public host key value. For more information, see Connect a remote repository through SSH.
   5. Enter a name for the deployment.
   6. Select a region for the deployment.

   7. In the Schedule frequency section, do the following:

      1. In the Repeats menu, select the frequency of scheduled pipeline runs.
      2. In the At time field, enter the time for scheduled pipeline runs.
      3. In the Timezone menu, select the time zone for the schedule.

      Alternatively, select On-demand to run your deployment on demand.

   8. Optional: Enter the Google Cloud project ID.

   9. Optional: Enter a schema suffix.

   10. Optional: Enter a table prefix.

   11. Optional: Enter the default schema.

   12. Optional: Enter compilation variables.

   13. Click Deploy.

List deployments

To list the created deployments, do the following:

1. In the Google Cloud console, go to the Pipelines & Connections page.

   Go to Pipelines & Connections

2. View the list of deployments.

Create a deployment schedule

To create a schedule for a deployment, do the following:

1. In the Google Cloud console, go to the Pipelines & Connections page.

   Go to Pipelines & Connections

2. Click the Deployments tab, and then select a deployment.

3. On the Deployment details page, in the Triggers (workflow configurations) section, click Create.

4. In the Create workflow configuration pane, follow the steps to create a workflow configuration.

Run a deployment schedule

To run a schedule for a deployment, do the following:

1. In the Google Cloud console, go to the Pipelines & Connections page.

   Go to Pipelines & Connections

2. Click the Deployments tab and then select a deployment.

3. In the Triggers (workflow configurations) section, next to the deployment schedule that you want to run, click View actions > Run now.

Delete a deployment schedule

To delete a schedule for a deployment, do the following:

1. In the Google Cloud console, go to the Pipelines & Connections page.

   Go to Pipelines & Connections

2. Click the Deployments tab, and then select a deployment.

3. On the Deployment details page, in the Triggers (workflow configurations) section, next to the deployment schedule that you want to delete, click View actions > Delete.

Configure alerts

To receive notifications when a workflow run fails, configure alerts for failed workflow invocations.

What's next

• Learn how to create a repository.
• Learn how to schedule runs.
• Learn how to configure alerts.