# Run a pipeline by using the job builder

This quickstart shows you how to run a Dataflow job by using the
[Dataflow job builder](https://docs.cloud.google.com/dataflow/docs/guides/job-builder). The job
builder is a visual UI for building and running Dataflow
pipelines in the Google Cloud console, without writing any code.

In this quickstart, you load an example pipeline into the job builder, run a
job, and verify that the job created output.

## Before you begin

Complete the following steps before running your pipeline.

### Set up your project

1. Create a Cloud Storage bucket:
   1. In the Google Cloud console, go to the Cloud Storage **Buckets** page.

      [Go to Buckets](https://console.cloud.google.com/storage/browser)
   2. Click **Create**.
   3. On the **Create a bucket** page, enter your bucket information. To go to the next step, click **Continue** .
      1. For **Name your bucket**, enter a unique bucket name. Don't include sensitive information in the bucket name, because the bucket namespace is global and publicly visible.
      2. In the **Choose where to store your data** section, do the following:
         1. Select a [Location type](https://docs.cloud.google.com/storage/docs/locations).
         2. Choose a location where your bucket's data is permanently stored from the **[Location type](https://docs.cloud.google.com/storage/docs/locations#available-locations)** drop-down menu.
            - If you select the [dual-region](https://docs.cloud.google.com/storage/docs/locations#location-dr) location type, you can also choose to enable [turbo replication](https://docs.cloud.google.com/storage/docs/availability-durability#turbo-replication) by using the relevant checkbox.
         3. To set up [cross-bucket replication](https://docs.cloud.google.com/storage/docs/availability-durability#cross-bucket-replication), select **Add cross-bucket replication via Storage Transfer Service** and follow these steps:

            #### Set up cross-bucket replication

            1. In the **Bucket** menu, select a bucket.
            2. In the **Replication settings** section,
               click **Configure** to configure settings for the
               replication job.

               The **Configure cross-bucket replication** pane
               appears.
               - To filter objects to replicate by object name prefix, enter a prefix that you want to include or exclude objects from, then click **Add a prefix**.
               - To set a storage class for the replicated objects, select a storage class from the **Storage class** menu. If you skip this step, the replicated objects will use the destination bucket's storage class by default.
               - Click **Done**.
      3. In the **Choose how to store your data** section, do the following:
         1. In the **Set a default class** section, select the following: **Standard**.
         2. To enable [hierarchical namespace](https://docs.cloud.google.com/storage/docs/hns-overview), in the **Optimize storage for data-intensive workloads** section, select **Enable hierarchical namespace on this bucket** . **Note:** You cannot enable hierarchical namespace in existing buckets.
      4. In the **Choose how to control access to objects** section, select whether or not your bucket enforces [public access prevention](https://docs.cloud.google.com/storage/docs/public-access-prevention), and select an [access control method](https://docs.cloud.google.com/storage/docs/access-control) for your bucket's objects. **Note:** You cannot change the **Prevent public access** setting if this setting is enforced at an [organization policy](https://docs.cloud.google.com/storage/docs/org-policy-constraints#public-access-prevention).
      5. In the **Choose how to protect object data** section, do the following:
         - Select any of the options under **Data protection** that you want to set for your bucket.
           - To enable [soft delete](https://docs.cloud.google.com/storage/docs/soft-delete), click the **Soft delete policy (For data recovery)** checkbox, and specify the number of days you want to retain objects after deletion.
           - To set [Object Versioning](https://docs.cloud.google.com/storage/docs/object-versioning), click the **Object versioning (For version control)** checkbox, and specify the maximum number of versions per object and the number of days after which the noncurrent versions expire.
           - To enable the retention policy on objects and buckets, click the **Retention (For compliance)** checkbox, and then do the following:
             - To enable [Object Retention Lock](https://docs.cloud.google.com/storage/docs/object-lock), click the **Enable object retention** checkbox.
             - To enable [Bucket Lock](https://docs.cloud.google.com/storage/docs/bucket-lock), click the **Set bucket retention policy** checkbox, and choose a unit of time and a length of time for your retention period.
         - To choose how your object data will be encrypted, expand the **Data encryption** section (), and select a [**Data encryption** method](https://docs.cloud.google.com/storage/docs/encryption).
   4. Click **Create**.

<br />

### Required roles


To get the permissions that
you need to run this quickstart,

ask your administrator to grant you the
following IAM roles on your project:

- [Dataflow Developer](https://docs.cloud.google.com/iam/docs/roles-permissions/dataflow#dataflow.developer) (`roles/dataflow.developer`)
- [Service Account User](https://docs.cloud.google.com/iam/docs/roles-permissions/iam#iam.serviceAccountUser) (`roles/iam.serviceAccountUser`)


For more information about granting roles, see [Manage access to projects, folders, and organizations](https://docs.cloud.google.com/iam/docs/granting-changing-revoking-access).


You might also be able to get
the required permissions through [custom
roles](https://docs.cloud.google.com/iam/docs/creating-custom-roles) or other [predefined
roles](https://docs.cloud.google.com/iam/docs/roles-overview#predefined).


To ensure that the service accounts has the necessary
permissions to run this quickstart,

ask your administrator to grant the
following IAM roles to the service accounts on your project:

**Important:** You must grant these roles to the service accounts, *not* to your user account. Failure to grant the roles to the correct principal might result in permission errors.

- [Dataflow Worker](https://docs.cloud.google.com/iam/docs/roles-permissions/dataflow#dataflow.worker) (`roles/dataflow.worker`)
- [Storage Object Admin](https://docs.cloud.google.com/iam/docs/roles-permissions/storage#storage.objectAdmin) (`roles/storage.objectAdmin`)


For more information about granting roles, see [Manage access to projects, folders, and organizations](https://docs.cloud.google.com/iam/docs/granting-changing-revoking-access).


Your administrator might also be able to give the service accounts
the required permissions through [custom
roles](https://docs.cloud.google.com/iam/docs/creating-custom-roles) or other [predefined
roles](https://docs.cloud.google.com/iam/docs/roles-overview#predefined).

### VPC network

By default, each new project starts with a
[default network](https://docs.cloud.google.com/vpc/docs/vpc#default-network).
If the default network for your project is [disabled](https://docs.cloud.google.com/vpc/docs/vpc#org-policies)
or was deleted, you need to have a network in your project for which
your user account has the
[Compute Network User role](https://docs.cloud.google.com/compute/docs/access/iam#compute.networkUser)
(`roles/compute.networkUser`).

## Load the example pipeline

In this step, you load an example pipeline that counts the words in
Shakespeare's *King Lear*.

1. Go to the **Jobs** page in the Google Cloud console.

   [Go to Jobs](https://console.cloud.google.com/dataflow)
2. Click **Create job from
   template**.

3. Click **Job builder**.

4. Click **Load Blueprints**.

5. Click **Word Count**. The job builder is populated with a graphical
   representation of the pipeline.

For each pipeline step, the job builder displays a card that specifies the
configuration parameters for that step. For example, the first step reads text
files from Cloud Storage. The location of the source data is pre-populated in
the **Text location** box.

![A screenshot of the job builder](https://docs.cloud.google.com/static/dataflow/images/job-builder-quickstart.png)

## Set the output location

In this step, you specify a Cloud Storage bucket where the pipeline writes
output.

1. Locate the card titled **New sink**. You might need to scroll.

2. In the **Text location** box, click **Browse**.

3. Select the name of the Cloud Storage bucket that you created in
   [Before you begin](https://docs.cloud.google.com/dataflow/docs/quickstarts/create-pipeline-job-builder#before-you-begin).

4. Click **View child resources**.

5. In the Filename box, enter `words`.

6. Click **Select**.

## Run the job

Click **Run job** . The job builder creates a Dataflow job and then
navigates to the [job graph](https://docs.cloud.google.com/dataflow/docs/guides/job-graph). When the job
starts, the job graph shows a graphical representation of the pipeline, similar
to the one shown in the job builder. As each step of the pipeline runs, the
status is updated in the job graph.

The **Job info** panel shows the overall status of the job. If the job completes
successfully, the **Job status** field updates to `Succeeded`.

## Examine the job output

When the job completes, perform the following steps to see the output from the
pipeline:

1. In the Google Cloud console, go to the Cloud Storage **Buckets** page.

   [Go to Buckets](https://console.cloud.google.com/storage/browser)
2. In the bucket list, click the name of the bucket that you created in
   [Before you begin](https://docs.cloud.google.com/dataflow/docs/quickstarts/create-pipeline-job-builder#before-you-begin).

3. Click the file named `words-00000-of-00001`.

4. In the **Object details** page, click the authenticated URL to view the
   pipeline output.

The output should look similar to the following:

    brother: 20
    deeper: 1
    wrinkles: 1
    'alack: 1
    territory: 1
    dismiss'd: 1
    [....]

## Clean up


To avoid incurring charges to your Google Cloud account for
the resources used on this page, follow these steps.

### Delete the project

The easiest way to eliminate billing is to delete the Google Cloud project
that you created for the quickstart.

> [!CAUTION]
> **Caution** : Deleting a project has the following effects:
>
> - **Everything in the project is deleted.** If you used an existing project for the tasks in this document, when you delete it, you also delete any other work you've done in the project.
> - **Custom project IDs are lost.** When you created this project, you might have created a custom project ID that you want to use in the future. To preserve the URLs that use the project ID, such as an `appspot.com` URL, delete selected resources inside the project instead of deleting the whole project.
>
>
> If you plan to explore multiple architectures, tutorials, or quickstarts, reusing projects
> can help you avoid exceeding project quota limits.

1. In the Google Cloud console, go to the **Manage resources** page.

   [Go to Manage resources](https://console.cloud.google.com/iam-admin/projects)
2. In the project list, select the project that you want to delete, and then click **Delete**.
3. In the dialog, type the project ID, and then click **Shut down** to delete the project.

<br />

### Delete the individual resources

If you want to keep the Google Cloud project that you used in this
quickstart, then delete the Cloud Storage bucket:

1. In the Google Cloud console, go to the Cloud Storage **Buckets** page.

   [Go to Buckets](https://console.cloud.google.com/storage/browser)
2. Click the checkbox for the bucket that you want to delete.
3. To delete the bucket, click **Delete**, and then follow the instructions.

## What's next

- [Use the job builder to create a pipeline](https://docs.cloud.google.com/dataflow/docs/guides/job-builder).
- [Read about the Apache Beam programming model](https://docs.cloud.google.com/dataflow/model/programming-model-beam).
- [Use Apache Beam to build pipelines](https://docs.cloud.google.com/dataflow/docs/guides/use-beam)