# Create a pipeline monitoring dashboard using Cloud Monitoring

Learn how to use Cloud Monitoring to create a dashboard to monitor pipelines.

*** ** * ** ***

To follow step-by-step guidance for this task directly in the
Google Cloud console, click **Guide me**:

[Guide me](https://console.cloud.google.com/freetrial?redirectPath=/?walkthrough_id%3Ddata-fusion--pipeline-monitoring-dashboard)

*** ** * ** ***

## Before you begin

1.


   Enable the Cloud Data Fusion, BigQuery, Cloud Storage, and Dataproc APIs, if any are not already enabled.


   **Roles required to enable APIs**


   To enable APIs, you need the `serviceusage.services.enable` permission. If you
   created the project, then you likely already have this permission through the
   Owner role (`roles/owner`). Otherwise, you can get this permission through the
   Service Usage Admin role (`roles/serviceusage.serviceUsageAdmin`).
   [Learn how to grant roles](https://docs.cloud.google.com/iam/docs/granting-changing-revoking-access).

   [Enable the APIs](https://console.cloud.google.com/apis/enableflow?apiid=datafusion.googleapis.com,bigquery.googleapis.com,storage.googleapis.com,dataproc.googleapis.com)
2. To create custom dashboards, you must be granted the
   [Monitoring Editor](https://docs.cloud.google.com/iam/docs/roles-permissions/monitoring#monitoring.editor)
   (`roles/monitoring.editor`) IAM role on the service account.

   For more information about granting roles, see
   [Manage access](https://docs.cloud.google.com/iam/docs/granting-changing-revoking-access).

## Create a Cloud Data Fusion instance with Cloud Logging enabled

To use Cloud Logging with your Cloud Data Fusion pipeline, create a
Cloud Data Fusion instance with Cloud Logging enabled:

1. Go to the Cloud Data Fusion **Instances** page and
   click **Create instance**.


   [Create an instance](https://console.cloud.google.com/data-fusion/instance-create)

   <br />

   <br />

2. In the **Instance name** field, enter a name for your new instance.

3. From the **Region** drop-down, select the Google Cloud region in which you
   want to create the instance.

4. From the **Version** drop-down, select a Cloud Data Fusion version.

5. Select an **Edition**.

6. Expand **Advanced options**.

7. In the **Logging and monitoring** section, select
   **Enable Stackdriver logging service**.

8. Click **Create**.

After you create an instance, you can't enable Cloud Logging in the Google Cloud console.

<br />

## Create a log-based metric

1. Go to the Cloud Logging **Log-based metrics** page:


   [Go to Log-based metrics](https://console.cloud.google.com/logs/metrics)

   <br />

   <br />

2. Click **Create metric**.

3. On the **Create a metric** page, do the following:

   1. For **Metric type** , select **Counter**.
   2. In the **Log-based metric name** field, enter `pipeline_logs`.
   3. In the **Units** field, enter `1`.
   4. In the **Build filter** field, enter the following:

          resource.type="cloud_dataproc_cluster"
          log_name=~"projects/.*/logs/datafusion-pipeline-logs"

   5. In the **Labels** section, click **Add label** and create the following
      labels. After entering each label, click **Done** , and click
      **Add label** again to create the next label.

      | Label name | Label type | Field name |
      |---|---|---|
      | `Project` | `STRING` | `resource.labels.project_id` |
      | `Message` | `STRING` | `jsonPayload.message` |
      | `LoggerName` | `STRING` | `labels.loggerName` |
      | `ClusterName` | `STRING` | `resource.labels.cluster_name` |
      | `SparkPhase` | `STRING` | `labels.".workflowSparkId"` |
      | `Region` | `STRING` | `resource.labels.region` |
      | `Pipeline` | `STRING` | `labels.".applicationId"` |
      | `RunId` | `STRING` | `labels.".runId"` |
      | `Namespace` | `STRING` | `labels.".namespaceId"` |
      | `LogLevel` | `STRING` | `labels.levelName` |

   6. Click **Create metric**.

      The newly created metric appears in the user-defined metrics table.
      If the metric isn't immediately visible, refresh the page.

      The dashboard contains the following charts:
      - All pipelines
      - Completed pipelines
      - Failed pipelines
      - All pipeline runs
      - Completed pipeline runs
      - Failed pipeline runs
      - Managed Service for Apache Spark clusters for runs

      After a metric is created, it might take up to 24 hours to start
      displaying the time series data.

## Install the dashboard

1. [Download the JSON file](https://docs.cloud.google.com/static/data-fusion/docs/tutorials/sample_datasets/datafusion-pipelines-overview.json) to your local machine.

2. Go to the Cloud Monitoring **Dashboards** page:


   [Go to Monitoring dashboards](https://console.cloud.google.com/monitoring/dashboards)

   <br />

   <br />

3. Click **Create dashboard**.

4. Click **Dashboard settings**
   **\> JSON \> JSON editor**.

5. In a text editor, open the JSON file that you downloaded.

6. Copy the content of the downloaded JSON file and paste it into the JSON
   editor, replacing the content that the JSON editor contains by default.

7. Click **Apply changes**.

   This refreshes the dashboard. The Cloud Data Fusion pipelines run
   after the metric was created, appear in the dashboard. If no pipelines were
   run after the metric was created, the dashboard will be empty.
8. Autosave is enabled by default. If autosave is disabled, click **Save**
   to save the dashboard.

9. Click **Close editor**.

   Your new dashboard appears in the list of dashboards on the
   **Monitoring overview** page.

## Clean up


To avoid incurring charges to your Google Cloud account for
the resources used on this page, follow these steps.

### Delete the Cloud Data Fusion instance

Follow these instructions to
[delete your Cloud Data Fusion instance](https://docs.cloud.google.com/data-fusion/docs/how-to/delete-instance).

### Delete the project


The easiest way to eliminate billing is to delete the project that you
created for the tutorial.

To delete the project:

> [!CAUTION]
> **Caution** : Deleting a project has the following effects:
>
> - **Everything in the project is deleted.** If you used an existing project for the tasks in this document, when you delete it, you also delete any other work you've done in the project.
> - **Custom project IDs are lost.** When you created this project, you might have created a custom project ID that you want to use in the future. To preserve the URLs that use the project ID, such as an `appspot.com` URL, delete selected resources inside the project instead of deleting the whole project.
>
>
> If you plan to explore multiple architectures, tutorials, or quickstarts, reusing projects
> can help you avoid exceeding project quota limits.

1. In the Google Cloud console, go to the **Manage resources** page.

   [Go to Manage resources](https://console.cloud.google.com/iam-admin/projects)
2. In the project list, select the project that you want to delete, and then click **Delete**.
3. In the dialog, type the project ID, and then click **Shut down** to delete the project.

<br />

<br />

## What's next

- Learn more about [Cloud Monitoring](https://docs.cloud.google.com/monitoring/docs/monitoring-overview).