Metrics overview

This page describes the metrics that help you monitor the health and performance of your Cloud Data Fusion instances and pipelines. Use Cloud Monitoring to monitor these metrics. The metrics provide insights into pipeline runs, instance details, API requests, and authorization checks.

The metrics are categorized as either pipeline metrics or instance metrics:

  • Pipeline metrics provide data about individual pipeline runs, such as run status, duration, latency, and data throughput.
  • Instance metrics provide aggregated information about the pipelines within an instance, including service availability, the number of deployed pipelines, and API request counts.

Filter and aggregate Cloud Data Fusion pipeline and instance metrics in Monitoring using metric and monitored-resource labels. When you customize your metrics views, you can use one or both of these label types.

Cloud Data Fusion Pipeline monitored-resource labels

Filter and aggregate the metrics with the following Cloud Data Fusion Pipeline monitored-resource labels:

Label name Description
resource_container The ID of the customer project.
org_id The ID of the organization that the customer project belongs to.
location The zone or region where the instance is hosted.
edition The edition of the Cloud Data Fusion instance.
is_private_ip_enabled Whether the instance uses an internal IP address.
version The Cloud Data Fusion data plane version of the instance.
instance_id The Cloud Data Fusion instance ID.
namespace The namespace of the pipeline.
pipeline_id The pipeline ID.
run_id The run ID for the pipeline.

Pipeline metric labels

Filter and aggregate the metrics with the following Cloud Data Fusion metric labels in Monitoring:

Name Metric Description Metric labels
Pipeline run status datafusion.googleapis.com/pipeline/v2/runs_completed_count The cumulative count of pipelines that have completed a run.
  • complete_state
  • previous_state
  • program
  • provisioner
  • cluster_state
  • compute_profile_id
  • enable_rbac
  • private_service_connect_enabled
Pipeline run time datafusion.googleapis.com/pipeline/v2/pipeline_duration Time taken to complete the pipeline run.
  • complete_state
  • program
  • provisioner
  • cluster_state
  • compute_profile_id
  • enable_rbac
  • private_service_connect_enabled
Pipeline start latency datafusion.googleapis.com/pipeline/v2/pipeline_start_latency The time taken for the pipeline to reach Running state.
  • program
  • provisioner
  • cluster_state
  • compute_profile_id
  • complete_state
  • enable_rbac
  • private_service_connect_enabled
Provisioning latency datafusion.googleapis.com/pipeline/v2/dataproc/provisioning_latency The Dataproc cluster provisioning latency.
  • provisioner
  • enable_rbac
  • private_service_connect_enabled
Dataproc API requests datafusion.googleapis.com/pipeline/v2/dataproc/api_request_count The cumulative count of Dataproc API requests.
  • provisioner
  • method
  • response_code
  • region
  • launch_mode
  • image_version
  • enable_rbac
  • private_service_connect_enabled
Pipeline preview run time datafusion.googleapis.com/pipeline/v2/preview_duration Time taken to complete preview.
  • complete_state
  • enable_rbac
  • private_service_connect_enabled
Pipeline bytes written datafusion.googleapis.com/pipeline/v2/write_bytes_count The cumulative count of bytes written by a pipeline.
  • enable_rbac
  • private_service_connect_enabled
Pipeline bytes read datafusion.googleapis.com/pipeline/v2/read_bytes_count The cumulative count of bytes read by a pipeline.
  • enable_rbac
  • private_service_connect_enabled
Pipeline bytes shuffled datafusion.googleapis.com/pipeline/v2/shuffle_bytes_count The cumulative count of bytes shuffled in a pipeline.
  • enable_rbac
  • private_service_connect_enabled
Plugin records processed in datafusion.googleapis.com/pipeline/v2/plugin/incoming_records_count Cumulative count of records entering a plugin.
  • enable_rbac
  • private_service_connect_enabled
  • stage_name
Plugin records processed out datafusion.googleapis.com/pipeline/v2/plugin/outgoing_records_count The cumulative count of records exiting a plugin.
  • enable_rbac
  • private_service_connect_enabled
  • stage_name

Cloud Data Fusion Instance monitored-resource labels

Starting with Cloud Data Fusion version 6.11.1.1, the InstanceV3 (datafusion.googleapis.com/InstanceV3) monitored resource is the default resource type for instance-level metrics. All new instances and instances upgraded to version 6.11.1.1 or later automatically emit metrics and logs using InstanceV3. The InstanceV3 resource structure differs from the previous version by removing the org_id and namespace labels.

By default, the emission of InstanceV2 metrics is disabled in Cloud Data Fusion version 6.11.1.1 and later. However, you can re-enable InstanceV2 emission alongside InstanceV3 using the Cloud Data Fusion REST API if you require backward compatibility for existing dashboards or queries.

You can filter and aggregate the metrics with the following Cloud Data Fusion Instance monitored-resource labels.

InstanceV3 monitored-resource labels

Label name Description
resource_container The ID of the customer project.
location The zone or region where the instance is hosted.
edition The edition of the instance.
is_private_ip_enabled Whether the instance uses an internal IP address.
version The Cloud Data Fusion data plane version of the instance.
instance_id The Cloud Data Fusion instance ID.

InstanceV2 monitored-resource labels

Label name Description
resource_container The ID of the customer project.
org_id The ID of the organization that the customer project belongs to.
location The zone or region where the instance is hosted.
edition The edition of the instance.
is_private_ip_enabled Whether the instance uses an internal IP address.
version The Cloud Data Fusion data plane version of the instance.
instance_id The Cloud Data Fusion instance ID.
namespace The namespace name.

Instance metric labels

Filter and aggregate the metrics with the following Cloud Data Fusion metric labels in Monitoring.

InstanceV3 metric labels

Name Metric Description Metric labels
Service status datafusion.googleapis.com/instance/v3/service_available The availability of Cloud Data Fusion services.
  • service
  • enable_rbac
  • private_service_connect_enabled
Deployed pipeline count datafusion.googleapis.com/instance/v3/pipelines The number of deployed pipelines.
  • enable_rbac
  • private_service_connect_enabled
  • maintenance_window_enabled
Concurrent pipelines running count datafusion.googleapis.com/instance/v3/concurrent_pipelines_running The number of pipelines running concurrently.
  • enable_rbac
  • private_service_connect_enabled
Concurrent pipeline launches count datafusion.googleapis.com/instance/v3/concurrent_pipelines_launched The number of pipelines in either Provisioning or Starting state.
  • enable_rbac
  • private_service_connect_enabled
CDAP REST API requests received datafusion.googleapis.com/instance/v3/api_request_count The cumulative count of REST API requests received by a service in the backend.
  • service
  • handler
  • method
  • enable_rbac
  • private_service_connect_enabled
CDAP REST API responses sent datafusion.googleapis.com/instance/v3/api_response_count The cumulative count of REST API responses sent by a service in the backend.
  • service
  • handler
  • method
  • response_code
  • enable_rbac
  • private_service_connect_enabled
Authorization check count datafusion.googleapis.com/instance/v3/authorization_check_count The cumulative count of authorization checks made by the access enforcer.
  • enable_rbac
  • type
  • private_service_connect_enabled
Authorization check time datafusion.googleapis.com/instance/v3/authorization_check_time The latency of authorization checks made by the access enforcer.
  • enable_rbac
  • type
  • private_service_connect_enabled
Draft pipeline count datafusion.googleapis.com/instance/v3/draft_pipelines The number of draft pipelines.
  • enable_rbac
  • private_service_connect_enabled
Namespace count datafusion.googleapis.com/instance/v3/namespaces The number of namespaces.
  • enable_rbac
  • private_service_connect_enabled

InstanceV2 metric labels

Name Metric Description Metric labels
Service status datafusion.googleapis.com/instance/v2/service_available The availability of Cloud Data Fusion services.
  • service
  • enable_rbac
  • private_service_connect_enabled
Deployed pipeline count datafusion.googleapis.com/instance/v2/pipelines The number of deployed pipelines.
  • enable_rbac
  • private_service_connect_enabled
  • maintenance_window_enabled
Concurrent pipelines running count datafusion.googleapis.com/instance/v2/concurrent_pipelines_running The number of pipelines running concurrently.
  • enable_rbac
  • private_service_connect_enabled
Concurrent pipeline launches count datafusion.googleapis.com/instance/v2/concurrent_pipelines_launched The number of pipelines in either Provisioning or Starting state.
  • enable_rbac
  • private_service_connect_enabled
CDAP REST API requests received datafusion.googleapis.com/instance/v2/api_request_count The cumulative count of REST API requests received by a service in the backend.
  • service
  • handler
  • method
  • enable_rbac
  • private_service_connect_enabled
CDAP REST API responses sent datafusion.googleapis.com/instance/v2/api_response_count The cumulative count of REST API responses sent by a service in the backend.
  • service
  • handler
  • method
  • response_code
  • enable_rbac
  • private_service_connect_enabled
Authorization check count datafusion.googleapis.com/instance/v2/authorization_check_count The cumulative count of authorization checks made by the access enforcer.
  • enable_rbac
  • type
  • private_service_connect_enabled
Authorization check time datafusion.googleapis.com/instance/v2/authorization_check_time The latency of authorization checks made by the access enforcer.
  • enable_rbac
  • type
  • private_service_connect_enabled
Draft pipeline count datafusion.googleapis.com/instance/v2/draft_pipelines The number of draft pipelines.
  • enable_rbac
  • private_service_connect_enabled
Namespace count datafusion.googleapis.com/instance/v2/namespaces The number of namespaces.
  • enable_rbac
  • private_service_connect_enabled

Manage InstanceV2 metric emission

For Cloud Data Fusion instances running version 6.11.1.1 or later, InstanceV2 metric emission is disabled by default. If you need to maintain backward compatibility with dashboards or alerts using the old format, you can re-enable InstanceV2 metrics using the Cloud Data Fusion REST API.

Enable InstanceV2 metrics

To enable the emission of InstanceV2 metrics, use the instances.patch method with enable_instance_v2_metrics set to true:

  curl -X PATCH \
  -H "Authorization: Bearer $(gcloud auth print-access-token)" \
  -H "Content-Type: application/json" \
  "https://datafusion.googleapis.com/v1/projects/PROJECT_ID/locations/LOCATION/instances/INSTANCE_ID?updateMask=monitoring_config" \
  --data '{"monitoringConfig": {"enable_instance_v2_metrics": true}}'

Replace the following:

  • PROJECT_ID: the Google Cloud project ID
  • LOCATION: the location of your instance
  • INSTANCE_ID: the ID of your Cloud Data Fusion instance

Disable InstanceV2 metrics

To disable InstanceV2 metrics and revert to the default behavior (emitting only InstanceV3 metrics), use the instances.patch method with enable_instance_v2_metrics set to false:

  curl -X PATCH \
  -H "Authorization: Bearer $(gcloud auth print-access-token)" \
  -H "Content-Type: application/json" \
  "https://datafusion.googleapis.com/v1/projects/PROJECT_ID/locations/LOCATION/instances/INSTANCE_ID?updateMask=monitoring_config" \
  --data '{"monitoringConfig": {"enable_instance_v2_metrics": false}}'

Replace the following:

  • PROJECT_ID: the Google Cloud project ID
  • LOCATION: the location of your instance
  • INSTANCE_ID: the ID of your Cloud Data Fusion instance

Migrate Cloud Monitoring queries from InstanceV2 to InstanceV3

Starting with Cloud Data Fusion version 6.11.1.1, the InstanceV3 (datafusion.googleapis.com/InstanceV3) monitored resource is the default resource type for instance-level metrics. If you have existing Monitoring dashboards, charts, or alert policies that target InstanceV2, you can update them to use the InstanceV3 resource type and metric paths.

To migrate your queries, follow these steps:

  1. Change resource type: Update resource.type from datafusion.googleapis.com/InstanceV2 to datafusion.googleapis.com/InstanceV3.

  2. Update metric names: Change the metric paths from .../instance/v2/... to .../instance/v3/....

  3. Remove labels: Remove any filters or aggregations based on resource.labels.org_id or resource.labels.namespace, as these labels are not present in InstanceV3.

For example, if the following is your existing InstanceV2 query:

fetch datafusion.googleapis.com/InstanceV2
| metric 'datafusion.googleapis.com/instance/v2/pipelines'
| filter resource.labels.instance_id == 'my-instance'
| group_by 1m, [value_pipelines_mean: mean(value.pipelines)]
| every 1m

Update it as follows:

fetch datafusion.googleapis.com/InstanceV3
| metric 'datafusion.googleapis.com/instance/v3/pipelines'
| filter resource.labels.instance_id == 'my-instance'
| group_by 1m, [value_pipelines_mean: mean(value.pipelines)]
| every 1m

What's next