`MonitoringTarget` custom resources in Google Distributed Cloud (GDC) air-gapped appliance let you
label scraped metrics to differentiate metric characteristics and filter time
series data. In addition to
[collecting metrics](https://docs.cloud.google.com/distributed-cloud/hosted/docs/latest/appliance/platform/pa-user/collect-metrics), you can
configure relabeling rules to route metrics to another project, add custom
key-value labels, or rename metrics before ingestion.

Each unique combination of key-value label pairs creates a new time series,
which can significantly increase data storage. Avoid using labels for
high-cardinality dimensions with many possible values, such as user IDs or email
addresses. For more information about metric relabeling, see
[How relabeling in Prometheus works](https://grafana.com/blog/2022/03/21/how-relabeling-in-prometheus-works/).

## Before you begin

To manage `MonitoringTarget` custom resources, request the necessary permissions from your administrator.

### Required IAM roles

Contact your Project IAM Admin to request the following roles:

- **MonitoringTarget Creator** (`monitoringtarget-creator`): create `MonitoringTarget` custom resources in the project namespace.
- **MonitoringTarget Editor** (`monitoringtarget-editor`): edit or modify `MonitoringTarget` custom resources in the project namespace.
- **MonitoringTarget Viewer** (`monitoringtarget-viewer`): view `MonitoringTarget` custom resources in the project namespace.

## Send metrics to another project

Container authors can code specific metrics to be sent to a different project, even to projects they don't own. To send a metric to a specific project, add the
`_gdch_project` label to the metric in your code, setting its value to the destination project name.

Alternatively, use the `MonitoringTarget` custom resource to send *all* collected metric data to another project. Set the `_gdch_project` label as a target label in the `metricsRelabelings` field, with the new project name as the replacement value.

The following code sample shows how to set the `_gdch_project` target label in the `MonitoringTarget` custom resource to send metrics to the project named `another-project-name`:

    apiVersion: monitoring.gdc.goog/v1
    kind: MonitoringTarget
    metadata:
      # Choose the same namespace as the workload pods.
      namespace: PROJECT_NAMESPACE
      name: MONITORING_TARGET_NAME
    spec:
      [...]
      podMetricsEndpoints:
        [...]
        metricsRelabelings:
          - action: replace
            targetLabel: _gdch_project
            replacement: another-project-name
          [...]

Replace the following:

- `PROJECT_NAMESPACE`: your project namespace.
- `MONITORING_TARGET_NAME`: the name of the `MonitoringTarget` definition file.

## Add new labels to metrics

Labels help with observability and monitoring by identifying metric characteristics. You can add labels that the scraped containers don't initially expose. In the `MonitoringTarget` custom resource, set the new label as a target label and provide a replacement value. This key-value pair is added to all metrics the
resource collects.

The following code sample shows how to set the `my_new_label` target label and configure its value to `my_label_value` in the `MonitoringTarget` custom resource:

    apiVersion: monitoring.gdc.goog/v1
    kind: MonitoringTarget
    metadata:
      # Choose the same namespace as the workload pods.
      namespace: PROJECT_NAMESPACE
      name: MONITORING_TARGET_NAME
    spec:
      [...]
      podMetricsEndpoints:
        [...]
        metricsRelabelings:
          - action: replace
            targetLabel: my_new_label
            replacement: my_label_value
          [...]

Replace the following:

- `PROJECT_NAMESPACE`: your project namespace.
- `MONITORING_TARGET_NAME`: the name of the `MonitoringTarget` definition file.

## Rename a metric

You can rename a container's metric using the `MonitoringTarget` resource. Set
the original metric name and its replacement in the `metricsRelabelings` field.
The original name is treated as a regular expression (regex) and is replaced by
the new name.

The following code sample shows how to replace the `cpu_usage` metric name
(regex) with `cpu_usage_new_name` in the `MonitoringTarget` custom resource:

    apiVersion: monitoring.gdc.goog/v1
    kind: MonitoringTarget
    metadata:
      # Choose the same namespace as the workload pods.
      namespace: PROJECT_NAMESPACE
      name: MONITORING_TARGET_NAME
    spec:
      [...]
      podMetricsEndpoints:
        [...]
        metricsRelabelings:
          - action: replace
            regex: cpu_usage
            replacement: cpu_usage_new_name
            sourceLabels: [__name__]
            targetLabel: __name__
          [...]

Replace the following:

- `PROJECT_NAMESPACE`: your project namespace.
- `MONITORING_TARGET_NAME`: the name of the `MonitoringTarget` definition file.