Google Distributed Cloud (GDC) air-gapped lets you scrape and collect metrics from your
workloads over time to monitor application performance and maintain data
observability. To initiate metric collection, you deploy a `MonitoringTarget`
custom resource to your project namespace on the management API server.

The `MonitoringTarget` custom resource directs the monitoring pipeline to scrape
designated pods that expose an HTTP endpoint in a Prometheus exposition format,
such as OpenMetrics. You can customize the scrape frequency, endpoint settings,
labels, and annotations for your target pods. Your project's Grafana instance
displays the collected metrics so that you can analyze your application's
operational state.

## Before you begin

To manage `MonitoringRule` custom resources, request the necessary permissions from your administrator.

### Required IAM roles

Contact your IAM Project Admin to request the following roles:

- **MonitoringRule Creator** (`monitoringrule-creator`): create `MonitoringRule` custom resources in the project namespace.
- **MonitoringRule Editor** (`monitoringrule-editor`): edit or modify `MonitoringRule` custom resources in the project namespace.
- **MonitoringRule Viewer** (`monitoringrule-viewer`): view `MonitoringRule` custom resources in the project namespace.

## Configure the `MonitoringTarget` custom resource

The `MonitoringTarget` custom resource tells the monitoring platform where to
collect metrics from. You can specify the pods for which you are collecting
metrics, the metrics endpoint of those pods, the scrapping frequency, and any
additional settings.

This resource defines the following configurations:

- **Targets**: The pods and their endpoints within your project that expose metrics.
- **Scrape interval**: How frequently you want to pull metrics from the selected endpoints.
- **Label customizations**: Optional rules with any label modifications for metrics.

Choose one of the following methods for specifying metrics endpoints in the
`MonitoringTarget` custom resource:

- **Static endpoints** : You explicitly declare the endpoint (port, path, scheme) in the `MonitoringTarget` configuration.
- **Annotations** : The pod metric endpoint information is retrieved from
  annotations within the container's `Deployment` file. This method offers more
  flexibility if each pod has different endpoints.

### Static endpoints

Follow these steps to expose metrics from your selected pods on a statically
defined endpoint:

1. Determine the Distributed Cloud project from which you want to gather
   metrics for monitoring.

2. In your pod's specification, declare the port that serves metrics in the
   `containerPort` field. The following example shows how to declare port
   `2112` in the pod's specification:

   ```yaml
   # ...
   spec:
     template:
       spec:
         containers:
         - name: your-container-name
           ports:
           - containerPort: 2112
   # ...
   ```
3. In your `MonitoringTarget` configuration, specify the endpoint details
   (port, path, scheme) in the `podMetricsEndpoints` section to match the
   port you exposed in the pod's specification.

   The following YAML file shows an example of a `MonitoringTarget`
   configuration where every selected pod has to expose metrics on the same
   endpoint, `http://your-container-name:2112/metrics`:

   ```yaml
   apiVersion: monitoring.gdc.goog/v1
   kind: MonitoringTarget
   metadata:
     # Choose the same namespace as the workload pods.
     namespace: your-project-namespace
     name: your-container-name
   spec:
     selector:
         # Choose pod labels to consider for this job.
         # Optional: Map of key-value pairs.
         # Default: No filtering by label.
         # To consider every pod in the project namespace, remove selector fields.
       matchLabels:
         app: your-app-label
     podMetricsEndpoints:
       port:
         value: 2112
       path:
         # Choose any value for your endpoint.
         # The /metrics value is an example.
         value: /metrics
       scheme:
         value: http
   ```
4. Apply the `MonitoringTarget` configuration to the Management API server
   within the same namespace as your target pods:

   ```bash
   kubectl --kubeconfig KUBECONFIG_PATH apply -f MONITORING_TARGET_NAME.yaml
   ```

   Replace the following:
   - `KUBECONFIG_PATH`: the path to the kubeconfig file for the Management API server.
   - `MONITORING_TARGET_NAME`: the name of the `MonitoringTarget` definition file.

The monitoring platform begins collecting metrics.

### Annotations

Follow these steps to expose metrics using annotations if each pod has
different endpoints:

1. Determine the Distributed Cloud project from which you want to gather
   metrics for monitoring.

2. Add the following annotations to the `annotations` section of your
   container's `Deployment` file:

   - `prometheus.io/path`
   - `prometheus.io/port`
   - `prometheus.io/scheme`

   The following example shows annotations for metrics on port `2112`:

   ```yaml
   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: your-container-name
     namespace: your-project-namespace
     labels:
       app: your-app-label
     annotations:
       # These annotations are not required. They demonstrate selecting
       # pod metric endpoints through annotations.
       prometheus.io/path: /metrics
       prometheus.io/port: \"2112\"
       prometheus.io/scheme: http
   ```
3. In your `MonitoringTarget` configuration, specify the annotations you
   added to the container's `Deployment` file in the `podMetricsEndpoints`
   section. This specification tells the custom resource to gather the
   metrics endpoint information from annotations on the selected pods.

   The following YAML file shows an example of a `MonitoringTarget`
   configuration using annotations:

   ```yaml
   apiVersion: monitoring.gdc.goog/v1
   kind: MonitoringTarget
   metadata:
   metadata:
     # Choose the same namespace as the workload pods.
     namespace: your-project-namespace
     name: your-container-name
   spec:
     selector:
       matchLabels:
         app: your-app-label
     podMetricsEndpoints:
       port:
         annotation: prometheus.io/port
       path:
         annotation: prometheus.io/path
       scheme:
         annotation: prometheus.io/scheme
   ```
4. Apply the `MonitoringTarget` configuration to the Management API server
   within the same namespace as your target pods:

   ```bash
   kubectl --kubeconfig KUBECONFIG_PATH apply -f MONITORING_TARGET_NAME.yaml
   ```

   Replace the following:
   - `KUBECONFIG_PATH`: the path to the kubeconfig file for the Management API server.
   - `MONITORING_TARGET_NAME`: the name of the `MonitoringTarget` definition file.

The monitoring platform begins collecting metrics.

Refer to the [complete `MonitoringTarget` specification](https://docs.cloud.google.com/distributed-cloud/hosted/docs/latest/gdcag/platform/pa-user/collect-metrics.md.txt#monitoringtarget-specification)
and the [API reference documentation](https://docs.cloud.google.com/distributed-cloud/hosted/docs/latest/gdcag/apis/service/monitoring/v1/monitoring-v1.md.txt#monitoringtarget)
for additional fields and options.

## Complete `MonitoringTarget` specification

The following YAML file shows an example for the complete specification of the
`MonitoringTarget` custom resource. For more information and a complete
description of fields, see the
[API reference documentation](https://docs.cloud.google.com/distributed-cloud/hosted/docs/latest/gdcag/apis/service/monitoring/v1/monitoring-v1.md.txt#monitoringtarget).

```yaml
apiVersion: monitoring.gdc.goog/v1
kind: MonitoringTarget
metadata:
  # Choose the same namespace as the workload pods.
  namespace: PROJECT_NAMESPACE
  name: MONITORING_TARGET_NAME
spec:
  # Choose matching pattern that identifies pods for this job.
  # Optional
  # Relationship between different selectors: AND
  selector:
    # Choose clusters to consider for this job.
    # Optional: List
    # Default: All clusters applicable to this project.
    # Relationship between different list elements: OR
    matchClusters:
    - string

    # Choose pod labels to consider for this job.
    # Optional: Map of key-value pairs.
    # Default: No filtering by label.
    # Relationship between different pairs: AND
    matchLabels:
      key1: value1

    # Choose annotations to consider for this job.
    # Optional: Map of key-value pairs
    # Default: No filtering by annotation
    # Relationship between different pairs: AND
    matchAnnotations:
      key1: value1

  # Configure the endpoint exposed for this job.
  podMetricsEndpoints:
    # Choose a port either through static value or annotation.
    # Optional
    # Annotation takes priority.
    # Default: static port 80
    port:
      value: integer
      annotation: string

    # Choose a path either through static value or annotation.
    # Optional
    # Annotation takes priority
    # Default: static path /metrics
    path:
      value: string
      annotation: string

    # Choose a scheme either through a static value (http or https) or annotation.
    # Optional
    # Annotation takes priority
    # Default: static scheme http
    scheme:
      value: string
      annotation: string

    # Choose the frequency to scrape the metrics endpoint defined in podMetricsEndpoints
    # Optional
    # Default: 60s
    scrapeInterval: string

    # Dynamically rewrite the label set of a target before it gets scraped.
    # https://prometheus.io/docs/prometheus/latest/configuration/configuration/#relabel_config
    # Optional
    # Default: No filtering by label
    metricsRelabelings:
    - sourceLabels:
      - string
      separator: string
      regex: string
      action: string
      targetLabel: string
      replacement: string
```

Replace the following:

- `PROJECT_NAMESPACE`: your project namespace.
- `MONITORING_TARGET_NAME`: the name of the `MonitoringTarget` definition file.