You can instrument your [App Hub](https://docs.cloud.google.com/app-hub/docs/overview) application so that Google Cloud Observability attaches application-specific labels to the metrics and trace data that your application generates. For workloads that run on Google Kubernetes Engine, you can also export an OpenTelemetry HTTP server metric that Application Monitoring uses to report the traffic level, server error rate, and latency for HTTP requests.

<br />

## About application-specific labels

Application-specific labels refer to labels or attributes that are injected
into log, metric, or trace data. These labels identify the service or workload
that generated the data. You can use application-specific labels like any other
labels. For example, you can filter data by the ID of an application.
Telemetry generated by
[supported infrastructure](https://docs.cloud.google.com/monitoring/docs/application-monitoring-services)
automatically includes these labels. With instrumentation, metrics and traces
written by your application can include these labels.

In this section,
interpret `a.b.{x,y}` as `a.b.x` and `a.b.y`.

Metric data from instrumentation you added to your applications can include the following
metric labels:

- `metric.labels.apphub_application_{container,id,location}`
- `metric.labels.apphub_workload_{criticality_type,environment_type,id}`

Trace spans generated by instrumentation you added to your applications can include the following
resource attributes:

- `gcp.apphub.application.{container,id,location}`
- `gcp.apphub.{workload,service}.{criticality_type,environment_type,id}`

## About OpenTelemetry HTTP server metrics

There are no system metrics for workloads that run on Google Kubernetes Engine
that can report the traffic level, server error rate, or
latency for HTTP requests. However, the values for these golden signals
can be derived from the
[`http.server.request.duration`](https://opentelemetry.io/docs/specs/semconv/http/http-metrics/#metric-httpserverrequestduration)
metric, which is automatically collected by the
[OpenTelemetry](https://opentelemetry.io/docs/what-is-opentelemetry/) HTTP client
library.

The `googlemanagedprometheus` exporter converts the
`http.server.request.duration` OpenTelemetry metric into a Prometheus metric that
has the following properties:

- Name: `prometheus/http_server_request_duration_seconds/histogram`
- Metric: `prometheus.googleapis.com/http_server_request_duration_seconds/histogram`
- Resource types: [`prometheus_target`](https://docs.cloud.google.com/monitoring/api/resources#tag_prometheus_target)
- Unit: `s`
- Kind: [`CUMULATIVE`](https://docs.cloud.google.com/monitoring/api/v3/kinds-and-types#metric-kinds)
- Value type: [`DISTRIBUTION`](https://docs.cloud.google.com/monitoring/api/v3/kinds-and-types#value-types)

When your Google Cloud project contains the `http_server_request_duration_seconds`
metric, then your Application Monitoring dashboards display the traffic level,
server error rate, and latency for HTTP requests.

To learn how to configure your application to export the
`http_server_request_duration_seconds` metric,
see the [Use OpenTelemetry on Kubernetes](https://docs.cloud.google.com/monitoring/docs/instrument-for-application-monitoring#otel-gke) section of this document.

## Add application labels to metric data

Google Cloud Observability attempts to identify the source of Prometheus metrics by comparing
the attributes attached to Prometheus metrics that are sent to your project
with data returned from the App Hub API.
The remainder of this section lists configurations that let
Google Cloud Observability identify the App Hub application.

### Use OpenTelemetry on Kubernetes

> [!NOTE]
> **Note:** We recommend that you use this approach with the Google-Built OpenTelemetry Collector. If you run workloads on Google Kubernetes Engine, then this approach exports the HTTP server metric that Application Monitoring uses to generate values for the traffic level, server error rate, and latency for HTTP requests.

To have Google Cloud Observability attach application labels to metric data generated by your
application's workloads that run in Google Kubernetes Engine, do the following:

1. Instrument the application with OpenTelemetry.

   - For general information, see [Observability for application developers](https://docs.cloud.google.com/stackdriver/docs/opentelemetry/application-developers).
   - For links to Go, Java, Node.js, and Python samples along with information about those samples, see [Instrumentation samples overview](https://docs.cloud.google.com/stackdriver/docs/instrumentation/setup/sample-overview).
2. Deploy either the Google-built Collector or the OpenTelemetry Collector, and
   configure the collector as follows:

   - Export metric data with the `googlemanagedprometheus` exporter.
   - Configure the `k8sattributes` processor to extract metadata, like the `namespace`, from the environment.
   - Configure the `transform/collision` processor to set the `project_id`, `location`, `cluster`, and `namespace` attributes.
   - Configure the `transform/aco-gke` processor to populate the `top_level_controller_name` and `top_level_controller_type` labels.

   For an example, see
   [`otlp-k8s-ingest/config/collector.yaml`](https://github.com/GoogleCloudPlatform/otlp-k8s-ingest/blob/main/config/collector.yaml),
   which is the configuration file for the Google-built Collector. To learn
   more about this collector, see
   [Deploy Google-Built OpenTelemetry Collector on GKE](https://docs.cloud.google.com/stackdriver/docs/instrumentation/opentelemetry-collector-gke).
3. Register your application's workloads with
   [App Hub](https://docs.cloud.google.com/app-hub/docs/overview).

### Use Google Cloud Managed Service for Prometheus on GKE

To have Google Cloud Observability attach application labels to metric data generated by your
application's workloads that run on Google Kubernetes Engine clusters, do the following:

1. Use
   [Google Cloud Managed Service for Prometheus with managed collection](https://docs.cloud.google.com/stackdriver/docs/managed-prometheus/setup-managed).

2. Deploy your workloads on a Google Kubernetes Engine cluster whose version is at
   least `1.32.1-gke.1439000`.

3. Register your application's workloads with
   [App Hub](https://docs.cloud.google.com/app-hub/docs/overview).

[Google Cloud Managed Service for Prometheus](https://docs.cloud.google.com/stackdriver/docs/managed-prometheus) discovers
the values of the application labels through [service discovery](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#pod)
metadata and then adds the `top_level_controller_{name,type}` labels to the
[`targetLabels.metadata`](https://github.com/GoogleCloudPlatform/prometheus-engine/blob/v0.15.1/doc/api.md#monitoring.googleapis.com/v1.TargetLabels).
During metric ingestion, Google Cloud Observability uses the
`top_level_controller_{name,type}` labels and the App Hub API to identify
your App Hub application and to add the appropriate labels to your
metric data.

### Use Cloud Run

To have Google Cloud Observability attach application labels to metric data generated by your
Cloud Run workloads, do the following:

1. Instrument your application by using either OpenTelemetry or the
   Managed Service for Prometheus sidecar for Cloud Run.
   For information about these approaches, see the following documents:

   - [Deploy Google-Built OpenTelemetry Collector on Cloud Run](https://docs.cloud.google.com/stackdriver/docs/instrumentation/opentelemetry-collector-cloud-run)
   - [Use the Prometheus sidecar for Cloud Run](https://docs.cloud.google.com/stackdriver/docs/managed-prometheus/cloudrun-sidecar)
2. Register your application's workloads with
   [App Hub](https://docs.cloud.google.com/app-hub/docs/overview).

### Verify metric labels

<br />

To verify that your application is sending Prometheus metrics to your project,
do the following:

1. Verify that your application is sending Prometheus metrics to your project:
   1. In the Google Cloud console, go to the
      **Metrics explorer** page:

      [Go to **Metrics explorer**](https://console.cloud.google.com/monitoring/metrics-explorer)

      <br />

      If you use the search bar to find this page, then select the result whose subheading is
      **Monitoring**.
   2. In the toolbar of the Google Cloud console, select your Google Cloud project. For [App Hub](https://docs.cloud.google.com/app-hub/docs/overview) configurations, select the App Hub host project or the app-enabled folder's management project.
   3. In the **Metric** element, expand the **Select a metric** menu, enter `Prometheus Target` in the filter bar, and then use the submenus to select a specific resource type and metric:
      1. In the **Active resources** menu, select **Prometheus Target**.
      2. To select a metric, use the **Active metric categories** and **Active metrics** menus.
      3. Click **Apply**.
   4. To add filters, which remove time series from the query results, use the
      [**Filter** element](https://docs.cloud.google.com/monitoring/charts/metrics-selector#filter-option).

   5. Configure how the data is viewed.   

      To view labels, expand the **Filter** menu.
      Each entry in the **Filter** menu corresponds to a label.

      For more information about configuring a chart, see
      [Select metrics when using Metrics Explorer.](https://docs.cloud.google.com/monitoring/charts/metrics-selector)

   If you aren't seeing any Prometheus metrics, then review your configuration.

If your Prometheus metrics don't contain application labels, then do the
following:

1. Verify that you registered your workload or service with
   [App Hub](https://docs.cloud.google.com/app-hub/docs/overview).

2. Examine your logs to determine whether there are any errors.

   For example, if you deploy an OpenTelemetry Collector or the
   Google-built Collector and are running on Google Kubernetes Engine, then you might do
   the following:
   1. In the Google Cloud console, go to the **Workloads** page:

      [Go to **Workloads**](https://console.cloud.google.com/kubernetes/workload/overview)

      <br />

      If you use the search bar to find this page, then select the result whose subheading is
      **Kubernetes Engine**.
   2. Select your collector deployment, and then select the **Logs** tab.
3. If you are using an OpenTelemetry Collector or the Google-built Collector, then
   review the configuration of your collector. Your collector must do the
   following:

   - Export metric data with the `googlemanagedprometheus` exporter.
   - Configure the `k8sattributes` processor to extract metadata, like the `namespace`, from the environment.
   - Configure the `transform/collision` processor to set the `project_id`, `location`, `cluster`, and `namespace` attributes.
   - Configure the `transform/aco-gke` processor to populate the `top_level_controller_name` and `top_level_controller_type` labels.

   For an example, see the Google-built Collector configuration file:
   [`otlp-k8s-ingest/config/collector.yaml`](https://github.com/GoogleCloudPlatform/otlp-k8s-ingest/blob/main/config/collector.yaml).

## Add application attributes to trace spans

You don't instrument your application such that application-specific attributes
are attached to the spans that are sent to your Google Cloud project. Instead,
Google Cloud Observability uses [OpenTelemetry resource attributes](https://github.com/open-telemetry/opentelemetry-proto/blob/main/opentelemetry/proto/resource/v1/resource.proto#L32)
and the App Hub API to identify the
App Hub application associated with a span, and then it adds the
application-specific attributes before storing the span.

To have Google Cloud Observability attach application-specific resource attributes to trace
data generated by your application's services and workloads, do all of the
following:

- Instrument your application with OpenTelemetry and configure your exporter to
  send trace data to the
  [Telemetry (OTLP) API](https://docs.cloud.google.com/stackdriver/docs/reference/telemetry/overview)
  endpoint.

  We recommend a collector-based export. The following documents describe
  collectors that attach all required attributes to the span data:
  - [Deploy Google-Built OpenTelemetry Collector on Google Kubernetes Engine](https://docs.cloud.google.com/stackdriver/docs/instrumentation/opentelemetry-collector-gke).
  - [Deploy Google-Built OpenTelemetry Collector on Cloud Run](https://docs.cloud.google.com/stackdriver/docs/instrumentation/opentelemetry-collector-cloud-run).
  - [Deploy the OpenTelemetry Collector on Container-Optimized OS](https://docs.cloud.google.com/stackdriver/docs/instrumentation/opentelemetry-collector-cos).

  If you don't use one of the previous collectors, then you must configure
  your instrumentation or collector to attach all
  [required attributes](https://docs.cloud.google.com/monitoring/docs/instrument-for-application-monitoring#required-attributes).
- Deploy your services and workloads on
  [supported infrastructure](https://docs.cloud.google.com/monitoring/docs/application-monitoring-services).

- Register your services and workloads with
  [App Hub](https://docs.cloud.google.com/app-hub/docs/overview).

For more information about these attributes, see
[Google Cloud App Hub semantic conventions](https://github.com/open-telemetry/semantic-conventions/blob/main/docs/resource/cloud-provider/gcp/apphub.md).

### Required attributes

> [!NOTE]
> **Note:** For applications deployed to Google Kubernetes Engine, we recommend that you use the Google-Built OpenTelemetry Collector.

This section applies to configurations where you don't use one of the
recommended collectors. For example, you might want to write your own
collector, or you might prefer to
[directly export your trace data](https://docs.cloud.google.com/stackdriver/docs/instrumentation/migrate-to-otlp-endpoints).

#### Attributes that identify your project and resource location

Set the following attributes:

- `cloud.provider`: Set to `"gcp"`.
- `cloud.account.id`: Set to your project ID.
- `cloud.availability_zone` or `cloud.region`: Set to the [zone or region](https://docs.cloud.google.com/docs/geography-and-regions) where your resources are located. You can set both attributes.

Resource detectors can help you set the previous attributes:

- [`resourcedetectionprocessor`](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/resourcedetectionprocessor)
- [Go: Google Cloud resource detector](https://pkg.go.dev/go.opentelemetry.io/contrib/detectors/gcp)
- [Java: `opentelemetry-java-contrib/gcp-resources`](https://github.com/open-telemetry/opentelemetry-java-contrib/tree/main/gcp-resources#gcp-resource-detectors-for-opentelemetry)
- [Node.js: `resource-detector-gcp`](https://www.npmjs.com/package/@opentelemetry/resource-detector-gcp/v/0.29.12)
- [Python: `opentelemetry-resourcedetector-gcp`](https://pypi.org/project/opentelemetry-resourcedetector-gcp/)

#### Attributes for Google Kubernetes Engine workloads

For a Google Kubernetes Engine workload, configure your collector or your instrumentation
to do the following:

- Extract metadata, like the `namespace`, from the environment.
- Extract relevant metadata from the `k8sattributes` processor.
- Set the `project_id`, `location`, `cluster`, and `namespace` attributes for metrics.
- Set the `top_level_controller_name` and `top_level_controller_type` labels for metrics.

For a collector-based example, see
[`otlp-k8s-ingest/config/collector.yaml`](https://github.com/GoogleCloudPlatform/otlp-k8s-ingest/blob/main/config/collector.yaml).
That collector uses the
[`k8sattributesprocessor`](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/k8sattributesprocessor)
processor.

#### Attributes for VM workloads

If you deploy your application on a VM, then that VM must be associated with
a [Managed Instance Group (MIG)](https://docs.cloud.google.com/compute/docs/instance-groups).
You must also attach the following attributes:

- `gcp.gce.instance_group_manager.name`
- `gcp.gce.instance_group_manager.region` or `gcp.gce.instance_group_manager.zone`