Monitoring and logging tools in Google Distributed Cloud (GDC) air-gapped let you monitor
your services and troubleshoot performance across your
Vertex AI environment. You can query and visualize logs and
metrics to observe the Vertex AI platform and pre-trained
services, such as Optical Character Recognition (OCR), Speech-to-Text, and
Vertex AI Translation, in

the monitoring instance of GDC.


For more information about logging and monitoring solutions in
GDC, see
[Monitor metrics and logs](https://docs.cloud.google.com/distributed-cloud/hosted/docs/latest/gdcag/platform/pa-user/obs-overview.md.txt).

## Before you begin

To get the permissions you need to view logs and metrics from
Vertex AI, ask your IAM Project Admin to grant you the
Project Grafana Viewer (`project-grafana-viewer`) role in your project
namespace.

For more information about this role, see [Prepare IAM permissions](https://docs.cloud.google.com/distributed-cloud/hosted/docs/latest/gdcag/platform/pa-user/obs-iam-permissions.md.txt#p-gr-v).

## Visualize logs and metrics in dashboards

You can view Vertex AI metrics and logs in dashboards. For
example, you can create a query to view how Vertex AI affects CPU
usage.

Follow these steps to view Vertex AI logs and metrics in
dashboards:

1. [Sign in to the GDC console and select your project](https://docs.cloud.google.com/distributed-cloud/hosted/docs/latest/gdcag/application/ao-user/iam/sign-in.md.txt).

2. In the navigation menu, click **Vertex AI** \> **Pre-trained APIs**.

3. On the **Pre-trained APIs** page, make sure that [the Vertex AI API for the service you want to monitor is enabled](https://docs.cloud.google.com/distributed-cloud/hosted/docs/latest/gdcag/application/ao-user/vertex-ai-enable-pre-trained-apis.md.txt).

4. Click **Monitor services in Grafana** to open the Grafana home page.

5. In the navigation menu of the home page, click explore **Explore**
   to open the **Explore** page.

6. From the menu at the **Explore** page, select one of the following data
   sources:

   - **Operational Logs**: retrieve operational logs.
   - **Audit Logs**: retrieve audit logs.
   - **Prometheus**: retrieve metrics.
7. If you want to display metrics, enter a query using
   [PromQL](https://prometheus.io/docs/prometheus/latest/querying/basics/)
   (Prometheus Query Language) expressions.

8. If you want to display logs, enter a query using
   [LogQL](https://grafana.com/docs/loki/latest/logql/) (Log Query
   Language) expressions.

The page displays the metrics or logs matching your query.

![The Prometheus option is selected on the Explore page to obtain metrics.](https://docs.cloud.google.com/static/distributed-cloud/hosted/docs/latest/gdcag/images/metrics-ui.png)

**Figure 1.** Menu option in Grafana to query metrics.

In figure 1, the **Prometheus** option displays the interface that lets you
build queries to retrieve metrics.

## Sample queries

The following table contains sample queries to monitor the
Vertex AI platform in your environment:

| **Vertex AI platform** ||||
|---|---|---|---|
| **Data source** | **Description** | **Component** | **Query** |
| Metrics | CPU usage percentage of the container | Level 1 operator | `rate(container_cpu_usage_seconds_total{namespace="ai-system",container="l1operator"}[30s]) * 100` |
| Metrics | CPU usage percentage of the container | Level 2 operator | `rate(container_cpu_usage_seconds_total{namespace="ai-system",container="l2operator"}[30s]) * 100` |
| Metrics | Memory usage in MB of the container | Level 1 operator | `container_memory_usage_bytes{namespace="ai-system",container="l1operator"} * 1e-6` |
| Metrics | Memory usage in MB of the container | Level 2 operator | `container_memory_usage_bytes{namespace="ai-system",container="l2operator"} * 1e-6` |
| Operational logs | L1 operator logs | Level 1 operator | `{service_name="vai-l1operator"}` |
| Operational logs | L2 operator logs | Level 2 operator | `{service_name="vai-l2operator"}` |
| Audit logs | Platform frontend audit logs | Vertex AI web plugin frontend | `{service_name="istio"} |~ upstream_cluster:.*(vai-web-plugin-frontend)` |
| Audit logs | Platform backend audit logs | Vertex AI web plugin backend | `{service_name="istio"} |~ upstream_cluster:.*(vai-web-plugin-backend)` |

The following table contains sample queries to monitor
Vertex AI API services, such as OCR,
Speech-to-Text, and Vertex AI Translation, in your environment:

| **Vertex AI services** ||||
|---|---|---|---|
| **Data source** | **Description** | **Service** | **Query** |
| Metrics | The effect of a pre-trained API on CPU usage. | OCR | `rate(container_cpu_usage_seconds_total{namespace="g-vai-ocr-sie",container="CONTAINER_NAME"}[30s]) * 100 CONTAINER_NAME values: vision-extractor | vision-frontend | vision-vms-ocr` |
| Metrics | The effect of a pre-trained API on CPU usage. | Speech-to-Text | `rate(container_cpu_usage_seconds_total{namespace="g-vai-speech-sie",container="CONTAINER_NAME"}[30s]) * 100` |
| Metrics | The effect of a pre-trained API on CPU usage. | Vertex AI Translation | `rate(container_cpu_usage_seconds_total{namespace="g-vai-translation-sie",container="CONTAINER_NAME"}[30s]) * 100 CONTAINER_NAME values: translation-aligner | translation-frontend | translation-prediction` |
| Metrics | Use the `destination_service` filter label to get the error rate over the last 60 minutes. | OCR | `rate(istio_requests_total{destination_service=~".*g-vai-ocr-sie.svc.cluster.local",response_code=~"[4-5][0-9][0-9]"}[60m])` |
| Metrics | Use the `destination_service` filter label to get the error rate over the last 60 minutes. | Speech-to-Text | `rate(istio_requests_total{destination_service=~".*g-vai-speech-sie.svc.cluster.local",response_code=~"[4-5][0-9][0-9]"}[60m])` |
| Metrics | Use the `destination_service` filter label to get the error rate over the last 60 minutes. | Vertex AI Translation | `rate(istio_requests_total{destination_service=~".*g-vai-translation-sie.svc.cluster.local",response_code=~"[4-5][0-9][0-9]"}[60m])` |
| Operational logs | Operational logs from Vertex AI services > [!NOTE] > **Note:** Use the namespace of the service to specify the main filter. > You can create more granular results by adding additional labels, such as `service_name` or `pod`, to your query. | OCR | `{namespace="g-vai-ocr-sie"}` |
| Operational logs | Operational logs from Vertex AI services > [!NOTE] > **Note:** Use the namespace of the service to specify the main filter. > You can create more granular results by adding additional labels, such as `service_name` or `pod`, to your query. | Speech-to-Text | `{namespace="g-vai-speech-sie"}` |
| Operational logs | Operational logs from Vertex AI services > [!NOTE] > **Note:** Use the namespace of the service to specify the main filter. > You can create more granular results by adding additional labels, such as `service_name` or `pod`, to your query. | Vertex AI Translation | `{namespace="g-vai-translation-sie"}` |
| Audit logs | Audit logs from Vertex AI services | OCR | `{service_name="istio"} |= "vision-frontend-server"` |
| Audit logs | Audit logs from Vertex AI services | Speech-to-Text | `{service_name="istio"} |= "speech-frontend-server"` |
| Audit logs | Audit logs from Vertex AI services | Vertex AI Translation | `{service_name="istio"} |= "translation-frontend-server"` |