Alert monitoring in Google Distributed Cloud (GDC) air-gapped lets you track and
troubleshoot active notifications across your project after
[creating alert rules](https://docs.cloud.google.com/distributed-cloud/hosted/docs/latest/gdcag/platform/pa-user/create-alert-rules.md.txt). You
can inspect alert severity, duration, status, and labels in the
GDC console. For scripting, cron jobs, and custom integrations, query
your project's Cortex Alertmanager HTTP endpoint with `curl`.

## Before you begin

To get the permissions that you need to query and visualize alerts, ask your
IAM Project Admin to grant you one of the associated Project Cortex Alertmanager
roles in your project namespace. Depending on the level of access and
permissions you need, you might obtain editor or viewer roles for this resource
in a project.

The following table summarizes the `Role` requirements for `PA and AO persona`.

| Persona | Object | Cluster | Role | Namespace | Group/User | Config |
|---|---|---|---|---|---|---|
| PA | grafana | org-admin | `project-cortex-alertmanager-viewer` | platform-obs | Group | 1 |
| PA | grafana | org-admin | `project-cortex-alertmanager-viewer` | platform-obs | User | 2 |
| AO | alertmanager | org-admin | `project-cortex-alertmanager-viewer` | PROJECT_NAMESPACE | Group | 1 |
| AO | alertmanager | org-admin | `project-cortex-alertmanager-viewer` | PROJECT_NAMESPACE | User | 2 |

Replace the following variables appropriately:

| Variable | Description |
|---|---|
| `KUBECONFIG` | You will need the kubeconfig for the specific cluster that contains the <var translate="no">NAMESPACE</var> where this RoleBinding will be applied. |
| `RULE_NAME` | The unique name for this `RoleBinding` resource within the namespace. For example, `pa-cortex-alertmanager-viewer`. |
| `NAMESPACE` | The Kubernetes namespace where this `RoleBinding` will be created and apply. Look for the `Namespace` column in the previous table. |
| `EMAIL_ADDRESS` | The identifier for the user being granted the role. This is often an email address. For example, `infrastructure-operator@example.com`. |
| `ROLE` | The name of the `Role` that contains the permissions you want to grant to the user. Look for the roles available in the previous table |
| `GROUP_NAME` | The name of the `Group` to which PA/AO user belongs to. For example, `pa-group`. |
| `ZONE` | Name of the Zone |

> [!IMPORTANT]
> **Important:** Before committing configuration, ensure the `kustomization.yaml` file exists in the folder where you are create the yaml file.

### Config 1

This configuration grants permissions to a user group to access PA/AO alerts in `org` cluster. After setting up this configuration, refer [console](https://docs.cloud.google.com/distributed-cloud/hosted/docs/latest/gdcag/platform/pa-user/query-and-view-alerts.md.txt#console) section of this document to view alerts on the GDC console.

- **Kubectl Command**

  This is the generic command format:

  ```bash
  kubectl --kubeconfig `KUBECONFIG` create rolebinding `RULE_NAME` -n `NAMESPACE` --group=`GROUP_NAME` --role=project-cortex-alertmanager-viewer
  ```

  **Example:**

  ```bash
  kubectl --kubeconfig <path-to-kubeconfig> create rolebinding project-cortex-alertmanager-viewer-binding --role=project-cortex-alertmanager-viewer --group=my-team --namespace=platform-obs
  ```
- **IAC File path**

  ```bash
  /infrastructure/zonal/zones/`ZONE`/org-admin/rolebindings/`GROUP_NAME`/<YAML_FILE>
  ```
- **Yaml File**

  ```yaml
  apiVersion: rbac.authorization.k8s.io/v1
  kind: RoleBinding
  metadata:
    name: RULE_NAME
    namespace: NAMESPACE
  subjects:
  - kind: Group
    name: GROUP_NAME
    apiGroup: rbac.authorization.k8s.io
  roleRef:
    kind: Role
    name: project-cortex-alertmanager-viewer
    apiGroup: rbac.authorization.k8s.io
  ```

### Config 2

This configuration is for the `PA` persona, targeting the `grafana` object in the `org-admin` cluster.
It grants the `project-cortex-alertmanager-viewer` role within the `platform-obs` namespace to a `User`.

- **Kubectl Command**

  This is the generic command format:

  ```bash
  kubectl --kubeconfig `KUBECONFIG` create rolebinding `RULE_NAME` -n platform-obs --user=`EMAIL_ADDRESS` --role=project-cortex-alertmanager-viewer
  ```

  **Example:**

  ```bash
  kubectl --kubeconfig <path-to-kubeconfig> create rolebinding project-cortex-alertmanager-viewer-binding --role=project-cortex-alertmanager-viewer --user=my-email@example.com --namespace=platform-obs
  ```
- **IAC File path**

  ```bash
  /infrastructure/zonal/zones/`ZONE`/org-admin/rolebindings/`EMAIL_ADDRESS`/<YAML_FILE>
  ```
- **Yaml File**

  ```yaml
  apiVersion: rbac.authorization.k8s.io/v1
  kind: RoleBinding
  metadata:
    name: RULE_NAME
    namespace: platform-obs
  subjects:
  - kind: User
    name: EMAIL_ADDRESS
    apiGroup: rbac.authorization.k8s.io
  roleRef:
    kind: Role
    name: project-cortex-alertmanager-viewer
    apiGroup: rbac.authorization.k8s.io
  ```

  For more information about these roles, see
  [Prepare IAM permissions](https://docs.cloud.google.com/distributed-cloud/hosted/docs/latest/gdcag/platform/pa-user/obs-iam-permissions.md.txt).

## View and filter open alerts

Select one of the following methods to query and filter open alerts from your
project namespace:

### Console

View the open alerts in a project from the GDC console:

1. [Sign in to the GDC console](https://docs.cloud.google.com/distributed-cloud/hosted/docs/latest/gdcag/platform/pa-user/iam/sign-in.md.txt).
2. In the GDC console, select your project.
3. In the navigation menu, select **Operations** \> **Alerting**.
4. Select the **Alerts** tab.
5. View the list of alerts.
6. On the **Alerts opened** section, click **Filter** to only display open alerts. You can also filter alerts by other property names or values.
7. Click an alert name to view the alert details.

### Cortex endpoint

This section describes how to access alerts using your Cortex Alertmanager
endpoint.

#### Identify your Cortex endpoint

The following URL is the endpoint of the Cortex instance of your project:

```bash
  https://GDC_URL/PROJECT_NAMESPACE/cortex/alertmanager/
```

Replace the following:

- `GDC_URL`: the URL of your organization in GDC.
- `PROJECT_NAMESPACE`: your project namespace.

  For example, the Cortex endpoint for the `platform-obs` project in the
  `org-1` organization is `https://org-1/platform-obs/cortex/alertmanager/`.

  > [!NOTE]
  > **Note:** You can directly open the URL in a web browser to query alerts. The instructions in the following sections use the `curl` command to call the endpoint, but the same applies to a web browser.

#### Authenticate the `curl` request

1. [Download and install the gdcloud CLI](https://docs.cloud.google.com/distributed-cloud/hosted/docs/latest/gdcag/resources/gdcloud-overview.md.txt).
2. [Set](https://docs.cloud.google.com/distributed-cloud/hosted/docs/latest/gdcag/resources/gdcloud-reference/gdcloud-config-set.md.txt) the
   gdcloud `core/organization_console_url` property:

   ```bash
   gdcloud config set core/organization_console_url
   https://GDC_URL
   ```
3. [Sign in with the configured identity provider](https://docs.cloud.google.com/distributed-cloud/hosted/docs/latest/gdcag/resources/gdcloud-auth.md.txt#sign-in-with-idp):

   ```bash
   gdcloud auth login
   ```
4. Use your username and password to authenticate and sign in.

   When the login is successful, you can use the authorization header in your
   curl request through the `gdcloud auth print-identity-token` command. For
   more information, see
   [gdcloud auth](https://docs.cloud.google.com/distributed-cloud/hosted/docs/latest/gdcag/resources/gdcloud-reference/gdcloud-auth.md.txt).

#### Call the Cortex endpoint

Complete the following steps to reach the Cortex endpoint using the `curl`
tool:

1. [Authenticate the `curl` request](https://docs.cloud.google.com/distributed-cloud/hosted/docs/latest/gdcag/platform/pa-user/query-and-view-alerts.md.txt#auth-curl-request).
2. Use `curl` to call the [Cortex endpoint](https://docs.cloud.google.com/distributed-cloud/hosted/docs/latest/gdcag/platform/pa-user/query-and-view-alerts.md.txt#id-cortex-endpoint) and
   extend the URL using the standard Alertmanager API specification
   (<https://prometheus.io/docs/prometheus/latest/querying/api/#alertmanagers>)
   to query alerts.

   The following is an example of a `curl` request:

         curl https://GDC_URL/PROJECT_NAME/cortex/alertmanager/api/v1/alertmanagers \
         -H "Authorization: Bearer $(gdcloud auth print-identity-token \
         --audiences=https://GDC_URL)"

   You obtain the output following the command. The API response is in JSON
   format.

   > [!NOTE]
   > **Note:** Add the `api/v1/alertmanagers` endpoint to query active and eliminated alerts from the URL.