Monitoring (MON) audit logs in Google Distributed Cloud (GDC) air-gapped appliance let you track
PromQL query execution along with dashboard and alert configuration changes. The
observability audit logger and Kubernetes audit logs record metadata fields and
JSON payloads for these data plane and control plane operations.

|---|---|
| **Audit log source** | - [Observability audit logger](https://docs.cloud.google.com/distributed-cloud/hosted/docs/latest/appliance/platform-application/pa-ao-operations/audit-observability-logger) - [Kubernetes audit logs](https://docs.cloud.google.com/distributed-cloud/hosted/docs/latest/appliance/platform-application/pa-ao-operations/audit-kubernetes) |
| **Audited operations** | - [Run PromQL queries using the user interface of the monitoring instance](https://docs.cloud.google.com/distributed-cloud/hosted/docs/latest/appliance/platform-application/pa-ao-operations/audit-mon#promql-monitoring-instance) - [Run PromQL queries using the HTTP API](https://docs.cloud.google.com/distributed-cloud/hosted/docs/latest/appliance/platform-application/pa-ao-operations/audit-mon#promql-http) - [Perform dashboard CRUD operations](https://docs.cloud.google.com/distributed-cloud/hosted/docs/latest/appliance/platform-application/pa-ao-operations/audit-mon#dashboard-crud) - [Perform alert CRUD operations](https://docs.cloud.google.com/distributed-cloud/hosted/docs/latest/appliance/platform-application/pa-ao-operations/audit-mon#alert-crud) |

## Run PromQL queries using the user interface of the monitoring instance

|---|---|
| **Audit log source** | Proxy server |
| **Log type** | Data plane |

| Fields in the log entry that contain audit information |||
| Audit metadata | Audit field name | Value |
|---|---|---|
| **User or service identity** | `user` | For example, ```json "user":{ "identity":"fop-cluster-admin@example.com", "issuer":"https://ais-core.org-1.zone1.google.gdch.test" } ``` |
| **Target** *(Fields and values that call the API)* | `resource` | For example, `"resource": "/infra-obs/grafana/api/ds/query"` |
| **Action** *(Fields containing the performed operation)* | `action` | Possible values: - `"action": "QUERY"` - `"action": "CREATE"` - `"action": "READ"` - `"action": "UPDATE"` - `"action": "DELETE"` - `"action": "CREATE/UPDATE"` |
| **Event timestamp** | `time` | For example, `"time": "2022-12-05T14:39:15.713354008Z"` |
| **Source of action** | - `sourceIPs` - `_gdch_service_name` | For example, ```json "sourceIPs":[ "10.253.166.214", "127.0.0.6" ], "_gdch_service_name":"grafana" ``` |
| **Outcome** | `response` | For example, `"response": "Successful: 200 OK"` |
| **Other fields** | `description` | The `description` value contains the complete query. For more information. see the **Example log**. |

**Example log**

    {
      "resource":"/infra-obs/grafana/api/ds/query",
      "response":"Successful: 200 OK",
      "_gdch_service_tenant":"infra-obs",
      "sourceIPs":[
        "10.253.166.214",
        "127.0.0.6"
      ],
      "_gdch_namespace":"infra-obs-obs-system",
      "userAgent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:107.0) Gecko/20100101 Firefox/107.0",
      "time":"2022-12-05T14:39:15.713354008Z",
      "auditID":"6bba5ff1-97d9-4bf8-92af-4f63049448cf",
      "numBytesSent":1821,
      "action":"QUERY",
      "_gdch_service_name":"grafana",
      "numBytesReceived":2827,
      "description":"{
          \"queries\":[{
          \"refId\":\"A\",
          \"expr\":\"{container=\\\"grafana-proxy-server\\\"},
          \"queryType\":\"range\",
          \"datasource\":{\"uid\":\"P982945308D3682D1\",\"type\":\"loki\"},
          \"key\":\"Q-c63373da-dec2-49c3-aa6c-4e5ba07ec8de-0\",
          \"editorMode\":\"builder\",
          \"maxLines\":1000,
          \"legendFormat\":\"\",
          \"datasourceId\":2,
          \"intervalMs\":1000,
          \"maxDataPoints\":2493
        }],
        \"range\":{
          \"from\":\"2022-12-05T13:39:15.461Z\",
          \"to\":\"2022-12-05T14:39:15.461Z\",
          \"raw\":{\"from\":\"now-1h\",\"to\":\"now\"}
        },
        \"from\":\"1670247555461\",
        \"to\":\"1670251155461\"
      }",
      "user":{
        "identity":"fop-cluster-admin@example.com",
        "issuer":"https://ais-core.org-1.zone1.google.gdch.test"
      },
      "_gdch_cluster":"org-1-admin",
      "_gdch_fluentbit_pod":"anthos-audit-logs-forwarder-whltm"
    }

## Run PromQL queries using the HTTP API

|---|---|
| **Audit log source** | Proxy server |
| **Log type** | Data plane |

| Fields in the log entry that contain audit information |||
| Audit metadata | Audit field name | Value |
|---|---|---|
| **User or service identity** | `user` | For example, ```json "user":{ "issuer":"https://ais-core.org-1.zone1.google.gdch.test", "identity":"fop-cluster-admin@example.com" } ``` |
| **Target** *(Fields and values that call the API)* | `resource` | For example, `"resource":"/alertmanager/api/v2/alerts/groups?silenced=false&inhibited=false&active=true"` |
| **Action** *(Fields containing the performed operation)* | `action` | Possible values: - `"action": "QUERY"` - `"action": "CREATE"` - `"action": "READ"` - `"action": "UPDATE"` - `"action": "DELETE"` - `"action": "CREATE/UPDATE"` |
| **Event timestamp** | `time` | For example, `"time": "2022-12-05T18:20:50.616925009Z"` |
| **Source of action** | - `sourceIPs` - `_gdch_service_name` | For example, ```json "sourceIPs":[ "10.200.0.1", "127.0.0.6" ], "_gdch_service_name":"cortex" ``` |
| **Outcome** | `response` | For example, `"response": "Successful: 200 OK"` |
| **Other fields** | *Not applicable* | *Not applicable* |

**Example log**

    {
      "user":{
        "issuer":"https://ais-core.org-1.zone1.google.gdch.test",
        "identity":"fop-cluster-admin@example.com"
      },
      "_gdch_service_tenant":"infra-obs",
      "_gdch_service_name":"cortex",
      "resource":"/alertmanager/api/v2/alerts/groups?silenced=false&inhibited=false&active=true",
      "time":"2022-12-05T18:20:50.616925009Z",
      "action":"READ",
      "numBytesReceived":2376,
      "sourceIPs":[
        "10.200.0.1",
        "127.0.0.6"
      ],
      "_gdch_namespace":"obs-system",
      "userAgent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:107.0) Gecko/20100101 Firefox/107.0",
      "numBytesSent":173,
      "auditID":"8451a7b3-77f9-4878-9308-641b55a83865",
      "response":"Successful: 200 OK",
      "_gdch_cluster":"org-1-admin",
      "_gdch_fluentbit_pod":"anthos-audit-logs-forwarder-2wqxp"
    }

## Perform dashboard CRUD operations

|---|---|
| **Audit log source** | [Kubernetes audit logs](https://docs.cloud.google.com/distributed-cloud/hosted/docs/latest/appliance/platform-application/pa-ao-operations/audit-kubernetes) |
| **Log type** | Control plane |

| Fields in the log entry that contain audit information |||
| Audit metadata | Audit field name | Value |
|---|---|---|
| **User or service identity** | `user` | For example, ```json "user":{ "extra":{ "authentication.kubernetes.io/pod-name":["fleet-admin-controller-875778d98-dnkj2"], "authentication.kubernetes.io/pod-uid":["caa4df7a-ae04-458e-a616-1c6893ce6e46"] }, "username":"system:serviceaccount:gpc-system:fleet-admin-controller", "groups":[ "system:serviceaccounts", "system:serviceaccounts:gpc-system", "system:authenticated" ], "uid":"0b93d757-e3be-440a-b18a-4a2b524de156" } ``` |
| **Target** *(Fields and values that call the API)* | - `requestURI` - `objectRef` | For example, ```json "requestURI":"/apis/observability.gdc.goog/v1/namespaces/alice-obs-system/dashboards", "objectRef":{ "apiVersion":"v1", "apiGroup":"observability.gdc.goog", "resource":"dashboards", "namespace":"alice-obs-system" } ``` |
| **Action** *(Fields containing the performed operation)* | `verb` | Possible values: - `"verb": "create"` - `"verb": "delete"` - `"verb": "get"` - `"verb": "list"` - `"verb": "patch"` - `"verb": "update"` - `"verb": "watch"` |
| **Event timestamp** | `requestReceivedTimestamp` | For example, `"requestReceivedTimestamp": "2022-12-05T15:36:24.980257Z"` |
| **Source of action** | - `sourceIPs` - `_gdch_service_name` | For example, ```json "sourceIPs":["10.253.166.100"], "_gdch_service_name":"apiserver" ``` |
| **Outcome** | `responseStatus` | For example, ```json "responseStatus":{ "code":201, "metadata":{} } ``` |
| **Other fields** | *Not applicable* | *Not applicable* |

**Example log**

    {
      "user":{
        "extra":{
          "authentication.kubernetes.io/pod-name":["fleet-admin-controller-875778d98-dnkj2"],
          "authentication.kubernetes.io/pod-uid":["caa4df7a-ae04-458e-a616-1c6893ce6e46"]
        },
        "username":"system:serviceaccount:gpc-system:fleet-admin-controller",
        "groups":[
          "system:serviceaccounts",
          "system:serviceaccounts:gpc-system",
          "system:authenticated"
        ],
        "uid":"0b93d757-e3be-440a-b18a-4a2b524de156"
      },
      "kind":"Event",
      "_gdch_fluentbit_pod":"anthos-audit-logs-forwarder-2wqxp",
      "apiVersion":"audit.k8s.io/v1",
      "_gdch_cluster":"org-1-admin",
      "level":"Metadata",
      "stageTimestamp":"2022-12-05T15:36:24.980257Z",
      "auditID":"a060d80a-4a47-4490-a859-5d3ccff36d3d",
      "requestReceivedTimestamp":"2022-12-05T15:36:24.980257Z",
      "userAgent":"fleet-admin-cm/v0.0.0 (linux/amd64) kubernetes/$Format",
      "stage":"RequestReceived",
      "requestURI":"/apis/observability.gdc.goog/v1/namespaces/alice-obs-system/dashboards",
      "objectRef":{
        "apiVersion":"v1",
        "apiGroup":"observability.gdc.goog",
        "resource":"dashboards",
        "namespace":"alice-obs-system"
      },
      "verb":"create",
      "sourceIPs":["10.253.166.100"],
      "_gdch_service_name":"apiserver"
    }

## Perform alert CRUD operations

|---|---|
| **Audit log source** | [Kubernetes audit logs](https://docs.cloud.google.com/distributed-cloud/hosted/docs/latest/appliance/platform-application/pa-ao-operations/audit-kubernetes) |
| **Log type** | Control plane |

| Fields in the log entry that contain audit information |||
| Audit metadata | Audit field name | Value |
|---|---|---|
| **User or service identity** | `user` | For example, ```json "user":{ "username":"kubernetes-admin", "groups":[ "system:masters", "system:authenticated" ] } ``` |
| **Target** *(Fields and values that call the API)* | - `requestURI` - `objectRef` | For example, ```json "requestURI":"/apis/monitoring.gdc.goog/v1/namespaces/alice/monitoringrules?fieldManager=kubectl-client-side-apply&fieldValidation=Strict", "objectRef":{ "apiVersion":"v1", "apiGroup":"monitoring.gdc.goog", "name":"obs-test-alert-sequel", "namespace":"alice", "resource":"monitoringrules" } ``` |
| **Action** *(Fields containing the performed operation)* | `verb` | Possible values: - `"verb": "create"` - `"verb": "delete"` - `"verb": "get"` - `"verb": "list"` - `"verb": "patch"` - `"verb": "update"` - `"verb": "watch"` |
| **Event timestamp** | `requestReceivedTimestamp` | For example, `"requestReceivedTimestamp": "2022-12-05T16:28:50.619659Z"` |
| **Source of action** | - `sourceIPs` - `_gdch_service_name` | For example, ```json "sourceIPs":["10.200.0.6"], "_gdch_service_name":"apiserver" ``` |
| **Outcome** | `responseStatus` | For example, ```json "responseStatus":{ "code":201, "metadata":{} } ``` |
| **Other fields** | *Not applicable* | *Not applicable* |

**Example log**

    {
      "level":"Metadata",
      "sourceIPs":[
        "10.200.0.6"
      ],
      "auditID":"753c3370-d3a5-4717-b84e-00fd56883fc4",
      "requestURI":"/apis/monitoring.gdc.goog/v1/namespaces/alice/monitoringrules?fieldManager=kubectl-client-side-apply&fieldValidation=Strict",
      "apiVersion":"audit.k8s.io/v1",
      "_gdch_fluentbit_pod":"anthos-audit-logs-forwarder-fgkth",
      "user":{
        "username":"kubernetes-admin",
        "groups":[
          "system:masters",
          "system:authenticated"
        ]
      },
      "userAgent":"kubectl/v1.25.4 (linux/amd64) kubernetes/872a965",
      "verb":"create",
      "stage":"ResponseComplete",
      "stageTimestamp":"2022-12-05T16:28:50.636050Z",
      "_gdch_cluster":"org-1-admin",
      "objectRef":{
        "apiVersion":"v1",
        "apiGroup":"monitoring.gdc.goog",
        "name":"obs-test-alert-sequel",
        "namespace":"alice",
        "resource":"monitoringrules"
      },
      "responseStatus":{
        "code":201,
        "metadata":{}
      },
      "kind":"Event",
      "annotations":{
        "authorization.k8s.io/reason":"",
        "authorization.k8s.io/decision":"allow"
      },
      "requestReceivedTimestamp":"2022-12-05T16:28:50.619659Z",
      "_gdch_service_name":"apiserver"
    }