Manage incidents for SQL-based alerting policies

An incident is a record of when the condition of an alerting policy is met. Typically, Cloud Monitoring opens an incident and sends a notification when the condition of the alerting policy is met. However, incidents aren't created under the following circumstances:

  • The policy is snoozed or disabled.
  • The number of alerting policies or incidents exceeds existing limits for alerting.

For each incident, Monitoring creates an Incident details page that lets you manage the incident, and that reports incident information that can help you troubleshoot the failure. For example, the Incident details page shows lists of SQL query result summaries and related incidents.

This document describes how you can find your incidents. It also describes how you can use the Incident details page to manage incidents for SQL-based alerting policies, which evaluate the results of a SQL query run against data from groups of log entries.

This feature is supported only for Google Cloud projects. For App Hub configurations, select the App Hub host project or management project.

Before you begin

To get the permissions that you need to view and manage incidents, ask your administrator to grant you the following IAM roles on your project:

For more information about granting roles, see Manage access to projects, folders, and organizations.

You might also be able to get the required permissions through custom roles or other predefined roles.

For more information about Cloud Monitoring roles, see Control access with Identity and Access Management.

View incidents

To view incidents in your project, use the Google Cloud console, the gcloud CLI (Public Preview), or the Monitoring API (Public Preview).

Google Cloud console

To list the incidents in your Google Cloud project, do the following:

  1. In the Google Cloud console, go to the  Alerting page:

    Go to Alerting

    If you use the search bar to find this page, then select the result whose subheading is Monitoring.

  2. In the toolbar of the Google Cloud console, select your Google Cloud project. For App Hub configurations, select the App Hub host project or management project.

    The Alerting page displays information about your alerting policies, snoozes, and incidents:

    • The Summary pane lists the number of open incidents.
    • The Incidents table displays the most recent open incidents. To list the most recent incidents in the table, including those that are closed, click Show closed incidents.

  3. To view the details of a specific incident, select the incident in the list.

    The Incident details page opens. For more information about the Incident details page, see the Investigate an incident section of this document.

Find older incidents

The Incidents table on the Alerting page shows the most recent open incidents. To view older incidents, do one of the following:

  • To page through the entries in the Incidents table, click  Newer or  Older.

  • To open a page that lets you list and filter your incidents, click See all incidents. The Incidents page opens. From that page, do the following:

    • Show all incidents, including closed incidents. To show all incidents, click Show closed incidents.
    • Filter incidents. For information about adding filters, see Filter incidents.
    • Acknowledge or close an incident, or snooze its alerting policy. To access these options, click  More options in the incident's row, and make a selection from the menu. For more information, see Manage incidents.

Filter incidents

To restrict the incidents that the table shows, add filters. If you add multiple filters, the table displays only incidents that satisfy all the filters.

To filter the table of incidents, do the following:

  1. On the Incidents page, click  Filter table and then select a filter property. Filter properties include the following:

    • State of the incident
    • Name of the alerting policy
    • When the incident was opened or closed

  2. Select a value from the secondary menu or enter a value in the filter bar.

    The Incidents table then lists the filter.

gcloud

You can use the gcloud to get incidents and list incidents.

Get incident

Before using any of the command data below, make the following replacements:

  • ALERT_NAME: The resource name of the alert. For example, projects/my-project/alerts/my-alert-id.

Execute the gcloud alpha monitoring alerts describe command:

Linux, macOS, or Cloud Shell

gcloud alpha monitoring alerts describe ALERT_NAME

Windows (PowerShell)

gcloud alpha monitoring alerts describe ALERT_NAME

Windows (cmd.exe)

gcloud alpha monitoring alerts describe ALERT_NAME
 The command returns a response with alert details such as alert state, attached labels, and the source alerting policy. Note that the labels in the response are subject to change while this feature is in preview. The response looks similar to the following: 
{
  "name": "projects/my-project/alerts/my-alert-id",
  "state": "OPEN",
  "open_time": "2025-06-11T09:53:46Z",
  "resource": {
    "type": "sql_alert"
  },
  "policy": {
    "name": "projects/my-project/alertPolicies/POLICY_1",
    "displayName": "test-policy"
  },
  "summaryText": "The row count SQL Condition for \"test-policy\" is above the threshold of 1."
}

List incidents

You can list incidents in your project and filter the results using the gcloud CLI.

Before using any of the command data below, make the following replacements:

  • PROJECT_ID: The identifier of the project.

Execute the gcloud alpha monitoring alerts list command:

Linux, macOS, or Cloud Shell

gcloud alpha monitoring alerts list

Windows (PowerShell)

gcloud alpha monitoring alerts list

Windows (cmd.exe)

gcloud alpha monitoring alerts list
 The command returns a response with alert details such as alert state, attached labels, and the source alerting policy. Note that the labels in the response are subject to change while this feature is in preview. The response looks similar to the following: 
{
  "alerts": [
    {
      "name": "projects/my-project/alerts/my-alert-id",
      "state": "OPEN",
      "open_time": "2025-06-11T09:53:46Z",
      "resource": {
        "type": "sql_alert"
      },
      "policy": {
        "name": "projects/my-project/alertPolicies/POLICY_1",
        "displayName": "test-policy"
      },
      "summaryText": "The row count SQL Condition for \"test-policy\" is above the threshold of 1."
    }
  ]
}
You can also use the following optional flags to filter, sort, or modify the output:

  • --filter: Provide a filter expression to filter alerts by time or by label. For example, filter by time with --filter='close_time>="2025-09-10T00:00:00Z"', or filter by label with --filter='resource.labels.key="value"'.

  • --sort-by: A comma-separated list of fields to sort the output by. For example, --sort-by=open_time.

  • --uri: Command outputs a list of resource URIs instead of the default output.

  • --limit: Set this flag to 2 or greater to limit the number of alerts returned. Don't use this flag in conjunction with the --filter flag.

Monitoring API

You can use the Monitoring API to get incidents and list incidents.

Get incident

To get details on a single incident with the Monitoring API, use the alerts.get method.

Before using any of the request data, make the following replacements:

  • PROJECT_ID: The identifier of the project.
  • ALERT_ID: The ID of the alert.

HTTP method and URL: 

GET https://monitoring.googleapis.com/v3/projects/PROJECT_ID/alerts/ALERT_ID

To send your request, expand one of these options:

The command returns a response with alert details such as alert state, attached labels, and the source alerting policy. Note that the labels in the response are subject to change while this feature is in preview. The response looks similar to the following: 
{
  "name": "projects/my-project/alerts/my-alert-id",
  "state": "OPEN",
  "open_time": "2025-06-11T09:53:46Z",
  "resource": {
    "type": "sql_alert"
  },
  "policy": {
    "name": "projects/my-project/alertPolicies/POLICY_1",
    "displayName": "test-policy"
  },
  "summaryText": "The row count SQL Condition for \"test-policy\" is above the threshold of 1."
}

List incidents

To list incidents in your project and filter the results with the Monitoring API, use the alerts.list method.

Before using any of the request data, make the following replacements:

  • PROJECT_ID: The identifier of the project.

HTTP method and URL: 

GET https://monitoring.googleapis.com/v3/projects/PROJECT_ID/alerts

To send your request, expand one of these options:

The command returns a response with alert details such as alert state, attached labels, and the source alerting policy. Note that the labels in the response are subject to change while this feature is in preview. The response looks similar to the following: 
{
  "alerts": [
    {
      "name": "projects/my-project/alerts/my-alert-id",
      "state": "OPEN",
      "open_time": "2025-06-11T09:53:46Z",
      "resource": {
        "type": "sql_alert"
      },
      "policy": {
        "name": "projects/my-project/alertPolicies/POLICY_1",
        "displayName": "test-policy"
      },
      "summaryText": "The row count SQL Condition for \"test-policy\" is above the threshold of 1."
    }
  ]
}
Curl users can add the --data-urlencode flag followed by a filter expression to filter alerts by time or label. See the following examples:

List alerts that were opened in the last hour:

--data-urlencode "filter=(open_time>=\"`date -u -d "1 hour ago" +"%Y-%m-%dT%H:%M:%SZ"`\")"

List & filter alerts that are open from the last day:

--data-urlencode "filter=(open_time>=\"`date -u -d "1 DAY ago" +"%Y-%m-%dT%H:%M:%SZ"`\" AND state=open)"

List & filter alerts that were opened between two periods:

--data-urlencode "filter=(open_time>=\"`date -u -d "2 DAY ago" +"%Y-%m-%dT%H:%M:%SZ"`\" AND open_time<=\"`date -u -d "1 DAY ago" +"%Y-%m-%dT%H:%M:%SZ"`\")"

List & filter alerts by user label. In this example, filter by a user label with the name app and the value my-gke-app:

--data-urlencode "filter=(policy.user_labels.app=\"my-gke-app\")"
Powershell users can use the following example to add a time-based filter to your request url: 
$baseUrl = "https://monitoring.googleapis.com/v3/projects/my-project/alerts"
$filterValue = (Get-Date).AddHours(-1).ToString("yyyy-MM-ddTHH:mm:ssZ")
$filter = 'open_time >= "' + $filterValue + '"'
$encodedFilter = [System.Uri]::EscapeDataString($filter)
$url = $baseUrl + "?filter=" + $encodedFilter

Investigate an incident

The Incident details page contains information that may help you identify the cause of an incident.

Explore query results

The Total slot time consumed per day pane shows the amount of time that your reserved BigQuery slots spent running the SQL queries for the alerting policy over the last 24 hours.

The SQL query results pane shows a list of query result summaries from each time Log Analytics ran the SQL query from the alerting policy's condition. By default, the list is filtered to show only queries that matched the condition of the alerting policy.

  • To view the query and the table of query results from a specific time that Log Analytics ran the query, click a value from the Query run time column.
  • To toggle between showing only query results that matched the condition of the alerting policy and all queries that Log Analytics ran from the alerting policy, click Show only queries matching alert conditions.

View supplementary information

The Documentation section shows the documentation template for notifications that you provided when creating the alerting policy. This information might include a description of what the alerting policy monitors as well as tips for mitigation. For more information, see Annotate notifications with user-defined documentation.

If you didn't configure documentation for your alerting policy, then the Documentation pane shows "No documentation is configured."

To help you discover underlying issues across your application, you can explore incidents related to other alerting policy conditions.

The Related incidents section shows a list of other incidents that were created when the condition of the alerting policy was met.

Manage incidents

Incidents are in one of the following states:

  •  Open: The condition of the SQL-based alerting policy was met, and the incident is still open. If the same condition is met again and there is already an incident open, then a new incident isn't opened.

  •  Acknowledged: The incident is open and has manually been marked as acknowledged. Typically, this status indicates that the incident is being investigated.

  •  Closed: You have manually closed the incident, or it was automatically closed after the auto-close period expired.

Acknowledge incidents

We recommend that you mark an incident as acknowledged when you begin investigating the cause of the incident.

To mark an incident as acknowledged, do the following:

  1. In the Incidents table of the Alerting page, click See all incidents.

  2. On the Incidents page, find the incident that you want to acknowledge, and then do one of the following:

    • Click  More options and then select Acknowledge.
    • Open the details page for the incident and then click Acknowledge incident.

Snooze an alerting policy

To prevent Monitoring from creating incidents and sending notifications during a specific time period, snooze the related alerting policy. When you snooze an alerting policy, incidents related to the alerting policy remain open but don't cause further notifications. The incidents close based on the alerting policy auto-close duration.

To create a snooze for an incident that you are viewing, do the following:

  1. On the Incident details page, click Snooze Policy.

  2. Select the snooze duration. After you select the snooze duration, the snooze begins immediately.

You can also snooze an alerting policy from the Incidents page by finding the incident that you want to snooze, clicking  More options, and then selecting Snooze. You can snooze alerting policies during outages to prevent further notifications during the troubleshooting process.

Close incidents

You can let Monitoring close an incident for you, or you can close the incident.

Monitoring automatically closes an incident when the auto-close duration for the alerting policy expires. By default, the auto-close duration is 7 days. The minimum auto-close duration is 30 minutes.

The auto-close duration specifies the time that must elapse, without a repeat of the cause of the incident, before the incident closes. For this reason, when an incident is open and its cause reoccurs, the incident can stay open longer than the auto-close duration.

To close an incident, do the following:

  1. In the Incidents table of the Alerting page, click See all incidents.

  2. On the Incidents page, find the incident that you want to close, and then do one of the following:

    • Click  View more and then select Close incident
    • Open the Incident details page for that incident and then click Close incident.

If you see the message Unable to close incident, try again in a few minutes. You can't close a new incident immediately because the conditions that caused the incident are still considered active by the alerting system.

Data retention and limits

For information about limits and about the retention period of incidents, see Limits for alerting.

What's next