Troubleshoot missing notifications

You can diagnose and resolve missing Cloud Monitoring alert notifications by inspecting notification_channel_events logs in Logs Explorer. You can then fix delivery and configuration issues across webhook, Pub/Sub, and SMS channels, or adjust alerting policies for virtual machine (VM) shutdowns and request-count metrics.

Notifications aren't received

If you don't receive any notifications through any configured notification channels, then do the following:

  1. In the Google Cloud console, go to the Logs Explorer page:

    Go to Logs Explorer

    If you use the search bar to find this page, then select the result whose subheading is Logging.

  2. Select the appropriate Google Cloud project.

  3. Query the logs for notification channel events:

    1. Expand the Log name menu and select notification_channel_events.
    2. Expand the Severity menu and select Error.
    3. Optional: To select a custom time range, use the time-range selector.
    4. Click Run query.

    The previous steps create the following query:

    resource.type:"stackdriver_notification_channel"
logName="projects/PROJECT_ID/logs/monitoring.googleapis.com%2Fnotification_channel_events"
severity=ERROR

    The summary line and the jsonPayload field typically contain failure information. For example, when a gateway error occurs, the summary line includes "failed with 502 Bad Gateway".

Webhook notifications aren't received

This section applies when you don't receive notifications through a configured webhook notification channel.

Private endpoint

If you have a private endpoint, then use Pub/Sub notifications combined with a pull subscription to that notification topic. You can't use webhooks for notifications to private endpoints.

When you configure a Pub/Sub notification channel, alert notifications are sent to a Pub/Sub queue that has Identity and Access Management controls. Any service that can query for, or listen to, a Pub/Sub topic can consume these notifications. For example, applications running on App Engine, Cloud Run, or Compute Engine virtual machines can consume these notifications.

If you use a pull subscription, then a request is sent to Google that waits for a message to arrive. These subscriptions require access to Google, but they don't require rules for firewalls or inbound access.

Public endpoint

To identify why the delivery failed, examine your Cloud Logging log entries for failure information.

For example, you can search for log entries for the notification channel resource by using the Logs Explorer, with a filter like the following:

resource.type="stackdriver_notification_channel"

Pub/Sub notifications aren't received

If you don't receive notifications through a configured Pub/Sub notification channel, then the logs can help you diagnose and resolve the failure. This section describes the log entries that the system writes when a notification fails to be sent. You can use the information in this section to resolve the reason for the failure.

Logs include a Failed to authenticate as service account error

When you query the notification_channel_events log for errors, you might find a log entry whose summary field includes the following message:

Failed to authenticate as service account service-PROJECT_NUMBER@gcp-sa-monitoring-notification.iam.gserviceaccount.com

This error occurs when the notifications service account doesn't exist. As a result, notifications aren't sent.

To verify that the service account exists, do the following:

  1. In the Google Cloud console, go to the IAM page:

    Go to IAM

    If you use the search bar to find this page, then select the result whose subheading is IAM & Admin.

  2. Search for a service account that has the following naming convention:

    service-PROJECT_NUMBER@gcp-sa-monitoring-notification.iam.gserviceaccount.com

    If this service account isn't listed, then select Include Google-provided role grants.

To trigger Monitoring to create the notifications service account, begin the process of creating a Pub/Sub notification channel:

  1. In the Google Cloud console, go to the  Alerting page:

    Go to Alerting

    If you use the search bar to find this page, then select the result whose subheading is Monitoring.

  2. Click Edit notification channels.

  3. In the Pub/Sub section, click Add new.

    Monitoring creates the notifications service account when one doesn't exist. The Create Pub/Sub Channel dialog shows the name of the notifications service account.

  4. If you don't want to add a notification channel, click Cancel. Otherwise, finish creating the notification channel and click Add channel.

  5. Grant the service account permissions to publish to your Pub/Sub topics:

    1. In a new browser tab, open the Create a notification channel document.
    2. Select the Pub/Sub tab, and then follow the steps in the Authorize service account section of the page.

Logs include a PERMISSION_DENIED error

When you query the notification_channel_events log for errors, you might find a log entry whose summary field includes the following message:

An error occurred while publishing notification to Cloud Pub/Sub topic TOPIC_ID. Possible causes: 1) you don't have the required role (https://cloud.google.com/monitoring/support/notification-options#pubsub); or 2) Cloud Pub/Sub API is not enabled in your project: PERMISSION_DENIED

This error occurs when the notifications service account hasn't been authorized to send notifications for the Pub/Sub topics of interest, or when the Pub/Sub API isn't enabled in your project.

To view the permissions for a service account, you can use the Google Cloud console or the Google Cloud CLI command:

  • The IAM page in the Google Cloud console lists the roles for each service account.
  • The Pub/Sub Topics page in the Google Cloud console lists each topic. When you select a topic, the Permissions tab lists the roles granted to service accounts.

  • To list all service accounts and their roles, run the following Google Cloud CLI command:

    gcloud projects get-iam-policy PROJECT_ID

    The following is a partial response for this command:

    serviceAccount:service-PROJECT_NUMBER@gcp-sa-monitoring-notification.iam.gserviceaccount.com
   role: roles/monitoring.notificationServiceAgent
   - members:
   [...]
   role: roles/owner
   - members:
   - serviceAccount:service-PROJECT_NUMBER@gcp-sa-monitoring-notification.iam.gserviceaccount.com
   role: roles/pubsub.publisher

    The command response includes only roles; it doesn't include per-topic authorization.

  • To list the IAM bindings for a specific topic, run the following command:

    gcloud pubsub topics get-iam-policy TOPIC_ID

    The following is a sample response for this command:

    bindings:
- members:
   - serviceAccount:service-PROJECT_NUMBER@gcp-sa-monitoring-notification.iam.gserviceaccount.com
   role: roles/pubsub.publisher
etag: BwXPRb5WDPI=
version: 1

For information about how to authorize the notifications service account, see Authorize service account.

Logs include a FAILED_PRECONDITION error

When you query the notification_channel_events log for errors, you might find a log entry whose summary field includes the following message:

An error occurred while publishing notification to Cloud Pub/Sub topic TOPIC_ID: FAILED_PRECONDITION

This error occurs when Pub/Sub rejects a publish request from Monitoring because the target topic doesn't meet a required precondition. Common causes for this error include the following:

  • Message storage policy and data residency restrictions: Your topic's message storage policy restricts the Google Cloud regions where messages can be processed or stored:

    • In-transit enforcement in a disallowed region: The topic's message storage policy has enforceInTransit set to true, and the publish request arrives in a region that isn't in allowedPersistenceRegions. When enforceInTransit is true, Pub/Sub rejects requests from disallowed regions instead of redirecting them. If the topic uses AI inference single message transforms, then the publish region must also be allowed by the transform.
    • Restricted allowed persistence regions: The topic has enforceInTransit set to false and the publish request arrives outside allowedPersistenceRegions, but Pub/Sub can't redirect the request because no viable cluster is available and allowedPersistenceRegions contains only private or restricted regions.
    • Cross-region synchronous replication without a valid secondary region: The topic uses cross-region synchronous replication and has a message storage policy, but allowedPersistenceRegions doesn't include a valid secondary replication region for the publish location.
    • Regional endpoint without in-transit enforcement: The publish request is routed to a regional endpoint, but the topic's message storage policy has enforceInTransit set to false.

    To resolve message storage policy errors, update your topic's message storage policy or single message transform region configuration so that allowedPersistenceRegions includes the publish region and any required secondary replication region, or set enforceInTransit to false if in-transit regional enforcement isn't required.

  • Customer-managed encryption key (CMEK) errors: Your topic is configured to use a customer-managed encryption key from Cloud Key Management Service (Cloud KMS), and Pub/Sub can't encrypt the published message:

    • Permission denied on the Cloud KMS key: Cloud KMS returns a PERMISSION_DENIED error when Pub/Sub attempts to encrypt the message, which Pub/Sub reports as FAILED_PRECONDITION. For example, this error occurs when the Pub/Sub service agent doesn't have the CryptoKey Encrypter/Decrypter (roles/cloudkms.cryptoKeyEncrypterDecrypter) role on the key.
    • Unusable Cloud KMS key state: Cloud KMS returns a FAILED_PRECONDITION error because the key version is disabled or destroyed, or because an external key manager key is unreachable.

    To resolve CMEK errors, verify that the Cloud KMS key version is enabled and reachable, and ensure that the Pub/Sub service agent has the roles/cloudkms.cryptoKeyEncrypterDecrypter role on the key.

You aren't notified when a VM shuts down

To get notified when a virtual machine (VM) shuts down, create an uptime check to periodically interrogate the VM and then create an alerting policy to monitor that uptime check. If you are using a Virtual Private Cloud (VPC), then you might need to create a private uptime check.

An alerting policy that monitors the compute.googleapis.com/instance/uptime metric won't notify you when the VM shuts down. For this metric, alerting policies only monitor time series for VM instances that are in the RUNNING state. If a VM is in any other state, such as STOPPED or DELETED, then it isn't monitored. For information on VM instance states, see VM instance life cycle.

Notifications for request-count alerting policies aren't received

If you don't receive notifications for an alerting policy that monitors the serviceruntime.googleapis.com/api/request_count metric, then make sure that the policy's alignment period is no more than 7 hours 30 minutes.

SMS notification messages or verification codes aren't received

If you don't receive SMS notifications or verification codes, then make sure that you haven't reached the SMS message limit. There might be logs that confirm this error. Check your logs for Denied quota token.

Note that SMS isn't a reliable notification channel type and might not be available in certain regions. Avoid relying solely on SMS channels for notifications. Instead, configure additional notification channels such as email.