Configure notification channels for alerts

Google Distributed Cloud (GDC) air-gapped appliance lets you configure custom notification channels to route triggered alerts to specific individuals, teams, or on-call rotations. Delivering alerts through channels such as PagerDuty, SMS, webhooks, or custom integrations helps reduce alert fatigue and supports timely incident response.

Because predefined notification channels such as Slack or email are unavailable in GDC, you must configure at least one custom channel to receive alerts. You define the channel type, destination endpoint, and authentication credentials in an Alertmanager ConfigMap object. You then import that configuration into an ObservabilityPipeline custom resource in your project namespace.

Before you begin

To manage ObservabilityPipeline custom resources, you must have the necessary permissions.

Request IAM organization roles

Contact your Organization IAM Admin to request the following roles:

• ObservabilityPipeline PA Creator (observabilitypipeline-pa-creator): create ObservabilityPipeline custom resources.
• ObservabilityPipeline PA Editor (observabilitypipeline-pa-editor): edit or delete ObservabilityPipeline custom resources.
• ObservabilityPipeline PA Viewer (observabilitypipeline-pa-viewer): view ObservabilityPipeline custom resources.
• ConfigMap Editor (observabilitypipeline-configmap-editor): create ConfigMap objects in the platform namespace.

Request IAM project roles

Contact your Project IAM Admin to request the following roles:

• ObservabilityPipeline Editor (observabilitypipeline-editor): edit or delete ObservabilityPipeline custom resources in the project namespace.
• ObservabilityPipeline Viewer (observabilitypipeline-viewer): view ObservabilityPipeline custom resources in the project namespace.

Configure notification channels

Important: Before configuring notification channels, ensure you have created alert rules in your project namespace. For more information, see Create alert rules.

Configure notification channels for alerts in your project namespace:

1. Define your custom configuration for notification channels in a YAML file named alertmanager.yml. You must follow the same syntax as the Alertmanager specification:

   https://prometheus.io/docs/alerting/latest/configuration/.

2. Create a ConfigMap object and include your custom configuration from the alertmanager.yml file in the data field.

   The following example shows how the ConfigMap object must look with the alertmanager.yml file:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: CONFIGMAP_NAME
     # The namespace must match your project namespace.
     namespace: PROJECT_NAMESPACE
   data:
     # The file name must be alertmanager.yml
     alertmanager.yml: |
       # Define your notification channels.
       # Add the custom configuration in the following lines of this file.
       # Follow the same syntax as in https://prometheus.io/docs/alerting/latest/configuration
   
     [...]
   

   Replace the following:

   • CONFIGMAP_NAME: the name of the ConfigMap definition file. You use this name later in the ObservabilityPipeline custom resource.
   • PROJECT_NAMESPACE: your project namespace.

3. Apply the ConfigMap object to the Management API server within the same namespace as your configured alerts:

   kubectl --kubeconfig KUBECONFIG_PATH apply -f CONFIGMAP_NAME.yaml
   

   Replace the following:

   • KUBECONFIG_PATH: the path to the kubeconfig file for the Management API server.
   • CONFIGMAP_NAME: the name of the ConfigMap definition file.

4. Edit the ObservabilityPipeline custom resource specification to import your configuration to the alertmanagerConfig field:

    # Configure the observability pipeline.
   apiVersion: observability.gdc.goog/v1
   kind: ObservabilityPipeline
   metadata:
     # The namespace must match your project namespace.
     namespace: PROJECT_NAMESPACE
     name: OBSERVABILITY_PIPELINE_NAME
   spec:
     # Configure alerts.
     alerting:
       # The alerting configuration must be in the ConfigMap.
       # The value must match the ConfigMap name exactly.
       alertmanagerConfig: CONFIGMAP_NAME
   
     [...]
   

   Replace the following:

   • PROJECT_NAMESPACE: your project namespace.
   • OBSERVABILITY_PIPELINE_NAME: the name of the ObservabilityPipeline custom resource.
   • CONFIGMAP_NAME: the name you used for the ConfigMap definition file. The name must match exactly.
   Note: In the alerting field of the ObservabilityPipeline custom resource, you can also define other configurations for the project namespace such as volumes (volumes) and volume mounts (volumeMounts). You reference volume mount names in the alertmanager.yml file.

5. Save the file.

6. Apply the changes of the ObservabilityPipeline custom resource to the Management API server within the same namespace as your configured alerts:

   kubectl --kubeconfig KUBECONFIG_PATH apply -f OBSERVABILITY_PIPELINE_NAME.yaml
   

   Replace the following:

   • KUBECONFIG_PATH: the path to the kubeconfig file for the Management API server.
   • OBSERVABILITY_PIPELINE_NAME: the name of the ObservabilityPipeline definition file.