Monitor ingestion data

Supported in:

This guide is for security engineers who want to monitor data ingestion health and troubleshoot issues within Google Security Operations. It explains how to use the Health Hub dashboard, configure Cloud Monitoring alerts to track the status of data sources and parsers, and use silent-host monitoring (SHM) to detect potential collector stoppages.

Using these monitoring workflows helps you identify, diagnose, and remediate data pipeline problems, distinguishing between user-fixable (actionable) and support-required (non-actionable) issues. Successful configuration ensures reliable data flow, which is crucial for effective security operations and analysis.

Common use cases

This section covers common use cases for ingestion monitoring.

Proactive health check

  • Objective: Review the status and health of all configured data sources.
  • Value: Detect potential ingestion problems before they impact security visibility.

Alert response

  • Objective: Investigate and fix a data source or parser issue flagged by an automated alert.
  • Value: Minimize data loss or delays, ensuring data is available for timely threat detection and response.

Key terminology

  • Health Hub: The central dashboard within Google SecOps for monitoring the status and health of all configured data sources and parsers.
  • Cloud Monitoring: Google Cloud service used to create alerting policies based on metrics, including those from Google SecOps ingestion.
  • Silent-host monitoring: Monitoring method to identify hosts in your environment that have gone silent.
  • Actionable issue: An ingestion problem that you can typically resolve yourself through configuration changes (for example, updating credentials).
  • Non-actionable issue: An ingestion problem that requires assistance from Google support to resolve (for example, an internal system error).
  • Parser: A component that normalizes raw log data into the Unified Data Model (UDM) structure.

Before you begin

Ensure you have the necessary Identity and Access Management (IAM) roles and permissions to access the Google SecOps instance, view the Health Hub dashboard, and configure alerts in Cloud Monitoring.

Monitor data ingestion

This section details various methods to monitor data ingestion.

Monitor using the Health Hub dashboard

Use the Health Hub dashboard to monitor overall data health at a glance and view the core health status for each feed, data source, and log type. This helps you identify irregular and failed sources and parsers without configuring custom external alerts.

  1. In the Google SecOps side navigation menu, click Health Hub.
  2. Review the Big Number widgets for Failed Sources and Failed Parsers to identify components requiring immediate attention.
  3. Inspect the Health Status by Data Source table. Check the Latest Issue Details column for error descriptions, such as, Config credential issue or Normalization issue.
  4. Click the Edit Data Source or Edit Parser links provided in the table to navigate directly to the respective configuration pages for remediation.
  5. Check timestamps, such as Last Event Time and Last Ingested, to verify that the data was ingested as expected.
  6. After applying a fix, monitor the Health Hub dashboard for the specific data source or parser to ensure the remediation was successful.

Set up automated ingestion alerts

Configure Monitoring to trigger alerts when ingestion values reach certain predefined levels. This lets you integrate email notifications into existing workflows to proactively fix broken feeds or detect silent log sources.

  1. In the Health Hub dashboard, click the Set Up Alerts link, which directs you to the Monitoring interface.
  2. Create an alerting policy:
    • Select metrics: Choose metrics under Chronicle Collector > Ingestion, such as Total ingested log count or Total ingested log size.
    • Add filters for collector_id or log_type to narrow the alert scope to specific sources.
  3. To detect silent forwarders, select Metric absence as the condition type. Configure it to trigger an alert if logs stop flowing for a specified duration (for example, 60 minutes).

Troubleshooting

This section details various methods to troubleshoot data ingestion issues.

Common issues

  • Pipeline latency: A significant delay between the Last Event Time and Last Ingested timestamp indicates potential latency. Health Hub exposes the 95th percentile of this delta. High values suggest pipeline latency, while normal values might mean the source is sending historical data.
  • Ingestion surges or drops: The system uses z-score standardization to flag anomalies. A drop is flagged if both daily and weekly standardized differences are less than -1.645.
  • Parsing failures: An alert is triggered if the proportion of parser errors relative to total ingested events increases by 5 percentage points or more compared to the previous day. Investigate the parser configuration using the link in Health Hub.

Latency, service quota, and limits

  • Data Refresh: The information on the Health Hub and Data Ingestion dashboards refreshes approximately every 15 minutes.

Error remediation

For a full list of error messages and solutions, see Troubleshoot ingestion.

Need more help? Get answers from Community members and Google SecOps professionals.