Monitor ingestion data

Supported in:

This guide is for a Security Engineer who wants to monitor data ingestion health and troubleshoot issues within Google Security Operations. It explains how to use the Health Hub dashboard and configure Cloud Monitoring alerts to track the status of data sources and parsers. By following this monitoring workflow, engineers can quickly identify, diagnose, and remediate data pipeline problems, distinguishing between user-fixable (actionable) and support-required (non-actionable) issues. Successful completion ensures reliable data flow, which is crucial for effective security operations and analysis.

The Health Hub dashboard is the central place in Google SecOps to monitor all configured data sources. This guide details how to interpret the metrics and take action on any failures.

Common use cases

This section covers common use cases.

Proactive health check

  • Objective: Regularly review the status and health of all configured data sources.
  • Value: Early detection of potential ingestion problems before they impact security visibility.

Alert response

  • Objective: Investigate and fix a data source or parser issue flagged by an automated alert.
  • Value: Minimize data loss or delays, ensuring data is available for timely threat detection and response.

Key terminology

  • Health Hub: The central dashboard within Google SecOps for monitoring the status and health of all configured data sources and parsers.
  • Cloud Monitoring: Google Cloud service used to create alerting policies based on metrics, including those from Chronicle ingestion.
  • Actionable issue: An ingestion problem that the user can typically resolve themselves through configuration changes (for example, updating credentials).
  • Non-actionable issue: An ingestion problem that likely requires assistance from Google support to resolve (for example, an internal system error).
  • Parser: A component that normalizes raw log data into the Unified Data Model (UDM) structure.

Before you begin

  • Permissions: Ensure you have the necessary IAM roles and permissions to access the Google SecOps instance, view the Health Hub, and configure alerts in Cloud Monitoring.
  • Environment check: This guide assumes you are operating within the Google SecOps environment. Feature availability may vary by region.

Monitor and troubleshoot data ingestion

This section details how to monitor data ingestion.

Monitor using Health Hub dashboard

  1. In the Google SecOps side navigation menu, click Health Hub.
  2. Review the "Big Number" widgets for Failed Sources and Failed Parsers to identify components requiring immediate attention.
  3. Inspect the Health Status by Data Source table. Check the Latest Issue Details column for error descriptions, such as, "Config credential issue" or "Normalization issue".
  4. Click the Edit Data Source or Edit Parser links provided in the table to navigate directly to the respective configuration pages for remediation.
  5. Check timestamps, such as Last Event Time and Last Ingested, to verify data flow.

Set Up automated ingestion alerts

  1. In the Health Hub dashboard, click the Set Up Alerts link, which directs you to the Cloud Monitoring interface.
  2. Create an alerting policy:
    • Select Metrics: Choose metrics under Chronicle Collector > Ingestion, such as Total ingested log count or Total ingested log size.
    • Add filters for collector\_id or log\_type to narrow the alert scope to specific sources.
  3. To detect silent forwarders, select Metric absence as the condition type. Configure it to trigger an alert if logs stop flowing for a specified duration (for example, 60 minutes).

Investigate common issues

  • Pipeline latency investigation: A significant delay between the Last Event Time and Last Ingested timestamp indicates potential latency. Health Hub exposes the 95th percentile of this delta. High values suggest pipeline latency, while normal values might mean the source is sending historical data.
  • Ingestion surges or drops: The system uses z-score standardization to flag anomalies. A drop is flagged if both daily and weekly standardized differences are less than -1.645.
  • Parsing failures: An alert is triggered if the proportion of parser errors relative to total ingested events increases by 5 percentage points or more compared to the previous day. Investigate the parser configuration using the link in Health Hub.

Troubleshooting

Latency, service quota, and limits

  • Data Refresh: Information on the Health Hub and Data Ingestion dashboards refreshes approximately every 15 minutes.
  • Pre-GA Offerings: Some Health Hub features may be under Pre-GA Offerings Terms, potentially with limited support.
  • Regional Availability: Features described may not be available to all customers in all regions.

Error remediation

Error code / Symptom Issue description Fix
Forbidden 403 Actionable Auth Error: Permission denied on the data source. Click Edit Data Source from Health Hub and update the service account credentials or permissions in the source system.
Internal_error Non-Actionable System Error. Open a support case with Google SecOps for backend investigation, providing details from Health Hub.
Normalization Issue Parsing Failures: Increased rate of errors in a specific parser. Includes normalization rate changes as well. Click Edit Parser from Health Hub. Review the parser code for errors, test with sample logs, and correct the configuration.

Validation and testing

  • After applying a fix, monitor the Health Hub dashboard for the specific data source or parser.
  • Confirm that the Last Ingested and Last Event Time timestamps update as expected, indicating successful data flow and normalization.
  • Check Cloud Monitoring to ensure any triggered alerts associated with the issue have been resolved.

Need more help? Get answers from Community members and Google SecOps professionals.