Google SecOps data ingestion

Supported in:

Google secops SIEM

Google Security Operations ingests customer logs, normalizes the data, and detects security alerts. It provides self-service features for data ingestion, threat detection, alerts, and case management. Google SecOps can also receive alerts from other SIEM systems and analyze them.

Note: Large files (5-10 GB or larger) can significantly delay data ingestion.

Google SecOps log ingestion

The Google SecOps ingestion service acts as a gateway for all data.

Google SecOps ingests data using the following systems:

• Google Cloud: Google SecOps retrieves data directly from your Google Cloud organization, which is the primary method for all standard Google Cloud logs (for example, Audit, VPC Flow, DNS, and Firewall). It is the most cost-effective and performant way to bring in Google Cloud telemetry to Google SecOps. For more information, see Ingest Google Cloud data to Google SecOps.

• Bindplane agent: This is a managed agent for collecting logs from on-premises environments and servers (Windows or Linux). Bindplane is a telemetry pipeline that can collect, refine, and export logs from any source into Google SecOps, and therefore provides flexibility in collecting different types of logs that don't work with other methods. You can use it for on-premises data, such as Firewall logs, Windows and Linux logs, or for cloud data you want to preprocess (for example, refine or filter) before ingesting into Google SecOps. You can also manage this agent by using the Bindplane OP Management console. For more information, see Use the Bindplane agent.

• Data feeds: Data feeds are used primarily for cloud-based logs where the third-party logs are already aggregated into an object store, such as Cloud Storage or Amazon S3, or when the third party supports 'push'-based methods, such as webhook. Data feeds also provide out-of-the-box support for a predefined set of API based integrations. Use Data feeds for cloud-based logs such as EDRs or any SaaS application and for the specific integrations predefined as Direct API. The data feeds send logs directly to the Google SecOps ingestion service. For more information, see the feed management documentation. Data feeds support log lines up to 4 MB in size.

• Ingestion APIs: Use the Ingestion API for custom, high-volume, or home-grown applications that don't fit into other methods. This method is slightly more complex to use than other ingestion methods. For more information, see the Ingestion API.

• Forwarders: The Forwarder is now end of life. Google recommends you use the Bindplane agent instead.

Parsers convert logs from customer systems into a Unified Data Model (UDM). Downstream systems within Google SecOps use the UDM to provide additional capabilities, including rules and UDM search.

Note: Ingestion supports reading parquet file format.

See Understand data availability for search for full details of the data ingestion lifecycle, including end-to-end data flow and latency, and how these factors impact the availability of recently ingested data for querying and analysis.

Note: To monitor the operational status and health of all of your data sources and parsers, go to the Health Hub. It includes information about anomalous sources and log types, and provides context to diagnose and remediate issues.

Note: Google SecOps recommends using UTC or ISO 8601 formats for log timestamps.

Types of Google SecOps ingestion

Google SecOps can ingest both logs and alerts, but supports only single-event alerts. You can use UDM search to find both ingested and built-in Google SecOps alerts.

Google SecOps supports the following types of data ingestion:

Raw logs

Google SecOps ingests raw logs using forwarders, the ingestion API, data feeds, or directly from Google Cloud.

Use a single-line JSON payload for raw log ingestion. For example, { "firstName": "Alex", "lastName": "N", "age": 30, "isStudent": false, "address": { "streetAddress": "1800 Amphibious Blvd", "city": "Anytown", "state": "CA", "postalCode": "94045" }, "phoneNumbers": [ { "type": "home", "number": "800-555-0199" }, { "type": "mobile", "number": "800-554-0199" } ], "hobbies": ["reading", "hiking", "cooking"]}

If you submit a multi-line payload, the system interprets each line as a separate log entry.

Note: If your data source produces multi-line JSON payloads, use a custom script or a data transformation tool to process them into a single-line format. After processing, send the payloads to the ingestion API. Alternatively, use the split delimiter parameter in a webhook feed to split the log lines.

Alerts from other SIEM systems

Google SecOps can ingest alerts from other SIEM systems, EDRs, or ticketing systems, as follows:

1. Receive alerts using Google SecOps connectors or Google SecOps webhooks.
2. Ingest the events associated with each alert and create a corresponding detection.
3. Process both the ingested events and detections.

You can create detection engine rules to identify patterns in the ingested events and generate additional detections.

Note: Detection engine rules don't identify patterns in alerts ingested from Google SecOps.

Need more help? Get answers from Community members and Google SecOps professionals.