Buffering and retry configurations for push-based systems

Supported in:

Google secops SIEM

The section contains example buffering and retry configurations for push-based systems. You can use the example configurations as starting points for your own system.

Google SecOps assigns burst limits to your Google SecOps tenants based on your purchased annual ingestion volume (purchased capacity) according to your Google SecOps license.

Important: You are solely responsible for independently configuring and testing your buffering and retry settings. Google is not liable for data loss resulting from burst limits due to improper system configuration. Google strongly recommends regular validation by your technical experts.

Bindplane (OpenTelemetry)

Bindplane agents push data. You must configure a persistent queue (file storage extension) in the collector configuration. Verify that the retry_on_failure policy is enabled in the exporter settings, with appropriate backoff intervals.

For more information, see the Bindplane documentation and Use Bindplane with Google SecOps.

Azure Event Hub (push scenario)

Note: If you use the standard Google SecOps feed for Azure Event Hub, it is pull-based.

If you write a custom function to push data from Azure Event Hub to the Google SecOps Ingestion API, your function must implement exponential backoff logic. If it receives a 429 error, it should wait and retry the batch, rather than discarding it.

HTTPS push ingestion using Amazon Kinesis, Pub/Sub, or webhooks

Make sure that the retention time is set to the maximum possible value. For example, to set the retention time for Pub/Sub, see Configure subscription message retention.

Google SecOps forwarder

If you use Google SecOps forwarder, use disk buffers to buffer data when you exceed your burst limit.

Caution: The forwarder is deprecated. The forwarder component is being phased out of the Google SecOps platform and will reach end-of-life (EOL) in January 2027. Apr 1, 2026: New Google SecOps customers cannot use the forwarder component. Jan 1, 2027: The forwarder is officially EOL. No further patches, including security patches, will be released. Apr 1, 2027: Data is no longer allowed to be ingested from the forwarder component. Google recommends using Bindplane for Google SecOps as the primary solution for log collection. For more information, see Use Bindplane with Google SecOps.

By default, the forwarder uses a small memory buffer (1 GB). If limits are hit, memory fills up and data drops. Configure disk_buffer_configuration to spill over to disk (for example, 100 GB) to survive longer pauses.

The maximum RAM size used by the collector is 4 GB. You can set this limit using the max_file_buffer_bytes setting in the collector configuration.

To buffer data more than 4 GB, use disk buffers. To decide on the disk buffer size, identify the rate at which forwarders are ingesting by using the following MQL query:

sum(rate(chronicle_googleapis_com:ingestion_log_bytes_count
{monitored_resource="chronicle.googleapis.com/Collector", collector_id!~ "
(COLLECTOR_ID
|COLLECTOR_ID
|COLLECTOR_ID)"}[5m]))

Replace COLLECTOR_ID with each of your collector IDs, as appropriate.

For example, if the rate of ingestion from the forwarder is 415 Kbps and the buffer compression efficiency is 70%:

• The buffer fill-up rate is calculated as 415 Kbps × (100% - 70%) = 124.5 Kbps.
• At this rate, a buffer size of 1 GB, which is the default in-memory buffer value, fills up in 2 hours and 20 minutes (1024 × 1024 / 124.5 = 8422.297 seconds).

If you have exceeded your burst limit, you need a 100-GB disk to buffer data for a day.

What's next

For information about quotas and burst limits, see Understand quotas and burst limits.

Need more help? Get answers from Community members and Google SecOps professionals.