Understand quotas and burst limits

This document describes quotas and burst limits in Google Security Operations.

Definition of burst limits

Burst limits are a form of service limits in Google SecOps that act as a speed limit for data ingestion, designed to protect the platform's shared infrastructure from sudden, massive spikes in traffic. A burst limit restricts the rate of ingestion (measured in megabytes per second—MBps, or gigabytes per second—GBps) within a rolling five-minute window.

Note: For the definition of service and other terms, see SecOps Service Specific Terms.

How burst limits are calculated

Google SecOps assigns burst limits to your Google SecOps tenants based on your purchased annual ingestion volume (purchased capacity) according to your Google SecOps license.

To accommodate expected variations and unplanned spikes in log traffic, your burst limit lets you ingest a daily volume of up to three times (3×) your expected daily average (that is, your purchased capacity divided by 365 days). This volume allowance can absorb standard ingestion spikes without disruption. For example, if your purchased capacity is 365 TB, your burst limit is set to allow data ingestion of up to 3 TB per day (that is, 365 TB/365 days ×3), and will fall within the range of 12 MBps - 36 MBps. Any further increase in your burst limit requires an increase in your purchased capacity.

Burst limits are enforced per Google SecOps customer tenant.

The following table shows how burst limits correspond to different amounts of purchased capacity:

Example purchased capacity	Burst limit range	5-minute burst limit	Ingestion at max burst limit (hourly)	Ingestion at max burst limit (daily)	Ingestion at max burst limit (annually)
100 TB	3 - 10 MBps	0.9 - 3 GB	~34 GB	~822 GB	300 TB
500 TB	16 - 48 MBps	4.8 - 14.4 GB	~171 GB	~4 TB	1.5 PB
1 PB	32 - 97 MBps	9.6 - 29 GB	~343 GB	~8 TB	3 PB
5 PB	158 - 476 MBps	47.4 - 143 GB	~1.7 TB	~41 TB	15 PB
30 PB	0.96 - 2.86 GBps	288 - 858 GB	~10.3 TB	~247 TB	90 PB

Note: A daily volume allowance is still subject to physical throughput constraints in the region. Although Google SecOps accommodates daily volume bursts up to the 3× limit, instantaneous data transfer rates (throughput) are actively managed to provide high availability and prevent resource starvation across the shared regional environment.

Ingestion traffic that includes extreme, sudden velocity spikes may be subject to dynamic rate limiting or temporary throttling to protect regional stability.

During these periods, data may experience ingestion delays until the spike subsides.

For ultra-high throughput requirements, see Custom capacity planning for ultra-high throughput.

Applicability of burst limits for pull-based feeds

Google SecOps also limits pull-based ingestion to one-third (33%) of the overall burst limit per logtype (across all feeds). This limit is in place to ensure that pull-based ingestion (usually from cloud sources) doesn't exhaust your tenant's overall burst limits and starve data ingestion using push-based methods (such as using Bindplane agents, forwarders, or direct ingestion to Google SecOps APIs).

Caution: The forwarder is deprecated. The forwarder component is being phased out of the Google SecOps platform and will reach end-of-life (EOL) in January 2027. Apr 1, 2026: New Google SecOps customers cannot use the forwarder component. Jan 1, 2027: The forwarder is officially EOL. No further patches, including security patches, will be released. Apr 1, 2027: Data is no longer allowed to be ingested from the forwarder component. Google recommends using Bindplane for Google SecOps as the primary solution for log collection. For more information, see Use Bindplane with Google SecOps.

Pull-based ingestion methods

Pull-based methods include ingestion methods (referred to as source types in Google SecOps) where Google SecOps actively reaches out to the source API to fetch data. This includes the following source types that are supported on Google SecOps:

• Third-party APIs
• Azure Event Hub
• Direct ingestion from Google Workspace and Google Cloud
• Cloud Storage
• Cloud Storage feed (event-driven)
• Amazon S3
• Amazon SQS
• Azure Blobstore
• SFTP request
• HTTP request

For example, if the burst limit for your tenant is set to 150 MBps, and your tenant is ingesting Okta user context logs using a third-party API connector (which is, a pull-based ingestion method), the system limits the ingestion rate of all Okta feeds combined to a maximum of [150/3 =] 50 MBps. This additional limit is applied even if your overall data ingestion rate is within your assigned burst limit.

Exceptions to logtype-level limits for pull-based ingestion methods

Although logtype-level limits generally apply to pull-based feeds, the following exceptions apply:

• HTTPS webhooks: this is a push-based method with logtype-level limits.
• Azure Event Hub: this is a pull-based method without logtype-level limits.

How burst limits are implemented

The system enforces burst limits in five-minute intervals. For example, if your burst limit is set to 50 MBps, you can ingest up to 15 GB every five minutes. If you ingest all 15 GB in the first two minutes, ingestion is blocked for the remaining three minutes of that window. This limit resets automatically at the start of the next five-minute interval.

Logtype-level limits are enforced in the same manner, but apply at the level of individual logtypes. For example, if you're allocated 5 GB for pull-based feeds every five minutes, and your total ingested data volume for any single logtype exceeds 5 GB in the first two minutes, ingestion is paused for the remaining three minutes of that window. The limit resets automatically at the start of the next five-minute interval.

What happens to your data if your exceed your burst limits

If you exceed your burst limit, Google SecOps pauses ingestion of additional data, and the following mechanisms are triggered—depending on whether your data is getting ingested using pull-based or push-based methods:

• Using pull-based methods: ingestion is buffered automatically and requires no additional configuration by you, the customer. The data remains stored in buffer storage until the limit resets and Google SecOps resumes data ingestion.
• Using push-based methods: Google SecOps temporarily rejects data ingestion with an HTTP 429 "Too Many Requests" error. This signals your ingestion mechanism to pause, buffer, and retry, ensuring no data is lost.

Using push-based ingestion methods, the responsibility to buffer and retry lies with you, the customer (see Customer responsibilities for data buffering and retry).

Burst limit rejections are not data loss

It's important to understand that burst limit rejections (HTTP 429) are not data-loss events. A burst limit rejection (HTTP 429 error) is a pause in data ingestion.

By ensuring your push-based systems have adequate disk buffering and retry logic, hitting a burst limit results only in a small delay (ingestion lag), never the permanent loss of security telemetry.

Data loss only occurs if the sending system (for example, the Bindplane agent, the forwarder, or the script) ignores the burst limit rejection error and deletes the log entry instead of storing it for a retry.

Customer responsibilities for data buffering and retry

Although Google SecOps automatically manages data buffering and retries for data being ingested using pull-based ingestion methods, you are responsible for data buffering and retrying data ingestion using push-based ingestion methods (such as HTTPS webhooks, Bindplane, forwarders, or Cribl).

You need to configure your systems to automatically buffer and resend data when your burst limit is reached to handle data overflow efficiently.

The following table highlights the key differences in how Google SecOps handles data ingestion when your burst limit is reached for both types of ingestion methods:

Feature	Pull-based ingestion	Push-based ingestion
How it works	Google SecOps actively reaches out to the source API to fetch data.	Your systems initiate the connection and send data to Google.
Responsibility for data buffering and retry	Google SecOps automatically manages buffering. When the burst limit is reached, Google SecOps pauses ingestion of additional data. The data remains stored in buffer storage until the limit resets and Google SecOps resumes fetching. Buffer storage only stores data for up to 90 days, after which the data is dropped.	The customer must manage buffering. When Google SecOps replies with HTTP 429, your sending system must catch this error, save the data to a local queue (disk or memory), and retry sending it later. If your sender is set to "drop on failure," data will be lost.
Data source types	Third-party API, Azure Event Hub, direct ingestion from Google Workspace and Google Cloud, Cloud Storage, Cloud Storage feed (event driven), Amazon S3, Amazon SQS, Azure Blobstore, SFTP request, HTTP request.	Google SecOps forwarder, Bindplane agent, Pub/Sub, Amazon Kinesis Firehose, HTTPS webhook, direct to ingestion API.
User action	Take steps to align your data ingestion volume with your purchased capacity.	Additionally, ensure that your ingestion sources are configured for data retention, buffering, and retrying. For more information, see Example buffering and retry configurations for push-based systems.

When does buffered data for pull-based feeds get backfilled?

For feeds using pull-based ingestion methods, when your burst limit window resets, Google SecOps backfills buffered data, prioritizing live data over buffered data. This mechanism ensures that your backlog of buffered data doesn't interfere with your incoming live data traffic (which can compound detection delays).

Note: During the backfilling of buffered data, you might observe your buffered data flowing in parallel with your live data.

How to view your assigned burst limit

To ascertain the burst limit assigned to your Google SecOps tenant, do the following:

1. In your Google SecOps console, go to Dashboards > Data Ingestion and Health.
2. View the Burst Limit Graph - Quota Limit. The graph displays your assigned limit (the flat line) relative to your actual ingestion rate.

Track if you are approaching or exceeding your burst limit

You can track utilization using the built-in dashboards or using Cloud Monitoring.

Use Google SecOps dashboards to track if you're approaching or exceeding your burst limit

• Go to Dashboards > Data Ingestion and Health and view the following:

  • Ingestion rate graph: shows your current throughput.
  • Burst rejection graph: shows the volume of logs rejected (HTTP 429 errors) due to exceeding the burst limit.

Use Cloud Monitoring to track if you're approaching or exceeding your burst limit

You can use the Metrics Explorer in Google Cloud to create custom alerts. We recommend creating an ingestion alert that notifies you when the volume of the bytes ingested exceeds the burst limit threshold.

Relevant metrics include the following:

• Ingested volume: chronicle.googleapis.com/ingestion/log/bytes_count
• Rejected volume: chronicle.googleapis.com/ingestion/log/quota_rejected_bytes_count

The following sections contain example PromQL queries for monitoring and alerting.

View your burst limit usage

• To view your burst limit usage, use the following PromQL query:

  100 * sum(rate(chronicle_googleapis_com:ingestion_log_bytes_count{monitored_resource="chronicle.googleapis.com/Collector"}[10m]))/min(min_over_time(chronicle_googleapis_com:ingestion_quota_limit{monitored_resource="chronicle.googleapis.com/Collector"}[10m]))

View the number of bytes rejected after exceeding the burst limit

• To view the number of bytes rejected after exceeding the burst limit, use the following PromQL query:

  topk(5, sum by ("collector_id","log_type")(rate({"__name__"="chronicle.googleapis.com/ingestion/log/quota_rejected_bytes_count","monitored_resource"="chronicle.googleapis.com/Collector","quota_type"="SHORT_TERM_DATA_RATE"}[${__interval}])))

Trigger an alert when you reach 70% of your burst limit

• To trigger an alert when you reach 70% of your burst limit, use the following PromQL query:

  100 * topk(5, sum by ("collector_id","log_type")(rate({"__name__"="chronicle.googleapis.com/ingestion/log/quota_rejected_bytes_count","monitored_resource"="chronicle.googleapis.com/Collector","quota_type"="SHORT_TERM_DATA_RATE"}[${__interval}]))) > 70

Note: The exact query syntax may vary based on your specific monitored resource configuration.

For more information about setting up ingestion alerts, see Using Cloud Monitoring for ingestion for ingestion insights.

Handle burst limit rejections caused by push-based methods

If you encounter rejection errors (HTTP 429) due to hitting your burst limit for incoming data using push-based methods, we recommend that you take the following steps:

• Verify buffering: make sure that your ingestion sources are buffering data and retrying.
• Optimize ingestion: review ingestion scripts and make sure they aren't sending unnecessary data or storming the API with massive batches all at once. Spread out historical data uploads if possible. Filter out redundant data using the data processing pipeline feature.
• Wait: for temporary spikes, it's often sufficient to wait for the five-minute window to reset and then try again.

Example buffering and retry configurations for push-based systems

The example configurations in this section are intended as useful starting points.

Make sure that your system configurations are independently assessed and tested from time to time by relevant experts to prevent the possibility of data loss in the event you reach your burst limit.

Bindplane (OpenTelemetry)

Bindplane agents push data. You must configure a persistent queue (file storage extension) in the collector configuration. Verify that the retry_on_failure policy is enabled in the exporter settings, with appropriate backoff intervals.

For more information, see the Bindplane documentation and Use Bindplane with Google SecOps.

Azure Event Hub (push scenario)

Note: If you use the standard Google SecOps feed for Azure Event Hub, it is pull-based.

If you write a custom function to push data from Azure Event Hub to the Google SecOps Ingestion API, your function must implement exponential backoff logic. If it receives a 429 error, it should wait and retry the batch, rather than discarding it.

HTTPS push ingestion using Amazon Kinesis, Pub/Sub, or webhooks

Make sure that the retention time is set to the maximum possible value. For example, to set the retention time for Pub/Sub, see Configure subscription message retention.

Google SecOps forwarder

If you use Google SecOps forwarder, use disk buffers to buffer data when you exceed your burst limit.

By default, the forwarder uses a small memory buffer (1 GB). If limits are hit, memory fills up and data drops. Configure disk_buffer_configuration to spill over to disk (for example, 100 GB) to survive longer pauses.

The maximum RAM size used by the collector is 4 GB. You can set this limit using the max_file_buffer_bytes setting in the collector configuration.

To buffer data more than 4 GB, use disk buffers. To decide on the disk buffer size, identify the rate at which forwarders are ingesting by using the following MQL query:

sum(rate(chronicle_googleapis_com:ingestion_log_bytes_count
{monitored_resource="chronicle.googleapis.com/Collector", collector_id!~ "
(COLLECTOR_ID
|COLLECTOR_ID
|COLLECTOR_ID)"}[5m]))

Replace COLLECTOR_ID with each of your collector IDs, as appropriate.

For example, if the rate of ingestion from the forwarder is 415 Kbps and the buffer compression efficiency is 70%:

• The buffer fill-up rate is calculated as 415 Kbps × (100% - 70%) = 124.5 Kbps.
• At this rate, a buffer size of 1 GB, which is the default in-memory buffer value, fills up in 2 hours and 20 minutes (1024 × 1024 / 124.5 = 8422.297 seconds).

If you have exceeded your burst limit, you need a 100-GB disk to buffer data for a day.

Custom capacity planning for ultra-high throughput

Notwithstanding anything described in the other sections of this document, data ingestion throughput that exceeds 3 GBps is considered as ultra-high throughput. If you are planning large-scale data migrations, anticipating sustained ultra-high throughput, or running architectures that consistently generate massive ingestion bursts, you must engage your account team for custom capacity provisioning.

Because dedicated regional capacity expansion can take several weeks to deploy, please notify Google Cloud support at least 90 days in advance of anticipated extreme ingestion events to make sure your throughput requirements can be addressed.

Frequently asked questions

The following sections provide answers to frequently asked questions.

Can I increase my burst limit?

If you expect your data ingestion volume to permanently increase, you can increase your purchased capacity by contacting your Google SecOps sales representative.

Can I increase the logtype-level limits for pull-based feeds?

You can increase your logtype-level limits for a specific logtype by raising a request using Google SecOps technical support in advance.

Increasing the logtype-level limit for one logtype doesn't change the limit applied to other logtypes or your overall burst limit.

Is it possible to track my data backlog?

Not at this time.

What are possible ways to clear my data backlog?

If you've accumulated a very large data backlog, and want to clear that backlog to free up your burst limit quota, you can do following:

• Purchase additional capacity to increase your limits.
• Disable specific feeds that have unexpectedly spiked in volume.

• Request Google SecOps technical support to drop your backlog.

  To drop your backlog, your data feed is temporarily disabled until all retry requests for backfilled data are successfully processed. During this time, you'll not be able to ingest any new data.

  Once your backlog is cleared, your feed is re-enabled, upon which you'll see new data flowing in. Depending on the size of your backlog, this can take anywhere from a few minutes to a few hours.

Do burst limits also apply to data ingestion into the data processing pipeline?

The ingestion rate limits applicable to data feeds sending raw logs data into the data processing pipeline of Google SecOps are set to be higher than your tenant's burst limit.

If you exceed your burst limit, the data processing pipeline stops accepting additional requests, according to the following:

• Using pull-based methods: ingestion is buffered automatically and requires no additional configuration.
• Using push-based methods: Google SecOps temporarily rejects the data with an HTTP 429 "Too Many Requests" error.

Any data that has been transformed after the burst limit is triggered is temporarily buffered in an internal queue until the limit is reset in the subsequent five-minute window.