UDM search compared with Rule results

Supported in:

When you investigate security telemetry in Google Security Operations, you might notice differences between the event counts returned by a Unified Data Model (UDM) search query and the detections generated by a live YARA-L 2.0 detection rule.

Although search queries and detection rules both use UDM and share core YARA-L 2.0 syntax, they run on different execution engines, evaluate data across distinct time windows, and handle repeated fields and deduplication differently.

The following sections explain why search results differ from detection rule alerts and how to reconcile result counts during investigations.

Summary of key differences

The following table compares the behavior of UDM search queries with live YARA-L 2.0 detection rules:

Attribute UDM search and statistical queries YARA-L 2.0 detection rules
Execution model Ad-hoc query against historical indexed logs at search time. Continuous evaluation against streaming ingested data or scheduled replays.
Supported sections Filter statements, optional match, outcome, optional dedup, order, and limit. Doesn't use events, condition, or option sections. meta, events, optional match, outcome, condition (required), and optional option.
Lookback period Scans historical indexed data up to 90 days in the past per search execution. Evaluates events continuously as they ingest. Multi-event rules evaluate within the configured match window (up to 24 hours for late-arriving events unless run using Retrohunt).
Result row limits Returns up to 10,000 rows per statistical search execution. No hard cap on generated detections over time, but each detection samples up to 10 underlying events per match variable in the user interface.
Repeated fields (any) Repeated fields implicitly use the any evaluation behavior. A condition matches if any element in the array satisfies the expression. Repeated fields unnest into individual rows. Multi-value repeated fields require explicitly specifying the any operator to match any item.
Windowed aggregations Functions such as avg(), stddev(), max(), and sum() don't require the window. prefix in the outcome section. Statistical and ordering functions in outcome require the window. prefix (for example, window.avg()) to bind to the match window.
Time granularity (by** versus over)** Supports tumbling group-by time granularity (by <duration> or over every <duration>, such as by 1h). Doesn't support sliding window correlation (over with before or after). Supports hop (over <duration>), tumbling (by <duration>), and sliding (over <duration> before/after $pivot) correlation windows.
Deduplication behavior Returns all matching events or grouped statistical rows unless an explicit dedup section is defined. Automatically deduplicates detections across adjacent windows with identical match variables to prevent alert storms.
Parser and schema version Evaluates data using the parser normalization and schema representation active when the search runs. Operates on the event representation generated by the parser version active when the event ingested.

Repeated fields and unnesting behavior

One of the most common reasons UDM search returns more matching events than a YARA-L detection rule is how repeated fields (arrays such as principal.ip, target.file.md5, and security_result.action) are evaluated. For syntax details, see Repeated fields.

  • UDM search: Repeated fields implicitly use any semantics. If you filter for principal.ip = "10.0.0.1", the search returns any event where "10.0.0.1" appears anywhere inside the principal.ip array.
  • YARA-L detection rules: Repeated fields unnest into separate evaluation rows. When you write a detection rule condition without explicit any quantifier syntax, equality checks against multi-value repeated fields can behave differently than in search.

Example event condition with explicit any operator

To make sure your detection rule matches the same repeated-field conditions as UDM search, use the explicit any operator in your event criteria:

events:
  // Match if any IP in the repeated array equals the target IP
  any $e.principal.ip = "10.0.0.1"

Windowed aggregations and syntax differences

When you migrate a YARA-L 2.0 statistical query from search to a detection rule, adjust the syntax in your outcome and match sections:

  • The window. prefix: In UDM search, calling statistical functions like $avg_bytes = avg(network.sent_bytes) works directly. In a YARA-L detection rule, statistical functions (avg, stddev, percentile, and variance) and ordering functions (first and last) require the window. prefix (for example, window.avg($e.network.sent_bytes)) to bind the calculation to the rule's match window. Basic aggregate functions (max, min, sum, and count) don't require the prefix. Learn more in Outcome section syntax.
  • Event variables: Detection rules require assigning events to an event variable (such as $e.metadata.event_type = "NETWORK_CONNECTION"), whereas UDM search references UDM fields directly without variable prefixes (metadata.event_type = "NETWORK_CONNECTION").
  • Sliding event windows: Detection rules support complex sliding correlation windows ($e1.metadata.event_timestamp.seconds > $e2.metadata.event_timestamp.seconds), whereas UDM search groups datasets by fixed time intervals (by 1h or over every 1d). Learn more in Windowing logic.

Deduplication and alert suppression

A UDM search reports every matching event or aggregated bucket within your selected time range. In contrast, the live detection engine applies automatic deduplication and sampling:

  • Identical match variables: If a detection rule triggers multiple times within adjacent time windows for the same match variable combination (for example, the same $hostname and $user), the rule engine suppresses duplicate alerts to reduce analyst fatigue. For more information, see Use deduplication in search and dashboards.
  • Test rule versus live streaming engine: Running a manual test rule execution in the rules editor uses less restrictive deduplication than the live streaming engine. Testing a rule over historical data might show a higher detection count than the live alerts generated when those events ingested.

Lookback windows and ingestion delays

  • Ingestion delay: UDM search statistical aggregations run over indexed log data. Very recently ingested events might experience a brief indexing delay before appearing in aggregate search queries.
  • Historical ingestion windows: Multi-event detection rules evaluate streaming data over rolling windows. If events ingest out of order or arrive more than 24 hours after their occurrence timestamp, live multi-event rules might not generate detections. To scan older historical data against a new or updated rule, run a retrohunt job. For instructions, see Run a rule against historical data.
  • Parser changes over time: If a parser is updated or corrected, historical scans in UDM search immediately reflect the updated normalization. Live rule detections reflect the exact parsed state of the log when ingestion occurred.

Reconciling counts during investigations

If you observe differences between a UDM search query and a detection rule, follow these steps to reconcile the results:

  1. Check for repeated fields: Verify whether your event criteria filter on repeated fields and make sure that your rule explicitly uses any where appropriate.
  2. Align time windows: Make sure your search query time range matches the start and end timestamps of the rule detection period.
  3. Account for deduplication: Remember that search counts represent raw event volumes or unsuppressed groupings, whereas detection counts reflect deduped alerts on unique match keys.
  4. Verify parser versions: If old events show different UDM field values in search than in an old alert, check whether parser updates occurred since the original event was ingested.

Need more help? Get answers from Community members and Google SecOps professionals.