Search best practices
This document describes Google's recommended best practices for using the Search feature in Google Security Operations. Searches can require substantial computational resources if they're not carefully constructed. Performance also varies depending on the size and complexity of the data in your Google SecOps instance.
Use specific filters in queries for maximum speed and performance
The single most effective way to improve search performance is to build queries using specific, optimized Unified Data Model (UDM) fields. These fields are optimized for fast retrieval, ensuring that your searches run quickly and use fewer computational resources.
The following sections list the high-performance UDM fields to use as filters in your queries.
Metadata fields
metadata.event_timestamp.secondsmetadata.event_typemetadata.log_typemetadata.product_event_typemetadata.product_namemetadata.vendor_name
Principal fields
principal.applicationprincipal.asset_idprincipal.file.md5principal.file.sha1principal.file.sha256principal.hostnameprincipal.ipprincipal.macprincipal.process.file.md5principal.process.file.sha1principal.process.file.sha256principal.process.pidprincipal.user.email_addressesprincipal.user.userid
Source fields
src.asset_idsrc.hostnamesrc.ipsrc.macsrc.user.userid
Target fields
target.applicationtarget.asset_idtarget.file.md5target.file.sha1target.file.sha256target.hostnametarget.iptarget.mactarget.process.file.md5target.process.file.sha1target.process.file.sha256target.user.email_addressestarget.user.employee_idtarget.user.useridtarget.user.windows_sid
Network fields
network.dhcp.chaddrnetwork.dhcp.ciaddrnetwork.dhcp.client_hostnamenetwork.dhcp.yiaddrnetwork.dns.answers.classnetwork.dns.answers.namenetwork.dns.answers.typenetwork.dns.questions.classnetwork.dns.questions.namenetwork.dns.questions.typenetwork.email.fromnetwork.email.tonetwork.http.method
Security result fields
security_result.actionsecurity_result.categorysecurity_result.rule_name
How to query entity and context data
If you search for entity or context log types (such as AZURE_AD_CONTEXT or WORKSPACE_USERS) using metadata.log_type = "<LOG_TYPE>", the search returns no results even if raw logs are visible in Raw Log Search. This is because UDM search only queries UDM event records.
To search for entity and context data, use the graph syntax:
To search for entity or context log type, query the
graphmetadata fields. For example:graph.metadata.event_metadata.log_type = "<LOG_TYPE>"To search for entity fields, use the prefix
graph.entity.noun.field. For example:graph.entity.user.email_addresses = "foo_email"
For more information, see Conduct a search for entity context data.
Construct effective search queries for performance
Writing optimized queries is key to maximize speed and minimize resource consumption across your security data. All query conditions must strictly adhere to this fundamental structure:
udm-field operator value
For example:
principal.hostname = "win-server"
Narrow the time range for your search
Because Google SecOps can ingest a large amount of data during a search, minimizing the time range and narrowing the scope of your query can improve search performance.
Use regular expressions in search query
You can use standard logical and comparison operators when constructing your UDM search queries to build complex expressions:
- Logical operators: Use
AND,OR, andNOTto combine conditions.ANDis assumed if you omit an operator between two conditions. - Operator precedence: Use parentheses () to override the default order of precedence. There is a maximum limit
of 169 logical operators (
OR,AND,NOT) that you can use within parentheses. - Comparison operators: Depending on the UDM field type (string, integer, timestamp), field operators can include:
=,!=,>=,>,<,<=
Google SecOps uses the RE2 regular expression engine.
Alternatively, for efficient searching of a large set of values, you can use the reference lists.
Use nocase as a search modifier
You can append the nocase modifier to a string comparison condition to make the search case-insensitive, which ignores capitalization.
For example, the following search ignores the case and matches all combinations that contain tim.smith regardless of the case:
target.user.userid = "TIM.SMITH" nocase
Avoid using regular expressions in enumerated fields
You can't use regular expressions when searching enumerated fields (fields with a range of
predefined values) like metadata.event_type or
network.ip_protocol
The following example is an invalid search:
metadata.event_type = /NETWORK_*/
Whereas, the following example is a valid search:
(metadata.event_type = "NETWORK_CONNECTION" or metadata.event_type = "NETWORK_DHCP")
Use any and all operators in the Events field
In Search, some UDM fields (such as the principal.ip, target.file.md5,
or security_result fields) are labeled as repeated because they hold a
list of values or message types within a single event. By default, unmodified
expressions on repeated fields use the any operator. To avoid unexpected match
behavior when evaluating a repeated field, explicitly use the any or all
modifiers.
When the any operator is used (or when a repeated field expression is
unmodified), the predicate evaluates to true if any value in the repeated
field satisfies the condition. Each element in the repeated field is treated
individually, which can cause unexpected results when you search for empty
values or use the != operator:
- If an event includes both
principal.ip = "192.0.2.1"andprincipal.ip = "192.0.2.2", a search forprincipal.ip != "192.0.2.1"matches the event because index1("192.0.2.2") satisfies the condition. - If an event has multiple
security_resultelements and only one element populates therule_namefield, the event matches bothsecurity_result.rule_name = ""andsecurity_result.rule_name != "".
To check that every element in a repeated field satisfies the condition, use the
all operator (for example, all principal.ip != "192.0.2.1" or
all security_result.rule_name = ""). For more information, see
Repeated fields.
Use Unix epoch time for timestamps or use YARA-L functions for date conversion
Timestamp fields are matched using Unix epoch time (the total number of seconds that have passed since Thursday, 1 January 1970 00:00:00 UTC), or you can use YARA-L functions for date conversion.
When searching for a specific timestamp, the following (in epoch time) is valid:
metadata.ingested_timestamp.seconds = 1660784400
The following timestamp is invalid:
metadata.ingested_timestamp = "2022-08-18T01:00:00Z"
When searching for a specific timestamp, the following (using a YARA-L function for date conversion) is valid:
metadata.event_type = "NETWORK_CONNECTION"
timestamp.get_date(metadata.ingested_timestamp.seconds) = "2026-03-15"
The following YARA-L example uses timestamp functions to check for ingestion time ranges:
metadata.event_type = "NETWORK_CONNECTION"
$date = timestamp.get_date(metadata.ingested_timestamp.seconds)
$date > "2026-03-15" AND $date < "2026-03-17"
How to query newly ingested data with older timestamps
You can't search newly ingested events that have an older timestamp unless you specify the event time range to query. This is because the time range is based on the timestamp of the parsed event, as opposed to the ingestion timestamp of the raw log event.
To search UDM events for ingested logs with older event timestamps, you can use All time option to search and query over metadata.ingested_timestamp.
Excluded fields from filters
The following fields are intentionally excluded from search filters:
metadata.idmetadata.product_log_id*.timestamp
Although these fields contain important metadata, their unique values introduce high cardinality and variance in the statistics, which negatively impacts search performance.
Troubleshooting
If you receive a generic error message, such as "Error: Search has encountered an error and couldn't load data", take the following steps to resolve the issue. If the error persists, contact Customer Support.
- Connect to Google SecOps from a different network. For example, use a cloud VM to help identify any network issues.
- Make sure Chronicle API calls are allowed by your organization's firewall or proxy configuration or policy.
- Verify that no data limit is configured for Search API calls, as searches can return large datasets.
- Search queries can run asynchronously and might require more time to return data. You can view previously run searches in the search history and select them to view the results later.
Need more help? Get answers from Community members and Google SecOps professionals.