# Parser syntax reference

This document describes the functions, parsing patterns, and syntax supported in
data mapping instructions. For a conceptual overview of how Google Security Operations
parses raw logs to the Unified Data Model (UDM) format, see [Overview of parsing](https://docs.cloud.google.com/chronicle/docs/event-processing/parsing-overview).

Default parsers, customer-specific parsers, and 'code snippet' parser extensions
use code-like data mapping instructions to convert original raw log fields to
UDM format. The parser syntax is similar to Logstash, but not identical. For
more information, see the [Logstash documentation](https://github.com/elastic/logstash).

> [!IMPORTANT]
> **Important:** Ingestion parsers and parser extensions use Logstash Grok and configuration syntax with filters (for example, `grok`, `mutate`, `json`, `date`, and `base64`). They run at ingestion time to parse raw logs into the Unified Data Model (UDM) and can't use YARA-L syntax, outcome blocks, or generator functions (such as `yield`). YARA-L 2.0 is strictly a detection engine query language evaluated downstream on normalized UDM events for threat alerting and can't parse raw logs, mutate UDM fields, or drop incoming log traffic.

UDM logs categorize each record by UDM event type. Each event type has required
and optional fields. You can find the mandatory fields for each event type in:
[Required and optional fields for each event type](https://docs.cloud.google.com/chronicle/docs/unified-data-model/udm-usage#required_and_optional_fields).

To build an efficient and accurate parser, Google recommends that you follow a structured, phased approach---completing the following core steps and incorporating [best practices](https://docs.cloud.google.com/chronicle/docs/reference/parser-syntax#best-practices):

1. [**Initialize fields**](https://docs.cloud.google.com/chronicle/docs/reference/parser-syntax#initialize-fields): Set fields to empty (`""`) at the beginning of the parser to ensure clean conditional checks later before mapping.
2. [**Extract data**](https://docs.cloud.google.com/chronicle/docs/reference/parser-syntax#extract-data): Identify the log format and select the appropriate filter ([Grok](https://docs.cloud.google.com/chronicle/docs/reference/parser-syntax#extract-data-using-grok), [JSON](https://docs.cloud.google.com/chronicle/docs/reference/parser-syntax#extract-json-logs), [XML](https://docs.cloud.google.com/chronicle/docs/reference/parser-syntax#extract-xml-logs), [KV](https://docs.cloud.google.com/chronicle/docs/reference/parser-syntax#extract-key-value-logs), or [CSV](https://docs.cloud.google.com/chronicle/docs/reference/parser-syntax#extract-csv-logs)) to extract the raw data into fields. You might also need to know how to [loop over a JSON array using a `for` loop](https://docs.cloud.google.com/chronicle/docs/reference/parser-syntax#for_loop), [loop over key-value pairs of a JSON object using `map`](https://docs.cloud.google.com/chronicle/docs/reference/parser-syntax#map_keyword), and [handle hybrid log formats](https://docs.cloud.google.com/chronicle/docs/reference/parser-syntax#hybrid-log-formats).
3. [**Transform and map data to UDM**](https://docs.cloud.google.com/chronicle/docs/reference/parser-syntax#transform-and-map-data-to-udm): Plan how to map the extracted data into the [Unified Data Model (UDM)](https://docs.cloud.google.com/chronicle/docs/event-processing/udm-overview) schema, grouping related fields and following a strict hierarchical order.

## Initialize fields

Before extracting any data, start by initializing the fields you plan to use to empty values (`""`). This is important because during the mapping phase, you'll frequently write conditional statements to check if a piece of data exists before attempting to map it (for example, `if [source_ip] != ""{ ... }`). If a field isn't initialized and the extraction phase fails to populate it, referencing that missing field later can cause the parser to fail or produce unintended results. Initializing establishes a clean baseline.

## Extract data

After you have identified the log format and selected the appropriate filter, use the information in the relevant section ([Grok](https://docs.cloud.google.com/chronicle/docs/reference/parser-syntax#extract-data-using-grok), [JSON](https://docs.cloud.google.com/chronicle/docs/reference/parser-syntax#extract-json-logs), [XML](https://docs.cloud.google.com/chronicle/docs/reference/parser-syntax#extract-xml-logs), [KV](https://docs.cloud.google.com/chronicle/docs/reference/parser-syntax#extract-key-value-logs), [CSV](https://docs.cloud.google.com/chronicle/docs/reference/parser-syntax#extract-csv-logs)) to extract the raw data into UDM fields.

This section also includes information about how to [loop over a JSON array using a `for` loop](https://docs.cloud.google.com/chronicle/docs/reference/parser-syntax#for_loop), [loop over key-value pairs of a JSON object using `map`](https://docs.cloud.google.com/chronicle/docs/reference/parser-syntax#map_keyword), and [handle hybrid log formats](https://docs.cloud.google.com/chronicle/docs/reference/parser-syntax#hybrid-log-formats).

### Extract data using the Grok function

With Grok, you can use [predefined patterns](https://github.com/elastic/logstash/blob/v1.4.2/patterns/grok-patterns)
and regular expressions to match log messages, and extract values from the
log message into tokens. Grok data extraction requires that field labels are
defined as part of the data extraction process.

> [!IMPORTANT]
> **Important:** In customer-specific parsers and 'code snippet' parser extensions, don't use an `EMAILADDRESS` Grok pattern to extract email addresses that start with a numeral. As a workaround, you can use the `DATA` Grok pattern and then apply a regular expression validation at the mapping stage.

#### Syntax for predefined patterns in Grok

    %{pattern:token}

    %{IP:hostip}
    %{NUMBER:event_id}

#### Syntax for regular expressions in Grok

The following regular expression pattern examples can be used to extract values from log messages.

| Regex | Data |
|---|---|
| \\s | Space |
| \\S | Not space |
| \\d | Digit |
| \\D | Not digit |
| \\w | Word |
| \\W | Not word |

> [!NOTE]
> **Note:** Regular expression syntax for parsers uses two backslashes instead of the single backslash for typical regular expression pattern matches.

    (?P<token>regex_pattern)
    (?P<eventId>\\S+)

#### Sample log message and Grok patterns

Example original raw log

    Mar 15 11:08:06 hostdevice1: FW-112233: Accepted connection TCP 10.100.123.45/9988 to 8.8.8.8/53

Grok pattern to extract data from the log

    %{SYSLOGTIMESTAMP:when} %{DATA:deviceName}: FW-%{INT:messageid}: (?P<action>Accepted|Denied) connection %{WORD:protocol} %{IP:srcAddr}/%{INT:srcPort} to %{IP:dstAddr}/%{INT:dstPort}

The following table describes available tokens and values that can be used in Grok patterns.

| **Token** | **Value** |
|---|---|
| when | Mar 15 11:08:06 |
| deviceName | hostdevice1 |
| messageid | 112233 |
| action | Accepted |
| protocol | TCP |
| srcAddr | 10.100.123.45 |
| srcPort | 9988 |
| dstAddr | 8.8.8.8 |
| dstPort | 53 |

#### Grok extraction syntax

    grok {
     match => {
     "message" => "<grok pattern>"
     }
    }

#### Grok overwrite option

The overwrite option used with the Grok syntax lets you overwrite a field
that already exists. This feature can be used to replace a default value with
the value extracted by the Grok pattern.

    If [field] != "" {
      mutate {
        replace => {
          "udm_mapping" => "%{field}"
        }
        on_error => "field_empty" 
      }
    }

    grok {
     match => { "message" => ["(?P<fieldName>.*)"] }
     overwrite => ["fieldName"]
     on_error => "grok_Failure"
    }

#### Grok match_all option

By default, the `grok` filter extracts only the first successful pattern match. To extract all non-overlapping occurrences of a pattern within a field, set the `match_all` option to `true`. This option aggregates the captured values into a map structure where keys are string representations of the match index (starting at "0"). You can iterate over these values using a `for` loop with the `map` keyword.

**Example:**

The following example shows the grok filter extracting all the IP addresses and mapping them to `principal.ip`:

    filter {
        grok {
            match => {"message": "%{IP:ips}"}
            match_all => true
        }
        mutate {
            replace => {
                "event.idm.read_only_udm.metadata.event_type" => "GENERIC_EVENT"
            }
        }
        for index, ip in ips map {
            mutate {
                merge => {
                    "event.idm.read_only_udm.principal.ip" => "ip"
                }
            }
        }
        mutate {
            merge => {
                "@output" => "event"
            }
        }
    }

#### Grok best practices

To maintain performance and stability within your parsers, adhere to the following principles when writing Grok filters:

- **Use intent-based naming** : All Grok variable names should clearly state their intent (for example, use `source_ip` or `target_port` rather than vague names like `ip1` or `val2`).
- **Initialize and overwrite** : All Grok variables must be initialized (set to `""`) at the top of the parser. Furthermore, these variables must be explicitly declared in the overwrite array of the `grok` filter block. This prevents flakiness and ensures the variables are explicitly overwritten with the newly fetched values from the Grok extraction.
- **Use predefined patterns** : Always use existing Grok variables (such as `%{SPACE}`, `%{WORD}`, or `%{IP}`) whenever possible, rather than writing custom regular expressions to handle spaces or common data types. This improves parsing efficiency and readability.

#### Google SecOps constraint: The double backslash rule

Because Logstash and the Google SecOps parser engine interpret string literals before passing them to the regex engine, you must *escape the escape character*. This means any regex token that normally uses a single backslash (\\) requires a double backslash (\\\\) in Google SecOps, as shown in the following examples:

- **Standard regex space** : `\s` requires the Google SecOps regex: `\\s`
- **Standard regex digit** : `\d` requires the Google SecOps regex: `\\d`
- **Standard literal period** : `\.` requires the Google SecOps regex: `\\.`
- **Standard literal bracket** : `\[` requires the Google SecOps regex: `\\[`

| Regex | Data |
|---|---|
| `\\s` | Space |
| `\\S` | Not space |
| `\\d` | Digit |
| `\\D` | Not digit |
| `\\w` | Word |
| `\\W` | Not word |

### Extract JSON formatted logs

#### JSON extraction syntax

    json {
      source => "message"
      on_error => "json_failure"
    }

#### Manipulating JSON arrays

JSON arrays can be accessed by adding an array_function parameter.

    json {
      source => "message"
      on_error => "json_failure"
      array_function => "split_columns"
    }

The split_columns function makes elements of an array accessible through an
index. So, if you have an array that looks like the following:

    { "ips" : ["1.2.3.4","1.2.3.5"] .. }

you will be able to access the two values using `ips.0` and `ips.1` tokens.

If you have nested arrays or multiple arrays, the function will unnest arrays
recursively. Here is a nested array example.

    {
      "devices": [
        {
          "ips": ["1.2.3.4"]
        }
      ]
    }

In this case, you access the IP address using `devices.0.ips.0` token.
Because `devices.1` doesn't exist, it will behave the same as other non-existing
elements of JSON.

If an element in a JSON doesn't exist, then:

- You cannot access it using an `if` statement unless you initialize the token to an empty string, "", before calling the JSON filter.
- You cannot use it in the mutate plugin's replace filter because it will cause an error.
- You can use it in the mutate plugin's rename filter because these will be ignored.
- You can use it in the mutate plugin's merge filter because these will be ignored.

> [!NOTE]
> **Note:** This can create a naive implementation of merge, where you attempt to merge a finite number of values, and merge only succeeds with those that exist. If an array only has two elements and the instruction tries to access a third, merge will not fail.

    mutate {
      merge => {
        "event.idm.read_only_udm.observer.ip" => "ips.0"  }
    }

    mutate {
      merge => {
        "event.idm.read_only_udm.observer.ip" => "ips.1"  }
    }

    # this doesn't fail even though this element doesn't exist.
    mutate {
      merge => {
        "event.idm.read_only_udm.observer.ip" => "ips.2"  }
    }

    # this doesn't fail even though this element doesn't exist.
    mutate {
      merge => {
        "event.idm.read_only_udm.observer.ip" => "ips.3"  }
    }

#### Handle fluctuating nested JSON objects

Because log payloads fluctuate, if you have an object inside a JSON that varies from log to log, you must not map it directly to UDM or use it in direct conditional logic. Attempting to directly reference a deeply nested object (for example, `network.source.hostname`) that happens to be missing in the current log will cause parsing problems.

The safe solution is to extract the JSON object into a safe intermediate variable using a `replace` (which fails gracefully). Then, perform a null-check on that intermediate variable before mapping it to your final UDM schema.

##### The unsafe way: Direct conditional logic

If the network or source object doesn't exist in the current log, evaluating the `if` statement or attempting the direct mapping will crash the parser:

**Example of a dangerous handling (if the network/source is missing, the parser fails here)**:


    if [network][source][hostname] != "" {
        replace => {
            "source_hostname" => "%{network.source.hostname}"
        }
        on_error => "source_hostname_not_found"
    }

##### The safe way: Intermediate extraction and null check

1. **Safely extract the nested object to an intermediate variable** : Using `replace` with `on_error` ensures that if the nested path is missing, it fails gracefully without crashing the pipeline, while tagging the error.

       mutate {
           replace => {
               "source_hostname" => "%{network.source.hostname}"
           }
           on_error => "network_source_hostname_not_found"
       }

2. **Perform conditional logic on the *safe* intermediate variable and safely map to the final UDM schema** : Using a merge handles array populations safely, and adding a second `on_error` protects against type-mismatches during UDM assignment.

       if [source_hostname] != "" {
           mutate {
               replace => {
                   "event.idm.read_only_udm.principal.hostname" => "%{source_hostname}"
               }
               on_error => "invalid_hostname_mapping"
           }
       }

### Extracting XML formatted logs

#### XML extraction syntax

Define the path to the field in the original log using
[XPath expression syntax](https://en.wikipedia.org/wiki/XPath).

    xml {
      source => "message"
        xpath => {
          "/Event/System/EventID" => "eventId"
          "/Event/System/Computer" => "hostname"
     }
    }

#### Manipulating XML with iteration

For example if the sample looks like this:

    message -
    <Event>
        <HOST_LIST>
            <HOST>
                <ID>iD1</ID>
                <IP>iP1</IP>
            </HOST>
            <HOST>
                <ID>iD2</ID>
                <IP>iP2</IP>
            </HOST>
        </HOST_LIST>
    </Event>

If we want to iterate over the above sample log then use the following:

    for index, _ in xml(message,/Event/HOST_LIST/HOST){
       xml {
        source => "message"
        xpath => {
          "/Event/HOST_LIST/HOST[%{index}]/ID" => "IDs"
          "/Event/HOST_LIST/HOST[%{index}]/IP" => "IPs"
        }
      }
    }

We also support nested for loop. See the sample below:

    message -
    <Event>
        <HOST_LIST>
            <HOST>
                <ID>id1</ID>
                <IP>ip1</IP>
                <Hashes>
                    <Hash>hash1</Hash>
                    <Hash>hash2</Hash>
                </Hashes>
            </HOST>
            <HOST>
                <ID>id2</ID>
                <IP>ip2</IP>
                <Hashes>
                    <Hash>hash1</Hash>
                    <Hash>hash2</Hash>
                </Hashes>
            </HOST>
        </HOST_LIST>
    </Event>
    "

    for index, _ in xml(message, /Event/HOST_LIST/HOST){
        xml {
          source => "message"
          xpath => {
            "/Event/HOST_LIST/HOST[%{index}]/ID" => "IDs"
          }
        }
        for i, _ in xml(message, /Event/HOST_LIST/HOST[%{index}]/Hashes/Hash) {
          xml {
            source => "message"
            xpath => {
              "/Event/HOST_LIST/HOST[%{index}]/Hashes/Hash[%{i}]" => "data"
            }
          }
        }
      }

**Note** - index starts with 1.

### Extract key-value formatted logs

#### Key-value extraction syntax

    kv {
      source => "message"
      field_split => "|"
      value_split => ":"
      whitespace => "strict"
      trim_value => "\""
    }

#### Key-value extraction example

    # initialize the token
    mutate {
      replace => {
        "destination" => ""
        }
    }

    # use the kv filter to split the log.
    kv {
      source => "message"
      field_split => " "
      trim_value => "\""
      on_error = "kvfail"
    }

    # assigned one of the field values to a UDM field
    mutate {
      replace => {
        "event.idm.read_only_udm.target.hostname" => "%{destination}"
      }
    }

The kv filter includes the following options:

- **field_split option**: The field_split option lets you extract key-value pairs from a string, for example extracting parameters from a URL query string or key-value pairs from the log. Specify the delimiter that separates each key-value pair.
- **value_split option**: Use the value_split option to identify the delimiter between the key and the value.
- **whitespace option**: Use the whitespace option to handle acceptance of unnecessary whitespace around the key-value pair. The default option is lenient which ignores the surrounding whitespace. If you have a situation where the whitespace shouldn't be ignored, set the option to "strict".
- **trim_value option**: Use the trim_value option to remove extraneous leading and trailing characters from the value, such as quotation marks.

### Extract CSV formatted logs

    # parse the message into individual variables, identified as column1, column2, column3, etc.
    csv {
      source => "message"
      separator => ","
      on_error => "csv_failed"
    }


    # assign each value to a token
      mutate {
        replace => {
          "resource_id" => "%{column1}"
          "principal_company_name" => "%{column3}"
          "location" => "%{column4}"
          "transaction_amount" => "%{column6}"
          "status" => "%{column9}"
          "meta_description" => "%{column11}"
          "target_userid" => "%{column24}"
          "target_company_name" => "%{column13}"
          "principal_userid" => "%{column15}"
          "date" => "%{column16}"
          "time" => "%{column17}"
        }
    }

### Loop over a JSON array using a `for` loop

You can use a `for` loop to iterate over a JSON array. The syntax is as follows:

    for <item> in <array> {
        ...
      }

#### Loop over an array

The following is a sample log containing a `businessPhones` field, which is an array containing phone numbers.

    entries: <
        data: "{\\"businessPhones\\":\[\\"(123) 234-2320\\", \\"(123) 234-2321\\"\],
        \"displayName\":\"Quinn\",\"employeeId\":null,
        \"givenName\":\"Quinn\",\"id\":\"abc-123\",
        \"jobTitle\":\"Technician\",\"mail\":\"quinn@altostrat.com\",
        \"onPremisesSecurityIdentifier\":\"A-1-5-21\",}"

The JSON representation of the `businessPhones` field is as follows:

    {
       "data":{
          "businessPhones":[
             "(123) 234-2320",
             "(123) 234-2321"
          ]
       }
    }

You can use the `for` keyword to iterate over phone numbers:

    filter {
      json {
        source => "message"
        on_error => "json_failure
        array_function => "split_columns"
      }
      for phoneNumber in businessPhones {
          mutate {
            merge => {
              "event.idm.read_only_udm.principal.resource.attribute.labels" => "phoneNumber"
            }
          }
        }
    }

The `statedump` output (abbreviated version) looks like the following:

    "event": {
        "idm": {
          "read_only_udm": {
            "principal": {
              "resource": {
                "attribute": {
                  "labels": [
                    "(123) 234-2320",
                    "(123) 234-2321"
                  ]
                }
              }
            }
          }
        }
      }

#### Get the index of an array

You can also get the index value of an array. The value of the index starts with 0. The syntax is as follows:

    for index, <item> in <array> {
        ...
      }

In the following example, you can extract the index of each element in the `businessPhones` array:

    {
       "data":{
          "businessPhones":[
             "(123) 234-2320",
             "(123) 234-2321"
          ]
       }
    }

In the following parser code, the `for` loop iterates over each element of the `buinessPhones` array and extracts its index value:

    filter {
     json {
       source => "message"
       on_error => "json_failure
       array_function => "split_columns"
     }
     for index, phoneNumber in businessPhones {
       mutate {
         convert => {
           "index" => "string"
         }
       }
       mutate {
         replace => {
           "phoneNumber_label" => ""
         }
       }
       mutate {
         replace => {
           "phoneNumber_label.key" => "phoneNumber %{index}"
           "phoneNumber_label.value" => "%{phoneNumber}"
         }
         on_error => "phoneNumber_invalid"
       }
       if ![phoneNumber_invalid] {
         mutate {
           merge => {
             "event.idm.read_only_udm.principal.resource.attribute.labels" => "phoneNumber_label"
           }
           on_error => "phoneNumber_label_merge_failed"
         }
       }
       statedump {}
     }
     mutate {
       replace => {
         "event.idm.read_only_udm.metadata.event_type" => "GENERIC_EVENT"
       }
     }
     mutate {
       merge => {
         "@output" => "event"
       }
     }
    }

The `statedump` output (abbreviated version) looks like the following:

    "event": {
       "idm": {
         "read_only_udm": {
           "principal": {
             "resource": {
               "attribute": {
                 "labels": [
                   {
                     "key": "phoneNumber 0",
                     "value": "(123) 234-2320"
                   },
                   {
                     "key": "phoneNumber 1",
                     "value": "(123) 234-2321"
                   }
                 ]
               }
             }
           }
         }
       }
     }

#### Access nested arrays

You can use a nested `for` loop to access nested arrays.

In the following sample log, `resourceIdentifiers` is an array of resource identifier objects. The resource identifier object contains three keys---`tenantid`, `type`, and `subnet`, which is a nested array of objects.

    {
      "records": {
        "hostname": "host"
      },
      "resourceIdentifiers": [
        {
          "tentantid": "a123",
          "type": "access",
          "subnet": \[
    {
    "ip": "10.1.1.1"
    },
    {
    "ip": "10.1.1.2"
    }
    \]
        }
      ]
    }

In the following parser code, the nested `for` loops iterate over an array of objects extracting key-value pairs and organizing those key-value pairs into UDM fields.

    filter {
     json {
       source => "message"
       on_error => "json_failure
       array_function => "split_columns"
     }

     for index, resourceId in resourceIdentifiers {
       for key, value in resourceId map {
           mutate {
             replace => {
               "resId_map_label" => ""
             }
           }
           if [key] != "" {
             mutate {
               replace => {
                 "resId_map_label.key" => "resourceId %{key}"
               }
               on_error => "key_invalid"
             }
             if ![key_invalid] and [value] != "" {
               mutate {
                 replace => {
                   "resId_map_label.value" => "%{value}"
                 }
                 # Because the key with the name "subnet" is an array, it will produce an error.
                 # Hence, the value inside the "subnet" key will not show in the output.
                 # To map the array values inside the "subnet" key, we need to run one more for loop over the value of the "subnet" key.
                 on_error => "value_nested"
               }
               if ![value_nested] {
                 mutate {
                   merge => {
                     "event.idm.read_only_udm.principal.resource.attribute.labels" => "resId_map_label"
                   }
                   on_error => "resId_map_label_merge_failed"
                 }
               }
               else {
                 for sub_index, subnet in value {
                   mutate {
                     convert => {
                       "sub_index" => "string"
                     }
                   }
                   for subnet_key, subnet_value in subnet map {
                     mutate {
                       replace => {
                         "subnet_map_label" => ""
                       }
                     }
                     mutate {
                       replace => {
                         "subnet_map_label.key" => "%{key} %{subnet_key} %{sub_index}"
                         "subnet_map_label.value" => "%{subnet_value}"
                       }
                       on_error => "subnet_value_invalid"
                     }
                     if ![subnet_value_invalid] {
                       mutate {
                         merge => {
                           "event.idm.read_only_udm.principal.resource.attribute.labels" => "subnet_map_label"
                         }
                         on_error => "subnet_map_label_merge_failed"
                       }
                     }
                   }
                 }
                 statedump {}
               }
             }
           }
       }
     }
     mutate {
       replace => {
         "event.idm.read_only_udm.metadata.event_type" => "GENERIC_EVENT"
       }
     }
     mutate {
       merge => {
         "@output" => "event"
       }
     }
    }

The `statedump` output (abbreviated version) looks like the following:

    events_for_log_entry:  {
     events:  {
       timestamp:  {
         seconds:  1709619033
         nanos:  818679197
       }
       idm:  {
         read_only_udm:  {
           metadata:  {
             event_timestamp:  {
               seconds:  1709619033
               nanos:  818679197
             }
             event_type:  GENERIC_EVENT
           }
           principal:  {
             resource:  {
               attribute:  {
                 labels:  {
                   key:  "subnet ip 0"
                   value:  "10.1.1.1"
                 }
                 labels:  {
                   key:  "subnet ip 1"
                   value:  "10.1.1.2"
                 }
                 labels:  {
                   key:  "resourceId tentantid"
                   value:  "a123"
                 }
                 labels:  {
                   key:  "resourceId type"
                   value:  "access"
                 }
               }
             }
           }
         }
       }
     }
    }

The following are some of the salient aspects of the parser code that is used to access nested arrays:

- **Outer `for` loop** ---`for index, resourceId in resourceIdentifiers {...}`:

  - This loop iterates over an array called `resourceIdentifiers`, which contains objects representing resource identifiers.
- **Inner `for` loop** ---`for key, value in resourceId map {...}`:

  - This loop iterates over each key-value pair within each `resourceId` object. It does this using the `map` keyword. To learn more about the `map` keyword, see [Loop over a JSON object key-value pairs](https://docs.cloud.google.com/chronicle/docs/reference/parser-syntax#map_keyword).

  - Create labels to store key-value pairs using the [`replace` function](https://docs.cloud.google.com/chronicle/docs/reference/parser-syntax#replace_function).

  - Handle potential errors using [`on_error` flags](https://docs.cloud.google.com/chronicle/docs/reference/parser-syntax#error_handling_-_on_error)
    (for example, `key_invalid` and `value_nested`).

  - Iterate over the nested `subnet` array---`for sub_index, subnet in value {...}`: This `for` loop is used to iterate over the `subnet` key, which contains an array of subnet objects. Each key-value pair in the `subnet` array is then iterated over using the `map` keyword---`for subnet_key, subnet_value in subnet map {...}`.

  - Merge transformed data: Using the [`merge` function](https://docs.cloud.google.com/chronicle/docs/reference/parser-syntax#merge_function), join the transformed data back into the event under a specific path
    (`event.idm.read_only_udm.principal.resource.attribute.labels`).

### Loop over key-value pairs of a JSON object using `map`

You can use the `map` keyword to loop over key-value pairs of a JSON object. The syntax is as follows:

    for key, value in <object> map {
        ...
      }

Both key and value are of the `string` data type.

#### Loop over key-value pairs of an object

The following is a sample log containing details of a Kubernetes resource.

    entries: <
        message: "{\"resource\": {\"type\": \"k8s_container\",\"labels\":
        {\"container_name\": \"test-container\",\"namespace_name\": \"default\",
        \"location\": \"us-west1-a\",\"project_id\": \"abc-123\",
        \"cluster_name\": \"test-cluster\",\"pod_name\":
        \"test-pod-123\"}}}"

The JSON representation of the log is as follows:

    {
       "resource":{
          "type":"k8s_container",
          "labels":{
    "container_name":"test-container",
    "namespace_name":"default",
    "location":"us-west1-a",
    "project_id":"abc-123",
    "cluster_name":"test-cluster",
    "pod_name":"test-pod-123"
    }
       }
    }

The `map` keyword is added as a suffix to the existing `for` loop expression.
This lets you iterate over key-value pairs of an object. In the following
parser code, `map` is used to iterate over the key-value pairs of the
`resource.labels` object.

    filter {
      json {
        source => "message"
        on_error => "json_failure
        array_function => "split_columns"
      }
      for key, value in resource.labels map {
        mutate {
          replace => {
            "test.key" => "%{key}"
            "test.value" => "%{value}"
          }
        }
      statedump {}
      }
    }

The output for each iteration is as follows:

First Iteration:

    "test": {
        "key": "cluster_name",
        "value": "test-cluster"
      }

Second Iteration:

    "test": {
        "key": "container_name",
        "value": "test-container"
      }

Third Iteration:

     "test": {
        "key": "location",
        "value": "us-west1-a"
     }

Fourth Iteration:

    "test": {
        "key": "namespace_name",
        "value": "default"
      }

Fifth Iteration:

    "test": {
        "key": "pod_name",
        "value": "test-pod-123"
      }

Sixth Iteration:

     "test": {
        "key": "project_id",
        "value": "abc-123"
      }

#### Loop over key-value pairs of a nested object

You can use the `map` keyword to loop over key-value pairs of nested objects.

The following is a sample log containing details of a Kubernetes resource.

    entries: <
        message: "{\"textPayload\": \"2023-07-18 18:01:27,259 INFO kube_hunter.
        modules.report.collector Found open service \\\"Kubelet API\\\" at 10.64.
        3.1:10250\",\"insertId\": \"3ki6ud2owr8d4afy\",\"resource\": {\"type\":
        \"k8s_container\",\"labels\": {\"container_name\": \"test-container\",
        \"namespace_name\": \"default\",\"location\": {\"country\" : \"US\",
        \"code\" : \"us-west1-a\"},\"cluster_name\": \"test-cluster\",
        \"pod_name\": \"test-pod-123\"}}}"

The JSON representation of the log is as follows:

    {
       "resource":{
          "type":"k8s_container",
          "labels":{
             "container_name":"test-container",
             "namespace_name":"default",
             "location":{
    "code":"us-west1-a",
    "country":"US"
    },
             "cluster_name":"test-cluster",
             "pod_name":"test-pod-123"
          }
       }
    }

In the following parser code, the `map` keyword is used to iterate over the key-value pairs of the nested object `resource.labels.location`.

    filter {
      json {
        source => "message"
        on_error => "json_failure
        array_function => "split_columns"
      }
      for key, value in resource.labels map {
        mutate {
          replace => {
            "test.key" => "%{key}"
          }
        }
        mutate {
          replace => {
            "test.value" => "%{value}"
          }
          on_error => "nested_key"
        }
        if [test][key] == "location" {
          for nestedKey, nestedValue in value map {
            mutate {
              replace => {
                "locationLabel.key" => "%{nestedKey}"
                "locationLabel.value" => "%{nestedValue}"
              }
            }
            statedump {}
          }
        }
      }
    }

The `statedump` output (abbreviated version) for `resource.labels.location` looks like the following:

    "key": "location",
    "locationLabel": {
        "key": "code",
        "value": "us-west1-a"
      }
      .
      .
      .
    "locationLabel": {
        "key": "country",
        "value": "US"
      }

### Handle hybrid log formats

Frequently, logs arrive in a hybrid format, combining a standard unstructured header with a structured payload (for example, SYSLOG + JSON, or SYSLOG + KV). In these scenarios, a single filter is insufficient. The parser must isolate the distinct data components and apply the appropriate filters sequentially.

**Example: Parsing SYSLOG + KV**

If a log contains a standard SYSLOG header followed by a KV payload, you must use a multi-step extraction strategy:

1. **Initial extraction (`grok`)** : First, apply a `grok` filter to parse the unstructured SYSLOG header (extracting metadata such as the timestamp and host). Within this same Grok pattern, capture the entire remaining KV message into a single, dedicated intermediate variable (for example, `%{GREEDYDATA:kv_data}`).
2. **Secondary extraction (`kv`)** : Next, apply the `kv` filter, configuring it to target specifically the `kv_data` variable rather than the entire raw log. This efficiently parses the remaining key-value pairs into individual fields without requiring complex, computationally heavy regular expressions.

## Transform and map data to UDM

This section provides information about how to transform raw data and map it to UDM records.

### Transform data using the *mutate* plugin

Use the mutate filter plugin to transform and consolidate data into a single
block or to break the data into separate mutate blocks. When using a single
block for the mutate functions, be aware that the mutations are executed in the
order described in the
[Logstash mutate plugin documentation](https://www.elastic.co/guide/en/logstash/current/plugins-filters-mutate.html).

#### Convert functions

Use the convert function to transform values to different data types. This
conversion is needed to assign values into fields within the respective data
type schemas. Legacy proto definitions (EDR, Webproxy, etc.) require data type
conversions to match the target data type. For example, IP address fields need
to be converted before assigning the value to the target schema field. UDM allows
for handling of most fields as strings, including IP address fields.

> [!NOTE]
> **Note:** Google SecOps requires IPv4 and IPv6 addresses to adhere to their standard formats. Other formats, including IP addresses with leading zeros (for example, `10.29.015.03`), hexadecimal values (for example, `0xC0A80201`), or decimal integers (for example, `3232236033`) are rejected.

Supported data types include the following:

- boolean
- float
- hash
- integer
- ipaddress
- macaddress
- string
- uinteger
- hextodec
- hextoascii

**Convert example**

    mutate {
      convert => {
        "jsonPayload.packets_sent" => "uinteger"
      }
    }

#### Gsub function

Match a regular expression against a field value and replace all matches with a
replacement string. This applies only to string fields.

**Gsub syntax**

This configuration takes an array consisting of 3 elements per
field/substitution. In other words, for every substitution you want to make you
add three elements to the gsub configuration array, specifically the name of the
field, the regular expression to replace, and the substitution string.

The gsub function supports [re2
syntax](https://github.com/google/re2/wiki/Syntax). You can
use simple strings most of the time as long as they don't contain characters
that have a special meaning, such as brackets ( `[` or `]` ). If you need special
characters to be interpreted literally, they must be *escaped* by preceding each
character with a backslash (\\).

One important exception to this rule is
escaping a backslash itself where if you want to search for a literal backslash
within your text, you need four backslashes (\\\\\\\\).

> [!NOTE]
> **Note:** This is because there are two levels of string interpretation that happen when the gsub parser code is interpreted---the double-quoted string interpretation and the regular expression interpretation. Double-quoted strings transform pairs of backslashes (\\\\\\\\) into single backslashes (\\\\). The regular expression engine then interprets the first backslash in \\\\ as an escape character, leading to a final interpretation of a single, literal backslash (\\).

    mutate {
      gsub => [
        # replace all occurrences of the three letters "cat" with the three letters "dog"
        "fieldname1", "cat", "dog",
        # replace all forward slashes with underscore
        "fieldname2", "/", "_",
        # replace backslashes, question marks, hashes, and minuses  # with a dot "."
        "fieldname3", "[\\\\?#-]", "."
       ]
    }

#### Lowercase functions

The lowercase function is used to transform a value into a lowercase value.

**Lowercase syntax**

    mutate {
      lowercase => [ "token" ]
    }

**Lowercase example**

    mutate {
      lowercase => [ "protocol" ]
    }

#### Merge function

The merge function is used to join multiple fields. When parsing repeated fields,
such as ip_address fields, use the merge function to assign IP addresses
to the token. Additionally, the merge function is used to generate the
normalized output message that is ingested in Google SecOps and can be used to
generate multiple events from the same log line.

**Merge syntax**

    mutate {
      merge => {
        "destinationToken" => "addedToken"
       }
    }

**Merge function example - using a repeated field**

    mutate {
      merge => {
        "event.idm.read_only_udm.target.ip" => "dstAddr"
      }


**Merge function example - output to a UDM record**

    mutate {
      merge => {
        "@output" => "event"
      }
    }

#### Rename function

The rename function renames a token and assigns the value to a new token. Use
this function when the tokenized value can be directly assigned to the schema
defined token. The original token and new token must be of the same data type
before performing the rename transformation. By using the rename function, the
original token is destroyed and replaced with the new token.

**Rename function syntax**

    mutate {
      rename => {
        "originalToken" => "newToken"
      }
    }

**Rename function example**

    mutate {
      rename => {
        "proto" => "event.idm.read_only_udm.network.ip_protocol"
        "srcport" => "event.idm.read_only_udm.network.target.port"
      }
    }

#### Replace function

The replace function assigns a value to a token. The assignment can be based on
constants, existing field values or a combination of values. The replace
function can also be used to define a token declaration. This function can only
be used for string values.

**Replace function syntax - assign a constant**

    mutate {
      replace => {
        "token" => "newConstantValue"
      }
    }

**Replace function syntax - assign a variable value**

    mutate {
      replace => {
        "token" => "%{otherTokenValue}"
      }
    }

**Replace function example - assign a constant**

    mutate {
      replace => {
        "event.idm.read_only_udm.security_result.action" => "ALLOW"
      }
    }

**Replace function example - assign a variable value**

    mutate {
      replace => {
        "shost" => "%{dhost}"
      }
    }

#### Uppercase function

The uppercase function is used to transform a value into an uppercase value.

**Uppercase function syntax**

    mutate {
      uppercase => [ "token" ]
    }

**Uppercase function example**

    mutate {
      uppercase => [ "protocol" ]
    }

#### RemoveField Function

The remove_field function destroys a token. The name of the token to be
destroyed can be either static or dynamic using existing token values. No action
is performed if the token doesn't exist.

**RemoveField function syntax - remove a static token**

    mutate {
      remove_field => [ "token" ]
    }

**RemoveField function syntax - remove a dynamic token**

    mutate {
      remove_field => [ "%{someTokenValue}" ]
    }

**RemoveField Function Example - remove a static token**

    mutate {
      remove_field => [ "event.webproxy.protocol" ]
    }

**RemoveField Function Example - remove a dynamic token**

    mutate {
      remove_field => [ "network.%{application_protocol}" ]
    }

#### Copy Function

The copy function deep copies the value of source token into destination token.
There is no restriction on the type of value that can be copied. After copying, any
change to destination token's value will have no effect on source token's value
and vice-versa because the value is deepcopied. Before applying a copy function,
the source token must exist. Also, if the destination token does not exist, a
new token is created, otherwise the value of the old token is overridden.

**Copy syntax**

    mutate {
      copy => {
        "destinationToken" => "sourceToken"
      }
    }

#### Split Function

The split function splits a string into an iterable array.

    mutate {
        split => {
           source => "src_field"
           separator => ","
           target => "target_field"
        }
      }

### Transform data using other functions

#### base64 function

The `base64` function converts a `base64` encoded value to a string.
This function is based on the Go language `base64` package.

The `source` field identifies the variable where the input value is stored.
The `target` value identifies the variable where to store the output.
By default, the function uses `Standard` decoding, but can be configured to use `URL` decoding.

    base64 {
      source => "ip_address"
      target => "ip_address_string"
      encoding => "RawStandard"
    }

The following example converts a base64 encoded IP address. The example checks if the `ip_address` variable is populated, decodes the value, and then stores the decoded value in the `ip_address_string` variable. The value stored in the `ip_address_string` variable is appended to the `event.idm.read_only_udm.target.ip` UDM field.

    if [ip_address] != "" {
      base64 {
        source => "ip_address"
        target => "ip_address_string"
      }
      mutate {
        merge => {
          "event.idm.read_only_udm.target.ip" => "%{ip_address_string}"
        }
      }
    }

#### Date function

The `date` function is required to handle the date and timestamp from the log
extraction. UDM fields that store a Timestamp require a properly normalized date
value. The date function supports a variety of date formats, including ISO8601,
UNIX and others along with custom-defined date and time formats.

Google SecOps supports the following predefined date formats:

- ISO8601
- RFC3339
- UNIX
- UNIX_MS
- yyyy-MM-dd HH:mm:ss
- yyyy/MM/dd HH:mm:ss
- yyyy-MM-ddTHH:mm:ss
- yyyy-MM-dd HH:mm:ss Z
- yyyy-MM-dd HH:mm:ss.SSS
- dd/MMM/yyyy:HH:mm:ss Z
- yyyy-MM-dd HH:mm:ss ZZ
- yyyy-MM-ddTHH:mm:ssZZ
- HH:mm:ss
- yyyy-MM-ddTHH:mm:ss.SSSZ
- yyyy-MM-dd HH:mm:ss.SSS Z
- dd MMM yyyy
- yyyy-MM-dd HH:mm:ssZ
- EEE MMM dd HH:mm:ss yyyy
- yyyy-MM-ddTHH:mm:ss.SSS
- yyyy-MM-ddHH:mm:ss
- MM/dd/yyyy HH:mm:ss A
- yyyy MMM d HH:mm:ss Z
- MMM d HH:mm:ss yyyy
- yyyy-MM-dd HH:mm:ss.SSSZ
- MMM d yyyy HH:mm:ss
- yyyy-mm-ddTHH:mm:ss.SSSZ
- yyyy-MMM-dd HH:mm:ss
- yyyy-MM-ddTHH:mm:ss.SSSSSSSSSZ
- MM-dd-yyyy HH:mm:ss
- yy-MM-dd HH:mm:ss
- yyyy-MM-dd HH:mm:ss.S
- MMM dd yyyy HH:mm:ss

Google SecOps also provides a set of system-supplied timestamps
that can be used in mapping instructions.

- `@createTimestamp`: Always included and represents the time Google SecOps received the logs.
- `@timestamp`: Optional. The timestamp provided by Splunk or PCAP collection, if it exists.
- `@collectionTimestamp`: Optional value at the log entry level. This timestamp represents the time the Forwarder collected the log entry. However, this value might not be present for logs that are ingested using the out-of-band processor.

At the end of processing, Google SecOps uses the timestamp present in the
@timestamp field as the timestamp value for all events. By default, the date
filter takes precedence to populate the @timestamp field. However, if there is a
need to use the log receipt time as the event timestamp, you can use the rename
function to rename the @createTimestamp to @timestamp. Best practice is to use
the log message and extract the date value. If the log doesn't include a
timestamp value, you might need to use createTimestamp for log ingestion.

For UNIX and UNIX-MS date formats, use the `on_error` statement to handle errors.

**Date function syntax**

    date {
      match => ["token", "format"]
      on_error => "no_match"
    }

**Date function example**

    date {
      match => ["when", "yyyy-MM-dd HH:mm:ss"]
      on_error => "no_match"
    }

**Date example using a timezone specification**

    date {
      match => ["logtime", "yyyy-MM-dd HH:mm:ss"]
      timezone => "America/New_York"
      on_error => "no_match"
    }

**Specify multiple date formats**

    date {
      match => ["ts", "yyyy-MM-dd HH:mm:ss", "UNIX", "ISO8601", "UNIX_MS"] 
      on_error => "no_match"}

**Handle timestamps without a year value - rebase option**

The rebase option for the date filter supports the ability to handle timestamps
without a year value. It sets the year based on the time the data was ingested.

**rebase option example**

    date {
      match => ["when", "MMM dd HH:mm:ss"]
      rebase => true
      on_error => "no_match"
    }

**Example: Extract a timestamp from a raw log**

You can extract a timestamp from a raw log and store the value to a UDM field, such as `metadata.collected_timestamp`.

Consider the following 1Password raw log containing a timestamp field.

    entries:  {
        data:  "{\"country\":\"US\",\"target_user\":
        {\"uuid\":\"FTASPXQHWRF3XMJDLGKWBMZ2LI\",\"name\":\"John Doe\",
        \"email\":\"abc.def.@demo.com\"},\"location\":{\"country\":\"US\",
        \"region\":\"California\",\"city\":\"Hawthorne\",\"latitude\":33.9168,
        \"longitude\":-118.3432},\"category\":\"success\",\"type\":\"mfa_ok\",
        \"details\":null,\"client\":{\"os_name\":\"Windows\",\"os_version\":\"10.
        0\",\"ip_address\":\"2603:8000:7600:c4e1:4db:400b:ff2:6626\",
        \"app_name\":\"1Password Browser Extension\",\"app_version\":\"20216\",
        \"platform_name\":\"Chrome\",\"platform_version\":\"89.0.4389.82\"},
        \"uuid\":\"EPNGUJLHFVHCXMJL5LJQGXTENA\",
        \"session_uuid\":\"UYA65VLTKZAMJAYVODY6BJ36VE\",
        \"timestamp\\":\\"2022-07-27T22:46:30.312374636Z\\"}"
      }

The following is a JSON representation of the raw log.

    {
      "country": "US",
      "target_user": {
        "uuid": "FTASPXQHWRF3XMJDLGKWBMZ2LI",
        "name": "Stephanie Badum",
        "email": "abc.def.@demo.com"
      },
      "location": {
        "country": "US",
        "region": "California",
        "city": "Hawthorne",
        "latitude": 33.9168,
        "longitude": -118.3432
      },
      "category": "success",
      "type": "mfa_ok",
      "details": null,
      "client": {
        "os_name": "Windows",
        "os_version": "10.0",
        "ip_address": "2603:8000:7600:c4e1:4db:400b:ff2:6626",
        "app_name": "1Password Browser Extension",
        "app_version": "20216",
        "platform_name": "Chrome",
        "platform_version": "89.0.4389.82"
      },
      "uuid": "EPNGUJLHFVHCXMJL5LJQGXTENA",
      "session_uuid": "UYA65VLTKZAMJAYVODY6BJ36VE",
      "timestamp": "2022-07-27T22:46:30.312374636Z"
    }

In the following example, the `grok` filter is used to extract an ISO 8601 timestamp from the `timestamp` field in a 1Password raw log and store it in `EventTime`. The `date` filter then parses the value in `EventTime` according to the ISO 8601 date format. The `target` field is used to designate where the result of the date parsing should be saved. In this case, the parsed date is stored in the `event.idm.read_only_udm.metadata.collected_timestamp` field.

    filter {
      json {
        source => "message"
        on_error => "json_failure
        array_function => "split_columns"
      }
      grok {
          match => {
            "timestamp" => "%{TIMESTAMP_ISO8601:EventTime}"
          }
          on_error => "time_stamp_failure"
        }
        if [EventTime] != "" {
          date {
            match => ["EventTime", "ISO8601"]
            target => "event.idm.read_only_udm.metadata.collected_timestamp"
          }
        }
    }

If the target field is not specified, then the timestamp is mapped to the `metadata.event_timestamp` UDM field.

#### Drop function

This function drops all messages that reach this filter logic from the normalization pipeline, suppressing UDM event generation only.

> [!NOTE]
> **Note:** When a parser `drop {}` filter drops a log, Google SecOps doesn't create a UDM record, and detection rules or UDM searches don't evaluate the event. However, Google SecOps retains the original raw log in immutable raw storage for compliance and audit retention, where it remains searchable through Raw Log Search. To drop raw logs before storage, configure filtering upstream at the collector or forwarder layer.

**Drop syntax**

    drop {
      tag => "TAG_MALFORMED_MESSAGE"
    }

**Drop example**

    if [domain] == "-" {
      drop {
        tag => "TAG_MALFORMED_MESSAGE"
      }
    }

#### Conditional logic

Conditionals are consistent with the Logstash documentation concerning the
usage of conditional statements. However, with parser syntax, only use conditionals as part of the filter logic for event transformation. The only conditional logic statements available are if, if/else, and if/else if/else.

| **Operator** | **Syntax** |
|---|---|
| Equal | == |
| Not equal | != |
| Less than | \< |
| Greater than | \> |
| Less than or equal | \<= |
| Greater than or equal | \>= |
| Regular expression match | =\~ |
| Regular expression does not match | !\~ |

**Conditional syntax - If**

    if [token] == "value" {
     <code block>
    }

**Conditional syntax - if/else**

    if [token1] == "value1" and [token2] == "value2" {
      <code block 1>
    } else {
      <code block 2>
    }

**Conditional syntax - if/else if/else**

    if [token] == "value1" {
      <code block 1>
    } else if [token] == "value2" {
      <code block 2>
    } else {
      <code block 3>
    }

**Conditional syntax - If example**

    if [protocol] in ["tcp", "udp", "icmp"] {
      mutate {
       uppercase => [ "protocol" ]
     }
    }

**Conditional syntax - if/else if/else example**

    if [action] == "drop" or [action] == "deny" or [action] == "drop ICMP" {
      mutate {
        replace => {
        "event.idm.read_only_udm.security_result.action" => "BLOCK"
        }
      }
    } else if [action] == "allow" {
      mutate {
        replace => {
          "event.idm.read_only_udm.security_result.action" => "ALLOW"
    }
      }
    } else {
      mutate {
        replace => {
          "event.idm.read_only_udm.security_result.action" => "UNKNOWN_ACTION"
        }
      }
    }

#### Error handling: `on_error`

Set the on_error property on any filter to catch errors. This property sets a
value to true if an error was encountered, and false otherwise.

**on_error syntax**

    on_error => "<value>"

**`on_error` function example**

This example uses the `on_error` function to check if a value is an IP
address without causing a failure. A use case for this is to determine
if the field value is an IP address or hostname, and then handle the field value
appropriately.

> [!NOTE]
> **Note:** Google SecOps requires IPv4 and IPv6 addresses to adhere to their standard formats. Other formats, including IP addresses with leading zeros (for example, `10.29.015.03`), hexadecimal values (for example, `0xC0A80201`), or decimal integers (for example, `3232236033`) are rejected.

    mutate {
      convert => {
        "host" => "ipaddress"
      }
      on_error => "is_not_ip"
    }

    if [is_not_ip] {
      # This means it's not an IP
    }

### Output data to a UDM record

Use the merge function to generate output. It is possible to generate more than
one event message based on a single log line. For example, you might want to
create the web proxy event message, but also generate an alert message.

#### Generating output - single event

    mutate {
      merge => {
        "@output" => "event"
     }
    }

#### Generating output - multiple events

    if [event1] != "" {
        mutate {
          merge => {
            "@output" => "event1"
          }
        }
      }
      if [event2] != "" {
        mutate {
          merge => {
            "@output" => "event2"
          }
        }
      }
    }

When generating multiple events as output, instead of assigning field values to
`event.*`, use a designation such as `event1.*` and `event2.*` to differentiate
between the values assigned to the first event versus the second event.

### Validate data using statedump plugin

Use the statedump filter plugin to validate the internal state of a parser.
The filter shows all the values set by the parser during troubleshooting.
You can use multiple blocks of statedump filters using the `label` property.

You can use the statedump filter for troubleshooting only. You must remove the
statedump filter blocks before validation.

#### Statedump filter example with label

    statedump {
      label => "foo"
    }

Statedump output example:

    Internal State (label=foo):
    {
      "@createTimestamp": {
        "nanos": 0,
        "seconds": 1693549534
      },
      "@enableCbnForLoop": true,
      "@onErrorCount": 0,
      "@output": [],
      "@timezone": "",
      "event": {
        "idm": {
          "read_only_udm": {
            "metadata": {
              "event_type": "GENERIC_EVENT"
            }
          }
        }
      },
      "message": "my sample log"
    }

The statedump filter includes the following options:

- `@createTimestamp` - The time when this dump was created.

- `@enableCbnForLoop` - Internal flag.

- `@onErrorCount` - Number of errors discovered so far.

- `@output` - The final output from the parser.

- `@timezone` - The offset from UTC for log entries.

- `event` - The event mapping done by the parser.

- `message` - The log message which the parser is run against.

## Best practices

This section provides best practices for building a parser.

### Use the most accurate suitable filter

Use the most accurate filter specifically designed for your log format. For example, use a `kv` filter for KV logs, rather than writing heavy or inefficient `grok` expressions to parse them.

### Extract all available fields and parameters from the raw log

To maximize parsing efficiency and reduce downstream computation, extract all available fields and parameters from the raw log at the time of parsing a new log. By structuring the complete log upfront, users can immediately write detection rules against any field in the dataset without needing to update the parser, rewrite code, or replay historical logs when new threats emerge.

### Organize the schema mapping hierarchy properly

Organize your code sequentially and map fields to the UDM structure using this exact priority order:

1. `Metadata`
2. `Principal`
3. `Target`
4. `Intermediary`
5. `Observer`
6. `About`
7. `Network`
8. `Security_result`
9. `additional`
10. `metadata.event_type`
11. `metadata.product_name` and `metadata.vendor_name`

### Validate checks for `event_type` assignment

Always verify specific UDM validation rules before setting `metadata.event_type`. Certain event types strictly require specific fields to be populated; for example, setting `STATUS_UPDATE` requires `principal` data to be present. Use conditional blocks to check for these mandatory fields before applying the event type.

    if [has_principal] == "true" {
      mutate {
        replace => { "metadata.event_type" => "STATUS_UPDATE" }
      }
    }

### Use dynamic mapping for polymorphic variables

When a single log field can contain different types of data depending on the event (such as a `host` field arriving as a raw IP address in one log, but a hostname string in another), don't map it blindly. You must write conditional logic to validate the content type and route it to the correct target UDM attribute (for example, `principal.ip` versus `principal.hostname`).

The same instructions apply for email `address` and `userid`.

    if [remote_request_ip] != "" {
        grok {
          match => {
            "remote_request_ip" => "%{IP:remote_request_ip}"
          }
          overwrite => ["remote_request_ip"]
          on_error => "remote_request_ip_grok_failure"
        }
        if ![remote_request_ip_grok_failure] {
          mutate {
            merge => {
              "principal.ip" => "remote_request_ip"
            }
            on_error => "remote_request_ip_empty"
          }
        }
        else {
          mutate {
            replace => {
              "principal.hostname" => "%{remote_request_ip}"
            }
            on_error => "remote_request_ip_hostname_empty"
          }
        }
      }

### Use exhaustive enum mapping

Make sure that you account for all possible variations when mapping enum UDM fields based on the raw log field. For example, if you have a log with field `action` as `allow` and you're mapping `security_result.action` to `ALLOW`, additionally map `security_result.action` to `BLOCK` considering the raw log value contains terms like `block`, `blocked`, or `deny`.

### Validate fields before processing or mapping

For fields that must conform to a specific format---such as email addresses, IP addresses, MAC addresses, or cryptographic hashes---always implement structural checks before processing them or mapping them to the schema. This ensures malformed data is caught early and doesn't cause pipeline or index errors.

**Example:**

    if [email_field] =~ "^.+@.+$"{
      mutate {
        merge => { "principal.user.email_address" => "email_field" 
    }
    on_error => "invalid_email"
      }
    }

### Unify the mappings in the configuration file

If a single UDM field relies on mapping inputs from multiple source fields, group all the mappings in the same place in the configuration file. Don't scatter mappings for a single destination field across different code blocks.

**Example of bad practice: Mappings scattered across separate blocks with generic errors**

    if [src_ip] != "" {
      mutate {
        merge => { "principal.ip" => "src_ip" }
        on_error => "src_ip_merge_failed"
      }
    }

    // hundreds of lines of code in between

    if [client_ip] != "" {
      mutate {
        merge => { "principal.ip" => "client_ip" }
        on_error => "client_ip_merge_failed"
      }
    }

**Example of good practice: Consolidated mapping logic with unique on_error tracking**

    if [src_ip] != "" {
      mutate {
        merge => { "principal.ip" => "src_ip" }
        on_error => "err_principal_ip_src"
      }
    }
    if [client_ip] != "" {
      mutate {
        merge => { "principal.ip" => "client_ip" }
        on_error => "err_principal_ip_client"
      }
    }

### Declare and initialize at the beginning and use null checks

Declare and initialize all required fields at the very beginning of the filter configuration. Always implement explicit null or presence checks before attempting to map these values to the final schema.

### Use initialized grok variables

When using `grok` to extract variables, ensure they are initialized before. Follow up by systematically overwriting these variables as your logic progresses, ensuring that the final mapped state is accurate and free of stale extraction data.

### Reuse variables for efficient memory use

Favor tokens and variables that are already instantiated in memory instead of continuously creating redundant, temporary custom fields.

### Enforce the Single Responsibility Principle

Keep different operations separated into individual `mutate` blocks to enforce the Single Responsibility Principle (SRP). Never mix different actions like rename and replace inside a single `mutate` block.

### Adhere to DRY principles

Eliminate structural redundancies and duplicate extraction patterns to keep the configuration clean and maintainable. Strictly follow DRY principles.

### Don't use an underscore as a loop or iterator variable name

Avoid using a standalone underscore (_) as a loop or iterator variable name under any circumstances.

### Use lowercase naming conventions

Eliminate `camelCase` identifiers entirely. All user-defined tokens must use lowercase alphanumeric characters with `snake_case` if separators are required.

### Use self-explanatory, unique `on_error` flags

Declare a unique, highly descriptive token name for every individual `on_error` tag. Never reuse the same error flag token across multiple filter actions.

**Need more help?** [Get answers from Community members and Google SecOps professionals.](https://security.googlecloudcommunity.com/)