Data enrichment and aliasing overview

Supported in:

This document provides an overview of how Google Security Operations enriches raw logs after converting them to normalized Unified Data Model (UDM) events. Google SecOps provides different enrichment capabilities during ingestion and search.

Enrichment during ingestion

  • UDM enrichment: Merges normalized data from context sources into UDM event sources to create a single enriched UDM event. These pipelines operate in near-real time and re-enrichment pipelines handle late-arriving data.
  • Aliasing service: Keeps track of users and assets over time, merging multiple UDM Enrichment using aliasing to add context to a UDM indicator or event.
  • Entity Context Graph (ECG): Combines customer log data, asset information, user identity, and multiple sources of threat intelligence to construct both timed and timeless entities and computed attributes (for example, prevalence, first seen, or last seen).
  • Data tables: Act as lookup tables with defined columns. The data is stored in rows that are joined at query time with your UDM events. You can define the data in data tables and manage their lifecycle.

Aliasing, UDM enrichment, and ECG are derived from your parsed security data.

For more information on sources that can provide context data, see the following topics:

Viewing events

View events in the Event Viewer's Event Fields tab. This tab displays UDM event fields in a hierarchical tree structure, labeled Selected. Each UDM field is labeled with an icon indicating whether the field contains enriched or unenriched data. The icon labels are the following:

  • U: Unenriched fields pull values directly from the original raw log during the normalization process.
  • E: Enriched fields contain values that Google SecOps generates to provide additional context about artifacts in your environment.

Enriched fields show all associated sources to help you with validation, troubleshooting, auditing, and compliance. You also filter fields by their enrichment source to refine your view.

Supported aliasing and UDM enrichment capabilities

Google SecOps supports aliasing and enrichment for the following:

  • Assets
  • Users
  • Processes
  • File hash metadata (VirusTotal)
  • Geographic locations
  • Cloud resources

Supported ECG capabilities

Google SecOps supports the following ECG-related enrichments for search:

  • Prevalence
  • First seen and last seen
  • WHOIS data
  • VirusTotal relationship data
  • Tor exit node IP addresses
  • Google Threat Intelligence IOCs
  • Safe Browsing Threat lists

Enrichment during ingestion and search help you balance the trade-off between latency and accuracy for detections, searches, and dashboards. UDM enrichment and aliasing simplify the user experience by writing enrichments directly to UDM events. ECG and data tables provide flexibility and can be applied to specific use cases.

IP geolocation mapping methodology

When Google SecOps enriches data with geographic locations, you might observe discrepancies between Google's IP geolocation mapping and third-party databases.

These discrepancies occur because Google's IP mapping methodology is user-centric, whereas most third-party IP geolocation providers are machine-centric. Google focuses on the location of the user, while third parties typically focus on the location of the machine, such as a proxy or data center.

For example, data centers often host VPNs or proxies that serve users located in different geographic regions. In these cases, Google attempts to map the IP address to the user's remote location, whereas third-party tools tend to map the IP address to the data center's physical location. For IP addresses that don't have end-user traffic, such as backend server infrastructure, Google falls back to public registry information and network-level signals.

To resolve or manage IP geolocation discrepancies, consider the following options:

  • Publish a geofeed: Third-party IP owners, such as ISPs or cloud providers, can publish an RFC 8805 geofeed and register it using the Google ISP Portal. Geolocation updates typically take two to three weeks to propagate and apply only to newly ingested events going forward; previously ingested UDM events aren't retroactively updated.
  • Use custom geolocation data or enrichment blocks: To use your own geolocation data in Google SecOps, you can configure enrichment blocks to turn off default GeoIP enrichment for specific log types, or join your own IP location lookup data at query time using data tables.

For more information about how Google SecOps enriches IP geolocation fields, see IP geolocation enrichment.

Need more help? Get answers from Community members and Google SecOps professionals.