# Collect Gmail logs

**Parser Version:** 7.0
Supported in:

This document explains how to ingest Gmail logs to Google Security Operations using Google Cloud Storage V2.

Gmail is Google Workspace's email service that provides secure, intelligent email with built-in spam and phishing protection. Gmail logs capture detailed information about email delivery, security events, and message flow through the Gmail infrastructure.

## Before you begin

Ensure that you have the following prerequisites:

- A Google SecOps instance
- GCP project with Cloud Storage API and BigQuery API enabled
- Google Workspace account with appropriate edition (Enterprise Standard, Enterprise Plus, Education Standard, or Education Plus)
- Super administrator access to Google Workspace Admin console
- Permissions to create and manage GCS buckets
- Permissions to manage IAM policies on GCS buckets
- Permissions to create BigQuery scheduled queries
- The service account gapps-reports@system.gserviceaccount.com must have editor role on the BigQuery project

## Create Google Cloud Storage bucket

### Using Google Cloud Console

1. Go to the [Google Cloud Console](https://console.cloud.google.com/).
2. Select your project or create a new one.
3. In the navigation menu, go to **Cloud Storage \> Buckets**.
4. Click **Create bucket**.
5. Provide the following configuration details:

   | Setting | Value |
   |---|---|
   | **Name your bucket** | Enter a globally unique name (for example, `gmail-logs-export`) |
   | **Location type** | Choose based on your needs (Region, Dual-region, Multi-region) |
   | **Location** | Select the location (for example, `us-central1`) |
   | **Storage class** | Standard (recommended for frequently accessed logs) |
   | **Access control** | Uniform (recommended) |
   | **Protection tools** | Optional: Enable object versioning or retention policy |

6. Click **Create**.

### Using gcloud command-line tool

- Alternatively, create a bucket using the `gcloud` command:

      gcloud storage buckets create gs://gmail-logs-export \
          --location=us-central1 \
          --default-storage-class=STANDARD

  - Replace:
    - `gmail-logs-export`: Your desired bucket name (globally unique).
    - `us-central1`: Your preferred region (for example, `us-central1`, `europe-west1`).

## Configure Google Workspace to export logs to BigQuery

Google Workspace logs, including Gmail logs, are exported to BigQuery through the unified Workspace logs and reports feature.

### Enable BigQuery export for Workspace logs

1. Sign in with a super administrator account to the [Google Admin console](https://admin.google.com/).
2. Go to **Menu \> Reporting \> Data integrations**.
3. Point to the **BigQuery Export** card and click **Edit**.
4. Check the **Enable Google Workspace data export to Google BigQuery** box.
5. Under **BigQuery project ID**, select the project where you want to store the logs.
6. Under **New dataset within project** , enter the name of the dataset to use for storing the logs (for example, `workspace_logs`).
7. Optional: Check the **Restrict the dataset to a specific geographic location** box and select the location from the menu.
8. Click **Save**.

   > [!NOTE]
   > **Note:** The service account gapps-reports@system.gserviceaccount.com is automatically added as an editor to the dataset. Don't remove this service account.

After enabling the export, activity log events are typically available within 10 minutes. The data is exported to tables named `activity_YYYYMMDD` in the specified dataset.

## Create scheduled query to export from BigQuery to GCS

To automatically export Gmail logs from BigQuery to Cloud Storage on a recurring schedule, create a scheduled query using the EXPORT DATA statement.

### Using BigQuery Console

1. In the [Google Cloud Console](https://console.cloud.google.com/), go to **BigQuery**.
2. In the left navigation, click **Scheduled queries**.
3. Click **Create scheduled query**.
4. In the **Query editor**, enter the following SQL:

       EXPORT DATA OPTIONS(
           uri='gs://gmail-logs-export/gmail-logs/*.json',
           format='JSON',
           overwrite=false
       ) AS
       SELECT 
           *
       FROM 
           `PROJECT_ID.workspace_logs.activity_*`
       WHERE 
           record_type = 'gmail'
           AND _TABLE_SUFFIX = FORMAT_DATE('%Y%m%d', DATE_SUB(CURRENT_DATE(), INTERVAL 1 DAY))

   - Replace:
     - `gmail-logs-export`: Your GCS bucket name
     - `PROJECT_ID`: Your GCP project ID
     - `workspace_logs`: Your BigQuery dataset name
5. In the **Schedule options** section:

   - **Repeat** : Select **Hours**.
   - **Every** : Enter `1`.
   - **Start date and run time**: Select current date and time.
6. In the **Destination for query results** section:

   - **Dataset**: Select a dataset for query metadata (not the exported data).
7. Click **Save**.

### Using bq command-line tool

- Alternatively, create a scheduled query using the `bq` command:

      bq mk \
          --transfer_config \
          --project_id=PROJECT_ID \
          --data_source=scheduled_query \
          --display_name='Gmail Logs Export to GCS' \
          --schedule='every 1 hours' \
          --params='{
              "query":"EXPORT DATA OPTIONS(uri=\"gs://gmail-logs-export/gmail-logs/*.json\", format=\"JSON\", overwrite=false) AS SELECT * FROM `PROJECT_ID.workspace_logs.activity_*` WHERE record_type = \"gmail\" AND _TABLE_SUFFIX = FORMAT_DATE(\"%Y%m%d\", DATE_SUB(CURRENT_DATE(), INTERVAL 1 DAY))",
              "destination_table_name_template":"gmail_export_metadata",
              "write_disposition":"WRITE_TRUNCATE"
          }'

  - Replace:
    - `PROJECT_ID`: Your GCP project ID
    - `gmail-logs-export`: Your GCS bucket name
    - `workspace_logs`: Your BigQuery dataset name

  > [!NOTE]
  > **Note:** The EXPORT DATA statement exports data in newline-delimited JSON format. Each log entry is written as a single JSON object on one line.

## Retrieve the Google SecOps service account

Google SecOps uses a unique service account to read data from your GCS bucket. You must grant this service account access to your bucket.

### Configure a feed in Google SecOps to ingest Gmail logs

1. Go to **Settings \> Data Management \> Feeds** . If you are on a SIEM-only platform, go to **Settings \> Feeds**.
2. Click **Add New Feed**.
3. Click **Configure a single feed**.
4. In the **Feed name** field, enter a name for the feed (for example, `Gmail Logs`).
5. Select **Google Cloud Storage V2** as the **Source type**.
6. Select **GMAIL Logs** as the **Log type**.

7. Click **Get Service Account**. A unique service account email is displayed, for example:

       chronicle-12345678@chronicle-gcp-prod.iam.gserviceaccount.com

8. Copy this email address for use in the next step.

   > [!NOTE]
   > **Note:** Each Google SecOps instance has a unique service account. Don't use service accounts from other documentation or examples.

9. Click **Next**.

10. Specify values for the following input parameters:

    - **Storage bucket URL**: Enter the GCS bucket URI with the prefix path:

           gs://gmail-logs-export/gmail-logs/

      - Replace:
        - `gmail-logs-export`: Your GCS bucket name
        - `gmail-logs`: The prefix/folder path where logs are stored

    > [!NOTE]
    > **Note:** Always include the trailing slash (/) at the end of the URI.

    - **Source deletion option**: Select the deletion option according to your preference:

      - **Never**: Never deletes any files after transfers (recommended for testing).
      - **Delete transferred files**: Deletes files after successful transfer.
      - **Delete transferred files and empty directories**: Deletes files and empty directories after successful transfer.

        > [!NOTE]
        > **Note:** If you select a deletion option, the service account must have **Storage Object Admin** role instead of Storage Object Viewer. Update IAM permissions accordingly.

    - **Maximum File Age**: Include files modified in the last number of days. Default is 180 days.

    - **Asset namespace** : The [asset namespace](https://docs.cloud.google.com/chronicle/docs/investigation/asset-namespaces).

    - **Ingestion labels**: The label to be applied to the events from this feed.

11. Click **Next**.

12. Review your new feed configuration in the **Finalize** screen, and then click **Submit**.

## Grant IAM permissions to the Google SecOps service account

The Google SecOps service account needs **Storage Object Viewer** role on your GCS bucket.

### Using Google Cloud Console

1. Go to **Cloud Storage \> Buckets**.
2. Click your bucket name.
3. Go to the **Permissions** tab.
4. Click **Grant access**.
5. Provide the following configuration details:
   - **Add principals**: Paste the Google SecOps service account email.
   - **Assign roles** : Select **Storage Object Viewer**.
6. Click **Save**.

### Using gcloud command-line tool

- Alternatively, grant permissions using the `gcloud` command:

      gcloud storage buckets add-iam-policy-binding gs://gmail-logs-export \
          --member="serviceAccount:SECOPS_SERVICE_ACCOUNT_EMAIL" \
          --role="roles/storage.objectViewer"

  - Replace:
    - `gmail-logs-export`: Your bucket name.
    - `SECOPS_SERVICE_ACCOUNT_EMAIL`: The Google SecOps service account email.

### Using gsutil command-line tool (legacy)

- Grant the SecOps service account Object Viewer access to the Gmail logs export bucket.

      gsutil iam ch serviceAccount:SECOPS_SERVICE_ACCOUNT_EMAIL:objectViewer \
          gs://gmail-logs-export

### Verify permissions

- To verify the permissions were granted correctly:

      gcloud storage buckets get-iam-policy gs://gmail-logs-export \
          --flatten="bindings[].members" \
          --filter="bindings.role:roles/storage.objectViewer"

You should see the Google SecOps service account email in the output.

> [!NOTE]
> **Note:** If you plan to use the deletion option "Delete transferred files" or "Delete transferred files and empty directories," grant **Storage Object Admin** role instead of Storage Object Viewer.

## UDM mapping table

| Log Field | UDM Mapping | Logic |
|---|---|---|
| attachment.file_extension_type, attachment.sha256, attachment.file_name, domains | about | Merged with attachment details and link domains |
| message_info.post_delivery_info.action_type, gmail.message_info.post_delivery_info.action_type, event_info.mail_event_type, gmail.event_info.mail_event_type, tok.product_bucket, tok.scope_name, record_type, token.client_type, message_info.num_message_attachments, gmail.message_info.num_message_attachments | additional.fields | Merged with various additional metadata fields |
| action_type, description, rule_id | metadata.description | Set to specific descriptions based on action_type values, or from description, or "Objectionable content" if rule_id == 7 |
|   | metadata.event_type | Set to "EMAIL_TRANSACTION" |
| event_name | metadata.product_event_type | Value copied directly |
|   | metadata.product_name | Set to "GMAIL" |
|   | metadata.vendor_name | Set to "Google" |
| __incoming_message, __outcoming_message | network.direction | Set to "INBOUND" if incoming message detected, "OUTBOUND" if outcoming |
| destination.address | network.email.cc | Merged from destination addresses where index \> 0 |
| message_info.source.from_header_address, message_info.source.address | network.email.from | Value from from_header_address if not empty, else from source.address |
| message_info.rfc2822_message_id, gmail.message_info.rfc2822_message_id | network.email.mail_id | Extracted from rfc2822_message_id, removing \< \>, using grok |
| message_info.subject, gmail.message_info.subject | network.email.subject | Value from message_info.subject if not empty, else gmail.message_info.subject |
| destination.address | network.email.to | Merged from first destination address |
|   | network.ip_protocol | Set to "TCP" |
| _payload_size | network.received_bytes | Set to _payload_size if incoming message |
| _payload_size | network.sent_bytes | Set to _payload_size if outcoming message |
| token.app_name, message_info.source.service, message_info.source.selector, gmail.message_info.source.service, gmail.message_info.source.selector | principal.application | Value from token.app_name if not empty, else concatenated from source.service and source.selector |
| message_info.connection_info.client_host_zone, gmail.message_info.connection_info.client_host_zone | principal.asset.hostname | Value from client_host_zone |
| client_ip | principal.asset.ip | Value copied directly |
| message_info.connection_info.client_host_zone, gmail.message_info.connection_info.client_host_zone | principal.hostname | Value from client_host_zone |
| client_ip | principal.ip | Value copied directly |
| message_info.connection_info.ip_geo_country, gmail.message_info.connection_info.ip_geo_country | principal.location.country_or_region | Value from ip_geo_country |
| email, source_address | principal.user.email_addresses | Merged from email_address and source_address |
| token.client_id | principal.user.group_identifiers | Value copied directly |
| message_info.source.from_header_displayname, gmail.message_info.source.from_header_displayname | principal.user.user_display_name | Value from from_header_displayname |
| source_address | principal.user.userid | Value copied directly |
| action | security_result.action | Value copied directly |
| category | security_result.category | Value copied directly |
| category_details | security_result.category_details | Value copied directly |
| message_info.connection_info.smtp_response_reason, gmail.message_info.connection_info.smtp_response_reason, rule_description, reason | security_result.description | Set to smtp response reason, or rule description, or classification reason |
| stringMatch.predefined_detector_name, stringMatch.matched_string, stringMatch.match_expression, stringMatch.source, stringMatch.type | security_result.detection_fields | Merged with detection field objects |
| rule_id | security_result.rule_id | Value copied directly |
| rule_name | security_result.rule_name | Value copied directly |
| _err_summary, rule_id, description | security_result.summary | Set to error summary, or rule-specific summary, or description |
| _target_host | target.administrative_domain | Value copied directly |
| message_info.destination.0.service, message_info.destination.0.selector, gmail.message_info.destination.0.service, gmail.message_info.destination.0.selector | target.application | Concatenated from service and selector |
| `gmail.message_info.link_domain` | `about.labels` | Mapped from changelog |
| `gmail.message_info.description` | `metadata.description` | Mapped from changelog |
| `gmail.message_info.attachment.sha256` | `about.file.sha256` | Mapped from changelog |
| `gmail.message_info.destination.address` | `network.email.to` | Mapped from changelog |
| `gmail.message_info.attachment.file_name` | `about.file.full_path` | Mapped from changelog |
| `gmail.message_info.source.from_header_address` | `network.email.from` | Mapped from changelog |
| `gmail.message_info.attachment.file_extension_type` | `about.file.mime_type` | Mapped from changelog |
| `gmail.message_info.triggered_rule_info.rule_type` | `security_result.rule_id` | Mapped from changelog |
| `gmail.message_info.triggered_rule_info.rule_name` | `security_result.rule_name` | Mapped from changelog |
| `gmail.message_info.source.service", "gmail.message_info.source.selector` | `principal.application` | Mapped from changelog |
| `gmail.message_info.payload_size` | `network.sent_bytes` | Mapped from changelog |
| `gmail.message_info.payload_size` | `network.received_bytes` | Mapped from changelog |
| `gmail.message_info.destination.0.service", "gmail.message_info.destination.0.selector` | `target.application` | Mapped from changelog |
| `gmail.message_info.post_delivery_info.action_type", "gmail.message_info.num_message_attachments", and "gmail.event_info.mail_event_type` | `additional.fields` | Mapped from changelog |
| `event_info.success` | `security_result.action` | Mapped from changelog |
| `attachment.file_name` | `about.file.full_path` | Mapped from changelog |
| `message_info.post_delivery_info.action_type", "message_info.post_delivery_info.interaction.link_url", "event_info.mail_event_type` | `additional.fields` | Mapped from changelog |

## Change Log

[View the Change Log for this parser](https://docs.cloud.google.com/chronicle/docs/ingestion/parser-list/gmail-logs-changelog)

**Need more help?** [Get answers from Community members and Google SecOps professionals.](https://security.googlecloudcommunity.com/)