# Collect Virtru Email Encryption logs

Supported in:

> [!NOTE]
> **Note:** This feature is covered by [Pre-GA Offerings Terms](https://chronicle.security/legal/service-terms/) of the Google Security Operations Service Specific Terms. Pre-GA features might have limited support, and changes to pre-GA features might not be compatible with other pre-GA versions. For more information, see the [Google SecOps Technical Support Service guidelines](https://chronicle.security/legal/technical-support-services-guidelines/) and the [Google SecOps Service Specific Terms](https://chronicle.security/legal/service-terms/).  
> If this feature isn't visible in your environment, see [Manage preview features](https://docs.cloud.google.com/chronicle/docs/secops/preview-features-manage) or contact your system administrator.

This document explains how to ingest Virtru Email Encryption logs to Google Security Operations using Google Cloud Storage V2.

Virtru Email Encryption is an end-to-end email encryption platform with access control, data loss prevention, and audit logging for enterprise communications. A Cloud Run function polls the Virtru Audit API on a schedule, writes logs to a GCS bucket in NDJSON format, and Google SecOps ingests them through a GCS V2 feed.

## Before you begin

Make sure that you have the following prerequisites:

- A Google SecOps instance
- A GCP project with Cloud Storage API enabled
- Permissions to create and manage GCS buckets
- Permissions to manage IAM policies on GCS buckets
- Permissions to create Cloud Run services, Pub/Sub topics, and Cloud Scheduler jobs
- Privileged access to Virtru Dashboard with administrator permissions

## Create Google Cloud Storage bucket

1. Go to the [Google Cloud Console](https://console.cloud.google.com/).
2. Select your project or create a new one.
3. In the navigation menu, go to **Cloud Storage \> Buckets**.
4. Click **Create bucket**.
5. Provide the following configuration details:

   | Setting | Value |
   |---|---|
   | **Name your bucket** | Enter a globally unique name (for example, `virtru-email-encryption-logs`) |
   | **Location type** | Choose based on your needs (Region, Dual-region, Multi-region) |
   | **Location** | Select the location (for example, `us-central1`) |
   | **Storage class** | Standard (recommended for frequently accessed logs) |
   | **Access control** | Uniform (recommended) |
   | **Protection tools** | Optional: Enable object versioning or retention policy |

6. Click **Create**.

## Collect Virtru Email Encryption API credentials

### Generate API token

1. Sign in to the [Virtru Dashboard](https://dashboard.virtru.com/) as an administrator.
2. Go to **Settings \> API Keys**.
3. Click **Generate API Key**.
4. Enter a label for the key (for example, `Google Security Operations Integration`).
5. Copy and save the following details in a secure location:
   - **API Token**: The API bearer token for authentication.
6. Click **Create**.

### Verify permissions

To verify the account has the required permissions:

1. Sign in to the [Virtru Dashboard](https://dashboard.virtru.com/).
2. Go to **Settings \> API Keys**.
3. If you can see the **API Keys** section and generate keys, you have the required administrator permissions.
4. If you cannot see this option, contact your Virtru administrator to grant administrator access.

### Test API access

- Test your credentials before proceeding with the integration:

      # Replace with your actual credentials
      API_TOKEN="your-api-token"

      # Test API access
      curl -v -H "Authorization: Bearer ${API_TOKEN}" \
        "https://api.virtru.com/audit/api/messages?start=2024-01-01T00:00:00Z&end=2024-01-02T00:00:00Z&limit=1"

## Create service account for Cloud Run function

The Cloud Run function needs a service account with permissions to write to GCS bucket and be invoked by Pub/Sub.

### Create service account

1. In the **GCP Console** , go to **IAM \& Admin \> Service Accounts**.
2. Click **Create Service Account**.
3. Provide the following configuration details:
   - **Service account name** : Enter `virtru-logs-collector-sa`.
   - **Service account description** : Enter `Service account for Cloud Run function to collect Virtru Email Encryption logs`.
4. Click **Create and Continue**.
5. In the **Grant this service account access to project** section, add the following roles:
   1. Click **Select a role**.
   2. Search for and select **Storage Object Admin**.
   3. Click **+ Add another role**.
   4. Search for and select **Cloud Run Invoker**.
   5. Click **+ Add another role**.
   6. Search for and select **Cloud Functions Invoker**.
6. Click **Continue**.
7. Click **Done**.

These roles are required for:

- **Storage Object Admin**: Write logs to GCS bucket and manage state files
- **Cloud Run Invoker**: Allow Pub/Sub to invoke the function
- **Cloud Functions Invoker**: Allow function invocation

### Grant IAM permissions on GCS bucket

Grant the service account write permissions on the GCS bucket:

1. Go to **Cloud Storage \> Buckets**.
2. Click your bucket name (for example, `virtru-email-encryption-logs`).
3. Go to the **Permissions** tab.
4. Click **Grant access**.
5. Provide the following configuration details:
   - **Add principals** : Enter the service account email (for example, `virtru-logs-collector-sa@your-project.iam.gserviceaccount.com`).
   - **Assign roles** : Select **Storage Object Admin**.
6. Click **Save**.

## Create Pub/Sub topic

Create a Pub/Sub topic that Cloud Scheduler will publish to and the Cloud Run function will subscribe to.

1. In the **GCP Console** , go to **Pub/Sub \> Topics**.
2. Click **Create topic**.
3. Provide the following configuration details:
   - **Topic ID** : Enter `virtru-logs-trigger`.
   - Leave other settings as default.
4. Click **Create**.

## Create Cloud Run function to collect logs

The Cloud Run function will be triggered by Pub/Sub messages from Cloud Scheduler to fetch logs from Virtru Audit API and write them to GCS.

1. In the **GCP Console** , go to **Cloud Run**.
2. Click **Create service**.
3. Select **Function** (use an inline editor to create a function).
4. In the **Configure** section, provide the following configuration details:

   | Setting | Value |
   |---|---|
   | **Service name** | `virtru-logs-collector` |
   | **Region** | Select region matching your GCS bucket (for example, `us-central1`) |
   | **Runtime** | Select **Python 3.12** or later |

5. In the **Trigger (optional)** section:

   1. Click **+ Add trigger**.
   2. Select **Cloud Pub/Sub**.
   3. In **Select a Cloud Pub/Sub topic** , choose the topic `virtru-logs-trigger`.
   4. Click **Save**.
6. In the **Authentication** section:

   1. Select **Require authentication**.
   2. Check **Identity and Access Management (IAM)**.

   > [!NOTE]
   > **Note:** Pub/Sub will automatically handle authentication when invoking the function.

7. Scroll down and expand **Containers, Networking, Security**.

8. Go to the **Security** tab:

   - **Service account** : Select the service account `virtru-logs-collector-sa`.
9. Go to the **Containers** tab:

   1. Click **Variables \& Secrets**.
   2. Click **+ Add variable** for each environment variable:

   | Variable Name | Example Value | Description |
   |---|---|---|
   | `GCS_BUCKET` | `virtru-email-encryption-logs` | GCS bucket name |
   | `GCS_PREFIX` | `virtru` | Prefix for log files |
   | `STATE_KEY` | `virtru/state.json` | State file path |
   | `VIRTRU_API_TOKEN` | `your-api-token` | Virtru API bearer token |
   | `MAX_RECORDS` | `10000` | Max records per run |
   | `PAGE_SIZE` | `500` | Records per page |
   | `LOOKBACK_HOURS` | `24` | Initial lookback period |

10. Scroll down in the **Variables \& Secrets** tab to **Requests**:

    - **Request timeout** : Enter `600` seconds (10 minutes).
11. Go to the **Settings** tab in **Containers**:

    - In the **Resources** section:
      - **Memory** : Select **512 MiB** or higher.
      - **CPU** : Select **1**.
    - Click **Done**.
12. Scroll down to **Execution environment**:

    - Select **Default** (recommended).
13. In the **Revision scaling** section:

    - **Minimum number of instances** : Enter `0`.
    - **Maximum number of instances** : Enter `100` (or adjust based on expected load).
14. Click **Create**.

15. Wait for the service to be created (1-2 minutes).

16. After the service is created, the **inline code editor** will open automatically.

### Add function code

1. Enter **main** in **Function entry point**.
2. In the inline code editor, create two files:

   - First file: **main.py:**

       import functions_framework
       from google.cloud import https://docs.cloud.google.com/python/docs/reference/storage/latest
       import json
       import os
       import urllib3
       from datetime import datetime, timezone, timedelta
       import time

       # Initialize HTTP client with timeouts
       http = urllib3.PoolManager(
           timeout=urllib3.Timeout(connect=5.0, read=30.0),
           retries=False,
       )

       # Initialize Storage client
       storage_client = https://docs.cloud.google.com/python/docs/reference/storage/latest.https://docs.cloud.google.com/python/docs/reference/storage/latest/google.cloud.storage.client.Client.html()

       # Environment variables
       GCS_BUCKET = os.environ.get('GCS_BUCKET')
       GCS_PREFIX = os.environ.get('GCS_PREFIX', 'virtru')
       STATE_KEY = os.environ.get('STATE_KEY', 'virtru/state.json')
       API_TOKEN = os.environ.get('VIRTRU_API_TOKEN')
       MAX_RECORDS = int(os.environ.get('MAX_RECORDS', '10000'))
       PAGE_SIZE = int(os.environ.get('PAGE_SIZE', '500'))
       LOOKBACK_HOURS = int(os.environ.get('LOOKBACK_HOURS', '24'))

       API_BASE = 'https://api.virtru.com'

       def parse_datetime(value: str) -> datetime:
           """Parse ISO datetime string to datetime object."""
           if https://docs.cloud.google.com/python/docs/reference/storage/latest/google.cloud.storage.blob.ObjectCustomContextPayload.html#google_cloud_storage_blob_ObjectCustomContextPayload_value.endswith("Z"):
               value = value[:-1] + "+00:00"
           return datetime.fromisoformat(value)

       @functions_framework.cloud_event
       def main(cloud_event):
           """
           Cloud Run function triggered by Pub/Sub to fetch Virtru Email Encryption
           audit logs and write to GCS.

           Args:
               cloud_event: CloudEvent object containing Pub/Sub message
           """

           if not all([GCS_BUCKET, API_TOKEN]):
               print('Error: Missing required environment variables')
               return

           try:
               bucket = storage_client.https://docs.cloud.google.com/python/docs/reference/storage/latest/google.cloud.storage.client.Client.html#google_cloud_storage_client_Client_bucket(GCS_BUCKET)

               # Load state
               state = load_state(bucket, STATE_KEY)

               # Determine time window
               now = datetime.now(timezone.utc)
               last_time = None

               if isinstance(state, dict) and https://docs.cloud.google.com/python/docs/reference/storage/latest/google.cloud.storage.hmac_key.HMACKeyMetadata.html#google_cloud_storage_hmac_key_HMACKeyMetadata_state.get("last_event_time"):
                   try:
                       last_time = parse_datetime(state["last_event_time"])
                       last_time = last_time - timedelta(minutes=2)
                   except Exception as e:
                       print(f"Warning: Could not parse last_event_time: {e}")

               if last_time is None:
                   last_time = now - timedelta(hours=LOOKBACK_HOURS)

               print(f"Fetching logs from {last_time.isoformat()} to {now.isoformat()}")

               # Fetch logs
               records, newest_event_time = fetch_logs(
                   start_time=last_time,
                   end_time=now,
                   page_size=PAGE_SIZE,
                   max_records=MAX_RECORDS,
               )

               if not records:
                   print("No new log records found.")
                   save_state(bucket, STATE_KEY, now.isoformat())
                   return

               # Write to GCS as NDJSON
               timestamp = now.strftime('%Y%m%d_%H%M%S')
               object_key = f"{GCS_PREFIX}/logs_{timestamp}.ndjson"
               blob = bucket.blob(object_key)

               ndjson = '\n'.join([json.dumps(record, ensure_ascii=False) for record in records]) + '\n'
               blob.https://docs.cloud.google.com/python/docs/reference/storage/latest/google.cloud.storage.blob.Blob.html#google_cloud_storage_blob_Blob_upload_from_string(ndjson, content_type='application/x-ndjson')

               print(f"Wrote {len(records)} records to gs://{GCS_BUCKET}/{object_key}")

               # Update state with newest event time
               if newest_event_time:
                   save_state(bucket, STATE_KEY, newest_event_time)
               else:
                   save_state(bucket, STATE_KEY, now.isoformat())

               print(f"Successfully processed {len(records)} records")

           except Exception as e:
               print(f'Error processing logs: {str(e)}')
               raise

       def load_state(bucket, key):
           """Load state from GCS."""
           try:
               blob = bucket.blob(key)
               if blob.exists():
                   state_data = blob.https://docs.cloud.google.com/python/docs/reference/storage/latest/google.cloud.storage.blob.Blob.html#google_cloud_storage_blob_Blob_download_as_text()
                   return json.loads(state_data)
           except Exception as e:
               print(f"Warning: Could not load state: {e}")

           return {}

       def save_state(bucket, key, last_event_time_iso: str):
           """Save the last event timestamp to GCS state file."""
           try:
               state = {'last_event_time': last_event_time_iso}
               blob = bucket.blob(key)
               blob.https://docs.cloud.google.com/python/docs/reference/storage/latest/google.cloud.storage.blob.Blob.html#google_cloud_storage_blob_Blob_upload_from_string(
                   json.dumps(state, indent=2),
                   content_type='application/json'
               )
               print(f"Saved state: last_event_time={last_event_time_iso}")
           except Exception as e:
               print(f"Warning: Could not save state: {e}")

       def fetch_logs(start_time: datetime, end_time: datetime, page_size: int, max_records: int):
           """
           Fetch audit logs from Virtru API with pagination and rate limiting.

           Args:
               start_time: Start time for log query
               end_time: End time for log query
               page_size: Number of records per page
               max_records: Maximum total records to fetch

           Returns:
               Tuple of (records list, newest_event_time ISO string)
           """
           endpoint = f"{API_BASE}/audit/api/messages"

           headers = {
               'Authorization': f'Bearer {API_TOKEN}',
               'Accept': 'application/json',
               'User-Agent': 'GoogleSecOps-VirtruCollector/1.0'
           }

           records = []
           newest_time = None
           page_num = 0
           backoff = 1.0
           offset = 0

           while True:
               page_num += 1

               if len(records) >= max_records:
                   print(f"Reached max_records limit ({max_records})")
                   break

               start_iso = start_time.strftime('%Y-%m-%dT%H:%M:%SZ')
               end_iso = end_time.strftime('%Y-%m-%dT%H:%M:%SZ')
               current_limit = min(page_size, max_records - len(records))

               url = f"{endpoint}?start={start_iso}&end={end_iso}&limit={current_limit}&offset={offset}"

               try:
                   response = http.request('GET', url, headers=headers)

                   # Handle rate limiting with exponential backoff
                   if response.status == 429:
                       retry_after = int(response.headers.get('Retry-After', str(int(backoff))))
                       print(f"Rate limited (429). Retrying after {retry_after}s...")
                       time.sleep(retry_after)
                       backoff = min(backoff * 2, 30.0)
                       continue

                   backoff = 1.0

                   if response.status != 200:
                       print(f"HTTP Error: {response.status}")
                       response_text = response.data.decode('utf-8')
                       print(f"Response body: {response_text}")
                       return [], None

                   data = json.loads(response.data.decode('utf-8'))

                   page_results = data if isinstance(data, list) else data.get('messages', data.get('data', []))

                   if not page_results:
                       print(f"No more results (empty page)")
                       break

                   print(f"Page {page_num}: Retrieved {len(page_results)} events")
                   records.extend(page_results)

                   # Track newest event time
                   for event in page_results:
                       try:
                           event_time = event.get('timestamp') or event.get('date') or event.get('created')
                           if event_time:
                               if newest_time is None or parse_datetime(event_time) > parse_datetime(newest_time):
                                   newest_time = event_time
                       except Exception as e:
                           print(f"Warning: Could not parse event time: {e}")

                   # Check for more results
                   if len(page_results) < page_size:
                       print(f"Reached last page (size={len(page_results)} < limit={page_size})")
                       break

                   offset += len(page_results)

               except Exception as e:
                   print(f"Error fetching logs: {e}")
                   return [], None

           print(f"Retrieved {len(records)} total records from {page_num} pages")
           return records, newest_time

   - Second file: **requirements.txt:**

       functions-framework==3.*
       google-cloud-storage==2.*
       urllib3>=2.0.0

3. Click **Deploy** to save and deploy the function.

4. Wait for deployment to complete (2-3 minutes).

   > [!NOTE]
   > **Note:** The Pub/Sub trigger configuration automatically creates the necessary subscriptions and permissions.

## Create Cloud Scheduler job

Cloud Scheduler will publish messages to the Pub/Sub topic at regular intervals, triggering the Cloud Run function.

1. In the **GCP Console** , go to **Cloud Scheduler**.
2. Click **Create Job**.
3. Provide the following configuration details:

   | Setting | Value |
   |---|---|
   | **Name** | `virtru-logs-collector-hourly` |
   | **Region** | Select same region as Cloud Run function |
   | **Frequency** | `0 * * * *` (every hour, on the hour) |
   | **Timezone** | Select timezone (UTC recommended) |
   | **Target type** | Pub/Sub |
   | **Topic** | Select the topic `virtru-logs-trigger` |
   | **Message body** | `{}` (empty JSON object) |

4. Click **Create**.

### Schedule frequency options

Choose frequency based on log volume and latency requirements:

| Frequency | Cron Expression | Use Case |
|---|---|---|
| Every 5 minutes | `*/5 * * * *` | High-volume, low-latency |
| Every 15 minutes | `*/15 * * * *` | Medium volume |
| Every hour | `0 * * * *` | Standard (recommended) |
| Every 6 hours | `0 */6 * * *` | Low volume, batch processing |
| Daily | `0 0 * * *` | Historical data collection |

### Test the integration

1. In the **Cloud Scheduler** console, find your job (`virtru-logs-collector-hourly`).
2. Click **Force run** to trigger manually.
3. Wait a few seconds and go to **Cloud Run \> Services \> virtru-logs-collector \> Logs**.
4. Verify the function executed successfully. Look for:

       Fetching logs from YYYY-MM-DDTHH:MM:SS+00:00 to YYYY-MM-DDTHH:MM:SS+00:00
       Page 1: Retrieved X events
       Wrote X records to gs://virtru-email-encryption-logs/virtru/logs_YYYYMMDD_HHMMSS.ndjson
       Successfully processed X records

5. Check the GCS bucket (`virtru-email-encryption-logs`) to confirm logs were written.

If you see errors in the logs:

- **HTTP 401**: Check API token in environment variables
- **HTTP 403**: Verify account has administrator permissions in Virtru Dashboard
- **HTTP 429**: Rate limiting - function will automatically retry with backoff
- **Missing environment variables**: Check all required variables are set

## Configure a feed in Google SecOps to ingest Virtru Email Encryption logs

1. Go to **Settings \> Data Management \> Feeds** . If you are on a SIEM-only platform, go to **Settings \> Feeds**.
2. Click **Add New Feed**.
3. Click **Configure a single feed**.
4. In the **Feed name** field, enter a name for the feed (for example, `Virtru Email Encryption Logs`).
5. Select **Google Cloud Storage V2** as the **Source type**.
6. Select **Virtru Email Encryption** as the **Log type**.
7. Click **Get Service Account**. A unique service account email will be displayed, for example:

       chronicle-12345678@chronicle-gcp-prod.iam.gserviceaccount.com

8. Copy this email address. You will use it in the next step.

   > [!NOTE]
   > **Note:** Each Google SecOps instance has a unique service account. Do not use service accounts from other documentation or examples.

9. Click **Next**.

10. Specify values for the following input parameters:

    - **Storage bucket URL**: Enter the GCS bucket URI with the prefix path:

          gs://virtru-email-encryption-logs/virtru/

      - Replace:
        - `virtru-email-encryption-logs`: Your GCS bucket name.
        - `virtru/`: Prefix/folder path where logs are stored.

    > [!NOTE]
    > **Note:** Always include the trailing slash (/) at the end of the URI.

    - **Source deletion option**: Select the deletion option according to your preference:

      - **Never**: Never deletes any files after transfers (recommended for testing).
      - **Delete transferred files**: Deletes files after successful transfer.
      - **Delete transferred files and empty directories**: Deletes files and empty directories after successful transfer.

        > [!NOTE]
        > **Note:** If you select a deletion option, the service account must have **Storage Object Admin** role instead of **Storage Object Viewer**. Update IAM permissions accordingly.

    - **Maximum File Age**: Include files modified in the last number of days (default is 180 days).

    - **Asset namespace** : The [asset namespace](https://docs.cloud.google.com/chronicle/docs/investigation/asset-namespaces).

    - **Ingestion labels**: The label to be applied to the events from this feed.

11. Click **Next**.

12. Review your new feed configuration in the **Finalize** screen, and then click **Submit**.

## Grant IAM permissions to the Google SecOps service account

The Google SecOps service account needs **Storage Object Viewer** role on your GCS bucket.

1. Go to **Cloud Storage \> Buckets**.
2. Click your bucket name (`virtru-email-encryption-logs`).
3. Go to the **Permissions** tab.
4. Click **Grant access**.
5. Provide the following configuration details:
   - **Add principals**: Paste the Google SecOps service account email.
   - **Assign roles** : Select **Storage Object Viewer**.
6. Click **Save**.

   > [!NOTE]
   > **Note:** If you plan to use the deletion option "Delete transferred files" or "Delete transferred files and empty directories," grant **Storage Object Admin** role instead of **Storage Object Viewer**.

## UDM mapping table

| Log field | UDM mapping | Logic |
|---|---|---|
| `audit_RecordId` | `additional.fields` | Merged |
| `audit_RecordType` | `additional.fields` | Merged |
| `objid_label` | `additional.fields` | Merged |
| `objtype_label` | `additional.fields` | Merged |
| `orgId_label` | `additional.fields` | Merged |
| `platform_label` | `additional.fields` | Merged |
| `attributelabel.value` | `metadata.description` | Directly mapped |
| `desc` | `metadata.description` | Directly mapped |
| `timestamp` | `metadata.event_timestamp` | Parsed as `ISO8601` |
| `has_principal` | `metadata.event_type` | Mapped: `true` → `STATUS_UPDATE` |
| `has_user` | `metadata.event_type` | Mapped: `true` → `USER_UNCATEGORIZED` |
| `eventMetaData.auditDataType` | `metadata.product_event_type` | Directly mapped |
| `id` | `metadata.product_log_id` | Directly mapped |
| `object.name` | `network.email.subject` | Merged |
| `email` | `network.email.to` | Mapped: `^.+@.+$` → `email` |
| `clientInfo.userAgent` | `network.http.parsed_user_agent` | Renamed/mapped |
| `clientInfo.userAgent` | `network.http.user_agent` | Directly mapped |
| `prin_host` | `principal.asset.hostname` | Directly mapped |
| `clientInfo.requestIp` | `principal.asset.ip` | Merged |
| `prin_ip` | `principal.asset.ip` | Merged |
| `actorattributes.value` | `principal.group.email_addresses` | Merged |
| `prin_host` | `principal.hostname` | Directly mapped |
| `clientInfo.requestIp` | `principal.ip` | Merged |
| `prin_ip` | `principal.ip` | Merged |
| `pid` | `principal.process.pid` | Directly mapped |
| `actor_attri_label` | `principal.resource.attribute.labels` | Merged |
| `actor.id` | `principal.user.email_addresses` | Merged |
| `object.name` | `principal.user.email_addresses` | Merged |
| `actor.id` | `principal.user.user_display_name` | Directly mapped |
| `action_label` | `security_result.detection_fields` | Merged |
| `action_result_label` | `security_result.detection_fields` | Merged |
| `attributelabel.value` | `src.user.user_display_name` | Directly mapped |
| `object.name` | `target.file.full_path` | Directly mapped |
| `attributelabel.value` | `target.group.email_addresses` | Merged |
| `attrikey` | `target.group.email_addresses` | Mapped: `virtru:data:org:1e148cd2-139d-4d27-9891-98f7275550a2:group` → `attributelabel.value` |
| `attri_label` | `target.resource.attribute.labels` | Merged |
| `email` | `target.user.email_addresses` | Mapped: `^.+@.+$` → `email` |
| N/A | `metadata.event_type` | Constant: `EMAIL_TRANSACTION` |
| N/A | `metadata.product_name` | Constant: `virtru encrytion` |
| N/A | `metadata.vendor_name` | Constant: `virtru` |

## Change Log

[View the Change Log for this parser](https://docs.cloud.google.com/chronicle/docs/ingestion/parser-list/virtru-email-encryption-changelog)

**Need more help?** [Get answers from Community members and Google SecOps professionals.](https://security.googlecloudcommunity.com/)