# Collect WP Engine logs

Supported in:

> [!NOTE]
> **Note:** This feature is covered by [Pre-GA Offerings Terms](https://chronicle.security/legal/service-terms/) of the Google Security Operations Service Specific Terms. Pre-GA features might have limited support, and changes to pre-GA features might not be compatible with other pre-GA versions. For more information, see the [Google SecOps Technical Support Service guidelines](https://chronicle.security/legal/technical-support-services-guidelines/) and the [Google SecOps Service Specific Terms](https://chronicle.security/legal/service-terms/).  
> If this feature isn't visible in your environment, see [Manage preview features](https://docs.cloud.google.com/chronicle/docs/secops/preview-features-manage) or contact your system administrator.

This document explains how to ingest WP Engine logs to Google Security Operations using Google Cloud Storage V2.

WP Engine is a managed WordPress hosting platform that provides enterprise-grade hosting with built-in security, performance optimization, and CDN services. It generates access logs, error logs, and CDN event logs that can be collected via the WP Engine API.

## Before you begin

Make sure you have the following prerequisites:

- A Google SecOps instance
- A GCP project with Cloud Storage API enabled
- Permissions to create and manage GCS buckets
- Permissions to manage IAM policies on GCS buckets
- Permissions to create Cloud Run services, Pub/Sub topics, and Cloud Scheduler jobs
- Privileged access to the WP Engine User Portal with API access permissions
- A WP Engine account with API access enabled

## Create Google Cloud Storage bucket

1. Go to the [Google Cloud Console](https://console.cloud.google.com/).
2. Select your project or create a new one.
3. In the navigation menu, go to **Cloud Storage \> Buckets**.
4. Click **Create bucket**.
5. Provide the following configuration details:

   | Setting | Value |
   |---|---|
   | **Name your bucket** | Enter a globally unique name (for example, `wpengine-logs`) |
   | **Location type** | Choose based on your needs (Region, Dual-region, Multi-region) |
   | **Location** | Select the location (for example, `us-central1`) |
   | **Storage class** | Standard (recommended for frequently accessed logs) |
   | **Access control** | Uniform (recommended) |
   | **Protection tools** | Optional: Enable object versioning or retention policy |

6. Click **Create**.

## Collect WP Engine API credentials

### Generate API credentials

1. Sign in to the [WP Engine User Portal](https://my.wpengine.com).
2. Click your profile name, then go to **Profile \> API Access**.
3. Click **Generate Credentials**.
4. Copy and save the following details in a secure location:

   - **API Username**: The generated API username
   - **API Password**: The generated API password (shown only once)

   > [!NOTE]
   > **Note:** The API password is displayed only once at the time of creation. If you lose it, you must generate new credentials.

### Get install name

1. Sign in to the [WP Engine User Portal](https://my.wpengine.com).
2. Go to **Sites** in the navigation menu.
3. Click on the site you want to collect logs from.
4. Note the **Install name** displayed on the site overview page. Each environment (Production, Staging, Development) has a separate install name.

### Test API access

- Test your credentials before proceeding with the integration:

      # Replace with your actual credentials
      WPE_USER="your-api-username"
      WPE_PASSWORD="your-api-password"

      # Test API access - list installs
      curl -v -u "${WPE_USER}:${WPE_PASSWORD}" "https://api.wpengineapi.com/v1/installs"

## Create service account for Cloud Run function

The Cloud Run function needs a service account with permissions to write to GCS bucket and be invoked by Pub/Sub.

### Create service account

1. In the **GCP Console** , go to **IAM \& Admin \> Service Accounts**.
2. Click **Create Service Account**.
3. Provide the following configuration details:
   - **Service account name** : Enter `wpengine-logs-collector-sa`
   - **Service account description** : Enter `Service account for Cloud Run function to collect WP Engine logs`
4. Click **Create and Continue**.
5. In the **Grant this service account access to project** section, add the following roles:
   1. Click **Select a role**.
   2. Search for and select **Storage Object Admin**.
   3. Click **+ Add another role**.
   4. Search for and select **Cloud Run Invoker**.
   5. Click **+ Add another role**.
   6. Search for and select **Cloud Functions Invoker**.
6. Click **Continue**.
7. Click **Done**.

These roles are required for:

- **Storage Object Admin**: Write logs to GCS bucket and manage state files
- **Cloud Run Invoker**: Allow Pub/Sub to invoke the function
- **Cloud Functions Invoker**: Allow function invocation

### Grant IAM permissions on GCS bucket

Grant the service account write permissions on the GCS bucket:

1. Go to **Cloud Storage \> Buckets**.
2. Click on your bucket name (for example, `wpengine-logs`).
3. Go to the **Permissions** tab.
4. Click **Grant access**.
5. Provide the following configuration details:
   - **Add principals** : Enter the service account email (for example, `wpengine-logs-collector-sa@PROJECT_ID.iam.gserviceaccount.com`)
   - **Assign roles** : Select **Storage Object Admin**
6. Click **Save**.

## Create Pub/Sub topic

Create a Pub/Sub topic that Cloud Scheduler will publish to and the Cloud Run function will subscribe to.

1. In the **GCP Console** , go to **Pub/Sub \> Topics**.
2. Click **Create topic**.
3. Provide the following configuration details:
   - **Topic ID** : Enter `wpengine-logs-trigger`
   - Leave other settings as default
4. Click **Create**.

## Create Cloud Run function to collect logs

The Cloud Run function will be triggered by Pub/Sub messages from Cloud Scheduler to fetch logs from the WP Engine API and write them to GCS.

1. In the **GCP Console** , go to **Cloud Run**.
2. Click **Create service**.
3. Select **Function** (use an inline editor to create a function).
4. In the **Configure** section, provide the following configuration details:

   | Setting | Value |
   |---|---|
   | **Service name** | `wpengine-logs-collector` |
   | **Region** | Select region matching your GCS bucket (for example, `us-central1`) |
   | **Runtime** | Select **Python 3.12** or later |

5. In the **Trigger (optional)** section:

   1. Click **+ Add trigger**.
   2. Select **Cloud Pub/Sub**.
   3. In **Select a Cloud Pub/Sub topic** , choose the topic `wpengine-logs-trigger`.
   4. Click **Save**.
6. In the **Authentication** section:

   1. Select **Require authentication**.
   2. Check **Identity and Access Management (IAM)**.

   > [!NOTE]
   > **Note:** Pub/Sub will automatically handle authentication when invoking the function.

7. Scroll down and expand **Containers, Networking, Security**.

8. Go to the **Security** tab:

   - **Service account** : Select the service account `wpengine-logs-collector-sa`.
9. Go to the **Containers** tab:

   1. Click **Variables \& Secrets**.
   2. Click **+ Add variable** for each environment variable:

   | Variable Name | Example Value | Description |
   |---|---|---|
   | `GCS_BUCKET` | `wpengine-logs` | GCS bucket name |
   | `GCS_PREFIX` | `wpengine` | Prefix for log files |
   | `STATE_KEY` | `wpengine/state.json` | State file path |
   | `WPE_API_USER` | `your-api-username` | WP Engine API username |
   | `WPE_API_PASSWORD` | `your-api-password` | WP Engine API password |
   | `WPE_INSTALL_ID` | `myinstall` | WP Engine install name |
   | `MAX_RECORDS` | `5000` | Max records per run |
   | `PAGE_SIZE` | `100` | Records per page |
   | `LOOKBACK_HOURS` | `24` | Initial lookback period |

10. In the **Variables \& Secrets** section, scroll down to **Requests**:

    - **Request timeout** : Enter `600` seconds (10 minutes)
11. Go to the **Settings** tab:

    - In the **Resources** section:
      - **Memory** : Select **512 MiB** or higher
      - **CPU** : Select **1**
12. In the **Revision scaling** section:

    - **Minimum number of instances** : Enter `0`
    - **Maximum number of instances** : Enter `100` (or adjust based on expected load)
13. Click **Create**.

14. Wait for the service to be created (1-2 minutes).

15. After the service is created, the **inline code editor** will open automatically.

### Add function code

1. Enter **main** in the **Entry point** field.
2. In the inline code editor, create two files:

   - First file: **main.py:**

       import functions_framework
       from google.cloud import https://docs.cloud.google.com/python/docs/reference/storage/latest
       import json
       import os
       import urllib3
       from datetime import datetime, timezone, timedelta
       import time
       import base64

       # Initialize HTTP client with timeouts
       http = urllib3.PoolManager(
         timeout=urllib3.Timeout(connect=5.0, read=30.0),
         retries=False,
       )

       # Initialize Storage client
       storage_client = https://docs.cloud.google.com/python/docs/reference/storage/latest.https://docs.cloud.google.com/python/docs/reference/storage/latest/google.cloud.storage.client.Client.html()

       # Environment variables
       GCS_BUCKET = os.environ.get('GCS_BUCKET')
       GCS_PREFIX = os.environ.get('GCS_PREFIX', 'wpengine')
       STATE_KEY = os.environ.get('STATE_KEY', 'wpengine/state.json')
       WPE_API_USER = os.environ.get('WPE_API_USER')
       WPE_API_PASSWORD = os.environ.get('WPE_API_PASSWORD')
       WPE_INSTALL_ID = os.environ.get('WPE_INSTALL_ID')
       MAX_RECORDS = int(os.environ.get('MAX_RECORDS', '5000'))
       PAGE_SIZE = int(os.environ.get('PAGE_SIZE', '100'))
       LOOKBACK_HOURS = int(os.environ.get('LOOKBACK_HOURS', '24'))

       # WP Engine API base URL
       API_BASE = 'https://api.wpengineapi.com/v1'

       # Log types to fetch
       LOG_TYPES = ['access', 'error']

       def get_auth_header():
         """Generate HTTP Basic auth header for WP Engine API."""
         credentials = f"{WPE_API_USER}:{WPE_API_PASSWORD}"
         encoded = base64.b64encode(credentials.encode('utf-8')).decode('utf-8')
         return f"Basic {encoded}"

       @functions_framework.cloud_event
       def main(cloud_event):
         """
         Cloud Run function triggered by Pub/Sub to fetch WP Engine
         logs and write to GCS.

         Args:
           cloud_event: CloudEvent object containing Pub/Sub message
         """

         if not all([GCS_BUCKET, WPE_API_USER, WPE_API_PASSWORD, WPE_INSTALL_ID]):
           print('Error: Missing required environment variables')
           return

         try:
           bucket = storage_client.https://docs.cloud.google.com/python/docs/reference/storage/latest/google.cloud.storage.client.Client.html#google_cloud_storage_client_Client_bucket(GCS_BUCKET)

           # Load state
           state = load_state(bucket, STATE_KEY)

           # Determine time window
           now = datetime.now(timezone.utc)
           last_offsets = {}

           if isinstance(state, dict) and https://docs.cloud.google.com/python/docs/reference/storage/latest/google.cloud.storage.hmac_key.HMACKeyMetadata.html#google_cloud_storage_hmac_key_HMACKeyMetadata_state.get("last_offsets"):
             last_offsets = state["last_offsets"]

           print(f"Fetching logs for install: {WPE_INSTALL_ID}")

           auth_header = get_auth_header()
           all_records = []

           # Fetch both access and error log types
           for log_type in LOG_TYPES:
             last_offset = last_offsets.get(log_type, 0)

             records = fetch_logs(
               auth_header=auth_header,
               install_id=WPE_INSTALL_ID,
               log_type=log_type,
               start_offset=last_offset,
               page_size=PAGE_SIZE,
               max_records=MAX_RECORDS,
             )

             # Tag records with log type
             for record in records:
               record['_wpe_log_type'] = log_type

             all_records.extend(records)

             # Update offset for this log type
             if records:
               last_offsets[log_type] = last_offset + len(records)

             print(f"Fetched {len(records)} {log_type} log records")

           if not all_records:
             print("No new log records found.")
             save_state(bucket, STATE_KEY, last_offsets)
             return

           # Write to GCS as NDJSON
           timestamp = now.strftime('%Y%m%d_%H%M%S')
           object_key = f"{GCS_PREFIX}/logs_{timestamp}.ndjson"
           blob = bucket.blob(object_key)

           ndjson = '\n'.join([json.dumps(record, ensure_ascii=False) for record in all_records]) + '\n'
           blob.https://docs.cloud.google.com/python/docs/reference/storage/latest/google.cloud.storage.blob.Blob.html#google_cloud_storage_blob_Blob_upload_from_string(ndjson, content_type='application/x-ndjson')

           print(f"Wrote {len(all_records)} records to gs://{GCS_BUCKET}/{object_key}")

           # Update state
           save_state(bucket, STATE_KEY, last_offsets)

           print(f"Successfully processed {len(all_records)} records")

         except Exception as e:
           print(f'Error processing logs: {str(e)}')
           raise

       def load_state(bucket, key):
         """Load state from GCS."""
         try:
           blob = bucket.blob(key)
           if blob.exists():
             state_data = blob.https://docs.cloud.google.com/python/docs/reference/storage/latest/google.cloud.storage.blob.Blob.html#google_cloud_storage_blob_Blob_download_as_text()
             return json.loads(state_data)
         except Exception as e:
           print(f"Warning: Could not load state: {e}")

         return {}

       def save_state(bucket, key, last_offsets: dict):
         """Save the last offsets to GCS state file."""
         try:
           state = {'last_offsets': last_offsets}
           blob = bucket.blob(key)
           blob.https://docs.cloud.google.com/python/docs/reference/storage/latest/google.cloud.storage.blob.Blob.html#google_cloud_storage_blob_Blob_upload_from_string(
             json.dumps(state, indent=2),
             content_type='application/json'
           )
           print(f"Saved state: last_offsets={last_offsets}")
         except Exception as e:
           print(f"Warning: Could not save state: {e}")

       def fetch_logs(auth_header: str, install_id: str, log_type: str, start_offset: int, page_size: int, max_records: int):
         """
         Fetch logs from WP Engine API with offset-based pagination
         and rate limiting.

         Args:
           auth_header: HTTP Basic auth header
           install_id: WP Engine install name
           log_type: Log type to fetch (access or error)
           start_offset: Starting offset for pagination
           page_size: Number of records per page
           max_records: Maximum total records to fetch

         Returns:
           List of log records
         """
         headers = {
           'Authorization': auth_header,
           'Accept': 'application/json',
           'User-Agent': 'GoogleSecOps-WPEngineCollector/1.0'
         }

         records = []
         offset = start_offset
         page_num = 0
         backoff = 1.0

         while True:
           page_num += 1

           if len(records) >= max_records:
             print(f"Reached max_records limit ({max_records}) for {log_type}")
             break

           limit = min(page_size, max_records - len(records))
           url = f"{API_BASE}/installs/{install_id}/logs?type={log_type}&limit={limit}&offset={offset}"

           try:
             response = http.request('GET', url, headers=headers)

             # Handle rate limiting with exponential backoff
             if response.status == 429:
               retry_after = int(response.headers.get('Retry-After', str(int(backoff))))
               print(f"Rate limited (429). Retrying after {retry_after}s...")
               time.sleep(retry_after)
               backoff = min(backoff * 2, 30.0)
               continue

             backoff = 1.0

             if response.status != 200:
               print(f"HTTP Error: {response.status}")
               response_text = response.data.decode('utf-8')
               print(f"Response body: {response_text}")
               return []

             data = json.loads(response.data.decode('utf-8'))

             page_results = data.get('results', data.get('data', []))

             if not page_results:
               print(f"No more results (empty page) for {log_type}")
               break

             print(f"Page {page_num}: Retrieved {len(page_results)} {log_type} events")
             records.extend(page_results)

             offset += len(page_results)

             # If we got fewer results than requested, no more pages
             if len(page_results) < limit:
               print(f"Last page reached for {log_type}")
               break

           except Exception as e:
             print(f"Error fetching {log_type} logs: {e}")
             return []

         print(f"Retrieved {len(records)} total {log_type} records from {page_num} pages")
         return records

   - Second file: **requirements.txt:**

       functions-framework==3.*
       google-cloud-storage==2.*
       urllib3>=2.0.0

3. Click **Deploy** to save and deploy the function.

4. Wait for deployment to complete (2-3 minutes).

   > [!NOTE]
   > **Note:** The Pub/Sub trigger configuration automatically creates the necessary subscriptions and permissions.

## Create Cloud Scheduler job

Cloud Scheduler will publish messages to the Pub/Sub topic at regular intervals, triggering the Cloud Run function.

1. In the **GCP Console** , go to **Cloud Scheduler**.
2. Click **Create Job**.
3. Provide the following configuration details:

   | Setting | Value |
   |---|---|
   | **Name** | `wpengine-logs-collector-hourly` |
   | **Region** | Select same region as Cloud Run function |
   | **Frequency** | `0 * * * *` (every hour, on the hour) |
   | **Timezone** | Select timezone (UTC recommended) |
   | **Target type** | Pub/Sub |
   | **Topic** | Select the topic `wpengine-logs-trigger` |
   | **Message body** | `{}` (empty JSON object) |

4. Click **Create**.

### Schedule frequency options

Choose frequency based on log volume and latency requirements:

| Frequency | Cron Expression | Use Case |
|---|---|---|
| Every 5 minutes | `*/5 * * * *` | High-volume, low-latency |
| Every 15 minutes | `*/15 * * * *` | Medium volume |
| Every hour | `0 * * * *` | Standard (recommended) |
| Every 6 hours | `0 */6 * * *` | Low volume, batch processing |
| Daily | `0 0 * * *` | Historical data collection |

### Test the integration

1. In the **Cloud Scheduler** console, find your job.
2. Click **Force run** to trigger the job manually.
3. Wait a few seconds.
4. Go to **Cloud Run \> Services**.
5. Click on `wpengine-logs-collector`.
6. Click the **Logs** tab.
7. Verify the function executed successfully. Look for:

       Fetching logs for install: myinstall
       Page 1: Retrieved X access events
       Fetched X access log records
       Page 1: Retrieved X error events
       Fetched X error log records
       Wrote X records to gs://wpengine-logs/wpengine/logs_YYYYMMDD_HHMMSS.ndjson
       Successfully processed X records

8. Go to **Cloud Storage \> Buckets**.

9. Click on your bucket name (`wpengine-logs`).

10. Navigate to the `wpengine/` folder.

11. Verify that a new `.ndjson` file was created with the current timestamp.

If you see errors in the logs:

- **HTTP 401**: Check API credentials in environment variables
- **HTTP 403**: Verify API access is enabled in WP Engine User Portal
- **HTTP 429**: Rate limiting - function will automatically retry with backoff
- **Missing environment variables**: Check all required variables are set

## Configure a feed in Google SecOps to ingest WP Engine logs

1. Go to **Settings \> Data Management \> Feeds** . If you are on a SIEM-only platform, go to **Settings \> Feeds**.
2. Click **Add New Feed**.
3. Click **Configure a single feed**.
4. In the **Feed name** field, enter a name for the feed (for example, `WP Engine Logs`).
5. Select **Google Cloud Storage V2** as the **Source type**.
6. Select **WPEngine** as the **Log type**.
7. Click **Get Service Account**. A unique service account email will be displayed, for example:

       chronicle-12345678@chronicle-gcp-prod.iam.gserviceaccount.com

8. Copy this email address.

   > [!NOTE]
   > **Note:** Each Google SecOps instance has a unique service account. Do not use service accounts from other documentation or examples.

9. Click **Next**.

10. Specify values for the following input parameters:

    - **Storage bucket URL**: Enter the GCS bucket URI with the prefix path:

          gs://wpengine-logs/wpengine/

      - Replace:
        - `wpengine-logs`: Your GCS bucket name.
        - `wpengine`: Optional prefix/folder path where logs are stored (leave empty for root).

    > [!NOTE]
    > **Note:** Always include the trailing slash (/) at the end of the URI.

    - **Source deletion option**: Select the deletion option according to your preference:

      - **Never**: Never deletes any files after transfers (recommended for testing).
      - **Delete transferred files**: Deletes files after successful transfer.
      - **Delete transferred files and empty directories**: Deletes files and empty directories after successful transfer.

        > [!NOTE]
        > **Note:** If you select a deletion option, the service account must have **Storage Object Admin** role instead of **Storage Object Viewer**. Update IAM permissions accordingly.

    - **Maximum File Age**: Include files modified in the last number of days (default is 180 days)

    - **Asset namespace** : The [asset namespace](https://docs.cloud.google.com/chronicle/docs/investigation/asset-namespaces)

    - **Ingestion labels**: The label to be applied to the events from this feed

11. Click **Next**.

12. Review your new feed configuration in the **Finalize** screen, and then click **Submit**.

## Grant IAM permissions to the Google SecOps service account

The Google SecOps service account needs **Storage Object Viewer** role on your GCS bucket.

1. Go to **Cloud Storage \> Buckets**.
2. Click on your bucket name.
3. Go to the **Permissions** tab.
4. Click **Grant access**.
5. Provide the following configuration details:
   - **Add principals**: Paste the Google SecOps service account email
   - **Assign roles** : Select **Storage Object Viewer**
6. Click **Save**.

> [!NOTE]
> **Note:** If you plan to use the deletion option "Delete transferred files" or "Delete transferred files and empty directories," use the **Storage Object Admin** role instead of **Storage Object Viewer**.

## UDM mapping table

| Log Field | UDM Mapping | Logic |
|---|---|---|
| request, sig, blog_id, kind, name, slug, ver | additional.fields | Merged with labels from request (as request_label), sig (as sig_label), blog_id (as blog_id_label), kind (as kind_label), name (as name_label), slug (as slug_label), ver (as ver_label) if each is not empty |
| msg | metadata.description | Value copied directly |
|   | metadata.event_type | Set to "STATUS_UPDATE" if has_principal is true, else "GENERIC_EVENT" |
| protocol | network.application_protocol | Value copied directly |
| version | network.application_protocol_version | Converted to string |
| method | network.http.method | Value copied directly |
| user_agent | network.http.parsed_user_agent | Converted to parseduseragent |
| secure_url | network.http.referral_url | Value copied directly |
| response_code | network.http.response_code | Converted to string then to integer |
| user_agent | network.http.user_agent | Value copied directly |
| received_bytes | network.received_bytes | Converted to string then to uinteger |
| Hostname | principal.asset.hostname | Value copied directly |
| client_ip | principal.asset.ip | Value copied directly |
| Hostname | principal.hostname | Value copied directly |
| client_ip | principal.ip | Value copied directly |
| port | principal.port | Converted to string then to integer |
| pid | principal.process.pid | Converted to string |
| scan_type, scan_value | security_result.description | Value from scan_value if not empty, else from scan_type if not empty |

## Change Log

[View the Change Log for this parser](https://docs.cloud.google.com/chronicle/docs/ingestion/parser-list/wpengine-changelog)

**Need more help?** [Get answers from Community members and Google SecOps professionals.](https://security.googlecloudcommunity.com/)