# Collect UKG logs

Supported in:

This document explains how to ingest UKG logs to Google Security Operations using Google Cloud Storage V2.

UKG (Ultimate Kronos Group) is a human capital management platform that generates workforce management, authentication, and audit trail logs. UKG Pro provides a REST API for programmatic access to audit data.

## Before you begin

Make sure you have the following prerequisites:

- A Google SecOps instance
- A GCP project with Cloud Storage API enabled
- Permissions to create and manage GCS buckets
- Permissions to manage IAM policies on GCS buckets
- Permissions to create Cloud Run services, Pub/Sub topics, and Cloud Scheduler jobs
- Privileged access to UKG Pro with administrator permissions and API access

## Create Google Cloud Storage bucket

1. Go to the [Google Cloud Console](https://console.cloud.google.com/).
2. Select your project or create a new one.
3. In the navigation menu, go to **Cloud Storage \> Buckets**.
4. Click **Create bucket**.
5. Provide the following configuration details:

   | Setting | Value |
   |---|---|
   | **Name your bucket** | Enter a globally unique name (for example, `ukg-logs`) |
   | **Location type** | Choose based on your needs (Region, Dual-region, Multi-region) |
   | **Location** | Select the location (for example, `us-central1`) |
   | **Storage class** | Standard (recommended for frequently accessed logs) |
   | **Access control** | Uniform (recommended) |
   | **Protection tools** | Optional: Enable object versioning or retention policy |

6. Click **Create**.

## Collect UKG Pro API credentials

### Obtain API credentials

1. Sign in to the **UKG Pro** administration portal.
2. Go to **Menu \> Administration \> System Configuration \> Web Services**.
3. Create or locate your API credentials:
   - **API Key (US-Customer-Api-Key)**: The customer API key for authentication.
   - **Service Account Username**: The service account username.
   - **Service Account Password**: The service account password.
4. Note the **Service Host URL** (for example, `https://SERVICE_HOST`).

   > [!NOTE]
   > **Note:** Consult UKG Pro API documentation for setting up OAuth 2.0 authentication if using token-based access.

### Verify permissions

To verify the API credentials have the required access:

1. Sign in to the **UKG Pro** administration portal.
2. Go to **Menu \> Administration \> System Configuration \> Security \> Audit Trail**.
3. Verify you can view audit records.
4. If you cannot access audit data, contact your UKG Pro administrator to grant the required permissions.

### Test API access

- Test your credentials before proceeding with the integration:

      # Replace with your actual credentials
      ACCESS_TOKEN="your-access-token"
      API_KEY="your-api-key"
      SERVICE_HOST="your-service-host"

      # Test API access - fetch audit logs
      curl -v -H "Authorization: Bearer ${ACCESS_TOKEN}" \
        -H "US-Customer-Api-Key: ${API_KEY}" \
        "https://${SERVICE_HOST}/personnel/v1/audit-logs?startDate=2024-01-01&endDate=2024-01-02"

## Create a service account for Cloud Run function

The Cloud Run function needs a service account with permissions to write to GCS bucket and be invoked by Pub/Sub.

### Create service account

1. In the **GCP Console** , go to **IAM \& Admin \> Service Accounts**.
2. Click **Create Service Account**.
3. Provide the following configuration details:
   - **Service account name** : Enter `ukg-collector-sa`.
   - **Service account description** : Enter `Service account for Cloud Run function to collect UKG logs`.
4. Click **Create and Continue**.
5. In the **Grant this service account access to project** section, add the following roles:
   1. Click **Select a role**.
   2. Search for and select **Storage Object Admin**.
   3. Click **+ Add another role**.
   4. Search for and select **Cloud Run Invoker**.
   5. Click **+ Add another role**.
   6. Search for and select **Cloud Functions Invoker**.
6. Click **Continue**.
7. Click **Done**.

These roles are required for:

- **Storage Object Admin**: Write logs to GCS bucket and manage state files
- **Cloud Run Invoker**: Allow Pub/Sub to invoke the function
- **Cloud Functions Invoker**: Allow function invocation

### Grant IAM permissions on GCS bucket

Grant the service account write permissions on the GCS bucket:

1. Go to **Cloud Storage \> Buckets**.
2. Click your bucket name (for example, `ukg-logs`).
3. Go to the **Permissions** tab.
4. Click **Grant access**.
5. Provide the following configuration details:
   - **Add principals** : Enter the service account email (for example, `ukg-collector-sa@your-project.iam.gserviceaccount.com`).
   - **Assign roles** : Select **Storage Object Admin**.
6. Click **Save**.

## Create Pub/Sub topic

Create a Pub/Sub topic that Cloud Scheduler will publish to and the Cloud Run function will subscribe to.

1. In the **GCP Console** , go to **Pub/Sub \> Topics**.
2. Click **Create topic**.
3. Provide the following configuration details:
   - **Topic ID** : Enter `ukg-trigger`.
   - Leave other settings as default.
4. Click **Create**.

## Create Cloud Run function to collect logs

The Cloud Run function will be triggered by Pub/Sub messages from Cloud Scheduler to fetch logs from UKG Pro API and write them to GCS.

1. In the **GCP Console** , go to **Cloud Run**.
2. Click **Create service**.
3. Select **Function** (use an inline editor to create a function).
4. In the **Configure** section, provide the following configuration details:

   | Setting | Value |
   |---|---|
   | **Service name** | `ukg-collector` |
   | **Region** | Select region matching your GCS bucket (for example, `us-central1`) |
   | **Runtime** | Select **Python 3.12** or later |

5. In the **Trigger (optional)** section:

   1. Click **+ Add trigger**.
   2. Select **Cloud Pub/Sub**.
   3. In **Select a Cloud Pub/Sub topic** , choose the topic `ukg-trigger`.
   4. Click **Save**.
6. In the **Authentication** section:

   1. Select **Require authentication**.
   2. Check **Identity and Access Management (IAM)**.

   > [!NOTE]
   > **Note:** Pub/Sub will automatically handle authentication when invoking the function.

7. Scroll down and expand **Containers, Networking, Security**.

8. Go to the **Security** tab:

   - **Service account** : Select the service account `ukg-collector-sa`.
9. Go to the **Containers** tab:

   1. Click **Variables \& Secrets**.
   2. Click **+ Add variable** for each environment variable:

   | Variable Name | Example Value | Description |
   |---|---|---|
   | `GCS_BUCKET` | `ukg-logs` | GCS bucket name |
   | `GCS_PREFIX` | `ukg` | Prefix for log files |
   | `STATE_KEY` | `ukg/state.json` | State file path |
   | `SERVICE_HOST` | `your-service-host` | UKG Pro service host URL |
   | `UKG_USERNAME` | `your-service-username` | Service account username |
   | `UKG_PASSWORD` | `your-service-password` | Service account password |
   | `UKG_API_KEY` | `your-api-key` | Customer API key |
   | `MAX_RECORDS` | `10000` | Max records per run |
   | `PAGE_SIZE` | `100` | Records per page |
   | `LOOKBACK_HOURS` | `24` | Initial lookback period |

10. Scroll down in the **Variables \& Secrets** tab to **Requests**:

    - **Request timeout** : Enter `600` seconds (10 minutes).
11. Go to the **Settings** tab in **Containers**:

    - In the **Resources** section:
      - **Memory** : Select **512 MiB** or higher.
      - **CPU** : Select **1**.
12. In the **Revision scaling** section:

    - **Minimum number of instances** : Enter `0`.
    - **Maximum number of instances** : Enter `100` (or adjust based on expected load).
13. Click **Create**.

14. Wait for the service to be created (1-2 minutes).

15. After the service is created, the **inline code editor** will open automatically.

### Add function code

1. Enter **main** in **Function entry point**.
2. In the inline code editor, create two files:

   - First file: **main.py:**

       import functions_framework
       from google.cloud import https://docs.cloud.google.com/python/docs/reference/storage/latest
       import json
       import os
       import urllib3
       from datetime import datetime, timezone, timedelta
       import time
       import base64

       # Initialize HTTP client with timeouts
       http = urllib3.PoolManager(
           timeout=urllib3.Timeout(connect=5.0, read=30.0),
           retries=False,
       )

       # Initialize Storage client
       storage_client = https://docs.cloud.google.com/python/docs/reference/storage/latest.https://docs.cloud.google.com/python/docs/reference/storage/latest/google.cloud.storage.client.Client.html()

       # Environment variables
       GCS_BUCKET = os.environ.get('GCS_BUCKET')
       GCS_PREFIX = os.environ.get('GCS_PREFIX', 'ukg')
       STATE_KEY = os.environ.get('STATE_KEY', 'ukg/state.json')
       SERVICE_HOST = os.environ.get('SERVICE_HOST', '').rstrip('/')
       UKG_USERNAME = os.environ.get('UKG_USERNAME', '')
       UKG_PASSWORD = os.environ.get('UKG_PASSWORD', '')
       UKG_API_KEY = os.environ.get('UKG_API_KEY', '')
       MAX_RECORDS = int(os.environ.get('MAX_RECORDS', '10000'))
       PAGE_SIZE = int(os.environ.get('PAGE_SIZE', '100'))
       LOOKBACK_HOURS = int(os.environ.get('LOOKBACK_HOURS', '24'))

       def parse_datetime(value: str) -> datetime:
           """Parse ISO datetime string to datetime object."""
           if https://docs.cloud.google.com/python/docs/reference/storage/latest/google.cloud.storage.blob.ObjectCustomContextPayload.html#google_cloud_storage_blob_ObjectCustomContextPayload_value.endswith("Z"):
               value = value[:-1] + "+00:00"
           return datetime.fromisoformat(value)

       def get_token():
           """Get OAuth 2.0 access token from UKG Pro."""
           token_url = f"https://{SERVICE_HOST}/authentication/token"

           auth_string = f"{UKG_USERNAME}:{UKG_PASSWORD}"
           auth_bytes = auth_string.encode('utf-8')
           auth_b64 = base64.b64encode(auth_bytes).decode('utf-8')

           headers = {
               'Authorization': f'Basic {auth_b64}',
               'US-Customer-Api-Key': UKG_API_KEY,
               'Content-Type': 'application/x-www-form-urlencoded',
               'Accept': 'application/json'
           }

           body = 'grant_type=client_credentials'

           backoff = 1.0
           max_retries = 3

           for attempt in range(max_retries):
               response = http.request('POST', token_url, body=body.encode('utf-8'), headers=headers)

               if response.status == 429:
                   retry_after = int(response.headers.get('Retry-After', str(int(backoff))))
                   print(f"Rate limited (429) on token request. Retrying after {retry_after}s...")
                   time.sleep(retry_after)
                   backoff = min(backoff * 2, 30.0)
                   continue

               if response.status != 200:
                   raise RuntimeError(f"Failed to get access token: {response.status} - {response.data.decode('utf-8')}")

               data = json.loads(response.data.decode('utf-8'))
               return data['access_token']

           raise RuntimeError(f"Failed to get token after {max_retries} retries due to rate limiting")

       @functions_framework.cloud_event
       def main(cloud_event):
           """
           Cloud Run function triggered by Pub/Sub to fetch UKG logs and write to GCS.

           Args:
               cloud_event: CloudEvent object containing Pub/Sub message
           """

           if not all([GCS_BUCKET, SERVICE_HOST, UKG_USERNAME, UKG_PASSWORD, UKG_API_KEY]):
               print('Error: Missing required environment variables')
               return

           try:
               bucket = storage_client.https://docs.cloud.google.com/python/docs/reference/storage/latest/google.cloud.storage.client.Client.html#google_cloud_storage_client_Client_bucket(GCS_BUCKET)

               # Load state
               state = load_state(bucket, STATE_KEY)

               # Determine time window
               now = datetime.now(timezone.utc)
               last_time = None

               if isinstance(state, dict) and https://docs.cloud.google.com/python/docs/reference/storage/latest/google.cloud.storage.hmac_key.HMACKeyMetadata.html#google_cloud_storage_hmac_key_HMACKeyMetadata_state.get("last_event_time"):
                   try:
                       last_time = parse_datetime(state["last_event_time"])
                       last_time = last_time - timedelta(minutes=2)
                   except Exception as e:
                       print(f"Warning: Could not parse last_event_time: {e}")

               if last_time is None:
                   last_time = now - timedelta(hours=LOOKBACK_HOURS)

               print(f"Fetching logs from {last_time.isoformat()} to {now.isoformat()}")

               # Get access token
               token = get_token()

               # Fetch logs
               records, newest_event_time = fetch_logs(
                   token=token,
                   start_time=last_time,
                   end_time=now,
                   page_size=PAGE_SIZE,
                   max_records=MAX_RECORDS,
               )

               if not records:
                   print("No new log records found.")
                   save_state(bucket, STATE_KEY, now.isoformat())
                   return

               # Write to GCS as NDJSON
               timestamp = now.strftime('%Y%m%d_%H%M%S')
               object_key = f"{GCS_PREFIX}/logs_{timestamp}.ndjson"
               blob = bucket.blob(object_key)

               ndjson = '\n'.join([json.dumps(record, ensure_ascii=False) for record in records]) + '\n'
               blob.https://docs.cloud.google.com/python/docs/reference/storage/latest/google.cloud.storage.blob.Blob.html#google_cloud_storage_blob_Blob_upload_from_string(ndjson, content_type='application/x-ndjson')

               print(f"Wrote {len(records)} records to gs://{GCS_BUCKET}/{object_key}")

               if newest_event_time:
                   save_state(bucket, STATE_KEY, newest_event_time)
               else:
                   save_state(bucket, STATE_KEY, now.isoformat())

               print(f"Successfully processed {len(records)} records")

           except Exception as e:
               print(f'Error processing logs: {str(e)}')
               raise

       def load_state(bucket, key):
           """Load state from GCS."""
           try:
               blob = bucket.blob(key)
               if blob.exists():
                   state_data = blob.https://docs.cloud.google.com/python/docs/reference/storage/latest/google.cloud.storage.blob.Blob.html#google_cloud_storage_blob_Blob_download_as_text()
                   return json.loads(state_data)
           except Exception as e:
               print(f"Warning: Could not load state: {e}")

           return {}

       def save_state(bucket, key, last_event_time_iso: str):
           """Save the last event timestamp to GCS state file."""
           try:
               state = {'last_event_time': last_event_time_iso}
               blob = bucket.blob(key)
               blob.https://docs.cloud.google.com/python/docs/reference/storage/latest/google.cloud.storage.blob.Blob.html#google_cloud_storage_blob_Blob_upload_from_string(
                   json.dumps(state, indent=2),
                   content_type='application/json'
               )
               print(f"Saved state: last_event_time={last_event_time_iso}")
           except Exception as e:
               print(f"Warning: Could not save state: {e}")

       def fetch_logs(token: str, start_time: datetime, end_time: datetime, page_size: int, max_records: int):
           """
           Fetch audit logs from UKG Pro API with pagination and rate limiting.

           Args:
               token: OAuth 2.0 access token
               start_time: Start time for log query
               end_time: End time for log query
               page_size: Number of records per page
               max_records: Maximum total records to fetch

           Returns:
               Tuple of (records list, newest_event_time ISO string)
           """
           endpoint = f"https://{SERVICE_HOST}/personnel/v1/audit-logs"

           headers = {
               'Authorization': f'Bearer {token}',
               'US-Customer-Api-Key': UKG_API_KEY,
               'Accept': 'application/json',
               'User-Agent': 'GoogleSecOps-UKGCollector/1.0'
           }

           records = []
           newest_time = None
           page_num = 0
           backoff = 1.0
           page = 1

           start_date = start_time.strftime('%Y-%m-%d')
           end_date = end_time.strftime('%Y-%m-%d')

           while True:
               page_num += 1

               if len(records) >= max_records:
                   print(f"Reached max_records limit ({max_records})")
                   break

               url = f"{endpoint}?startDate={start_date}&endDate={end_date}&page={page}&per_page={page_size}"

               try:
                   response = http.request('GET', url, headers=headers)

                   if response.status == 429:
                       retry_after = int(response.headers.get('Retry-After', str(int(backoff))))
                       print(f"Rate limited (429). Retrying after {retry_after}s...")
                       time.sleep(retry_after)
                       backoff = min(backoff * 2, 30.0)
                       continue

                   backoff = 1.0

                   if response.status != 200:
                       print(f"HTTP Error: {response.status}")
                       response_text = response.data.decode('utf-8')
                       print(f"Response body: {response_text}")
                       return [], None

                   data = json.loads(response.data.decode('utf-8'))

                   # UKG may return results in different formats
                   if isinstance(data, list):
                       page_results = data
                   else:
                       page_results = data.get('data', data.get('results', data.get('auditLogs', [])))

                   if not page_results:
                       print(f"No more results (empty page)")
                       break

                   print(f"Page {page_num}: Retrieved {len(page_results)} events")
                   records.extend(page_results)

                   # Track newest event time
                   for event in page_results:
                       try:
                           event_time = event.get('auditDate') or event.get('dateTime') or event.get('timestamp')
                           if event_time:
                               if newest_time is None or parse_datetime(event_time) > parse_datetime(newest_time):
                                   newest_time = event_time
                       except Exception as e:
                           print(f"Warning: Could not parse event time: {e}")

                   if len(page_results) < page_size:
                       print(f"Reached last page (size={len(page_results)} < limit={page_size})")
                       break

                   page += 1

               except Exception as e:
                   print(f"Error fetching logs: {e}")
                   return [], None

           print(f"Retrieved {len(records)} total records from {page_num} pages")
           return records, newest_time

   - Second file: **requirements.txt:**

       functions-framework==3.*
       google-cloud-storage==2.*
       urllib3>=2.0.0

3. Click **Deploy** to save and deploy the function.

4. Wait for deployment to complete (2-3 minutes).

   > [!NOTE]
   > **Note:** The Pub/Sub trigger configuration automatically creates the necessary subscriptions and permissions.

## Create Cloud Scheduler job

Cloud Scheduler will publish messages to the Pub/Sub topic at regular intervals, triggering the Cloud Run function.

1. In the **GCP Console** , go to **Cloud Scheduler**.
2. Click **Create Job**.
3. Provide the following configuration details:

   | Setting | Value |
   |---|---|
   | **Name** | `ukg-collector-hourly` |
   | **Region** | Select same region as Cloud Run function |
   | **Frequency** | `0 * * * *` (every hour, on the hour) |
   | **Timezone** | Select timezone (UTC recommended) |
   | **Target type** | Pub/Sub |
   | **Topic** | Select the topic `ukg-trigger` |
   | **Message body** | `{}` (empty JSON object) |

4. Click **Create**.

### Schedule frequency options

Choose frequency based on log volume and latency requirements:

| Frequency | Cron Expression | Use Case |
|---|---|---|
| Every 5 minutes | `*/5 * * * *` | High-volume, low-latency |
| Every 15 minutes | `*/15 * * * *` | Medium volume |
| Every hour | `0 * * * *` | Standard (recommended) |
| Every 6 hours | `0 */6 * * *` | Low volume, batch processing |
| Daily | `0 0 * * *` | Historical data collection |

### Test the integration

1. In the **Cloud Scheduler** console, find your job (`ukg-collector-hourly`).
2. Click **Force run** to trigger manually.
3. Wait a few seconds and go to **Cloud Run \> Services \> ukg-collector \> Logs**.
4. Verify the function executed successfully. Look for:

       Fetching logs from YYYY-MM-DDTHH:MM:SS+00:00 to YYYY-MM-DDTHH:MM:SS+00:00
       Page 1: Retrieved X events
       Wrote X records to gs://ukg-logs/ukg/logs_YYYYMMDD_HHMMSS.ndjson
       Successfully processed X records

5. Check the GCS bucket (`ukg-logs`) to confirm logs were written.

If you see errors in the logs:

- **HTTP 401**: Check API credentials in environment variables
- **HTTP 403**: Verify service account has required permissions in UKG Pro
- **HTTP 429**: Rate limiting - function will automatically retry with backoff
- **Failed to get access token** : Verify `SERVICE_HOST`, `UKG_USERNAME`, `UKG_PASSWORD`, and `UKG_API_KEY` are correct

## Configure a feed in Google SecOps to ingest UKG logs

1. Go to **Settings \> Data Management \> Feeds** . If you are on a SIEM-only platform, go to **Settings \> Feeds**.
2. Click **Add New Feed**.
3. Click **Configure a single feed**.
4. In the **Feed name** field, enter a name for the feed (for example, `UKG Logs`).
5. Select **Google Cloud Storage V2** as the **Source type**.
6. Select **UKG** as the **Log type**.
7. Click **Get Service Account**. A unique service account email will be displayed, for example:

       chronicle-12345678@chronicle-gcp-prod.iam.gserviceaccount.com

8. Copy this email address. You will use it in the next step.

   > [!NOTE]
   > **Note:** Each Google SecOps instance has a unique service account. Do not use service accounts from other documentation or examples.

9. Click **Next**.

10. Specify values for the following input parameters:

    - **Storage bucket URL**: Enter the GCS bucket URI with the prefix path:

          gs://ukg-logs/ukg/

      - Replace:
        - `ukg-logs`: Your GCS bucket name.
        - `ukg`: Optional prefix/folder path where logs are stored (leave empty for root).

    > [!NOTE]
    > **Note:** Always include the trailing slash (/) at the end of the URI.

    - **Source deletion option**: Select the deletion option according to your preference:

      - **Never**: Never deletes any files after transfers (recommended for testing).
      - **Delete transferred files**: Deletes files after successful transfer.
      - **Delete transferred files and empty directories**: Deletes files and empty directories after successful transfer.

        > [!NOTE]
        > **Note:** If you select a deletion option, the service account must have **Storage Object Admin** role instead of **Storage Object Viewer**. Update IAM permissions accordingly.

    - **Maximum File Age**: Include files modified in the last number of days (default is 180 days).

    - **Asset namespace** : The [asset namespace](https://docs.cloud.google.com/chronicle/docs/investigation/asset-namespaces).

    - **Ingestion labels**: The label to be applied to the events from this feed.

11. Click **Next**.

12. Review your new feed configuration in the **Finalize** screen, and then click **Submit**.

## Grant IAM permissions to the Google SecOps service account

The Google SecOps service account needs **Storage Object Viewer** role on your GCS bucket.

1. Go to **Cloud Storage \> Buckets**.
2. Click your bucket name (`ukg-logs`).
3. Go to the **Permissions** tab.
4. Click **Grant access**.
5. Provide the following configuration details:
   - **Add principals**: Paste the Google SecOps service account email.
   - **Assign roles** : Select **Storage Object Viewer**.
6. Click **Save**.

   > [!NOTE]
   > **Note:** If you plan to use the deletion option "Delete transferred files" or "Delete transferred files and empty directories," grant **Storage Object Admin** role instead of **Storage Object Viewer**.

## UDM mapping table

| Log Field | UDM Mapping | Logic |
|---|---|---|
| integrationUserKey, userStatus | entity.user.attribute.labels | Merged as key-value labels with key set to field name and value from field |
| userName | entity.user.email_addresses | Merged into array if matches email regex |
| employeeId | entity.user.employee_id | Value copied directly if not empty |
| userLastLogonDate | entity.user.last_login_time | Converted to timestamp using date match with formats yyyy-MM-ddTHH:mm:ss.SS or ISO8601 |
| userLastPasswordChangeDate | entity.user.last_password_change_time | Converted to timestamp using date match with formats yyyy-MM-ddTHH:mm:ss.SSS or ISO8601 |
| userId | entity.user.product_object_id | Converted to string |
| fullName | entity.user.user_display_name | Value copied directly |
| userId | entity.user.userid | Converted to string |
| has_user | metadata.entity_type | Set to "USER" if has_user == "true", else "UNKNOWN_ENTITYTYPE" |

## Change Log

[View the Change Log for this parser](https://docs.cloud.google.com/chronicle/docs/ingestion/parser-list/ukg-changelog)

**Need more help?** [Get answers from Community members and Google SecOps professionals.](https://security.googlecloudcommunity.com/)