Collect Akamai DataStream 2 logs

Supported in:

Google secops SIEM

Note: This feature is covered by Pre-GA Offerings Terms of the Google Security Operations Service Specific Terms. Pre-GA features might have limited support, and changes to pre-GA features might not be compatible with other pre-GA versions. For more information, see the Google SecOps Technical Support Service guidelines and the Google SecOps Service Specific Terms.

This document explains how to ingest Akamai DataStream 2 logs to Google Security Operations using Amazon S3.

Before you begin

Make sure you have the following prerequisites:

• Google SecOps instance
• Privileged access to Akamai Control Center (DataStream 2 configuration access)
• Privileged access to AWS (S3, IAM)

Configure AWS S3 bucket and IAM for Google SecOps

1. Create Amazon S3 bucket following this user guide: Creating a bucket
2. Save bucket Name and Region for future reference (for example, akamai-cloud-monitor).
3. Create a user following this user guide: Creating an IAM user.
4. Select the created User.
5. Select the Security credentials tab.
6. Click Create Access Key in the Access Keys section.
7. Select Third-party service as the Use case.
8. Click Next.
9. Optional: add a description tag.
10. Click Create access key.
11. Click Download CSV file to save the Access Key and Secret Access Key for later use.
12. Click Done.
13. Select the Permissions tab.
14. Click Add permissions in the Permissions policies section.
15. Select Add permissions.
16. Select Attach policies directly
17. Search for and select the AmazonS3FullAccess policy.
18. Click Next.
19. Click Add permissions.

Configure the IAM policy and role for S3 uploads

1. In the AWS console, go to IAM > Policies > Create policy > JSON tab.

2. Enter the following policy:

   {
     "Version": "2012-10-17",
     "Statement": [
       {
         "Sid": "AllowAkamaiWriteToS3",
         "Effect": "Allow",
         "Action": ["s3:PutObject"],
         "Resource": "arn:aws:s3:::akamai-datastream-2-logs/akamai/datastream2/json/*"
       }
     ]
   }
   

   • Replace akamai-datastream-2-logs if you entered a different bucket name.

3. Click Next > Create policy.

4. Go to IAM > Users > Create user.

5. Name the user akamai-datastream-writer.

6. Attach the newly created policy.

7. Create access keys for this user to use in Akamai DataStream 2 configuration.

Configure Akamai DataStream 2 to deliver logs to Amazon S3

1. In Akamai Control Center go to DataStream 2.
2. Click Create a stream.
3. Select log type appropriate for your property (for example, Delivery, Edge DNS, GTM).
4. In Data sets, select the fields you require. Keep defaults unless you have a specific need.
5. Go to Delivery > Destination and select Amazon S3.
6. Fill the S3 destination details using the newly created bucket:
   • Bucket: akamai-datastream-2-logs
   • Folder path: akamai/datastream2/json/
   • Region: Your bucket region
   • Access key ID: The User access key created earlier
   • Secret access key: The User secret access key created earlier
7. Set Log format to JSON.
8. Optional: In Delivery options, set Push frequency to 30 seconds.
9. Click Validate & Save > Next > Activate.

Optional: Create read-only IAM user & keys for Google SecOps

1. Go to AWS Console > IAM > Users > Add users.
2. Click Add users.
3. Provide the following configuration details:
   • User: Enter secops-reader.
   • Access type: Select Access key – Programmatic access.
4. Click Create user.
5. Attach minimal read policy (custom): Users > secops-reader > Permissions > Add permissions > Attach policies directly > Create policy.

6. In the JSON editor, enter the following policy:

   {
     "Version": "2012-10-17",
     "Statement": [
       {
         "Effect": "Allow",
         "Action": ["s3:GetObject"],
         "Resource": "arn:aws:s3:::akamai-datastream-2-logs/*"
       },
       {
         "Effect": "Allow",
         "Action": ["s3:ListBucket"],
         "Resource": "arn:aws:s3:::akamai-datastream-2-logs"
       }
     ]
   }
   

7. Set the name to secops-reader-policy.

8. Go to Create policy > search/select > Next > Add permissions.

9. Go to Security credentials > Access keys > Create access key.

10. Download the CSV (these values are entered into the feed).

Configure a feed in Google SecOps to ingest Akamai DataStream 2 logs

1. Go to SIEM Settings > Feeds.
2. Click + Add New Feed.
3. In the Feed name field, enter a name for the feed (for example, Akamai DataStream 2 logs).
4. Select Amazon S3 V2 as the Source type.
5. Select Akamai DataStream 2 as the Log type.
6. Click Next.
7. Specify values for the following input parameters:
   • S3 URI: s3://akamai-datastream-2-logs/akamai/datastream2/json/
   • Source deletion options: Select deletion option according to your preference.
   • Maximum File Age: Include files modified in the last number of days. Default is 180 days.
   • Access Key ID: User access key with access to the S3 bucket.
   • Secret Access Key: User secret key with access to the S3 bucket.
   • Asset namespace: The asset namespace.
   • Ingestion labels: The label applied to the events from this feed.
8. Click Next.
9. Review your new feed configuration in the Finalize screen, and then click Submit.

Supported Akamai Datastream 2 Sample Logs

• JSON

{ "logName": "projects/monotaro-akamai-log/logs/akamai_data_stream", "resource": { "type": "cloud_run_revision", "labels": { "service_name": "akamai-datastream2", "configuration_name": "hhhhh", "location": "asia-northeast1", "project_id": "monotaro-akamai-log", "revision_name": "akamai-datastream2-brpwf" } }, "jsonPayload": { "cliip": "0.0.0.0", "cookie": "sid=MASKED_SESSION_ID", "reqtimesec": "1704844800.068", "trace": "projects/monotaro-akamai-log/traces/acc4d1aeffecdf96896e5aff5cf1fbf5", "errorcode": "-", "totalbytes": "27489", "reqport": "443", "overheadbytes": "1160", "reqhost": "www.example.com", "tlsversion": "TLSv1.3", "referer": "masked URL", "reqendtimemsec": "15", "acclang": "-", "reqid": "80c241bc", "city": "TOKYO", "maxagesec": "-", "cp": "1219543", "uncompressedsize": "-", "proto": "HTTP/2", "querystr": "s=MASKED_QUERY_PARAM&v=other_param", "range": "-", "objsize": "25510", "transfertimemsec": "1", "reqmethod": "GET", "rspcontentlen": "25510", "statuscode": "200", "dnslookuptimemsec": "-", "turnaroundtimemsec": "83", "xforwardedfor": "-", "ua": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) MASKED_UA", "cachestatus": "0", "rspcontenttype": "application/json", "bytes": "25510", "version": "3", "tlsoverheadtimemsec": "0", "country": "JP", "reqpath": "recsys/v1/recommend", "customfield": "MASKED_CUSTOM_FIELD_1,MASKED_CUSTOM_FIELD_2", "securityrules": "mntr_25916||" }, "timestamp": "2024-01-10T00:00:00.068+00:00", "receiveTimestamp": "2024-01-10T00:00:19.987565+00:00", "insertId": "148w3lkg16liikb" }

Need more help? Get answers from Community members and Google SecOps professionals.