Collect Akamai DataStream 2 logs
Parser Version: 2.0
This document explains how to ingest Akamai DataStream 2 logs to Google Security Operations using Amazon S3.
Akamai DataStream 2 captures performance and security logs from the Akamai Intelligent Edge Platform and delivers them in near real-time to a destination of your choice. You can use DataStream 2 to monitor, debug, and analyze CDN and security events across your Akamai properties.
Before you begin
Make sure you have the following prerequisites:
- A Google SecOps instance.
- Privileged access to Akamai Control Center with DataStream 2 configuration permissions.
- Privileged access to AWS (S3, IAM, and AWS KMS if the bucket is encrypted with a customer-managed KMS key).
Configure AWS S3 bucket and IAM for Google SecOps
- Create Amazon S3 bucket following this user guide: Creating a bucket.
- Save bucket Name and Region for future reference (for example,
akamai-datastream2-logs). - Create a User following this user guide: Creating an IAM user.
- Select the created User.
- Select Security credentials tab.
- Click Create Access Key in section Access Keys.
- Select Third-party service as Use case.
- Click Next.
- Optional: Add description tag.
- Click Create access key.
- Click Download .csv file to save the Access Key and Secret Access Key for future reference.
- Click Done.
- Select Permissions tab.
- Click Add permissions in section Permissions policies.
- Select Add permissions.
- Select Attach policies directly.
- Search for AmazonS3FullAccess policy.
- Select the policy.
- Click Next.
- Click Add permissions.
Configure the IAM policy for Akamai S3 uploads
- In the AWS console, go to IAM > Policies > Create policy > JSON tab.
- Copy and paste the policy below.
Policy JSON (replace
akamai-datastream2-logsif you used a different bucket name):{ "Version": "2012-10-17", "Statement": [ { "Sid": "AllowAkamaiDataStreamWrite", "Effect": "Allow", "Action": [ "s3:PutObject", "s3:GetObject", "s3:ListBucket" ], "Resource": [ "arn:aws:s3:::akamai-datastream2-logs", "arn:aws:s3:::akamai-datastream2-logs/*" ] } ] }Click Next.
Enter a policy name (for example,
AkamaiDataStreamS3Policy).Click Create policy.
Go to IAM > Users > Create user.
Enter a user name (for example,
akamai-datastream-writer).Click Next.
Select Attach policies directly.
Search for and select the policy created above.
Click Next > Create user.
Select the created user, go to Security credentials > Create access key.
Select Third-party service as Use case and click Next > Create access key.
Click Download .csv file to save the Access Key ID and Secret Access Key for Akamai DataStream 2 configuration.
Configure AWS KMS permissions for SSE-KMS buckets
Complete this section only if the bucket is encrypted with a customer managed AWS KMS key (SSE-KMS). Without these permissions, Akamai cannot write log files to the bucket and destination validation fails.
- In the AWS console, go to S3 > Buckets and select your bucket.
- Go to the Properties tab and check Default encryption:
- If Encryption type is Server-side encryption with Amazon S3 managed keys (SSE-S3), skip this section and the KMS section for the Google SecOps user.
- Otherwise, note the AWS KMS key that the bucket uses.
- Go to Key Management Service and find that key:
- If the key is listed under AWS managed keys (alias
aws/s3), skip this section and the KMS section for the Google SecOps user. Principals in your account can use theaws/s3key through Amazon S3 without extra permissions. - If the key is listed under Customer managed keys, select it and copy its ARN.
- If the key is listed under AWS managed keys (alias
- Go to IAM > Users and select
akamai-datastream-writer. - Go to the Permissions tab and click Add permissions > Create inline policy.
In the Policy editor, select JSON and paste the following policy:
{ "Version": "2012-10-17", "Statement": [ { "Sid": "AllowAkamaiDataStreamKmsKeyUse", "Effect": "Allow", "Action": [ "kms:GenerateDataKey", "kms:Decrypt" ], "Resource": "KMS_KEY_ARN" } ] }Replace
KMS_KEY_ARNwith the key ARN you copied (for example,arn:aws:kms:us-east-1:111122223333:key/1234abcd-12ab-34cd-56ef-1234567890ab).Click Next.
In Policy name, enter a name (for example,
akamai-datastream-kms) and click Create policy.Go to Key Management Service > Customer managed keys and select the key.
Go to the Key policy tab. If the tab shows the default view, click Switch to policy view. Check that the key policy allows IAM policies to grant access to the key:
- If the key policy contains a statement that allows the account root principal (
arn:aws:iam::ACCOUNT_ID:root) thekms:*action, no change is needed. Keys created in the AWS KMS console have this statement by default. The inline policy that you created grants access. - Otherwise, click Edit and add a statement that allows the user, then click Save changes:
{ "Sid": "AllowAkamaiDataStreamKmsKeyUse", "Effect": "Allow", "Principal": { "AWS": "arn:aws:iam::ACCOUNT_ID:user/akamai-datastream-writer" }, "Action": [ "kms:GenerateDataKey", "kms:Decrypt" ], "Resource": "*" }Replace
ACCOUNT_IDwith your 12-digit AWS account ID. In a key policy,"Resource": "*"refers to the key that the policy is attached to.- If the key policy contains a statement that allows the account root principal (
Configure Akamai DataStream 2 to deliver logs to Amazon S3
- Sign in to Akamai Control Center.
- Go to DataStream 2.
- Click Create a stream.
- Select a log type appropriate for your property (for example, Delivery, Edge DNS, GTM).
- In Data sets, select the fields you require. Keep defaults unless you have a specific need.
- Go to Destination and select Amazon S3.
- Fill in the S3 destination details:
- Name: Enter a descriptive label for the destination (for example,
SIEM S3 Destination). - Bucket: Enter your S3 bucket name (for example,
akamai-datastream2-logs). - Folder path: Enter a folder path (for example,
datastream2/logs/). Supports dynamic variables such as timestamps. - Region: Select the AWS region that matches your S3 bucket.
- Access Key ID: Enter the access key ID of the IAM user created for Akamai.
- Secret Access Key: Enter the secret access key of the IAM user created for Akamai.
- Name: Enter a descriptive label for the destination (for example,
Click Validate & Save.
Optional: In Delivery options, configure the following:
- Push frequency: Select 30 seconds or 60 seconds.
- Filename prefix and suffix: Customize as needed. Avoid using
.in prefixes.
Click Next > Activate.
(Optional) Create read-only IAM user for Google SecOps
If you prefer a separate read-only user for Google SecOps to ingest logs:
- Go to AWS Console > IAM > Policies > Create policy > JSON tab.
Copy and paste the policy below (replace
akamai-datastream2-logsif you used a different bucket name):{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "s3:GetObject" ], "Resource": "arn:aws:s3:::akamai-datastream2-logs/*" }, { "Effect": "Allow", "Action": [ "s3:ListBucket" ], "Resource": "arn:aws:s3:::akamai-datastream2-logs" } ] }Click Next.
Enter a policy name (for example,
secops-reader-policy).Click Create policy.
Go to IAM > Users > Create user.
Enter a username (for example,
secops-reader).Click Next.
Select Attach policies directly and select the policy created above.
Click Next > Create user.
Select the created user, go to Security credentials > Create access key.
Select Third-party service as Use case and click Next > Create access key.
Click Download .csv file to save the Access Key ID and Secret Access Key for the Google SecOps feed configuration.
Grant AWS KMS permissions to the Google SecOps user
Complete this section only if the bucket is encrypted with a customer managed AWS KMS key (SSE-KMS), as determined in Grant AWS KMS permissions for an SSE-KMS bucket. The feed reads the bucket with the access key of one IAM user: secops-reader if you created the optional read-only user, or otherwise the user from Configure AWS S3 bucket and IAM for Google SecOps. The AmazonS3FullAccess policy does not include AWS KMS permissions.
- Go to IAM > Users and select the user whose access key you use in the feed.
- Go to the Permissions tab and click Add permissions > Create inline policy.
In the Policy editor, select JSON and paste the following policy:
{ "Version": "2012-10-17", "Statement": [ { "Sid": "AllowSecOpsFeedKmsDecrypt", "Effect": "Allow", "Action": [ "kms:Decrypt" ], "Resource": "KMS_KEY_ARN" } ] }Replace
KMS_KEY_ARNwith the ARN of the customer managed key that the bucket uses.Click Next.
In Policy name, enter a name (for example,
secops-reader-kms) and click Create policy.If the key policy does not contain the default statement for the account root principal, add a statement for this user as described in Grant AWS KMS permissions for an SSE-KMS bucket, with the user ARN as
Principaland onlykms:DecryptasAction.
Configure a feed in Google SecOps to ingest Akamai DataStream 2 logs
- Go to Settings > Data Management > Feeds. If you are on a SIEM-only platform, go to Settings > Feeds.
- Click Add New Feed.
- On the next page, click Configure a single feed.
- Enter a unique name for the Feed name (for example,
Akamai DataStream 2 Logs). - Select Amazon S3 V2 as the Source type.
- Select Akamai DataStream 2 as the Log type.
- Click Next and then click Submit.
Specify values for the following fields:
- S3 URI:
s3://akamai-datastream2-logs/datastream2/logs/ - Source deletion option: Select the deletion option according to your preference.
- Maximum File Age: Include files modified in the last number of days (default is 180 days).
- Access Key ID: User access key with access to the S3 bucket.
- Secret Access Key: User secret key with access to the S3 bucket.
- Asset namespace: The asset namespace.
- Ingestion labels: The label to be applied to the events from this feed.
- S3 URI:
Click Next and then click Submit.
UDM mapping table
| Log Field | UDM Mapping | Logic |
|---|---|---|
billingRegion_label |
additional.fields |
Merged |
bytes_label |
additional.fields |
Merged |
cache_status_label |
additional.fields |
Merged |
customfield_label |
additional.fields |
Merged |
errorCode_label |
additional.fields |
Merged |
error_code_label |
additional.fields |
Merged |
fileSizeBucket_label |
additional.fields |
Merged |
jsonPayload_cookie_label |
additional.fields |
Merged |
jsonPayload_cp_label |
additional.fields |
Merged |
jsonPayload_overheadbytes_label |
additional.fields |
Merged |
objSize_label |
additional.fields |
Merged |
objsize_label |
additional.fields |
Merged |
overheadBytes_label |
additional.fields |
Merged |
rspContentLen_label |
additional.fields |
Merged |
rspcontentlen_label |
additional.fields |
Merged |
trace_label |
additional.fields |
Merged |
transferTimeMSec_label |
additional.fields |
Merged |
turnAroundTimeMSec_label |
additional.fields |
Merged |
has_principal |
metadata.event_type |
Mapped: true → NETWORK_HTTP, true → NETWORK_CONNECTION, true → STATUS_UPDATE |
has_user |
metadata.event_type |
Mapped: true → USER_UNCATEGORIZED |
insertId |
metadata.product_log_id |
Directly mapped |
proto_type |
network.application_protocol |
Directly mapped |
jsonPayload.reqmethod |
network.http.method |
Directly mapped |
reqMethod |
network.http.method |
Directly mapped |
jsonPayload_referer |
network.http.referral_url |
Directly mapped |
jsonPayload.statuscode |
network.http.response_code |
Directly mapped |
statusCode |
network.http.response_code |
Directly mapped |
UA |
network.http.user_agent |
Directly mapped |
jsonPayload.ua |
network.http.user_agent |
Directly mapped |
jsonPayload.reqid |
network.session_id |
Directly mapped |
reqId |
network.session_id |
Directly mapped |
jsonPayload.tlsversion |
network.tls.version |
Directly mapped |
tlsVersion |
network.tls.version |
Directly mapped |
version |
network.tls.version_protocol |
Directly mapped |
cliIP |
principal.asset.ip |
Merged |
jsonPayload.cliip |
principal.asset.ip |
Merged |
cliIP |
principal.ip |
Merged |
jsonPayload.cliip |
principal.ip |
Merged |
jsonPayload.city |
principal.location.city |
Directly mapped |
jsonPayload.country |
principal.location.country_or_region |
Directly mapped |
jsonPayload.totalbytes |
principal.network.received_bytes |
Renamed/mapped |
totalBytes |
principal.network.received_bytes |
Directly mapped |
logName |
security_result.category_details |
Merged |
asn_label |
security_result.detection_fields |
Merged |
querystr_label |
security_result.detection_fields |
Merged |
securityrules_label |
security_result.detection_fields |
Merged |
service_name_label |
security_result.detection_fields |
Merged |
jsonPayload.reqhost |
target.asset.hostname |
Directly mapped |
jsonPayload.reqpath |
target.file.full_path |
Directly mapped |
reqPath |
target.file.full_path |
Directly mapped |
jsonPayload.rspcontenttype |
target.file.mime_type |
Directly mapped |
rspContentType |
target.file.mime_type |
Directly mapped |
jsonPayload.reqhost |
target.hostname |
Directly mapped |
jsonPayload.reqport |
target.port |
Renamed/mapped |
reqPort |
target.port |
Renamed/mapped |
uncompressedsize |
target.process.file.size |
Directly mapped |
resource.type |
target.resource.resource_subtype |
Directly mapped |
| N/A | metadata.event_type |
Constant: NETWORK_HTTP |
| N/A | metadata.product_name |
Constant: Akamai Datastream |
| N/A | metadata.vendor_name |
Constant: AkamaiDatastream |
Change Log
View the Change Log for this parser
Need more help? Get answers from Community members and Google SecOps professionals.