Collect Fastly CDN logs

Supported in:

Google secops SIEM

Note: This feature is covered by Pre-GA Offerings Terms of the Google Security Operations Service Specific Terms. Pre-GA features might have limited support, and changes to pre-GA features might not be compatible with other pre-GA versions. For more information, see the Google SecOps Technical Support Service guidelines and the Google SecOps Service Specific Terms.

This document explains how to ingest Fastly CDN logs to Google Security Operations using Amazon S3.

Fastly is a content delivery network (CDN) and edge cloud platform that provides real-time content delivery, security, and edge computing services. Fastly's Real-Time Log Streaming feature can send CDN access logs, WAF events, and other telemetry data to various destinations including Amazon S3 for security monitoring and analysis.

Before you begin

Make sure that you have the following prerequisites:

• A Google SecOps instance
• Privileged access to Fastly control panel or API
• Privileged access to AWS (S3, IAM)
• A Fastly API token with global or global:read scope

Create Fastly API token

1. Sign in to the Fastly control panel at https://manage.fastly.com.
2. Click your user icon in the upper-right corner.
3. Go to Account > Personal API tokens.
4. Click Create Token.
5. In the Create API Token dialog, provide the following:
   • Name: Enter a descriptive name (for example, Google SecOps S3 Logging Configuration).
   • Type: Select User token.
   • Scope: Select Global access (required to configure logging endpoints).
   • Expiration: Select Never expire or set an appropriate expiration date.
6. Click Create Token.
7. Copy and save the API token securely.

Note: The API token is displayed only once. If you lose it, you must create a new token.

Configure AWS S3 bucket and IAM for Google SecOps

1. Create Amazon S3 bucket following this user guide: Creating a bucket.
2. Save bucket Name and Region for future reference (for example, fastly-cdn-logs).
3. Create a User following this user guide: Creating an IAM user.
4. Select the created User.
5. Select Security credentials tab.
6. Click Create Access Key in section Access Keys.
7. Select Third-party service as Use case.
8. Click Next.
9. Optional: Add description tag.
10. Click Create access key.
11. Click Download .csv file to save the Access Key and Secret Access Key for future reference.
12. Click Done.
13. Select the Permissions tab.
14. Click Add permissions in section Permissions policies.
15. Select Add permissions.
16. Select Attach policies directly.
17. Search for AmazonS3FullAccess policy.
18. Select the policy.
19. Click Next.
20. Click Add permissions.

Configure Fastly S3 logging endpoint

Option A: Configure via Fastly control panel

1. Sign in to the Fastly control panel at https://manage.fastly.com
2. Select the service you want to configure logging for.
3. Click Edit configuration and select Clone active version to create a new draft version.
4. Click Logging in the left navigation menu.
5. In the Amazon Web Services S3 section, click Create endpoint.

6. In the Create an Amazon S3 endpoint form, provide the following configuration details:

   • Basic settings:

     • Name: Enter a descriptive name (for example, secops-s3-logs).
     • Placement: Select Format Version Default (recommended).

     • Log format: Enter the following JSON format string:

       {"timestamp":"%{begin:%Y-%m-%dT%H:%M:%S%z}t","client_ip":"%{req.http.Fastly-Client-IP}V","geo_country":"%{client.geo.country_name}V","geo_city":"%{client.geo.city}V","geo_region":"%{client.geo.region}V","datacenter":"%{server.datacenter}V","host":"%{if(req.http.Fastly-Orig-Host, req.http.Fastly-Orig-Host, req.http.Host)}V","url":"%{json.escape(req.url)}V","request_method":"%{json.escape(req.method)}V","request_protocol":"%{json.escape(req.proto)}V","request_referer":"%{json.escape(req.http.referer)}V","request_user_agent":"%{json.escape(req.http.User-Agent)}V","response_state":"%{json.escape(fastly_info.state)}V","response_status":%{resp.status}V,"response_reason":%{if(resp.response, "\"%22\"+json.escape(resp.response)+\"%22\", \"null\")}V,"response_body_size":%{resp.body_bytes_written}V,"response_header_size":%{resp.header_bytes_written}V,"request_body_size":%{req.body_bytes_read}V,"request_header_size":%{req.header_bytes_read}V,"cache_status":"%{fastly_info.state}V","is_tls":"%{if(req.is_ssl, \"true\", \"false\")}V","tls_protocol":"%{cstr_escape(tls.client.protocol)}V","tls_cipher":"%{cstr_escape(tls.client.cipher)}V","server_identity":"%{json.escape(server.identity)}V","is_edge":%{if(fastly.ff.visits_this_service == 0, "true", "false")}V,"time_elapsed_usec":%{time.elapsed.usec}V,"time_start_sec":%{time.start.sec}V}
       

     Note: This format includes essential CDN fields, geographic data, TLS information, and timing metrics. You can customize this format based on your requirements using variables from the Useful variables to log documentation.
     • Timestamp format: Leave as default (strftime compatible string).

   S3 bucket configuration:

   • Bucket name: Enter the S3 bucket name (for example, fastly-cdn-logs).
   • Domain: Leave empty if your bucket is in US Standard region. For other regions, enter the appropriate S3 endpoint (for example, s3-us-west-2.amazonaws.com for us-west-2 region).
   • Access method: Select User Credentials
   • Access key: Enter the AWS access key from step 11.
   • Secret key: Enter the AWS secret access key from step 11.
   • Period: Enter 3600 (logs are finalized every hour).

7. Click Advanced options to expand additional settings:

   Advanced settings:

   • Path: Enter fastly-cdn/ (optional prefix for organizing logs within the bucket).
   • PGP public key: Leave empty (encryption not required for this integration).
   • Select a log line format: Select Blank (we're using custom JSON format).
   • Compression: Select gzip (recommended to reduce storage costs).
   • Redundancy level: Select Standard.
   • ACL: Select private (recommended).
   • Server side encryption: Select None (or configure as per your security requirements).
   • Maximum bytes: Enter 0 (no maximum file size limit).

8. Click Create to create the logging endpoint.

9. Click Activate in the top-right corner.

10. Select Activate on Production to deploy your configuration changes.

11. In the confirmation dialog, click Activate to confirm.

Note: Fastly will begin streaming logs to S3 immediately. Files are finalized and made available for reading based on the configured period (default 3600 seconds).

Option B: Configure via Fastly API

If you prefer to configure logging via API, use the following approach:

1. Get your Fastly service ID from the control panel or by listing services:

   curl -H "Fastly-Key: YOUR_FASTLY_TOKEN" https://api.fastly.com/service
   

2. Clone the active version to create a new draft version:

   curl -X PUT -H "Fastly-Key: YOUR_FASTLY_TOKEN" \
     "https://api.fastly.com/service/SERVICE_ID/version/ACTIVE_VERSION/clone"
   

3. Create the S3 logging endpoint on the new draft version:

   curl -X POST -H "Fastly-Key: YOUR_FASTLY_TOKEN" \
     -H "Content-Type: application/x-www-form-urlencoded" \
     "https://api.fastly.com/service/SERVICE_ID/version/NEW_VERSION/logging/s3" \
     --data-urlencode 'name=secops-s3-logs' \
     --data-urlencode 'bucket_name=fastly-cdn-logs' \
     --data-urlencode 'access_key=YOUR_AWS_ACCESS_KEY' \
     --data-urlencode 'secret_key=YOUR_AWS_SECRET_KEY' \
     --data-urlencode 'path=fastly-cdn/' \
     --data-urlencode 'period=3600' \
     --data-urlencode 'gzip_level=9' \
     --data-urlencode 'format_version=2' \
     --data-urlencode 'format={"timestamp":"%{begin:%Y-%m-%dT%H:%M:%S%z}t","client_ip":"%{req.http.Fastly-Client-IP}V","geo_country":"%{client.geo.country_name}V","geo_city":"%{client.geo.city}V","host":"%{if(req.http.Fastly-Orig-Host, req.http.Fastly-Orig-Host, req.http.Host)}V","url":"%{json.escape(req.url)}V","request_method":"%{json.escape(req.method)}V","response_status":%{resp.status}V,"cache_status":"%{fastly_info.state}V"}'
   

4. Activate the new version:

   curl -X PUT -H "Fastly-Key: YOUR_FASTLY_TOKEN" \
     "https://api.fastly.com/service/SERVICE_ID/version/NEW_VERSION/activate"
   

Note: Replace SERVICE_ID, ACTIVE_VERSION, NEW_VERSION, YOUR_FASTLY_TOKEN, YOUR_AWS_ACCESS_KEY, and YOUR_AWS_SECRET_KEY with your actual values.

Verify log delivery to S3

1. Wait approximately 1 hour (based on the configured period) for the first log file to be finalized.
2. In the AWS Console, go to S3 > Buckets.
3. Click your bucket name (for example, fastly-cdn-logs).
4. Navigate to the fastly-cdn/ prefix folder.

5. Verify that log files are being created with naming pattern similar to:

   fastly-cdn/YYYY-MM-DD-HH-MM-SS-RANDOM_STRING.log.gz
   

6. Download a sample log file and verify the JSON format matches your configuration.

Configure a feed in Google SecOps to ingest Fastly CDN logs

1. Go to SIEM Settings > Feeds.
2. Click Add New Feed.
3. On the next page, click Configure a single feed.
4. Enter a unique name for the Feed name.
5. Select Amazon S3 V2 as the Source type.
6. Select Fastly CDN as the Log type.
7. Click Next and then click Submit.

8. Specify values for the following fields:

   • S3 URI: s3://fastly-cdn-logs/fastly-cdn/.
   • Source deletion option: Select the deletion option according to your preference.
   • Maximum File Age: Include files modified in the last number of days (default is 180 days).
   • Access Key ID: User access key with access to the S3 bucket.
   • Secret Access Key: User secret key with access to the S3 bucket.
   • Asset namespace: The asset namespace.
   • Ingestion labels: The label to be applied to the events from this feed.
   Note: If you select the Delete transferred files or Delete transferred files and empty directories option, make sure that you granted appropriate permissions to the service account.

9. Click Next and then click Submit.

UDM mapping table

Log field	UDM mapping	Logic
request_protocol, response_state, response_reason, response_body_size, fastly_server, fastly_is_edge	additional.fields	Merged with labels created from each field if not empty
	metadata.event_type	Set to "GENERIC_EVENT", then to "STATUS_UPDATE" if principal_mid_present is true
request_method	network.http.method	Value copied directly
request_user_agent	network.http.parsed_user_agent	Converted to parseduseragent then renamed
request_referer	network.http.referral_url	Value copied directly
response_status	network.http.response_code	Converted to string then to integer
request_user_agent	network.http.user_agent	Value copied directly
host	principal.asset.hostname	Value copied directly
client_ip	principal.asset.ip	Value copied directly (after grok validation as IP)
host	principal.hostname	Value copied directly
client_ip	principal.ip	Value copied directly (after grok validation as IP)
geo_city	principal.location.city	Value copied directly
geo_country	principal.location.country_or_region	Value copied directly
security_result	security_result	Value copied directly
url	target.url	Value copied directly
	metadata.product_name	Set to "FASTLY_CDN"
	metadata.vendor_name	Set to "FASTLY_CDN"

Need more help? Get answers from Community members and Google SecOps professionals.