Collect Fivetran logs

Supported in:

Google secops SIEM

This document explains how to configure Fivetran to push logs to Google Security Operations using webhooks.

Fivetran is a data integration platform that automates data pipelines from various sources to data warehouses. Fivetran generates operational events including connector sync events, transformation events, and connection status changes. These events can be sent to external endpoints via outbound webhooks for monitoring, alerting, and security analysis.

Before you begin

Make sure you have the following prerequisites:

A Google SecOps instance.
A Fivetran account with admin or account-level permissions.
Access to the Google Cloud Console (for API key creation).
Fivetran account with REST API access enabled.

Create webhook feed in Google SecOps

Create the feed

Go to SIEM Settings > Feeds.
Click + Add New.
Select Configure a single feed.
In the Feed name field, enter a name (for example, Fivetran Events).
Select Webhook as the Source type.
Select Fivetran as the Log type.
Click Next.
Specify values for the following input parameters:
- Split delimiter (optional): Leave empty.
- Asset namespace: The asset namespace.
- Ingestion labels: The label to be applied to the events from this feed.
Click Next.
Review your new feed configuration and click Submit.

Generate and save secret key

On the feed details page, click Generate Secret Key.
A dialog displays the secret key.
Copy and save the secret key securely.

Get the feed endpoint URL

Go to the Details tab of the feed.
In the Endpoint Information section, copy the Feed endpoint URL.
Save this URL for the next steps.
Click Done.

Create Google Cloud API key

Google SecOps requires an API key for authentication.

Create the API key

Go to the Google Cloud Console Credentials page.
Select your project.
Click Create credentials > API key.
Click Edit API key to restrict the key.

Restrict the API key

In the API key settings:
- Name: Enter a descriptive name (for example, SecOps Webhook API Key).
Under API restrictions:
1. Select Restrict key.
2. In the dropdown, search for and select Google SecOps API (or Chronicle API).
Click Save.
Copy the API key value and save it securely.

Configure Fivetran webhook

Construct the webhook URL

Combine the endpoint URL and API key:
```
<ENDPOINT_URL>?key=<API_KEY>
```

Create webhook using Fivetran REST API

Get Fivetran API credentials

Sign in to your Fivetran account.
Go to Account Settings > API Config.
Click Generate API Key if you don't have one.
Copy the API Key and API Secret.

Create account-level webhook

Use this method to receive events from all connectors in your account.

Open a terminal and run the following command:

curl -X POST [https://api.fivetran.com/v1/webhooks/account](https://api.fivetran.com/v1/webhooks/account) \
        -u "API_KEY:API_SECRET" \
        -H "Content-Type: application/json" \
        -H "Accept: application/json" \
        -d '{
                "url": "[https://malachiteingestion-pa.googleapis.com/v2/unstructuredlogentries:batchCreate?key=YOUR_CHRONICLE_API_KEY](https://malachiteingestion-pa.googleapis.com/v2/unstructuredlogentries:batchCreate?key=YOUR_CHRONICLE_API_KEY)",
                "events": [
                        "sync_start",
                        "sync_end",
                        "transformation_start",
                        "transformation_succeeded",
                        "transformation_failed",
                        "connection_successful",
                        "connection_failure",
                        "create_connector",
                        "pause_connector",
                        "resume_connector",
                        "edit_connector",
                        "delete_connector",
                        "force_update_connector",
                        "resync_connector",
                        "resync_table"
                ],
                "active": true,
                "secret": "YOUR_CHRONICLE_SECRET_KEY"
        }'

Webhook details

Available webhook events

Event	Description
`sync_start`	Connector sync started
`sync_end`	Connector sync completed
`transformation_start`	Transformation started
`transformation_succeeded`	Transformation completed successfully
`transformation_failed`	Transformation failed
`connection_successful`	Connection test succeeded
`connection_failure`	Connection test failed
`create_connector`	New connector created

Webhook retry behavior

Fivetran automatically retries failed webhooks for up to 24 hours with the following schedule:

Retry	Time After Initial Attempt
1st retry	6 minutes
2nd retry	27 minutes
3rd retry	1 hour 45 minutes
4th retry	6 hours 25 minutes
5th retry	23 hours 13 minutes

UDM mapping table

Log Field	UDM Mapping	Logic
`jsonPayload.connector_id`	`additional.connector_id`	Value copied directly.
`jsonPayload.connector_type`	`additional.connector_type`	Value copied directly.
`jsonPayload.data.query`	`additional.query`	Value copied directly.
N/A	`metadata.event_type`	Set based on presence of principal and target.
`jsonPayload.event`	`metadata.product_event_type`	Value copied directly.
`jsonPayload.sync_id`	`metadata.product_log_id`	Value copied directly.
`jsonPayload.connector_name`	`principal.hostname`	Value copied directly.
`resource.labels.email_id`	`principal.user.email_addresses`	Mapped if format is valid email.
`resource.labels.unique_id`	`principal.user.userid`	Value copied directly.
`severity`	`security_result.severity`	"INFO" maps to INFORMATIONAL.
`logName`	`target.resource.name`	Value copied directly.
N/A	`target.resource.type`	Set to DATABASE.
N/A	`metadata.product_name`	Set to FIVETRAN.
N/A	`metadata.vendor_name`	Set to FIVETRAN.

Need more help? Get answers from Community members and Google SecOps professionals.