# Load Salesforce data into BigQuery
To schedule recurring data transfers from your Salesforce Sales Cloud
to BigQuery, create a transfer configuration to specify what data
objects to transfer, and how often to schedule the data transfer. You can create
a transfer configuration using either the Google Cloud console, the `bq`
command-line tool, or the BigQuery Data Transfer Service API. Once you have set up the
transfer configuration,
[BigQuery Data Transfer Service](https://docs.cloud.google.com/bigquery/docs/dts-introduction.md.txt)
transfers the latest data into a BigQuery table on the specified
schedule.
To learn about how a Salesforce transfer works, see
[Introduction to Salesforce data transfers](https://docs.cloud.google.com/bigquery/docs/salesforce-transfer-intro.md.txt).
## Limitations
Salesforce data transfers are subject to the following
limitations:
- The Salesforce connector only supports transfers from Salesforce Sales Cloud.
- The Salesforce connector only supports fields included in Salesforce Bulk API V1 version 64.0. Some fields that were included in previous versions of the Salesforce Bulk API might not be supported. For more information about these changes to the Salesforce connector, see [Salesforce Bulk API](https://docs.cloud.google.com/bigquery/docs/transfer-changes.md.txt#salesforce).
- The Salesforce connector uses Salesforce Bulk
API V1 to connect to the Salesforce Sales Cloud endpoint to
retrieve data.
- The Salesforce connector only supports the Salesforce Bulk API V1 to connect to the Salesforce instance, and only supports the transfer of entities which are supported by the Salesforce Bulk API. For more information about what entities are supported, see [Entity is not supported by the Bulk API](https://help.salesforce.com/s/articleView?id=000383508&type=1).
- The Salesforce connector does not support the transfer of
the following objects that have binary fields.
- `Attachment`
- `ContentVersion`
- `Document`
- `StaticResource`
- `Scontrol`
- `EmailCapture`
- `MailMergeTemplate`
- The minimum interval time between recurring data transfers is 15 minutes. The
default interval for a recurring transfer is 24 hours.
- Due to Salesforce processing limits, scheduling too many data
transfers at a time can lead to delays or failures. We recommend that you limit
Salesforce data transfers to the following:
- Have no more than 10 assets per transfer configuration.
- Across your different transfer configurations, have no more than 10 simultaneous transfer runs at a time.
- A single transfer configuration can only support one data transfer run at a
given time. In the case where a second data transfer is scheduled to run
before the first transfer is completed, then only the first data transfer
completes while any other data transfers that overlap with the first transfer
is skipped.
- To avoid skipped transfers within a single transfer configuration, we recommend that you increase the duration of time between large data transfers by configuring the **Repeat frequency**.
- If you are using network attachments with your data transfer, you must
[set up a public network address translation (NAT)](https://docs.cloud.google.com/nat/docs/set-up-manage-network-address-translation.md.txt)
with a static IP address. For more information, see
[Set up IP allowlist for Salesforce transfers](https://docs.cloud.google.com/bigquery/docs/salesforce-transfer.md.txt#salesforce-allowlist).
- If your network attachment was created in a Google Cloud project that is different from the Google Cloud project of the data transfer, you must grant the `compute.networkAttachments.get` and `compute.networkAttachments.update` permissions to the [service agent](https://docs.cloud.google.com/bigquery/docs/enable-transfer-service.md.txt#service_agent) that belongs to the Google Cloud project where the data transfer was created.
-
If your configured network attachment and virtual machine (VM) instance are
located in different regions, there might be cross-region data movement when
you transfer data from Salesforce.
### Incremental transfer limitations
Incremental Salesforce transfers are subject to the following limitations:
- You can only choose `TIMESTAMP` columns as watermark columns.
- Incremental ingestion is only supported for assets with valid watermark columns.
- Values in a watermark column must be monotonically increasing.
- Incremental transfers cannot sync delete operations in the source table.
- A single transfer configuration can only support either incremental or full ingestion.
- You cannot update objects in the `asset` list after the first incremental ingestion run.
- You cannot change the write mode in a transfer configuration after the first incremental ingestion run.
- You cannot change the watermark column or the primary key after the first incremental ingestion run.
- The destination BigQuery table is clustered using the provided primary key and is subject to [clustered table limitations](https://docs.cloud.google.com/bigquery/docs/clustered-tables.md.txt#limitations).
- When you update an existing transfer configuration to the incremental ingestion mode for the first time, the first data transfer after that update transfers all available data from your data source. Any subsequent incremental data transfers will transfer only the new and updated rows from your data source.
To learn about how incremental transfers work, see
[Full or incremental transfers](https://docs.cloud.google.com/bigquery/docs/salesforce-transfer-intro.md.txt#full-incremental-transfers).
## Before you begin
The following sections describe the steps that you need to take before you
create a Salesforce data transfer.
### Create a Salesforce Connected App
You must
[create a Salesforce Connected App](https://help.salesforce.com/s/articleView?id=sf.connected_app_create.htm&type=5)
with the following required configurations:
- [Configure the Basic Information](https://help.salesforce.com/s/articleView?id=sf.connected_app_create_basics.htm&type=5) in the Connected App. The **Connected App Name** and **Contact Email** fields are required for a Salesforce transfer.
- [Enable OAuth Settings](https://help.salesforce.com/s/articleView?id=sf.connected_app_create_api_integration.htm&type=5) with the following configurations:
- Select the **Enable OAuth Settings** checkbox.
- In the **Callback URL** field, enter the following:
- For a production environment, enter `https://login.salesforce.com/services/oauth2/token`.
- For a sandbox environment, enter `https://test.salesforce.com/services/oauth2/token`.
- Verify that the **Issue JSON Web Token (JWT)-based access tokens for named users** checkbox isn't selected.
- In the **Selected OAuth Scopes** section, select **Manage user data via APIs (api)**.
- Select the **Enable Client Credentials Flow** , then click **OK** on the notice that appears.
Once you have configured the Connected App with the required configurations,
click **Save**. You are redirected to the detail page of your newly created
Connected App.
Once you have created the Connected App, you must also configure the client
credentials flow by doing the following:
1. Click **Setup**.
2. In the search bar, search for *Connected Apps*.
3. Click **Manage Apps** \> **Connected Apps** . If you are using the Salesforce Lightning Experience, click **Manage Connected Apps**.
4. On the Connected App that you have created, click **Edit**.
5. The **App details** page appears. In the **Client Credentials Flow** section, enter your username in the **Run As** field. You can use the finder tool in this field to ensure that you have selected the correct user.
6. Click **Save**.
### Required Salesforce information
You must have the following Salesforce information when creating a
Salesforce data transfer:
| Parameter Name | Description |
|---|---|
| `myDomain` | Your [My Domain](https://help.salesforce.com/s/articleView?id=sf.domain_name_overview.htm) in Salesforce. |
| `clientId` | Consumer Key of the Salesforce connected application. |
| `clientSecret` | OAuth Client Secret or Consumer Secret of the Salesforce connected application. |
To obtain your `myDomain`, `clientID`, and `clientSecret` values,
select one of the following options:
### Salesforce Classic
### Retrieve `myDomain` details
To find your `myDomain`, do the following:
1. Sign in to the Salesforce platform.
2. Click **Setup**.
3. In the search bar, search for *My Domain*.
4. In the search results, click **Domain Management** \> **My Domain**.
In the **My Domain Details** section, your `myDomain` appears as
the prefix in **Current My Domain URL** . For example, if the My Domain URL is
`example.my.salesforce.com`, the `myDomain` value to use is `example`.
### Retrieve `ClientId` and `ClientSecret` details
To find your `ClientId` and `ClientSecret` values, do the following:
1. Sign in to the Salesforce platform.
2. Click **Setup**.
3. In the search bar, search for *Apps*.
4. In the **Build** section in the search results, click **Create** \> **Apps**.
5. Click a **Connected App Name**.
6. In the **Connected Apps** details page, click **Manage Consumer Details**.
7. Verify your identity using one of the registered methods. You can view the consumer details page for up to five minutes before you're prompted to verify your identity again.
8. In the **Consumer Details** page, the **Consumer Key** is your `ClientId` value. The **Customer Secret** is your `ClientSecret` value.
### Salesforce Lightning Experience
### Retrieve `myDomain` details
To find your `myDomain`, do the following:
1. Sign in to the Salesforce platform.
2. Click **Setup**.
3. In the search bar, search for *My Domain*.
4. In the search results, click **Company Settings** \> **My Domain**.
In the **My Domain Details** section, your `myDomain` appears as
the prefix in **Current My Domain URL** . For example, if the My Domain URL is
`example.my.salesforce.com`, the `myDomain` value to use is `example`.
### Retrieve `ClientId` and `ClientSecret` details
1. Sign in to the Salesforce platform.
2. Click **Setup**.
3. In the search bar, search for *Apps*.
4. In the search results, click **Apps** \> **App Manager**.
5. Find a connected app, then click **View**.
6. Click **Manage Consumer Details**.
7. Verify your identity using one of the registered methods. You can view the consumer details page for up to five minutes before you're prompted to verify your identity again.
8. In the **Consumer Details** page, the **Consumer Key** is your `ClientId` value. The **Customer Secret** is your `ClientSecret` value.
### Set up IP allowlist for Salesforce transfers
You must configure your Google Cloud environment and your Salesforce
account to add specific IP addresses to the allowlist for data transfers.
This ensures that Salesforce only accepts connections from a trusted,
static IP address. This step is required if you are using network attachments
with your data transfers.
To do so, you must first set up and configure your Google Cloud network to use
a static IP address:
1. [Set up a public network address translation (NAT)](https://docs.cloud.google.com/nat/docs/set-up-manage-network-address-translation.md.txt) with a static IP address within your Virtual Private Cloud network. The CloudNAT must be configured within the same region as the destination dataset for this data transfer.
2. [Set up a network attachment](https://docs.cloud.google.com/vpc/docs/create-manage-network-attachments.md.txt) within the same VPC network. This resource is used by the BigQuery Data Transfer Service to access private services.
Next, you must [configure the trusted IP ranges in
Salesforce](https://help.salesforce.com/s/articleView?id=xcloud.security_networkaccess.htm&type=5).
When adding the range of the IP address, use the static IP address from your
Google Cloud public NAT for both the beginning and ending IP addresses for the
IP range.
> [!NOTE]
> **Note:** For more granular control, you can apply IP restrictions at the profile level or on every API call. For more information, see [Restrict Login IP Addresses in Profiles](https://help.salesforce.com/s/articleView?id=platform.login_ip_ranges.htm&type=5).
Once you have set up the IP ranges, you can now specify the static IP when you
[set up your transfer configuration](https://docs.cloud.google.com/bigquery/docs/salesforce-transfer.md.txt#sf-transfer-setup)
by selecting your network attachment in the **Network attachment** field.
### BigQuery prerequisites
- Verify that you have completed all actions required to [enable the BigQuery Data Transfer Service](https://docs.cloud.google.com/bigquery/docs/enable-transfer-service.md.txt).
- [Create a BigQuery dataset](https://docs.cloud.google.com/bigquery/docs/datasets.md.txt) to store your data.
- If you intend to set up transfer run notifications for Pub/Sub, ensure that you have the `pubsub.topics.setIamPolicy` Identity and Access Management (IAM) permission. Pub/Sub permissions are not required if you only set up email notifications. For more information, see [BigQuery Data Transfer Service run notifications](https://docs.cloud.google.com/bigquery/docs/transfer-run-notifications.md.txt).
### Required BigQuery roles
To get the permissions that
you need to create a BigQuery Data Transfer Service data transfer,
ask your administrator to grant you the
[BigQuery Admin](https://docs.cloud.google.com/iam/docs/roles-permissions/bigquery.md.txt#bigquery.admin) (`roles/bigquery.admin`) IAM role on your project.
For more information about granting roles, see [Manage access to projects, folders, and organizations](https://docs.cloud.google.com/iam/docs/granting-changing-revoking-access.md.txt).
This predefined role contains
the permissions required to create a BigQuery Data Transfer Service data transfer. To see the exact permissions that are
required, expand the **Required permissions** section:
#### Required permissions
The following permissions are required to create a BigQuery Data Transfer Service data transfer:
- BigQuery Data Transfer Service permissions:
- `bigquery.transfers.update`
- `bigquery.transfers.get`
- BigQuery permissions:
- `bigquery.datasets.get`
- `bigquery.datasets.getIamPolicy`
- `bigquery.datasets.update`
- `bigquery.datasets.setIamPolicy`
- `bigquery.jobs.create`
You might also be able to get
these permissions
with [custom roles](https://docs.cloud.google.com/iam/docs/creating-custom-roles.md.txt) or
other [predefined roles](https://docs.cloud.google.com/iam/docs/roles-overview.md.txt#predefined).
For more information, see [Grant `bigquery.admin` access](https://docs.cloud.google.com/bigquery/docs/enable-transfer-service.md.txt#grant_bigqueryadmin_access).
## Set up a Salesforce data transfer
Add Salesforce data into BigQuery by setting up a
transfer configuration using one of the following options:
### Console
1. Go to the Data transfers page in the Google Cloud console.
[Go to Data transfers](https://console.cloud.google.com/bigquery/transfers)
2. Click **Create transfer**.
3. In the **Source type** section, for **Source** , choose **Salesforce**.
4. In the **Data source details** section, do the following:
- For **Network attachment** , select a network attachment from the list. You must [configure your public NAT and set up your IP allow list](https://docs.cloud.google.com/bigquery/docs/salesforce-transfer.md.txt#salesforce-allowlist) before you can use a network attachment with this data transfer.
- For **My Domain** , enter your Salesforce [My Domain](https://help.salesforce.com/s/articleView?id=sf.domain_name_overview.htm).
- For **Client ID**, enter the Salesforce connected application Consumer Key.
- For **Client secret**, enter the Salesforce connected application Consumer Secret.
- For **Ingestion type** , select **Full** or **Incremental** .
- If you select **Incremental** for **Write mode** , select either **Append** or **Upsert** . For more information about the different write modes, see [Full or incremental transfers](https://docs.cloud.google.com/bigquery/docs/salesforce-transfer-intro.md.txt#full-incremental-transfers).
- For **Salesforce objects to transfer** , click **Browse** :
- Select any objects to be transferred to the BigQuery destination dataset. You can also manually enter any objects to include in the data transfer in this field.
- If you have selected **Append** as your incremental write mode, you must select a column as the watermark column.
- If you have selected **Upsert** as your incremental write mode, you must select a column as the watermark column, and then select one or more columns as the primary key.
5. In the **Destination settings** section, for **Dataset**, choose the
dataset that you created to store your data.
6. In the **Transfer config name** section, for **Display name**, enter a
name for the data transfer.
7. In the **Schedule options** section:
- In the **Repeat frequency** list, select an option to specify how often this data transfer runs. To specify a custom repeat frequency, select **Custom** . If you select **On-demand** , then this transfer runs when you [manually trigger the transfer](https://docs.cloud.google.com/bigquery/docs/working-with-transfers.md.txt#manually_trigger_a_transfer).
- If applicable, select either **Start now** or **Start at set time**, and provide a start date and run time.
8. Optional: In the **Notification options** section, do the following:
- To enable email notifications, click the **Email notification** toggle. When you enable this option, the transfer administrator receives an email notification when a transfer run fails.
- To enable [Pub/Sub transfer run notifications](https://docs.cloud.google.com/bigquery/docs/transfer-run-notifications.md.txt) for this transfer, click the **Pub/Sub notifications** toggle. You can select your [topic](https://docs.cloud.google.com/pubsub/docs/overview.md.txt#types) name, or you can click **Create a topic** to create one.
9. Click **Save**.
### bq
Enter the [`bq mk` command](https://docs.cloud.google.com/bigquery/docs/reference/bq-cli-reference.md.txt#bq_mk)
and supply the transfer creation flag
`--transfer_config`:
```bash
bq mk
--transfer_config
--project_id=PROJECT_ID
--data_source=DATA_SOURCE
--display_name=NAME
--target_dataset=DATASET
--params='PARAMETERS'
```
Where:
- PROJECT_ID (optional): your Google Cloud project ID. If `--project_id` isn't supplied to specify a particular project, the default project is used.
- DATA_SOURCE: the data source --- `salesforce`.
- NAME: the display name for the data transfer configuration. The transfer name can be any value that lets you identify the transfer if you need to modify it later.
- DATASET: the target dataset for the transfer configuration.
- PARAMETERS: the parameters for the created transfer
configuration in JSON format. For example:
`--params='{"param":"param_value"}'`. The following are the parameters for
a Salesforce data transfer:
- `connector.authentication.oauth.clientId`: the Consumer Key of the Salesforce connected application.
- `connector.authentication.oauth.clientSecret`: OAuth Client Secret or Consumer Secret of the Salesforce connected application.
- `connector.authentication.oauth.myDomain`: the [Salesforce My Domain](https://help.salesforce.com/s/articleView?id=sf.domain_name_overview.htm). For example, if your domain URL is `example.my.salesforce.com`, then the value is `example`.
- `ingestionType`: specify either `full` or `incremental`. For more information, see [Full or incremental transfers](https://docs.cloud.google.com/bigquery/docs/salesforce-transfer-intro.md.txt#full-incremental-transfers).
- `writeMode`: specify either `WRITE_MODE_APPEND` or `WRITE_MODE_UPSERT`.
- `watermarkColumns`: specify columns in your table as watermark columns. This field is required for incremental transfers.
- `primaryKeys`: specify columns in your table as primary keys. This field is required for incremental transfers.
- `assets`: the path to the Salesforce objects to be transferred to BigQuery.
When specifying multiple assets during an incremental transfer, the values
of the `watermarkColumns` and `primaryKeys` fields correspond to the
position of values in the `assets` field. In the following example,
`Id` corresponds to the table `Account`, while `master_label`
and `type` corresponds to the table `CaseHistory`.
```bash
"primaryKeys":[['Id'], ['master_label','type']],
"assets":["Account","CaseHistory"],
```
The following command creates an incremental Salesforce data transfer in the
default project, and uses the `APPEND` write mode.
```bash
bq mk
--transfer_config
--target_dataset=mydataset
--data_source=salesforce
--display_name='My Transfer'
--params='{"assets": ["Account", "CaseHistory"]
"connector.authentication.oauth.clientId": "1234567890",
"connector.authentication.oauth.clientSecret":"ABC12345",
"connector.authentication.oauth.myDomain":"MyDomainName",
"connector.authentication.username":"user1@force.com",
"connector.authentication.password":"abcdef1234",
"ingestionType":"incremental",
"writeMode":"WRITE_MODE_UPSERT",
"watermarkColumns":["SystemModstamp","CreatedDate"]
"primaryKeys":[['Id'], ['master_label','type']]}'
```
### API
Use the [`projects.locations.transferConfigs.create`](https://docs.cloud.google.com/bigquery/docs/reference/datatransfer/rest/v1/projects.locations.transferConfigs/create.md.txt)
method and supply an instance of the [`TransferConfig`](https://docs.cloud.google.com/bigquery/docs/reference/datatransfer/rest/v1/projects.locations.transferConfigs.md.txt#TransferConfig)
resource.
When you save the transfer configuration, the Salesforce connector automatically triggers a transfer run according to your schedule option. With every transfer run, the Salesforce connector transfers all available data from Salesforce into BigQuery.
To manually run a data transfer outside of your regular schedule, you can start
a [backfill run](https://docs.cloud.google.com/bigquery/docs/working-with-transfers.md.txt#manually_trigger_a_transfer).
## Troubleshoot transfer setup
If you are having issues setting up your data transfer, see [Salesforce transfer issues](https://docs.cloud.google.com/bigquery/docs/transfer-troubleshooting.md.txt#salesforce-issues).
## What's next
- For an overview of the BigQuery Data Transfer Service, see [Introduction to BigQuery Data Transfer Service](https://docs.cloud.google.com/bigquery/docs/dts-introduction.md.txt).
- For information on using transfers including getting information about a transfer configuration, listing transfer configurations, and viewing a transfer's run history, see [Working with transfers](https://docs.cloud.google.com/bigquery/docs/working-with-transfers.md.txt).
- Learn how to [load data with BigQuery Omni operations](https://docs.cloud.google.com/bigquery/docs/load-data-using-cross-cloud-transfer.md.txt).