Introduction to the BigQuery Data Transfer Service
The BigQuery Data Transfer Service lets you automate scheduled, managed data transfers into BigQuery. After you configure a data transfer, the BigQuery Data Transfer Service automatically loads data into BigQuery on a regular basis. You can also initiate data backfills to recover from outages or gaps, perform dataset copies, and run scheduled queries. You can't use BigQuery Data Transfer Service to transfer data out of BigQuery.
To configure and manage transfers, you can use the Google Cloud console, the bq command-line tool, or the BigQuery Data Transfer Service API. The Pipelines & Connections page also provides a streamlined workflow (preview) for creating transfers. To receive service updates, subscribe to the BigQuery DTS announcements group.
Supported data sources
The BigQuery Data Transfer Service supports loading data from the following data sources:
| Category | Data source |
|---|---|
| Cloud storage | Amazon Simple Storage Service (Amazon S3) |
| Cloud storage | Azure Blob Storage |
| Cloud storage | Cloud Storage |
| Databases and data warehouses | Amazon Redshift |
| Databases and data warehouses | Apache Hive Metastore |
| Databases and data warehouses | Microsoft SQL Server (Preview) |
| Databases and data warehouses | MySQL |
| Databases and data warehouses | Oracle |
| Databases and data warehouses | PostgreSQL |
| Databases and data warehouses | Snowflake |
| Databases and data warehouses | Teradata |
| Google services | Campaign Manager |
| Google services | Comparison Shopping Service (CSS) Center (Preview) |
| Google services | Display & Video 360 |
| Google services | Google Ad Manager |
| Google services | Google Ads |
| Google services | Google Analytics 4 |
| Google services | Google Merchant Center (Preview) |
| Google services | Google Play |
| Google services | Search Ads 360 |
| Google services | YouTube Channel |
| Google services | YouTube Content Owner |
| Marketing platforms | Facebook Ads |
| Marketing platforms | HubSpot (Preview) |
| Marketing platforms | Klaviyo (Preview) |
| Marketing platforms | Mailchimp (Preview) |
| Payment platforms | PayPal (Preview) |
| Payment platforms | Shopify (Preview) |
| Payment platforms | Stripe (Preview) |
| SaaS platforms | Salesforce |
| SaaS platforms | Salesforce Marketing Cloud |
| SaaS platforms | ServiceNow |
Data delivery SLO considerations
The Data Delivery SLO applies to automatically scheduled data transfers using the BigQuery Data Transfer Service from sources within Google Cloud.
For data transfers involving third-party or non-Google Cloud sources, service outages with these sources can impact performance with the BigQuery Data Transfer Service. As such, the Data Delivery SLO does not apply to BigQuery Data Transfer Service data transfers from non-Google Cloud sources.
Supported regions
Like BigQuery, the BigQuery Data Transfer Service is a multi-regional resource, with many additional single regions available.
A BigQuery dataset's locality is specified when you create a destination dataset to store the data transferred by the BigQuery Data Transfer Service. When you set up a transfer, the transfer configuration itself is set to the same location as the destination dataset. The BigQuery Data Transfer Service processes and stages data in the same location as the destination dataset.
The BigQuery Data Transfer Service supports data transfers from any region where your data is stored to any location where your destination dataset is located.
For detailed information about transfers and region compatibility for BigQuery Data Transfer Service, see Dataset locations and transfers. For supported regions for BigQuery, see Dataset locations.
Using reservation slots with data transfers
Jobs triggered by the BigQuery Data Transfer Service only use reservation slots if the project, folder, or organization is assigned to a reservation with any of the following job types:
- Query jobs using
QUERY - Load jobs using
PIPELINE
Pricing
For information on BigQuery Data Transfer Service pricing, see the Pricing page.
Once data is transferred to BigQuery, standard BigQuery storage and query pricing applies.
Quotas
For information on BigQuery Data Transfer Service quotas, see the Quotas and limits page.
Troubleshoot transfer configurations
For information about resolving issues with BigQuery Data Transfer Service, see Troubleshoot transfer configurations.
What's next
- To learn how to create a transfer, see the documentation for your data source.