Unstructured data and storage migration

This page provides architectural guidance and migration strategies for transferring unstructured datasets, object storage, and file shares to Cloud Storage on Google Cloud.

If you are a cloud architect, data engineer, or storage administrator, use the information on this page to evaluate data transfer methods, choose online or offline transfer tools, ensure data integrity, and optimize network bandwidth during migration.

Before you read this page, you should know the total size of your source datasets, available network transfer bandwidth, and your target migration timelines.

Choose a data transfer method

When planning your data transfer, use the following table to choose a transfer method based on your dataset size and network bandwidth:

Scenario Dataset size Recommended solution Considerations
Online data transfer over high-bandwidth network Terabytes to petabytes Storage Transfer Service Automated, continuous online transfer with checksum verification.
Offline data transfer with limited bandwidth Multiple petabytes Transfer Appliance Secure, physical appliance shipment to bypass network constraints.

Online data transfer by using Storage Transfer Service

Storage Transfer Service automates online data transfers from on-premises storage, POSIX file systems, or third-party cloud providers to Cloud Storage.

Key capabilities of Storage Transfer Service

Storage Transfer Service provides the following capabilities:

  • Managed scalability: scale transfers automatically without provisioning or managing local servers.
  • Data integrity verification: verify data accuracy automatically during transfer by using end-to-end MD5 checksums.
  • Continuous incremental synchronization: synchronize modified files continuously to minimize cutover downtime.

For information about the end-to-end transfer workflow, see Create transfers.

Offline data transfer by using Transfer Appliance

For large datasets where network bandwidth is constrained, use Transfer Appliance to ship data physically to Google Cloud.

What's next