The size limit for manifest files used in agent-based transfers has been removed.

For more information, see Transfer specific files or objects using a manifest.

Organization Policy Service custom constraints are now available for Storage Transfer Service. You can use custom constraints to control how Storage Transfer Service is used in your organization. For example, you can restrict transfers to only allow Cloud Storage to Cloud Storage transfers, or restrict transfers to a specific list of approved source buckets.

See Custom organization policy constraints for details.

Storage Transfer Service event-driven transfers are now available for Azure Blob Storage and Data Lake Storage Gen2 sources. Event-driven transfers listen to Azure Event Grid notifications via Azure Storage Queues to automatically transfer new or updated objects from your Azure container to Cloud Storage.

For more information, see Event-driven transfers from Azure Blob Storage or Data Lake Storage Gen2.

You can now transfer data from AWS S3 or Azure Blob Storage to Cloud Storage over a private network connection, using Cross-Cloud Interconnect or Partner Interconnect. Transferring data over a private connection can optimize costs, provide dedicated bandwidth, and help meet compliance needs by keeping data off the public internet.

See Transfer from AWS or Azure over a customer-managed private network for details.

Storage Transfer Service agent-based transfers now support Cloud Storage regional endpoints. When you specify a Cloud Storage regional endpoint for a transfer agent, all data transfer traffic through that agent to or from a Cloud Storage bucket remains within the specified Google Cloud region.

See Regional endpoints for details.

You can now delegate permissions to a user-managed service account for agentless transfers, instead of using the default Google-managed service agent. This allows for more granular permission control by assigning specific bucket permissions to different user-managed service accounts, and restricting which users can use those service accounts for transfer jobs.

See Delegate service agent permissions to a user-managed service account for details.

Cloud Logging for agent-based transfers now logs skipped files. A skipped file is logged when the file already exists in the sink, and your transfer job is configured to ignore existing files.

See Cloud Logging for Storage Transfer Service for details.

Transfers over a Google-managed private network are now supported from more AWS regions, including regions in Europe and Asia. See the full list of supported regions.

When providing a URL list to transfer files, you can now host the list itself in an access-controlled Cloud Storage bucket. See Transfer from public URLs for full details.

We've released a lighter weight (~50% reduction in image size) and more secure version of the transfer agent container image. If you're running agents that were installed on or before June 17th, 2024, we highly recommend that you delete those agents and install new agents into your agent pool. Any existing transfers using that pool will resume once the new agents are installed.

Cloud Logging for Storage Transfer Service now supports transfers involving POSIX file systems.

See Cloud Logging for Storage Transfer Service for more details.

Feature

Storage Transfer Service now supports transfers from Amazon S3 over a Google-managed private network. Transfer jobs that select this option pay no AWS egress fees; instead, a flat per-GiB rate is charged by Google Cloud. This allows you to transfer data at a potentially lower overall cost.

Learn more about egress options for S3 transfers, including the managed private network.

Storage Transfer Service has added support for Shared Keys as an authentication method when transferring from Microsoft Azure Storage.

To use an Azure Shared Key, you must store the key value in Secret Manager. See Save your Microsoft credentials in Secret Manager for details.

Support for transfers from cloud and on-premises Hadoop Distributed File System (HDFS) sources is now generally available (GA).

HDFS support allows for use cases such as migrating from on-premises storage to Cloud Storage, archiving data to free up on-premises storage space, replicating data to Google Cloud for business continuity, or transferring data to Google Cloud for analysis and processing.

See Transfer from HDFS to Cloud Storage for details.

Storage Transfer Service has added preview support for transferring managed folders between Cloud Storage buckets. Permissions on managed folders are copied between buckets when using this option.

See Transfer Cloud Storage managed folders for details.

Storage Transfer Service supports transfers from cloud and on-premises Hadoop Distributed File System (HDFS) sources. This feature is in limited release; complete the form linked from the user guide at Transfer from HDFS to Cloud Storage to request access.

This new feature supports use cases such as migrating from on-premises storage to Cloud Storage, archiving data to free up on-premises storage space, replicating data to Google Cloud for business continuity, or transferring data to Google Cloud for analysis and processing.

Storage Transfer Service has updated transfer agent behavior when transferring to Cloud Storage. To align with Cloud Storage auto-scaling, agents now gradually ramp up the number of requests being made to Cloud Storage. Customers who transfer many small files may notice initially slower transfer speeds while the requests are ramping up, but increased performance across the duration of the transfer.

You can now transfer data from Amazon S3 via your CloudFront domain. Learn more.

Event-driven transfers are now generally available (GA).

Event-driven transfers provide serverless, real-time replication from AWS S3 to Cloud Storage, and between Cloud Storage buckets. With this new capability, you can accelerate your event-driven analytics pipeline, enable automatic replication across Cloud Storage buckets, create a backup copy of data in a different region or project, or perform live migration.

Learn more about Event-driven transfers.

Transfers from Amazon S3 no longer require s3:GetBucketLocation permission on the source bucket.

For a full list of required permissions, see Configure access to a source: Amazon S3.

Cloud Monitoring for Storage Transfer Service is now Generally Available (GA). This integration allows you to monitor the number of objects and data being transferred and to compute transfer speeds. This GA launch also adds monitoring of errors and error codes.

See Monitor transfer jobs for details.

Feature

Storage Transfer Service now offers Preview support for providing a Secret Manager secret when creating transfer jobs from AWS S3 or Azure storage.

Secret Manager provides strong encryption, role-based access control, and audit logging to protect your secrets.

For details, see the Secret Manager section of the following documents:

• Configure access to a source: Amazon S3
• Configure access to a source: Microsoft Azure Storage

We discovered a security vulnerability in the Storage Transfer Service agent container. We've fixed this issue with a container update that is more secure.

If you're running agents that were installed on or before February 17, 2023, you should follow the instructions in the Action required email sent to your account email address to update the container image.

Agents installed after February 17, 2023 do not need to be updated.

Storage Transfer Service now publishes the IP ranges from which it makes requests to your AWS or Azure storage resources when performing a transfer. This allows you to restrict your resources by IP, and still allow Storage Transfer Service access.

For details, see the IP restrictions section of the following documents:

• Configure access to a source: Amazon S3
• Configure access to a source: Microsoft Azure Storage

Storage Transfer Service can now optionally preserve UID, GID, and mode metadata for folders, and recreate empty folders, when transferring between file systems.

See Metadata preservation for details.

Transfers from S3-compatible storage to Cloud Storage are now generally available (GA). This feature builds on support for Multipart upload and List Object V2, which makes Cloud Storage suitable for running applications written for the S3 API.

With this new feature, customers can seamlessly copy data from self-managed object storage to Google Cloud Storage. For customers moving data from AWS S3 to Cloud Storage, this feature provides an option to control network routes to Google Cloud, resulting in considerably lower egress charges.

See Transfer from S3-compatible sources for details.

Support for Manifest in Storage Transfer Service is now generally available (GA). You can use Manifest to transfer a specific list of objects, object versions, and files from cloud and on-premises sources. Programmatic users can use the output of an upstream operation generating a list of files and objects as an input for Storage Transfer Service to act upon.

Temporary pricing for Storage Transfer Service has expired; refer to Pricing for current charges.

Customers under contract can reach out to Google Cloud representatives for details.

Storage Transfer Service now offers Preview support for tracking progress of a Transfer Job using Cloud Monitoring, allowing you to monitor the number of objects and amount of data copied by Storage Transfer Service in near real-time.

See Monitor transfer jobs for details.

Storage Transfer Service now offers GA Support for transferring data between file systems, including on-premises file systems and Filestore instances. This allows you to use the Transfer Service API, gcloud command line tool, or the Cloud console to migrate data from a self-managed file system to Filestore; accelerate data transfer from an on-premise file system to a cloud file system; or move data between on-premises systems.

You can also transfer specific files or objects using a manifest for file system to file system transfers.

Storage Transfer Service offers Preview support for event-driven transfers - serverless, real-time replication from AWS S3 to Cloud Storage, and between Cloud Storage buckets. With this new capability, you can accelerate your event-driven analytics pipeline, enable automatic replication across Cloud Storage buckets, create a backup copy of data in a different region or project, or perform live migration.

Learn more about Event-driven transfers.

Multipart upload for transfers originating from a file system is now generally available (GA) and enabled by default.

Multipart uploads can speed up transfers that include large files. You must grant additional permissions on the destination or intermediate bucket; see Permissions for file system transfers for details.