Event-driven transfers

Storage Transfer Service can listen to event notifications in AWS, Azure, or Google Cloud to automatically transfer data that has been added or updated in the source location. Event-driven transfers are supported from AWS S3, Azure, or Cloud Storage, and always use a Cloud Storage bucket as the destination.

Event-driven transfers listen to:

  • Amazon S3 Event Notifications sent to Amazon SQS for AWS S3 sources.
  • Azure Storage Queues for Azure sources.
  • Pub/Sub subscriptions for Cloud Storage sources.

Object deletions are not detected; deleting an object at the source does not delete the associated object in the destination bucket.

Benefits of event-driven transfers

Because event-driven transfers listen for changes to the source bucket, updates are copied to the destination in near-real time. Storage Transfer Service doesn't need to execute a list operation against the source, saving time and money.

Use cases include:

  • Event-driven analytics: Replicate data from AWS or Azure to Cloud Storage to perform analytics and processing.

  • Cloud Storage replication: Enable automatic, asynchronous object replication between Cloud Storage buckets.

    Event-driven transfers with Storage Transfer Service differ from typical Cloud Storage replication by creating a copy of your data in a different bucket.

    This provides benefits such as:

    • Keeping development and production data in separate namespaces.
    • Sharing data without providing access to the original bucket.
    • Backing up to a different continent, or to an area not covered by dual-region and multi-region storage.

  • DR/HA setup: Replicate objects from source to backup destination in order of minutes:

    • Cross-cloud backup: Create a copy of an AWS S3 or Azure backup on Cloud Storage.
    • Cross-region or cross-project backup: Create a copy of Cloud Storage bucket in a different region or project.

  • Live migration: Event-driven transfer can power low-downtime migration, on the order of minutes of downtime, as a follow up step to one-time batch migration.

Learn how