Filter source objects by storage class

For transfers from Amazon S3, you can filter objects by their storage class.

By default, when transferring from Amazon S3:

• Objects in standard and non-archival storage classes, including Glacier Instant Retrieval (GLACIER_IR), are transferred.
• Objects in Glacier Deep Archive (DEEP_ARCHIVE) are transferred only if they are restored.
• Objects in Glacier Flexible Retrieval (GLACIER) are skipped regardless of restore status.

Specify a storage class list

To specify the storage classes to include:

Google Cloud console

Follow the instructions in Transfer from Amazon S3 to Cloud Storage.

After you've specified the source and destination for the transfer, select Filter by storage class and select the storage classes to include.

gcloud CLI

This feature is not supported in gcloud CLI.

REST

Use the transferSpec.objectConditions.includeStorageClasses option to specify the storage classes to include.

For example, to transfer only STANDARD and GLACIER storage classes:

POST https://storagetransfer.googleapis.com/v1/transferJobs
{
  "description": "DESCRIPTION",
  "status": "ENABLED",
  "projectId": "PROJECT_ID",
  "transferSpec": {
      "awsS3DataSource": {
          "bucketName": "AWS_SOURCE_NAME",
          "awsAccessKey": {
            "accessKeyId": "AWS_ACCESS_KEY_ID",
            "secretAccessKey": "AWS_SECRET_ACCESS_KEY"
          }
      },
      "gcsDataSink": {
          "bucketName": "GCS_SINK_NAME"
      },
      "objectConditions": {
          "includeStorageClasses": ["STANDARD", "GLACIER"]
      }
  }
}

When includeStorageClasses is specified and is not blank, only objects belonging to the specified storage classes are transferred.

For transfers from Amazon S3, valid values are listed in the AWS documentation: https://docs.aws.amazon.com/AmazonS3/latest/userguide/sc-howtoset.html.

Archived objects

Objects in S3 Glacier Deep Archive (DEEP_ARCHIVE) and S3 Glacier Flexible Retrieval (GLACIER) must be restored before they can be transferred.

Storage Transfer Service provides counters to track archived objects that were skipped during the transfer process:

• unrestoredDeepArchiveObjectsSkippedCount: The number of Glacier Deep Archive (DEEP_ARCHIVE) objects that were skipped because they were not restored.
• unsupportedS3GlacierObjectsSkippedCount: The number of Glacier Flexible Retrieval (GLACIER) objects skipped because they were not restored, or because GLACIER was not specified as an included storage type.