Use object contexts

Preview

This product or feature is subject to the "Pre-GA Offerings Terms" in the General Service Terms section of the Service Specific Terms. Pre-GA products and features are available "as is" and might have limited support. For more information, see the launch stage descriptions.

This page describes how to attach and manage contexts on Cloud Storage objects in the form of key-value pairs.

Get the required roles

To get the permissions that you need to create and manage object contexts, ask your administrator to grant you the following IAM roles on object:

• Create objects with contexts: Storage Object Creator (roles/storage.objectCreator)
• Attach, update, view, and delete contexts from objects: Storage Object User (roles/storage.objectUser)
• View context keys and values attached to objects: Storage Object Viewer (roles/storage.objectViewer)

For more information about granting roles, see Manage access to projects, folders, and organizations.

These predefined roles contain the permissions required to create and manage object contexts. To see the exact permissions that are required, expand the Required permissions section:

Required permissions

The following permissions are required to create and manage object contexts:

• Create an object with object contexts:
  • storage.objects.create
  • storage.objects.createContext
• Attach, update, and delete object contexts:
  • storage.objects.update
  • storage.objects.createContext
  • storage.objects.updateContext
  • storage.objects.deleteContext
• View object contexts:
  • storage.objects.get
  • storage.objects.list

You might also be able to get these permissions with custom roles or other predefined roles.

Attach contexts to new objects

Attach contexts to objects when you upload new objects to Cloud Storage buckets. Each context consists of a key and a value.

Command line

To attach contexts when you upload objects with the gcloud alpha storage cp command, use the --custom-contexts flag:

gcloud alpha storage cp OBJECT_LOCATION gs://DESTINATION_BUCKET_NAME --custom-contexts=KEY=VALUE,...

Where:

• OBJECT_LOCATION is the local path to your object. For example, Desktop/dog.png.
• DESTINATION_BUCKET_NAME is the name of the bucket to which you are uploading your object. For example, my-bucket.
• KEY is the context key to attach to an object. For example, Department. You can specify multiple key-value pairs separated by commas.
• VALUE is the value to associate with the context key. For example, Human resources.

Alternatively, create a JSON file that contains the contexts you want to attach to the objects, and use the --custom-contexts-file flag:

  {
    "KEY": {
      "value": "VALUE"
    },
    ...
  }

Where:

• KEY is the context key to attach to an object. For example, Department. You can specify multiple key-value pairs.
• VALUE is the value to associate with the context key. For example, Human resources.

To attach contexts when you upload directories with the gcloud alpha storage rsync command, use the --custom-contexts flag or the --custom-contexts-file flag:

gcloud alpha storage rsync DIRECTORY_LOCATION gs://DESTINATION_BUCKET_NAME --recursive --custom-contexts=KEY=VALUE,...

Where:

• DIRECTORY_LOCATION is the local path to your directory. For example, ~/my_directory.
• DESTINATION_BUCKET_NAME is the name of the bucket to which you are uploading your directory. For example, my-bucket.
• KEY is the context key to attach to objects. For example, Department. You can specify multiple key-value pairs separated by commas.
• VALUE is the value to associate with the context key. For example, Human resources.

JSON API

To attach contexts to objects when you upload new objects, use any of the following methods:

• Multipart uploads

• Resumable uploads

As part of the object metadata in JSON format, include the contexts field:

  {
    "contexts": {
      "custom": {
        "KEY": {
          "value": "VALUE"
        },
        ...
      }
    }
  }

Where:

• KEY is the context key to attach to an object. For example, Department. You can specify multiple key-value pairs in the custom object.
• VALUE is the value to associate with the context key. For example, Human resources.

Attach or modify contexts to an existing object

You can attach new contexts to your existing objects in the Cloud Storage buckets.

Command line

Use the gcloud alpha storage objects update command:

gcloud alpha storage objects update gs://BUCKET_NAME/OBJECT_NAME CUSTOM_CONTEXTS_FLAG

Where:

• BUCKET_NAME is the name of the bucket that contains the object for which you want to edit context. For example, my-bucket.
• OBJECT_NAME is the name of the object. For example, pets/dog.png.

• CUSTOM_CONTEXTS_FLAG is any of the following flags:

  • To replace all existing contexts, use --custom-contexts=KEY=VALUE,... or --custom-contexts-file=CUSTOM_CONTEXTS_FILE

    Where:

    • KEY is the context key to attach to an object. For example, Department. You can specify multiple key-value pairs separated by commas.
    • VALUE is the value to associate with the context key. For example, Human resources.
    • CUSTOM_CONTEXTS_FILE is the path to the JSON or YAML file that contains the contexts you want to attach to the object.

  • To delete all existing contexts, use --clear-custom-contexts.

  • To add, modify, or delete individual contexts, use a combination of --update-custom-contexts=KEY=VALUE,... and --remove-custom-contexts=KEY,...

    Where:

    • KEY is the context key that you want to attach to or delete from an object. For example, Department.
    • VALUE is the value to associate with the context key that you want to attach to an object. For example, Human resources.

If successful, the response looks like the following example:

Patching gs://my-bucket/pets/dog.png#1560574162144861...
  Completed 1

Client libraries

Java

For more information, see the Cloud Storage Java API reference documentation.

To authenticate to Cloud Storage, set up Application Default Credentials. For more information, see Set up authentication for client libraries.

import com.google.cloud.storage.Blob;
import com.google.cloud.storage.BlobId;
import com.google.cloud.storage.BlobInfo;
import com.google.cloud.storage.BlobInfo.ObjectContexts;
import com.google.cloud.storage.BlobInfo.ObjectCustomContextPayload;
import com.google.cloud.storage.Storage;
import com.google.cloud.storage.StorageOptions;
import com.google.common.collect.Maps;
import java.util.Map;

public class SetObjectContexts {
  public static void setObjectContexts(
      String projectId, String bucketName, String objectName, String key, String value)
      throws Exception {
    // The ID of your GCP project
    // String projectId = "your-project-id";

    // The ID of your GCS bucket
    // String bucketName = "your-unique-bucket-name";

    // The ID of your GCS object
    // String objectName = "your-object-name";

    // The context key-value you want to add
    // String key = "your-context-key";
    // String value = "your-context-value";

    try (Storage storage =
        StorageOptions.newBuilder().setProjectId(projectId).build().getService()) {
      BlobId blobId = BlobId.of(bucketName, objectName);
      Blob blob = storage.get(blobId);
      if (blob == null) {
        System.out.println("The object " + objectName + " was not found in " + bucketName);
        return;
      }

      // Recommended: Set a generation-match precondition to avoid potential race
      // conditions and data corruptions. The request to update returns a 412 error if
      // the object's generation number does not match your precondition.
      Storage.BlobTargetOption precondition = Storage.BlobTargetOption.generationMatch();

      // This section demonstrates how to upsert, delete all, and delete a specific context.

      // To upsert a context (if the key already exists, its value is replaced;
      // otherwise, a new key-value pair is added):
      ObjectCustomContextPayload payload =
          ObjectCustomContextPayload.newBuilder().setValue(value).build();
      Map<String, ObjectCustomContextPayload> custom = Maps.newHashMap();
      custom.put(key, payload);
      ObjectContexts contexts = ObjectContexts.newBuilder().setCustom(custom).build();

      /*
       * To delete all existing contexts:
       * ObjectContexts contexts = ObjectContexts.newBuilder().setCustom(null).build();
       */

      /*
       * To delete a specific key from the context:
       * Map<String, ObjectCustomContextPayload> custom = Maps.newHashMap();
       * custom.put(key, null);
       * ObjectContexts contexts = ObjectContexts.newBuilder().setCustom(custom).build();
       */
      BlobInfo pendingUpdate = blob.toBuilder().setContexts(contexts).build();
      storage.update(pendingUpdate, precondition);

      System.out.println(
          "Updated custom contexts for object " + objectName + " in bucket " + bucketName);
    }
  }
}

JSON API

1. Have gcloud CLI installed and initialized, which lets you generate an access token for the Authorization header.

2. Create a JSON file that contains the settings for the object, which must include the contexts configuration fields for the object.

   To add, modify, or overwrite existing contexts, use the following format:

     {
       "contexts": {
         "custom": {
           "KEY": {
             "value": "VALUE"
           },
           ...
         }
       }
     }

   Where:

   • KEY is the context key to attach to an object. For example, Department. You can specify multiple key-value pairs in the custom object.
   • VALUE is the value to associate with the context key. For example, Human resources.

   To delete all existing contexts, use the following format:

     {
       "contexts": {
         "custom": null
       }
     }

   To delete a specific key from the context, use the following format:

     {
       "contexts": {
         "custom": {
           "KEY": null,
           ...
         }
       }
     }

   Where:

   KEY is the context key that you want to delete from an object. For example, Department. You can specify multiple keys to delete from the custom object.

3. Use cURL to call the JSON API with a PATCH Object request:

   curl -X PATCH --data-binary @JSON_FILE_NAME \
     -H "Authorization: Bearer $(gcloud auth print-access-token)" \
     -H "Content-Type: application/json" \
     "https://storage.googleapis.com/storage/v1/b/BUCKET_NAME/o/OBJECT_NAME"

   Where:

   • JSON_FILE_NAME is the path to the file that includes the object contexts information.
   • BUCKET_NAME is the name of the bucket that contains the object for which you want to edit context. For example, my-bucket.
   • OBJECT_NAME is the URL-encoded name of the object. For example, pets/dog.png is URL-encoded as pets%2Fdog.png.

Alternatively, you can replace an object's context with a PUT Object request. The PUT object request also replaces other object metadata. Therefore, we don't recommend using the PUT object request.

View object contexts

You can view an object's contexts by listing object metadata or describing a specific object.

Command line

Use the gcloud alpha storage objects describe command:

gcloud alpha storage objects describe gs://BUCKET_NAME/OBJECT_NAME

Where:

• BUCKET_NAME is the name of the bucket containing the object whose context you want to view. For example, my-bucket.
• OBJECT_NAME is the name of the object whose context you want to view. For example, pets/dog.png

If successful, the response looks similar to the following example:

bucket: my-bucket
contexts:
  Department:
    createTime: '2023-01-01T00:00:00.000000+00:00'
    type: CUSTOM
    updateTime: '2023-01-01T00:00:00.000000+00:00'
    value: HR
  DataClassification:
    createTime: '2023-01-01T00:00:00.000000+00:00'
    type: CUSTOM
    updateTime: '2023-01-01T00:00:00.000000+00:00'
    value: Confidential
name: employees.txt

Note: To limit the output to only the context keys and values, use --format=contextsonly.

Client libraries

Java

For more information, see the Cloud Storage Java API reference documentation.

To authenticate to Cloud Storage, set up Application Default Credentials. For more information, see Set up authentication for client libraries.

import com.google.cloud.storage.Blob;
import com.google.cloud.storage.BlobInfo.ObjectContexts;
import com.google.cloud.storage.BlobInfo.ObjectCustomContextPayload;
import com.google.cloud.storage.Storage;
import com.google.cloud.storage.StorageOptions;
import java.util.Map;

public class GetObjectContexts {
  public static void getObjectContexts(String projectId, String bucketName, String objectName)
      throws Exception {
    // The ID of your GCP project
    // String projectId = "your-project-id";

    // The ID of your GCS bucket
    // String bucketName = "your-unique-bucket-name";

    // The ID of your GCS object
    // String objectName = "your-object-name";

    try (Storage storage =
        StorageOptions.newBuilder().setProjectId(projectId).build().getService()) {

      Blob blob = storage.get(bucketName, objectName);
      if (blob == null) {
        System.out.println("The object " + objectName + " was not found in " + bucketName);
        return;
      }
      ObjectContexts objectContexts = blob.getContexts();

      if (objectContexts != null) {
        Map<String, ObjectCustomContextPayload> customContexts = objectContexts.getCustom();
        if (customContexts == null) {
          System.out.println("No custom contexts found for object: " + objectName);
          return;
        }
        // Print blob's object contexts
        System.out.println("\nCustom Contexts:");
        for (Map.Entry<String, ObjectCustomContextPayload> custom : customContexts.entrySet()) {
          System.out.println(custom.getKey() + "=" + custom.getValue());
        }
      }
    }
  }
}

JSON API

1. Have gcloud CLI installed and initialized, which lets you generate an access token for the Authorization header.

2. Use cURL to call the JSON API with a GET Object request:

   curl -X GET \
     -H "Authorization: Bearer $(gcloud auth print-access-token)" \
     "https://storage.googleapis.com/storage/v1/b/BUCKET_NAME/o/OBJECT_NAME"

   Where:

   • BUCKET_NAME is the name of the bucket containing the object whose context you want to view. For example, my-bucket.
   • OBJECT_NAME is the URL-encoded name of the object whose context you want to view. For example, pets/dog.png, URL-encoded as pets%2Fdog.png.

   If successful, the response looks similar to the following example:

     {
       "kind": "storage#object",
       "name": "employees.txt",
       "bucket": "my-bucket",
       "contexts": {
         "custom": {
           "Department": {
             "value": "HR",
             "createTime": "2023-01-01T00:00:00.000Z",
             "updateTime": "2023-01-01T00:00:00.000Z"
           },
           "DataClassification": {
             "value": "Confidential",
             "createTime": "2023-01-01T00:00:00.000Z",
             "updateTime": "2023-01-01T00:00:00.000Z"
           }
         }
       }
     }
     

Filter objects by contexts

Filter objects by the existence of object context keys or their specific values. Filtering objects by contexts helps to locate and manage particular groups of objects efficiently. For details, see Filter objects by contexts.

Manage object contexts during object operations

Object contexts are preserved by default when you copy, rewrite, compose, move, or restore objects. You can also modify contexts during the copy, rewrite, and compose operations.

Command line

The gcloud alpha storage cp command, gcloud alpha storage rsync command, and gcloud alpha storage mv command preserve contexts from the source object by default. To modify contexts during these operations, use any of the following flags:

• The --custom-contexts or --custom-contexts-file flags to set new contexts for the destination object.
• The --clear-custom-contexts flag to prevent contexts from the source object from being attached to the destination object.
• A combination of the --update-custom-contexts and --remove-custom-contexts flags to modify individual contexts from the source object before attaching them to the destination object.

The gcloud alpha storage objects compose command merges contexts from the source objects and attaches them to the destination objects by default. Cloud Storage resolves conflicts by prioritizing contexts from source objects processed later. For more information about the object context behavior during a compose operation, see Composite object contexts. You can also specify new contexts for the destination object using the --custom-contexts or --custom-contexts-file flags.

JSON API

• To modify contexts during a copy or a rewrite object operation, include the contexts.custom property in the request body. If you don't include this property, contexts from the source object are preserved by default.

• When you compose objects, contexts from source objects merge into the destination object by default. Cloud Storage resolves conflicts by prioritizing contexts from source objects processed later. For more information about the object context behavior during a compose operation, see Composite object contexts. You can also specify new contexts for the destination object in the destination.contexts.custom property.

What's next

• Learn about the object context properties in the Cloud Storage API documentation.