Troubleshooting Storage Intelligence issues

This document describes how to troubleshoot common issues with Storage Intelligence, Storage Insights inventory reports, Storage Insights datasets, and Storage batch operations.

Storage Intelligence configuration errors

The following sections describe errors you might encounter when configuring or managing Storage Intelligence for a resource.

400: Invalid bucket name

Issue: The request returns 400 Bad Request with the message The specified bucket is not valid.

Solution: The request is invalid. Ensure the request meets the following requirements:

• Use locations/global. Storage Intelligence doesn't support other locations.
• Ensure that bucket names or regular expressions in bucket_id_regexes are valid.

The following is an example of a valid request:

curl -X PATCH \
    -H "Authorization: Bearer $(gcloud auth print-access-token)" \
    -H "Content-Type: application/json" \
    -d '{
      "edition_config": "STANDARD",
      "filter": {
        "included_cloud_storage_buckets": {
          "bucket_id_regexes": [
            "my-bucket-name",
            "prod-data-.*"
          ]
        }
      }
    }' \
    "https://storage.googleapis.com/v2/projects/PROJECT_ID/locations/global/intelligenceConfig?updateMask=edition_config,filter"

400: Invalid argument - empty update mask

Issue: When you send a configuration or update request, the request returns 400 Bad Request with the message Empty UPDATE_MASK in the request.

Solution: Provide a non-empty UPDATE_MASK in your request. UPDATE_MASK specifies a comma-separated list of FieldMask fields in the IntelligenceConfig resource to update (such as updateMask=edition_config or updateMask=edition_config,filter).

400: Invalid update mask path

Issue: When updating a configuration, the request returns 400 Bad Request with the message Invalid UPDATE_MASK paths.

Solution: Verify that each field name in UPDATE_MASK matches a valid field in the IntelligenceConfig resource.

400: Field is not editable

Issue: When updating a configuration, the request returns 400 Bad Request with the message Invalid UPDATE_MASK: UPDATE_TIME field is not editable.

Solution: Remove uneditable system fields (such as UPDATE_TIME) from UPDATE_MASK. Only specify mutable fields defined in IntelligenceConfig.

400: Invalid value

Issue: The request returns 400 Bad Request with the message Invalid value at storage_intelligence.edition_config.

Solution: Set edition_config to a supported value: INHERIT, STANDARD, or DISABLED.

400: Non-empty filter

Issue: The request returns 400 Bad Request with the message Non-empty filter cannot be specified for INHERIT or DISABLED edition configuration.

Solution: Remove bucket filters from the request. Bucket filters aren't supported when edition_config is set to INHERIT or DISABLED.

400: Empty location or bucket values in filter

Issue: The request returns 400 Bad Request with the message Empty location or bucket values in filter.

Solution: Ensure that neither location nor bucket is an empty string in your bucket filter.

Storage Insights common issues

This section describes how to resolve common issues with inventory reports and datasets.

Multiple inventory reports generated daily

Issue: An inventory report configuration generates multiple report files each day.

Solution: Cloud Storage shards inventory reports for buckets with more than 1,000,000 objects, generating one shard per 1,000,000 objects. For example, a bucket with 3,500,000 objects generates four report shards and a manifest file that lists each shard.

Inventory reports not appearing in the destination bucket

Issue: Inventory reports don't appear in the destination bucket.

Solution: If reports aren't delivered to the destination bucket, verify the following:

• Ensure the configured start date has passed. For more information, see Create an inventory report configuration.

• View your inventory report history to check for failures and their root causes. To see your inventory report history, complete the following steps:

  1. In the Google Cloud console, go to the Cloud Storage Buckets page.
     

     Go to Buckets

  2. In the list of buckets, click the name of the source bucket that contains the inventory report configuration.

  3. On the Bucket details page, click the Inventory reports tab.

  4. In the list of inventory report configurations, click the UUID of the inventory report configuration that generated the reports you want to check.

  5. Check for failures in the Inventory report history section. You can hold the pointer over Help () to get details about why a failure occurred.

• Make sure the project-level service agent is granted the IAM roles required for reading and writing inventory reports. For more information, see Grant required roles to the service agent.

Inventory report delays

Issue: Inventory report generation is delayed.

Solution: Report generation times vary. Delays of up to 24 hours are normal.

Datasets not getting populated

Issue: Storage Insights dataset tables remain empty.

Solution: In your linked BigQuery dataset, check error_attributes_view for error codes. For more information, see Troubleshoot dataset errors.

Null values in 'ref' column when querying datasets

Issue: When querying Storage Insights datasets in BigQuery, the ref column returns null.

Solution: For objects ending in /, the ref column in datasets is null.

If the ref column returns null values when you query Storage Insights datasets in BigQuery, verify that you have granted the required connection permissions and roles, including access to Cloud Storage resources, as described in Analyze object data and metadata using BigQuery.

Storage batch operations job validation errors

This section describes validation errors that occur when submitting a batch operations job request to storagebatchoperations.googleapis.com.

400: Invalid job ID or resource name

Issue: The job creation request returns a 400 Bad Request (INVALID_ARGUMENT) response with the reason JOB_ID_INVALID or RESOURCE_NAME_TOO_LONG.

Solution: Verify that the job ID consists of 1–63 lowercase alphanumeric characters or hyphens ([a-z0-9]([-a-z0-9]*[a-z0-9])?) and that the full resource path doesn't exceed 1,024 bytes. For more information, see Job name.

400: Conflicting or missing transformation parameters

Issue: The job creation request returns a 400 Bad Request (INVALID_ARGUMENT) response with the reason TRANSFORMATION_NOT_SPECIFIED, REWRITE_OBJECT_MISSING_PARAMETERS, PUT_OBJECT_HOLD_MISSING_PARAMETERS, or PUT_METADATA_MISSING_PARAMETERS.

Solution: Specify exactly one transformation type with all required parameters. If configuring object retention, verify that Object Lock is enabled on the bucket and that timestamps use RFC 3339 UTC format. For more information about parameter requirements by transformation, see Job type.

400: Overlapping or duplicate object prefixes

Issue: The job creation request returns a 400 Bad Request (INVALID_ARGUMENT) response with the reason OBJECT_PREFIX_OVERLAP or DUPLICATE_OBJECT_PREFIX.

Solution: Remove duplicate prefixes and ensure that no prefix in included_object_prefixes is a prefix of another entry in the list. For more information, see Object prefixes.

400: Manifest file formatting and access issues

Issue: The job creation request returns a 400 Bad Request (INVALID_ARGUMENT) response with the reason MANIFEST_LOCATION_REQUIRED or MANIFEST_LOCATION_INVALID, or the job can't read the manifest.

Solution: Verify that the manifest URI is a valid CSV path (gs://<bucket_name>/<path>/<object_name>.csv) and that the Storage batch operations service agent has the roles/storage.objectViewer role on the manifest bucket. For more information about CSV formatting and schema requirements, see Manifest.

400: Storage Insights dataset discovery errors

Issue: Using an Storage Insights dataset for object discovery returns a 400 Bad Request (INVALID_ARGUMENT or FAILED_PRECONDITION) response with the reason BUCKET_DISCOVERY_SNAPSHOT_TOO_OLD, TARGET_LOCATIONS_REQUIRED_FOR_SNAPSHOT_TIME, or BUCKET_DISCOVERY_TOO_MANY_BUCKETS.

Solution: Verify that snapshot_time is within the last 48 hours, specify target_locations for the buckets, and ensure the discovery query matches no more than 1,000 buckets. For more information, see Create a manifest using Storage Insights datasets.

400: Storage class transformation fails on Autoclass-enabled buckets

Issue: The job creation request returns a 400 Bad Request (FAILED_PRECONDITION) response with the reason AUTOCLASS_STORAGE_CLASS_TRANSFORMATION_UNSUPPORTED.

Solution: You can't run storage class transformations on buckets with Autoclass enabled. Either target a bucket without Autoclass or remove the storage class transformation. For more information, see Autoclass restrictions.

400: Object ACL updates fail on uniform bucket-level access buckets

Issue: The job creation request returns a 400 Bad Request (FAILED_PRECONDITION) response with the reason UBLA_OBJECT_ACL_UPDATE_UNSUPPORTED.

Solution: You can't update object ACLs on buckets with uniform bucket-level access enabled. Manage access using IAM roles at the bucket or project level instead. For more information, see uniform bucket-level access.

Storage batch operations runtime and execution issues

This section describes issues that occur during the asynchronous execution of a batch operations job.

403: Permission errors during execution

Issue: A batch job fails during execution with 403 Forbidden (PERMISSION_DENIED).

Solution: Grant the Storage batch operations service agent ([e-mail]) the required IAM roles for your transformation type. For more information, see Grant permissions to the service agent.

CMEK encryption errors during object rewrite

Issue: Object rewrites fail with 400 Bad Request or 403 Forbidden due to Cloud KMS key status or permission errors.

Solution: Verify that the Cloud KMS key is Enabled and resides in the same region as the target bucket, and that the service agent has the roles/cloudkms.cryptoKeyEncrypterDecrypter role. For more information, see Job type: Rewrite object.

High failure count in error_summaries

Issue: A batch job completes with a non-zero counters.failed_object_count and error codes in error_summaries (such as 404 NOT_FOUND, 412 FAILED_PRECONDITION, or 403 PERMISSION_DENIED).

Solution: Run gcloud storage batch-operations jobs describe with the --location flag (for example, gcloud storage batch-operations jobs describe JOB_ID --location=LOCATION) to view the aggregated error breakdown, and check Cloud Logging for per-object error logs. For more information, see Get job details.

Storage batch operations job fails due to a snapshot that's over two days old

Issue: When you create a CEL filter-based storage batch operations job, job creation fails. The error message states that the snapshot time is older than two days.

Solution: To prevent actions on outdated object states, storage batch operations automatically fails job creation. This failure occurs if the selected snapshot is older than two days. Select one of the following methods to resolve this issue:

• Use a manifest file: Query your dataset manually in BigQuery. Export the results to a CSV manifest file, and upload the file to a Cloud Storage bucket. You can then create the batch operations job using the manifest method to avoid the two-day limit.
• Check dataset configurations: Verify that your dataset configurations are active and not paused. Confirm that dataset snapshots run successfully. For information about how to verify your configurations, see View a dataset configuration.
• Use target location and snapshot time overrides: Specify the --target-snapshot-time flag to bypass the two-day staleness failure by explicitly selecting a snapshot in RFC 3339 format. Specify the --target-locations flag to limit the operation to the locations where the snapshot exists. You can use these overrides to resolve synchronization delays that prevent the automated global snapshot from updating. Consequently, you can manually target a more recent regional snapshot. For the command syntax, see Create a job using advanced filters.

Caution: Providing a manual manifest file or overriding the snapshot age bypasses the safety checks designed to prevent actions on outdated object states. Under normal operations, Storage Insights updates dataset snapshots once every 24 hours. Before using these bypass methods, verify that targeting objects based on older metadata is safe for your workload.

CEL filter-based storage batch operations job fails on newly subscribed projects

Issue: Running a CEL filter-based storage batch operations job on a newly subscribed project fails because the system cannot find a valid snapshot.

Solution: After you enable the Storage Intelligence subscription, you must wait 24 hours before running CEL filter-based storage batch operations jobs. This delay lets the system perform the initial metadata snapshot and establish the starting snapshot time.

CEL filter-based storage batch operations job fails with permissions errors or throws runtime errors

Issue: A CEL filter-based storage batch operations job fails during execution or returns runtime permissions errors.

Solution: storage batch operations uses your user credentials to process objects. The job fails if you don't have the necessary IAM reader or writer permissions on the targeted buckets and objects. This issue occurs when your CEL filters select resources that you don't have access to. Confirm that your account has the Storage Admin (roles/storage.admin), Storage Object Admin (roles/storage.objectAdmin), or equivalent roles for all buckets and objects in the scope of the job. For instructions on granting roles, see Use IAM permissions.

Monitoring and log analysis

For more information about inspecting per-object execution failures and error payloads in Cloud Logging, see View storage batch operations logs.

What's next

• Overview of Storage Intelligence
• Configure and manage Storage Intelligence
• Create and manage batch operations jobs
• Analyze object data and metadata using BigQuery