Storage Insights datasets tables and schemas

This document describes the tables and their schemas included in Storage Insights datasets, which you can use to analyze operational patterns, monitor resource usage, and understand activity across your storage environment.

Dataset schema of metadata

The following sections describe the metadata fields included in datasets. For more information about BigQuery column modes, see Modes. The column modes determine how BigQuery stores and queries the data.

Bucket metadata

The bucket metadata schema includes the bucket_attributes_view and bucket_attributes_latest_snapshot_view tables. The bucket metadata tables have the following fields:

Metadata field Mode Type Description
snapshotTime NULLABLE TIMESTAMP The snapshotTime field stores the time of the bucket metadata snapshot refresh in RFC 3339 format.
name NULLABLE STRING The name of the bucket.
location NULLABLE STRING The location of the bucket. Object data for objects in the bucket resides in physical storage within this location.
project NULLABLE INTEGER The project number of the project the bucket belongs to.
storageClass NULLABLE STRING The bucket's default storage class.
public NULLABLE RECORD Deprecated. This field indicates whether a bucket is publicly accessible. Use iamConfiguration instead.
public.bucketPolicyOnly NULLABLE BOOLEAN Deprecated. This field, part of the public record, indicates whether uniform bucket-level access is enabled, which prevents access through object-level ACLs.
public.publicAccessPrevention NULLABLE STRING Deprecated. This field, part of the public record, indicates whether public access to the bucket is prevented.
autoclass NULLABLE RECORD The bucket's Autoclass configuration, which, when enabled, controls the storage class of objects based on how and when the objects are accessed.
autoclass.enabled NULLABLE BOOLEAN Whether or not Autoclass is enabled.
autoclass.toggleTime NULLABLE TIMESTAMP The time at which Autoclass was last enabled or disabled for this bucket, in RFC 3339 format.
versioning NULLABLE BOOLEAN Whether or not the bucket has versioning enabled. For more information, see Object Versioning.
lifecycle NULLABLE BOOLEAN Whether or not the bucket has a lifecycle configuration. See lifecycle management for more information.
metageneration NULLABLE INTEGER The metadata generation of this bucket.
timeCreated NULLABLE TIMESTAMP The creation time of the bucket in RFC 3339 format.
tags NULLABLE RECORD Deprecated. This field contains user-defined key-value pairs associated with the bucket. Use resourceTags instead.
tags.lastUpdatedTime NULLABLE TIMESTAMP Deprecated. This field indicates the last time the tags were updated.
tags.tagMap REPEATED RECORD Deprecated. This field, part of the tags record, contains the map of tag keys and values.
tags.tagMap.key NULLABLE STRING Deprecated. This field, part of the tags.tagMap record, represents the key of a tag.
tags.tagMap.value NULLABLE STRING Deprecated. This field, part of the tags.tagMap record, represents the value of a tag.
labels REPEATED RECORD User-provided bucket labels, in key-value pairs.
labels.key NULLABLE STRING An individual label entry.
labels.value NULLABLE STRING The label's value.
softDeletePolicy NULLABLE RECORD The bucket's soft delete policy, which defines the period of time during which objects in the bucket are retained in a soft-deleted state after being deleted. Objects in a soft-deleted state cannot be permanently deleted, and are restorable until their hardDeleteTime.
softDeletePolicy.effectiveTime NULLABLE DATETIME

The datetime at which the soft delete policy becomes effective, in RFC 3339 format.

softDeletePolicy.effectiveTime is updated whenever softDeletePolicy.retentionDurationSeconds is increased.
softDeletePolicy.retentionDurationSeconds NULLABLE LONG The period of time during which a soft-deleted object is retained and cannot be permanently deleted, in seconds. The value must be greater than or equal to 604800 (7 days) and less than 7776000 (90 days). The value can also be set to 0, which disables the soft delete policy.
iamConfiguration NULLABLE RECORD The IAM configuration for a bucket.
iamConfiguration.uniformBucketLevelAccess NULLABLE RECORD The bucket's uniform bucket-level access configuration.
iamConfiguration.uniformBucketLevelAccess.enabled NULLABLE BOOLEAN Whether or not the bucket uses uniform bucket-level access.
iamConfiguration.publicAccessPrevention NULLABLE STRING The bucket's public access prevention status, which is either "inherited" or "enforced".
resourceTags REPEATED RECORD The bucket's tags. For more information, see Cloud Resource Manager API.
resourceTags.key NULLABLE STRING The resourceTags key.
resourceTags.value NULLABLE STRING The resourceTags value.

Object metadata

The object metadata schema includes the object_attributes_view and object_attributes_latest_snapshot_view tables. The object metadata tables have the following fields:

Metadata field Mode Type Description
snapshotTime NULLABLE TIMESTAMP The snapshotTime field stores the time of the object metadata snapshot refresh in RFC 3339 format.
bucket NULLABLE STRING The name of the bucket containing this object.
location NULLABLE STRING The location of the bucket. Object data for objects in the bucket resides in physical storage within this location.
componentCount NULLABLE INTEGER Returned for composite objects only. Number of non-composite objects in the composite object. componentCount includes non-composite objects that were part of any composite objects used to compose the current object.
contentDisposition NULLABLE STRING Content-Disposition of the object data.
contentEncoding NULLABLE STRING Content-Encoding of the object data.
contentLanguage NULLABLE STRING Content-Language of the object data.
contentType NULLABLE STRING Content-Type of the object data.
crc32c NULLABLE INTEGER CRC32c checksum, as described in RFC 4960, Appendix B; encoded using base64 in big-endian byte order.
customTime NULLABLE TIMESTAMP A user-specified timestamp for the object in RFC 3339 format.
etag NULLABLE STRING HTTP 1.1 Entity tag for the object.
eventBasedHold NULLABLE BOOLEAN Whether or not the object is subject to an event-based hold.
generation NULLABLE INTEGER The content generation of this object.
md5Hash NULLABLE STRING MD5 hash of the data, encoded using base64. This field is not present for composite objects.
metadata REPEATED RECORD User-provided metadata, in key-value pairs.
metadata.key NULLABLE STRING An individual metadata entry.
metadata.value NULLABLE STRING The metadata value.
metageneration NULLABLE INTEGER The version of the metadata for this object at this generation.
name NULLABLE STRING The name of the object.
size NULLABLE INTEGER Content-Length of the data in bytes.
storageClass NULLABLE STRING Storage class of the object.
temporaryHold NULLABLE BOOLEAN Whether or not the object is subject to a temporary hold.
timeCreated NULLABLE TIMESTAMP The creation time of the object in RFC 3339 format.
timeDeleted NULLABLE TIMESTAMP The deletion time of the object in RFC 3339 format.
updated NULLABLE TIMESTAMP The modification time of the object metadata in RFC 3339 format.
timeStorageClassUpdated NULLABLE TIMESTAMP The time at which the object's storage class was last changed.
retentionExpirationTime NULLABLE TIMESTAMP The earliest time that the object can be deleted, which depends on any retention configuration set for the object and any retention policy set for the bucket that contains the object. The value for retentionExpirationTime is given in RFC 3339 format.
softDeleteTime NULLABLE DATETIME The time at which the object was soft deleted. Only available for objects in buckets with a soft delete policy.
hardDeleteTime NULLABLE DATETIME The time at which a soft-deleted object is permanently deleted and can no longer be restored. The value is the sum of the softDeleteTime value and the softDeletePolicy.retentionDurationSeconds value of the bucket. Only available for objects in buckets with a soft delete policy.
project NULLABLE INTEGER The project number of the project the bucket belongs to.

Latest bucket and object metadata snapshot

The linked dataset exposes the latest snapshot of the bucket and object metadata through the following dedicated views:

  • The bucket_attributes_latest_snapshot_view table provides the latest metadata for your Cloud Storage buckets. Its structure matches the Bucket metadata schema.

  • The object_attributes_latest_snapshot_view table provides the latest metadata for your Cloud Storage objects. Its structure matches the Object metadata schema.

Project metadata

The linked dataset exposes project metadata as a view named project_attributes_view:

Metadata field Mode Type Description
snapshotTime NULLABLE TIMESTAMP The snapshotTime field stores the time of the project metadata snapshot refresh in RFC 3339 format.
name NULLABLE STRING The name of the project.
id NULLABLE STRING The unique identifier for the project.
number NULLABLE INTEGER A numeric value associated with the project.

Dataset schema for activity data tables

Storage Insights datasets provide activity data, including operational metrics, error logs, and aggregated views with customizable scopes to support diverse analysis needs. The following sections describe the activity data schemas and the data available in them.

Object activity data

The object_events_view table in your Storage Insights dataset provides queryable records of create, update, and delete object operations, metadata, and details about the errors encountered during these operations. You can use the data to track and troubleshoot changes to your storage assets by querying the object_events_view view. The data in the object_events_view view is scoped to your Storage Intelligence subscription and lists all the actively enabled objects within it.

The object_events_view table includes the following fields:

Field Mode Type Description
requestCompletionTimestamp NULLABLE TIMESTAMP The timestamp when the request was completed, in RFC 3339 format.
project NULLABLE INTEGER The project number that owns the bucket resource.
bucketName NULLABLE STRING The name of the bucket specified in the request.
objectName NULLABLE STRING The name of the object specified in the request. This field can be null.
requestOperation NULLABLE STRING The Cloud Storage operation performed. This field is limited to write, update, and delete operations and can be null.
requestBytes NULLABLE INTEGER The number of bytes sent in the request.
responseBytes NULLABLE INTEGER The number of bytes sent in the response.
requestProcessingTimeMicros NULLABLE INTEGER The time in microseconds the server took to process the request. This time is measured from when the first byte is received until the response is sent. For resumable uploads, the response to the final upload request determines the endpoint.
requestId NULLABLE STRING The identifier for the request.
requestHost NULLABLE STRING The host specified in the original request.
requestHttpMethod NULLABLE STRING The HTTP method used for this request.
responseStatus NULLABLE INTEGER The HTTP status code sent in the server's response.
requestReferrer NULLABLE STRING The HTTP referrer for the request.
size NULLABLE INTEGER The size of the object in bytes.
generation NULLABLE INTEGER The content generation of the object, used for object versioning.
timeCreated NULLABLE TIMESTAMP The object's creation time, in RFC 3339 format.
storageClass NULLABLE STRING The storage class of the object.
errorReason NULLABLE STRING The reason for the error (limited to 429 errors).
location NULLABLE STRING The location of the object.

Project activity view

The project_activity_view table offers aggregated operational insights across your projects. The table summarizes total activity by counting operation types, bytes transferred, and response codes and errors. You can use the project activity insights for monitoring, tracking, and internal cost attribution. The data in the view is determined by your Storage Intelligence subscription scope. The view displays aggregated data only for the buckets and objects that are actively enabled under your subscription. For example, if you enable Storage Intelligence only for ten specific buckets, the view displays data exclusively for those ten buckets. Conversely, if you enable Storage Intelligence at a higher level (organization, folder, or project), all buckets and objects within that scope are automatically included, and the view displays the aggregated data for them.

The project_activity_view table includes the following fields:

Field Mode Type Description
snapshotStartTime NULLABLE TIMESTAMP The snapshot's start time, in RFC 3339 format.
snapshotEndTime NULLABLE TIMESTAMP The snapshot's end time, in RFC 3339 format.
project NULLABLE INTEGER The project number.
totalRequests NULLABLE INTEGER The total number of requests made across the project.
totalReads NULLABLE INTEGER The total number of read operations.
totalWrites NULLABLE INTEGER The total number of write operations.
totalUpdates NULLABLE INTEGER The total number of update operations.
totalDeletes NULLABLE INTEGER The total number of delete operations.
totalLists NULLABLE INTEGER The total number of list operations.
totalRequestBytes NULLABLE INTEGER The total number of request bytes transferred.
totalResponseBytes NULLABLE INTEGER The total number of response bytes transferred.
count4xx NULLABLE INTEGER The total number of 4xx errors.
count5xx NULLABLE INTEGER The total number of 5xx errors.
count2xx NULLABLE INTEGER The total number of 2xx success responses.
count400 NULLABLE INTEGER The total number of 400 errors for bad requests.
count401 NULLABLE INTEGER The total number of 401 errors for unauthorized requests.
count403 NULLABLE INTEGER The total number of 403 errors for forbidden requests.
count429 NULLABLE INTEGER The total number of 429 errors for usage limit exceeded requests.

Bucket activity view

The bucket_activity_view table offers aggregated operational insights across your buckets. The table summarizes total activity by counting operation types, bytes transferred, response codes, errors, and includes details about top prefixes. You can use the bucket activity insights for bucket-level monitoring, troubleshooting, metering, and analyzing your most and least active buckets. The data in bucket_activity_view table is scoped to your Storage Intelligence subscription and includes records for all actively enabled buckets.

The bucket_activity_view table includes the following fields:

Field Mode Type Description
snapshotStartTime NULLABLE TIMESTAMP The snapshot's start time, in RFC 3339 format.
snapshotEndTime NULLABLE TIMESTAMP The snapshot's end time, in RFC 3339 format.
name NULLABLE STRING The name of the bucket.
generation NULLABLE INTEGER The bucket's generation number.
project NULLABLE INTEGER The project number.
location NULLABLE STRING The location of the bucket.
totalRequests NULLABLE INTEGER The total number of requests made on the bucket.
totalReads NULLABLE INTEGER The total number of read operations.
totalWrites NULLABLE INTEGER The total number of write operations.
totalUpdates NULLABLE INTEGER The total number of update operations.
totalDeletes NULLABLE INTEGER The total number of delete operations.
totalLists NULLABLE INTEGER The total number of list operations.
totalRequestBytes NULLABLE INTEGER The total number of request bytes transferred.
totalResponseBytes NULLABLE INTEGER The total number of response bytes transferred.
count4xx NULLABLE INTEGER The total number of 4xx errors.
count5xx NULLABLE INTEGER The total number of 5xx errors.
count2xx NULLABLE INTEGER The total number of 2xx success responses.
count400 NULLABLE INTEGER The total number of 400 errors for bad requests.
count401 NULLABLE INTEGER The total number of 401 errors for unauthorized requests.
count403 NULLABLE INTEGER The total number of 403 errors for forbidden requests.
count429 NULLABLE INTEGER The total number of 429 errors for usage limit exceeded requests.
topPrefixes429Errors REPEATED RECORD The top 5 bucket prefixes with the most 429 errors, returned as a RECORD containing the prefix as a STRING and the error count as an INTEGER.
topPrefixesRequestBytes REPEATED RECORD The top 5 bucket prefixes with the highest number of requested bytes, returned as a RECORD containing the prefix as a STRING and the count as an INTEGER.
topPrefixesResponseBytes REPEATED RECORD The top 5 bucket prefixes with the highest number of response bytes, returned as a RECORD containing the prefix as a STRING and the count as an INTEGER.
topPrefixesHighestOperations REPEATED RECORD The top 5 bucket prefixes with the highest number of operations, returned as a RECORD containing the prefix as a STRING and the count as an INTEGER.

Bucket region activity view

The bucket_region_activity_view table summarizes the total bytes sent and received for a bucket, aggregated by destination region. Use the bucket activity regional data to identify the most active regions, helping you determine the optimal bucket location, analyze regional egress, and consider bucket relocation as needed to optimize performance, cost, or resilience. The data in bucket_region_activity_view table is scoped to your Storage Intelligence subscription and includes records for all actively enabled buckets.

Field Mode Type Description
snapshotStartTime NULLABLE TIMESTAMP The snapshot's start time, in RFC 3339 format.
snapshotEndTime NULLABLE TIMESTAMP The snapshot's end time, in RFC 3339 format.
project NULLABLE INTEGER The project number for the bucket.
name NULLABLE STRING The name of the bucket.
generation NULLABLE INTEGER The bucket's generation number.
requestLocation NULLABLE STRING The Google Cloud location where the request originated.
bucketLocation NULLABLE STRING The location of the bucket.
requestBytes NULLABLE INTEGER The total number of bytes transferred for requests between the specified bucket and location.
responseBytes NULLABLE INTEGER The total number of bytes transferred for responses between the specified bucket and location.

Dataset schema for events and errors

In the linked dataset, you can also view the snapshot processing events and errors in the events_view and error_attributes_view views. To troubleshoot the snapshot processing errors, see Troubleshoot dataset errors.

Events log

You can view event logs in the events_view view in the linked dataset:

Metadata field Mode Type Description
manifest.snapshotTime NULLABLE TIMESTAMP The time, in RFC 3339 format, when the snapshot of the events is refreshed.
manifest.viewName NULLABLE STRING The name of the view that refreshes.
manifest.location NULLABLE STRING The source location of the data that refreshes.
globalManifest.snapshotTime NULLABLE TIMESTAMP The ingestion completion time, in RFC 3339 format, for the bucket and object attributes tables across all source locations.
eventTime NULLABLE TIMESTAMP The time the event occurred.
eventCode NULLABLE STRING The event code associated with the corresponding entry. The eventCode values are as follows:
  • eventCode 1 indicates that the manifest.viewName view is refreshed with all entries for the source location (manifest.location) within the snapshot (manifest.snapshotTime).
  • eventCode 2 indicates that the dataset is refreshed with the bucket and object entries for all source locations. The refresh occurs within the snapshot (globalManifest.snapshotTime).
  • eventCode 3 indicates that the dataset is published each time data processing is complete for the activity views. The corresponding eventMessage is DATA_REFRESH_ACTIVITY_JOURNAL. The event code 3 refers to the activity manifest.viewName view being refreshed with all entries for the source location manifest.location within the log window.

Error codes

You can view error codes in the error_attributes_view table in the linked dataset:

Metadata field Mode Type Description
errorCode NULLABLE INTEGER The error code associated with this entry. For a list of valid values and how to resolve these errors, see Troubleshoot dataset errors.
errorSource NULLABLE STRING The source of the error. Valid value: CONFIGURATION_PREPROCESSING.
errorTime NULLABLE TIMESTAMP The time the error occurs.
sourceGcsLocation NULLABLE STRING The source Cloud Storage location of the error. For projects, this field is null because projects are not associated with a location.
bucketErrorRecord.bucketName NULLABLE STRING The name of the bucket involved in the error. You can use this information to debug a bucket error.
bucketErrorRecord.serviceAccount NULLABLE STRING The service account that needs permission to ingest objects from the bucket. You can use this information to debug a bucket error.
projectErrorRecord.projectNumber NULLABLE INTEGER The number of the project involved in the error. You can use this information to debug a project error.
projectErrorRecord.organizationName NULLABLE STRING The name of the organization the project must belong to for processing. A value of 0 indicates that the project is not in the organization. You can use this information to debug a project error.

Troubleshoot dataset errors

To troubleshoot the snapshot processing errors that are logged in the error_attributes_view view in the linked dataset, see the following table:

Error code Error case Error message Troubleshooting
1 Source project doesn't belong to the organization Source project projectErrorRecord.projectNumber doesn't belong to the organization projectErrorRecord.organizationName. Add source project projectErrorRecord.projectNumber to organization projectErrorRecord.organizationName. For instructions to migrate a project between organizations, see Migrate projects between organizations.
2 Bucket authorization error Permission denied for ingesting objects for bucket bucketErrorRecord.bucketName. Give service account bucketErrorRecord.serviceAccount Identity and Access Management (IAM) permissions to allow ingestion of objects for bucket bucketErrorRecord.bucketName. For more information, see Grant required permissions to the service agent.
3 Destination project doesn't belong to the organization Destination project projectErrorRecord.projectNumber not in organization projectErrorRecord.organizationName. Add destination project projectErrorRecord.projectNumber to organization projectErrorRecord.organizationName. For instructions to migrate a project between organizations, see Migrate projects between organizations.
4 Source project doesn't have Storage Intelligence configured. Source project projectErrorRecord.projectNumber doesn't have Storage Intelligence configured. Configure Storage Intelligence for the source project projectErrorRecord.projectNumber. For more information, see Configure and manage Storage Intelligence.
5 Bucket doesn't have Storage Intelligence configured. Bucket bucketErrorRecord.bucketName doesn't have Storage Intelligence configured. Configure Storage Intelligence for the bucket bucketErrorRecord.bucketName. For more information, see Configure and manage Storage Intelligence.
6 Bucket access authentication error ACTIVITY_BUCKET_ACCESS_AUTHORIZATION_ERROR for bucket bucketErrorRecord.bucketName. Grant service account bucketErrorRecord.serviceAccount Identity and Access Management (IAM) permissions to allow ingestion of activity for bucket bucketErrorRecord.bucketName. For more information, see Grant required permissions to the service agent.

What's next