Cloud Storage supports streaming object reads, which let your application or process consume object byte streams in real time directly over the network connection without saving a copy of the data to a local disk or persistent storage.
If you want to download an object and save it to your local file system, see Download objects.
Checksum validation for streaming reads
You shouldn't use a streaming read if you require checksum validation
prior to data processing. Because streaming reads typically use the HTTP Range
header, Cloud Storage doesn't return checksums that apply solely to
an arbitrary requested portion of object data. Server checksums validate the
complete object, meaning that your application might read and process corrupted
data before the full stream finishes and validates.
Required roles
To get the required permissions that you need to perform a streaming read of an
object, ask your administrator to grant you the Storage Object Viewer
(roles/storage.objectViewer) role on the bucket.
This role contains the permission required to stream reads. To see the exact permission that's required, expand the Required permissions section:
Required permissions
storage.objects.get
You might also be able to get this permission with other predefined roles or custom roles.
For instructions on granting roles on buckets, see Set and manage IAM policies on buckets.
Stream an object read
The following examples show how to perform a streaming read from a Cloud Storage object to a process:
Console
The Google Cloud console doesn't support streaming reads. Use the Google Cloud CLI instead.
Command line
Run the gcloud storage cp command using a hyphen (-) for the
destination URL, and then pipe the data to your process:
gcloud storage cp gs://BUCKET_NAME/OBJECT_NAME - | PROCESS_NAME
Replace the following:
BUCKET_NAME: the name of the bucket that contains the object that you want to read. For example,my_app_bucket.OBJECT_NAME: the name of the object that you want to stream to the process. For example,data_measurements.PROCESS_NAME: the name of the process into which you want to feed data. For example,analyze_data.
You can also stream data from a Cloud Storage object to a
standard Linux command such as sort:
gcloud storage cp gs://my_app_bucket/data_measurements - | sort
Client libraries
For more information, see the
Cloud Storage C++ API
reference documentation.
To authenticate to Cloud Storage, set up Application Default Credentials.
For more information, see
Set up authentication for client libraries.
For more information, see the
Cloud Storage C# API
reference documentation.
To authenticate to Cloud Storage, set up Application Default Credentials.
For more information, see
Set up authentication for client libraries.
For more information, see the
Cloud Storage Go API
reference documentation.
To authenticate to Cloud Storage, set up Application Default Credentials.
For more information, see
Set up authentication for client libraries.
For more information, see the
Cloud Storage Java API
reference documentation.
To authenticate to Cloud Storage, set up Application Default Credentials.
For more information, see
Set up authentication for client libraries.
For more information, see the
Cloud Storage Node.js API
reference documentation.
To authenticate to Cloud Storage, set up Application Default Credentials.
For more information, see
Set up authentication for client libraries.
For more information, see the
Cloud Storage PHP API
reference documentation.
To authenticate to Cloud Storage, set up Application Default Credentials.
For more information, see
Set up authentication for client libraries.
For more information, see the
Cloud Storage Python API
reference documentation.
To authenticate to Cloud Storage, set up Application Default Credentials.
For more information, see
Set up authentication for client libraries.
For more information, see the
Cloud Storage Ruby API
reference documentation.
To authenticate to Cloud Storage, set up Application Default Credentials.
For more information, see
Set up authentication for client libraries.
C++
C#
Go
Java
Node.js
PHP
Python
Ruby
Rust
REST APIs
JSON API
To perform a streaming read, follow the instructions for downloading an object with the following considerations:
Before beginning the read, retrieve the object's metadata and save the object's generation number. Include this generation number in each of your requests to ensure that you don't read data from two different generations if the original object is overwritten.
Use the
Rangeheader in your request to retrieve a piece of the overall object, which you can send to a local process.Continue making requests for successive pieces of the object until you retrieve the entire object.
XML API
To perform a streaming read, follow the instructions for downloading an object with the following considerations:
Before beginning the read, retrieve the object's metadata and save the object's generation number. Include this generation number in each of your requests to ensure that you don't read data from two different generations if the original object is overwritten.
Use the
Rangeheader in your request to retrieve a piece of the overall object, which you can send to a local process.Continue making requests for successive pieces of the object until you retrieve the entire object.
What's next
- Stream an upload.
- Learn more about decompressive transcoding.
- Learn more about object reads and downloads.