Object reads and downloads

This page discusses concepts related to reading and downloading objects in Cloud Storage.

Reading objects

Reading an object refers to streaming, consuming, or inspecting object data or a specific byte range over an open network connection directly in memory. Rather than being saved locally, the retrieved payload is ephemeral and lives in an application memory stream, buffer, pipeline, or directly in CPU/GPU memory. Reading objects is suitable for high-performance, throughput-intensive workloads, such as streaming training datasets and tensor batches into Artificial Intelligence (AI) and Machine Learning (ML) frameworks like Pytorch or Jax.

The following list describes the types of object reads available in Cloud Storage:

  • Reading objects into memory or interactive object reads: A method where data is consumed interactively or into memory. The client retrieves the entire object and loads all of its bytes into a contiguous variable, byte array, or in-memory string.

  • Streaming reads: A method where data is consumed as a stream of bytes directly as it's being received from the network, rather than waiting for the entire transfer to finish.

Downloading objects

Downloading an object is an end-to-end task that transfers object data from Cloud Storage into a persistent file on a local file system, disk, or target storage device. Unlike in-memory streaming reads, downloading encompasses both fetching the remote bytes and committing them to local storage. Downloading objects is primarily suited for general-purpose file management, local archiving, data backup, and offline utility usage.

When using REST APIs, downloads are performed using HTTP or HTTPS GET requests that can include an optional Range header, which defines a specific portion of the object to download to persistent or local storage. You can also use the Range header to resume interrupted downloads. When using gRPC, downloads use storage.v2.Storage/ReadObject or BidiReadObject streams.

The following list describes the types of object downloads available in Cloud Storage:

  • Simple downloads: The most basic form of object downloading. The client sends a single request to retrieve either the entire object or a specified byte range in one continuous transfer.

  • Sliced object downloads: A high-performance strategy where a single large object is divided into multiple smaller "slices" (also known as "chunks"). The client downloads these slices in parallel using multiple independent and concurrent requests, then reassembles the slices locally into a single file.

Tool support for reading and downloading objects

Console

When using the Google Cloud console, you can perform the following object reads and downloads:

Command line

You can perform the following object reads and downloads when using the Google Cloud CLI:

Client libraries

C++

You can perform the following types of reads and downloads when using the C++ client library:

C#

You can perform the following types of reads and downloads when using the C# client library:

Go

You can perform the following types of reads and downloads when using the Go client library:

Java

You can perform the following types of reads and downloads when using the Java client library:

Node.js

You can perform the following types of reads and downloads when using the Node.js client library:

PHP

You can perform the following types of reads and downloads when using the PHP client library:

Python

You can perform the following types of reads and downloads when using the Python client library:

Ruby

You can perform the following types of reads and downloads when using the Ruby client library:

REST APIs

If you use REST APIs to download objects, see Request endpoints for a complete discussion on the request endpoints you can use.

JSON API

You can perform the following types of downloads when using the JSON API:

XML API

You can perform the following types of downloads when using the XML API:

What's next