串流讀取

Cloud Storage 支援串流物件讀取作業，可讓應用程式或程序透過網路連線，直接即時使用物件位元組串流，而不必將資料副本儲存到本機磁碟或持久儲存空間。

如要下載物件並儲存至本機檔案系統，請參閱「將物件下載為檔案」一文。

在串流讀取時使用檢查碼驗證

如果您需要在處理資料前進行總和檢查碼驗證，請勿使用串流讀取。由於串流讀取通常會使用 HTTP Range 標頭，因此 Cloud Storage 不會傳回僅適用於任意要求部分物件資料的總和檢查碼。伺服器檢查碼會驗證完整物件，也就是說，系統可能會在完整串流完成並驗證前，讀取及處理已損毀的資料。

必要的角色

如要取得串流讀取所需的權限，請要求管理員授予您 bucket 的「Storage 物件檢視者」(roles/storage.objectViewer) 角色。

這個角色具備串流讀取所需的權限。如要查看確切的必要權限，請展開「Required permissions」(必要權限) 部分：

所需權限

  • storage.objects.get

您或許還可透過其他預先定義的角色或自訂角色取得這項權限。

如需授予 bucket 角色的操作說明，請參閱「在 bucket 上設定及管理 IAM 政策」。

串流物件讀取作業

下列範例說明如何從 Cloud Storage 物件串流讀取資料至程序：

控制台

Google Cloud 控制台不支援串流讀取作業。請改用 Google Cloud CLI。

指令列

  1. 執行 gcloud storage cp 指令，並對到達網頁網址使用連字號，然後將資料填入處理程序：

    gcloud storage cp gs://BUCKET_NAME/OBJECT_NAME - | PROCESS_NAME

    其中：

    • BUCKET_NAME 是包含物件的值區名稱。例如：my_app_bucket。
    • OBJECT_NAME 是您要串流至程序的物件名稱。例如：data_measurements。
    • PROCESS_NAME 是您要饋送資料的程序名稱。例如：analyze_data。

此外，您也可以將資料從 Cloud Storage 物件串流到標準的 Linux 指令，例如 sort：

gcloud storage cp gs://my_app_bucket/data_measurements - | sort

用戶端程式庫

C++

詳情請參閱「Cloud Storage C++ API 參考文件」。

如要向 Cloud Storage 進行驗證，請設定應用程式預設憑證。詳情請參閱「設定用戶端程式庫的驗證作業」。

namespace gcs = ::google::cloud::storage;
[](gcs::Client client, std::string const& bucket_name,
   std::string const& object_name) {
  gcs::ObjectReadStream stream = client.ReadObject(bucket_name, object_name);

  int count = 0;
  std::string line;
  while (std::getline(stream, line, '\n')) {
    ++count;
  }
  if (stream.bad()) throw google::cloud::Status(stream.status());

  std::cout << "The object has " << count << " lines\n";
}

C#

詳情請參閱「Cloud Storage C# API 參考文件」。

如要向 Cloud Storage 進行驗證，請設定應用程式預設憑證。詳情請參閱「設定用戶端程式庫的驗證作業」。


using Google.Cloud.Storage.V1;
using System;
using System.IO;

public class DownloadFileSample
{
    public void DownloadFile(
        string bucketName = "your-unique-bucket-name",
        string objectName = "my-file-name",
        string localPath = "my-local-path/my-file-name")
    {
        var storage = StorageClient.Create();
        using var outputFile = File.OpenWrite(localPath);
        storage.DownloadObject(bucketName, objectName, outputFile);
        Console.WriteLine($"Downloaded {objectName} to {localPath}.");
    }
}

Go

詳情請參閱「Cloud Storage Go API 參考文件」。

如要向 Cloud Storage 進行驗證，請設定應用程式預設憑證。詳情請參閱「設定用戶端程式庫的驗證作業」。


import (
	"context"
	"fmt"
	"io"
	"time"

	"cloud.google.com/go/storage"
)

// downloadFileIntoMemory downloads an object.
func downloadFileIntoMemory(w io.Writer, bucket, object string) ([]byte, error) {
	// bucket := "bucket-name"
	// object := "object-name"
	ctx := context.Background()
	client, err := storage.NewClient(ctx)
	if err != nil {
		return nil, fmt.Errorf("storage.NewClient: %w", err)
	}
	defer client.Close()

	ctx, cancel := context.WithTimeout(ctx, time.Second*50)
	defer cancel()

	rc, err := client.Bucket(bucket).Object(object).NewReader(ctx)
	if err != nil {
		return nil, fmt.Errorf("Object(%q).NewReader: %w", object, err)
	}
	defer rc.Close()

	data, err := io.ReadAll(rc)
	if err != nil {
		return nil, fmt.Errorf("io.ReadAll: %w", err)
	}
	fmt.Fprintf(w, "Blob %v downloaded.\n", object)
	return data, nil
}

Java

詳情請參閱「Cloud Storage Java API 參考文件」。

如要向 Cloud Storage 進行驗證，請設定應用程式預設憑證。詳情請參閱「設定用戶端程式庫的驗證作業」。


import com.google.cloud.ReadChannel;
import com.google.cloud.storage.BlobId;
import com.google.cloud.storage.Storage;
import com.google.cloud.storage.StorageOptions;
import com.google.common.io.ByteStreams;
import java.io.IOException;
import java.nio.channels.FileChannel;
import java.nio.file.Path;
import java.nio.file.Paths;
import java.nio.file.StandardOpenOption;

public class StreamObjectDownload {

  public static void streamObjectDownload(
      String projectId, String bucketName, String objectName, String targetFile)
      throws IOException {
    // The ID of your GCP project
    // String projectId = "your-project-id";

    // The ID of your GCS bucket
    // String bucketName = "your-unique-bucket-name";

    // The ID of your GCS object
    // String objectName = "your-object-name";

    // The path to the file to download the object to
    // String targetFile = "path/to/your/file";
    Path targetFilePath = Paths.get(targetFile);

    Storage storage = StorageOptions.newBuilder().setProjectId(projectId).build().getService();
    try (ReadChannel reader = storage.reader(BlobId.of(bucketName, objectName));
        FileChannel targetFileChannel =
            FileChannel.open(targetFilePath, StandardOpenOption.WRITE)) {

      ByteStreams.copy(reader, targetFileChannel);

      System.out.println(
          "Downloaded object "
              + objectName
              + " from bucket "
              + bucketName
              + " to "
              + targetFile
              + " using a ReadChannel.");
    }
  }
}

Node.js

詳情請參閱「Cloud Storage Node.js API 參考文件」。

如要向 Cloud Storage 進行驗證，請設定應用程式預設憑證。詳情請參閱「設定用戶端程式庫的驗證作業」。

/**
 * TODO(developer): Uncomment the following lines before running the sample.
 */
// The ID of your GCS bucket
// const bucketName = 'your-unique-bucket-name';

// The ID of your GCS file
// const fileName = 'your-file-name';

// The filename and file path where you want to download the file
// const destFileName = '/local/path/to/file.txt';

// Imports the Google Cloud client library
const {Storage} = require('@google-cloud/storage');

// Creates a client
const storage = new Storage();

async function streamFileDownload() {
  // The example below demonstrates how we can reference a remote file, then
  // pipe its contents to a local file.
  // Once the stream is created, the data can be piped anywhere (process, sdout, etc)
  await storage
    .bucket(bucketName)
    .file(fileName)
    .createReadStream() //stream is created
    .pipe(fs.createWriteStream(destFileName))
    .on('finish', () => {
      // The file download is complete
    });

  console.log(
    `gs://${bucketName}/${fileName} downloaded to ${destFileName}.`
  );
}

streamFileDownload().catch(console.error);

PHP

詳情請參閱「Cloud Storage PHP API 參考文件」。

如要向 Cloud Storage 進行驗證，請設定應用程式預設憑證。詳情請參閱「設定用戶端程式庫的驗證作業」。

use Google\Cloud\Storage\StorageClient;

/**
 * Download an object from Cloud Storage and save it as a local file.
 *
 * @param string $bucketName The name of your Cloud Storage bucket.
 *        (e.g. 'my-bucket')
 * @param string $objectName The name of your Cloud Storage object.
 *        (e.g. 'my-object')
 * @param string $destination The local destination to save the object.
 *        (e.g. '/path/to/your/file')
 */
function download_object(string $bucketName, string $objectName, string $destination): void
{
    $storage = new StorageClient();
    $bucket = $storage->bucket($bucketName);
    $object = $bucket->object($objectName);
    $object->downloadToFile($destination);
    printf(
        'Downloaded gs://%s/%s to %s' . PHP_EOL,
        $bucketName,
        $objectName,
        basename($destination)
    );
}

Python

詳情請參閱「Cloud Storage Python API 參考文件」。

如要向 Cloud Storage 進行驗證，請設定應用程式預設憑證。詳情請參閱「設定用戶端程式庫的驗證作業」。

from google.cloud import storage


def download_blob_to_stream(bucket_name, source_blob_name, file_obj):
    """Downloads a blob to a stream or other file-like object."""

    # The ID of your GCS bucket
    # bucket_name = "your-bucket-name"

    # The ID of your GCS object (blob)
    # source_blob_name = "storage-object-name"

    # The stream or file (file-like object) to which the blob will be written
    # import io
    # file_obj = io.BytesIO()

    storage_client = storage.Client()

    bucket = storage_client.bucket(bucket_name)

    # Construct a client-side representation of a blob.
    # Note `Bucket.blob` differs from `Bucket.get_blob` in that it doesn't
    # retrieve metadata from Google Cloud Storage. As we don't use metadata in
    # this example, using `Bucket.blob` is preferred here.
    blob = bucket.blob(source_blob_name)
    blob.download_to_file(file_obj)

    print(f"Downloaded blob {source_blob_name} to file-like object.")

    return file_obj
    # Before reading from file_obj, remember to rewind with file_obj.seek(0).

Ruby

詳情請參閱「Cloud Storage Ruby API 參考文件」。

如要向 Cloud Storage 進行驗證，請設定應用程式預設憑證。詳情請參閱「設定用戶端程式庫的驗證作業」。

# Downloads a blob to a stream or other file-like object.

# The ID of your GCS bucket
# bucket_name = "your-unique-bucket-name"

# Name of a file in the Storage bucket
# file_name   = "some_file.txt"

# The stream or file (file-like object) to which the contents will be written
# local_file_obj = StringIO.new

require "google/cloud/storage"

storage = Google::Cloud::Storage.new
bucket  = storage.bucket bucket_name
file    = bucket.file file_name

file.download local_file_obj, verify: :none

# rewind the object before starting to read the downloaded contents
local_file_obj.rewind
puts "The full downloaded file contents are: #{local_file_obj.read.inspect}"

Rust

use google_cloud_storage::client::Storage;

pub async fn sample(client: &Storage, bucket_id: &str) -> anyhow::Result<()> {
    const NAME: &str = "object-to-download.txt";
    let mut reader = client
        .read_object(format!("projects/_/buckets/{bucket_id}"), NAME)
        .send()
        .await?;
    println!("counting newlines {NAME} in bucket {bucket_id}");
    let mut count = 0;
    while let Some(buffer) = reader.next().await.transpose()? {
        count += buffer.into_iter().filter(|c| *c == b'\n').count();
    }
    println!("object {NAME} in bucket {bucket_id} contains {count} newlines");
    Ok(())
}

REST API

JSON API

如要執行串流讀取，請按照下載物件的操作說明，並注意下列事項：

  • 開始讀取前，請擷取物件的中繼資料，並儲存物件的產生編號。在每個要求中加入這個產生編號，確保您不會在原始資料遭到覆寫時，從兩個不同的產生編號讀取資料。

  • 在要求中使用 Range 標頭，擷取整體物件的一部分，然後傳送至本機程序。

  • 繼續要求物件的後續部分，直到擷取整個物件為止。

XML API

如要執行串流讀取，請按照下載物件的操作說明，並注意下列事項：

  • 開始讀取前，請擷取物件的中繼資料，並儲存物件的產生編號。在每個要求中加入這個產生編號，確保您不會在原始資料遭到覆寫時，從兩個不同的產生編號讀取資料。

  • 在要求中使用 Range 標頭，擷取整體物件的一部分，然後傳送至本機程序。

  • 繼續要求物件的後續部分，直到擷取整個物件為止。

後續步驟