Configure Apache Iceberg tables in BigQuery

Datastream lets you replicate data from your sources into Apache Iceberg tables in BigQuery. Apache Iceberg tables provide the same fully managed experience as standard BigQuery tables, while storing data in Parquet format inside your own Cloud Storage buckets. This setup lets you query and analyze your data using BigQuery while retaining full control over your underlying storage.

Datastream supports streaming to Apache Iceberg tables only in the append-only write mode. For more information, see Configure write mode.

Table metadata

Datastream appends a STRUCT column named datastream_metadata to each table that's written to the BigQuery destination.

The datastream_metadata column contains the following fields:

  • UUID: This field has the STRING data type.
  • SOURCE_TIMESTAMP: This field has the INTEGER data type.
  • CHANGE_SEQUENCE_NUMBER: This field has the STRING data type. It's an internal sequence number used by Datastream for each change event.
  • CHANGE_TYPE: This field has the STRING data type. It indicates the type of the change event. For the append-only write mode the value is INSERT.
  • SORT_KEYS: This field contains an array of STRING values. You can use the values to sort the change events.

Configure streaming to Apache Iceberg tables

To set up your stream to ingest data into Apache Iceberg tables:

  1. Create a Cloud Storage bucket where you want to store your data.
  2. Create a Cloud resource connection in BigQuery. For information about how to create this type of connection, see Create and set up a Cloud resource connection.

  3. Get the identifier of the connection service account:

    bq show --location=LOCATION --connection --project_id=PROJECT_ID
CONNECTION_NAME

  4. Grant your Cloud resource connection access to the Cloud Storage bucket that you created. To do this, add the storage.admin IAM permission to the connection service account:

    gcloud storage buckets add-iam-policy-binding gs://YOUR_GCS_BUCKET \
--member=serviceAccount:YOUR_SERVICE_ACCOUNT_ID \
--role=roles/storage.admin

  5. Create an Apache Iceberg tables stream.

    For information about how to create an Apache Iceberg tables stream using the Google Cloud console, see Create a stream.

    For information about how to create a request to stream data to Apache Iceberg tables using REST, Google Cloud CLI or Terraform, see Manage streams using the API.

What's next