Configure BigLake Iceberg tables in BigQuery

This page describes how to configure Datastream for replication to BigLake Iceberg tables in BigQuery.

BigLake Iceberg tables offer the same fully managed experience as standard BigQuery tables, but store data in customer-owned Cloud Storage buckets in the Apache Iceberg table format and Parquet file format. You can query and analyse data using BigQuery capabilities while keeping the data in your own storage buckets.

Table metadata

Datastream appends a STRUCT column named datastream_metadata to each table that's written to the BigQuery destination.

The datastream_metadata column contains the following fields:

  • UUID: This field has the STRING data type.
  • SOURCE_TIMESTAMP: This field has the INTEGER data type.
  • CHANGE_SEQUENCE_NUMBER: This field has the STRING data type. It's an internal sequence number used by Datastream for each change event.
  • CHANGE_TYPE: This field has the STRING data type. It indicates the type of the change event. For the append-only write mode the value is INSERT.
  • SORT_KEYS: This field contains an array of STRING values. You can use the values to sort the change events.

Configure streaming to BigLake Iceberg tables

To set up your stream to ingest data into BigLake Iceberg tables:

  1. Create a Cloud Storage bucket where you want to store your data.
  2. Create a Cloud resource connection in BigQuery. For information about how to create this type of connection, see Create and set up a Cloud resource connection.

  3. Get the identifier of the connection service account:

    bq show --location=LOCATION --connection --project_id=PROJECT_ID
CONNECTION_NAME

  4. Grant your Cloud resource connection access to the Cloud Storage bucket that you created. To do this, add the storage.admin IAM permission to the connection service account:

    gcloud storage buckets add-iam-policy-binding gs://YOUR_GCS_BUCKET \
--member=serviceAccount:YOUR_SERVICE_ACCOUNT_ID \
--role=roles/storage.admin

  5. Create a BigLake Iceberg tables stream.

    For information about how to create a BigLake Iceberg tables stream using the Google Cloud console, see Create a stream.

    For information about how to create a request to stream data to BigLake Iceberg tables using REST, Google Cloud CLI or Terraform, see Manage streams using the API.

What's next