About the custom Apache Iceberg catalog for BigQuery endpoint

The custom Apache Iceberg catalog for BigQuery endpoint connects open source query engines like Apache Spark and Apache Flink to the Lakehouse runtime catalog. By integrating a custom catalog plugin (BigQueryMetastoreCatalog), this endpoint lets you manage and query Apache Iceberg table metadata directly through BigQuery while storing data and metadata files in Cloud Storage.

How the custom Iceberg catalog works

The custom Apache Iceberg catalog uses a custom catalog implementation (org.apache.iceberg.gcp.bigquery.BigQueryMetastoreCatalog) provided in a JAR library. When your compute workloads run on Managed Service for Apache Spark, the engine uses this plugin to interact with BigQuery as the metadata store for your Apache Iceberg tables.

The workflow operates as follows:

  1. Catalog implementation: Query engines load the BigQuery Metastore catalog JAR and configure Spark session catalog properties to use org.apache.iceberg.gcp.bigquery.BigQueryMetastoreCatalog.
  2. Metadata management: When you create or modify tables using Spark SQL or DataFrame APIs, the plugin stores dataset and table definitions in BigQuery.
  3. Data storage: Apache Iceberg metadata files (metadata.json, manifest lists, manifests) and data files (such as Parquet files) are stored directly in your specified Cloud Storage warehouse path.
  4. Cross-engine access: Because metadata is registered with BigQuery, you can query tables both from open source engines like Spark and directly from BigQuery.

Resource hierarchy

The custom Apache Iceberg catalog for BigQuery endpoint organizes metadata into the following hierarchy:

Resource Description
Project The Google Cloud project that contains your BigQuery resources and Cloud Storage warehouse storage.
Namespace (Dataset) A BigQuery dataset configured to act as an Iceberg namespace, defining the default Cloud Storage location for tables created within it.
Table An Apache Iceberg table whose schema, snapshots, and metadata pointers are tracked in BigQuery and whose data files reside in Cloud Storage.

Compare custom Iceberg catalogs and Iceberg REST catalogs

The following table summarizes key differences between the custom Apache Iceberg catalog for BigQuery endpoint and the Apache Iceberg REST catalog endpoint:

Feature Custom Iceberg catalog for BigQuery Apache Iceberg REST catalog endpoint (Recommended)
Catalog API standard Custom plugin (BigQueryMetastoreCatalog) Open standard Apache Iceberg REST Catalog API
Catalog hierarchy Project > BigQuery Dataset > Table Project > Catalog > Namespace > Table (P.C.N.T)
Storage configuration Cloud Storage paths specified per dataset or table Multiple-bucket (bl://) or single-bucket (gs://) catalogs
Credential vending Not supported (uses direct IAM credentials) Supported (vends short-lived storage access tokens)
Disaster recovery Not supported Supported (cross-region replication and failover)
Recommended for Existing deployments and workflows New workloads and standard Iceberg REST client integrations

What's next