Use Apache Spark with the Lakehouse runtime catalog

Apache Spark connects to borderless Lakehouse through the Lakehouse runtime catalog. This integration uses the Apache Iceberg REST catalog endpoint to provide a standard interface for discovering and managing Apache Iceberg tables.

To get started, see the following guides:

Spark configuration properties

Configure your Spark session or job submission with the following properties to connect to the Apache Iceberg REST catalog endpoint:

Property Value Description
spark.sql.catalog.CATALOG_NAME org.apache.iceberg.spark.SparkCatalog Registers the Apache Iceberg Spark catalog implementation.
spark.sql.catalog.CATALOG_NAME.type rest Specifies the REST catalog type.
spark.sql.catalog.CATALOG_NAME.uri https://biglake.googleapis.com/iceberg/v1/restcatalog Specifies the Apache Iceberg REST catalog endpoint URI in the Lakehouse runtime catalog. To route requests to a regional endpoint, use https://biglake.LOCATION.rep.googleapis.com/iceberg/v1/restcatalog.
spark.sql.catalog.CATALOG_NAME.warehouse bl://projects/PROJECT_ID/catalogs/CATALOG_ID Specifies the warehouse path for a multiple-bucket catalog in the Lakehouse runtime catalog. For a single-bucket catalog, use gs://CLOUD_STORAGE_BUCKET_NAME.
spark.sql.catalog.CATALOG_NAME.header.x-goog-user-project PROJECT_ID Specifies the Google Cloud project ID for billing and quota on Lakehouse runtime catalog requests.
spark.sql.catalog.CATALOG_NAME.rest.auth.type org.apache.iceberg.gcp.auth.GoogleAuthManager Configures Google Cloud authentication for the Apache Iceberg REST catalog endpoint.
spark.sql.catalog.CATALOG_NAME.io-impl org.apache.iceberg.gcp.gcs.GCSFileIO Specifies the Cloud Storage FileIO implementation for Apache Iceberg. Required when using credential vending mode.
spark.sql.catalog.CATALOG_NAME.header.X-Iceberg-Access-Delegation vended-credentials Requests short-lived, downscoped storage credentials from the Lakehouse runtime catalog during table load when using credential vending mode.
spark.sql.extensions org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions Enables Apache Iceberg SQL extensions and optimizations in Spark.
spark.sql.defaultCatalog CATALOG_NAME Optional: sets the default catalog for the Spark session.

Replace the following:

  • CATALOG_NAME: the catalog name identifier configured in your Spark session (for example, my_catalog).

  • LOCATION: the Google Cloud region where your catalog is located when using a regional endpoint (for example, us-central1).

  • PROJECT_ID: your Google Cloud project ID.

  • CATALOG_ID: the ID of your catalog in the Lakehouse runtime catalog.

  • CLOUD_STORAGE_BUCKET_NAME: the name of the Cloud Storage bucket associated with a single-bucket catalog.

For more configuration details, see Configure client application and the official Apache Iceberg Spark documentation.