The *Apache Iceberg REST catalog endpoint* in the
*Lakehouse runtime catalog* lets you discover, read, and manage Apache
Iceberg tables in Lakehouse across multiple query engines. Built on the
*Apache Iceberg REST Catalog API*, this endpoint acts as a single source of
truth for open-source or third-party engines, such as Apache Spark,
that access data in Cloud Storage and interoperate with BigQuery.

The endpoint supports [credential vending](https://docs.cloud.google.com/lakehouse/docs/set-up-lakehouse-iceberg-rest-catalog.md.txt#create_a_catalog) for delegated storage access and
[cross-region replication and disaster recovery](https://docs.cloud.google.com/lakehouse/docs/about-managed-disaster-recovery.md.txt). While existing workflows
can continue using the earlier [*Custom Apache Iceberg catalog for
BigQuery* endpoint](https://docs.cloud.google.com/lakehouse/docs/about-custom-iceberg-catalog-endpoint), use the Apache Iceberg REST catalog endpoint
for new catalogs to get a more standardized and feature-rich interface.

## Before you begin

1.
   [Verify that billing is enabled for your Google Cloud project](https://docs.cloud.google.com/billing/docs/how-to/verify-billing-enabled.md.txt#confirm_billing_is_enabled_on_a_project).

2.


   Enable the BigLake API, if it is not already enabled.


   **Roles required to enable APIs**


   To enable APIs, you need the `serviceusage.services.enable` permission. If you
   created the project, then you likely already have this permission through the
   Owner role (`roles/owner`). Otherwise, you can get this permission through the
   Service Usage Admin role (`roles/serviceusage.serviceUsageAdmin`).
   [Learn how to grant roles](https://docs.cloud.google.com/iam/docs/granting-changing-revoking-access.md.txt).

   [Enable the API](https://console.cloud.google.com/apis/enableflow?apiid=biglake.googleapis.com)
3. Familiarize yourself with [Lakehouse runtime catalog](https://docs.cloud.google.com/lakehouse/docs/about-lakehouse-catalogs.md.txt) and the [Iceberg REST catalog endpoint overview](https://docs.cloud.google.com/lakehouse/docs/about-iceberg-rest-catalog-endpoint.md.txt) before continuing.
4. If you have existing version 1 (V1) Apache Iceberg tables, you must upgrade them before using them with the Apache Iceberg REST catalog endpoint. For more information, see [Upgrade Iceberg V1 tables to
   V2](https://docs.cloud.google.com/lakehouse/docs/update-tables).

### Required roles


To get the permissions that
you need to use the Apache Iceberg REST catalog endpoint in Lakehouse runtime catalog,

ask your administrator to grant you the
following IAM roles:

- Perform administrative tasks, such as managing catalog user access, storage access, and the catalog's credential vending mode:
  - [BigLake Admin](https://docs.cloud.google.com/iam/docs/roles-permissions/biglake.md.txt#biglake.admin) (`roles/biglake.admin`) on the project
  - [Storage Admin](https://docs.cloud.google.com/iam/docs/roles-permissions/storage.md.txt#storage.admin) (`roles/storage.admin`) on all associated Cloud Storage buckets.
- Create (register) tables in a Lakehouse runtime catalog: [BigLake Admin](https://docs.cloud.google.com/iam/docs/roles-permissions/biglake.md.txt#biglake.admin) (`roles/biglake.admin`) on the project.
- Read table data in credential vending mode: [BigLake Viewer](https://docs.cloud.google.com/iam/docs/roles-permissions/biglake.md.txt#biglake.viewer) (`roles/biglake.viewer`) on the project. If you use query engines such as Managed Service for Apache Spark or Dataflow to read table data, grant this role to the service account that you use to run jobs in that engine.
- Write table data in credential vending mode: [BigLake Editor](https://docs.cloud.google.com/iam/docs/roles-permissions/biglake.md.txt#biglake.editor) (`roles/biglake.editor`) on the project. If you use query engines such as Managed Service for Apache Spark or Dataflow to write table data, grant this role to the service account that you use to run jobs in that engine.
- Use the auto-provisioned Lakehouse runtime catalog service account in credential vending mode: [Storage Object User](https://docs.cloud.google.com/iam/docs/roles-permissions/storage.md.txt#storage.objectUser) (`roles/storage.objectUser`) on all associated Cloud Storage buckets. After creating the catalog, explicitly grant the Storage Object User role (`roles/storage.objectUser`) on all associated storage buckets to your catalog's auto-provisioned Lakehouse runtime catalog service account.
- Read catalog resources and table data in non-credential vending mode:
  - [BigLake Viewer](https://docs.cloud.google.com/iam/docs/roles-permissions/biglake.md.txt#biglake.viewer) (`roles/biglake.viewer`) on the project
  - [Storage Object Viewer](https://docs.cloud.google.com/iam/docs/roles-permissions/storage.md.txt#storage.objectViewer) (`roles/storage.objectViewer`) on all associated Cloud Storage buckets.
- Manage catalog resources and write table data in non-credential vending mode:
  - [BigLake Editor](https://docs.cloud.google.com/iam/docs/roles-permissions/biglake.md.txt#biglake.editor) (`roles/biglake.editor`) on the project
  - [Storage Object User](https://docs.cloud.google.com/iam/docs/roles-permissions/storage.md.txt#storage.objectUser) (`roles/storage.objectUser`) on all associated Cloud Storage buckets.


For more information about granting roles, see [Manage access to projects, folders, and organizations](https://docs.cloud.google.com/iam/docs/granting-changing-revoking-access.md.txt).


You might also be able to get
the required permissions through [custom
roles](https://docs.cloud.google.com/iam/docs/creating-custom-roles) or other [predefined
roles](https://docs.cloud.google.com/iam/docs/roles-overview#predefined).

## Limitations

The Apache Iceberg REST catalog endpoint is subject to the following
limitations:

**General limitations**

- Apache Iceberg V2 tables (GA) and V3 tables (Preview) are supported. Iceberg V1 tables aren't supported. Before you use existing V1 tables with the Apache Iceberg REST catalog endpoint, you must upgrade them to a supported version.
- When using credential vending mode, if the query engine allows you to set the `io-impl` property for a catalog connection, you must set it to `org.apache.iceberg.gcp.gcs.GCSFileIO`.

**Table limitations**

- Creating or modifying tables in the Apache Iceberg REST catalog endpoint using BigQuery data definition language (DDL) or data manipulation language (DML) statements is in [Preview](https://docs.cloud.google.com/products#product-launch-stages) and requires enabling the `gcp.biglake.bigquery-dml.enabled` table property. For more information, see [Configure table options](https://docs.cloud.google.com/lakehouse/docs/table-options.md.txt).
- Tables managed through the Apache Iceberg REST catalog endpoint don't support fine-grained access control (FGAC), such as row-level and column-level security.
- Setting the Iceberg table properties `write.data.path` or `write.metadata.path` to values other than their defaults is prohibited.
- Table paths must be nested within the parent namespace path (for example, `gs://{namespace_path}/.../{table_name}`). To prevent conflicts and improve security, a random string suffix is automatically appended to the resulting location (for example, `gs://{namespace_path}/{table_name}/{random_suffix}`).

**Data limitations**

- Only Parquet files are supported. For more details on how BigQuery handles Parquet files, see [Loading Parquet data
  from Cloud Storage](https://docs.cloud.google.com/bigquery/docs/loading-data-cloud-storage-parquet).
- The Iceberg `metadata.json` file size is limited to 1MB. To request an increase to this limit, contact your Google account team.

**Query limitations**

- Apache Iceberg metadata tables (such as `.snapshots` or `.files`) can't be queried in BigQuery using five-part name identifiers; you can query these tables using Spark.

## Considerations

Consider the following configurations when creating a catalog.

#### Bucket type

You can choose to create a multiple-bucket catalog or a single-bucket catalog.

- **Multiple-bucket catalog (recommended):** This configuration lets your catalog associate multiple buckets and lets you name your catalog independently of any bucket name.
- **Single-bucket:** This configuration restricts your catalog to a single bucket and locks the catalog name to the bucket name. It is discouraged for new projects.

For more information about the differences between these configurations,
see [catalog and storage locations](https://docs.cloud.google.com/lakehouse/docs/about-iceberg-rest-catalog-endpoint.md.txt#warehouse-storage-locations).

#### Credential mode (scope)

You can create a catalog that uses either end-user credentials or [credential
vending mode](https://docs.cloud.google.com/lakehouse/docs/credential-vending).

- **End-user credentials:** The catalog passes the identity of the end user
  accessing it to Cloud Storage for authorization checks.

- **Credential vending mode:** A storage access delegation mechanism that lets
  Lakehouse runtime catalog administrators control permissions
  directly on Lakehouse runtime catalog resources, eliminating
  the need for catalog users to have direct access to Cloud Storage
  buckets. It lets Google Cloud's *Lakehouse* administrators
  give users permissions on specific data files.

  The auto-provisioned Lakehouse runtime catalog service account requires the
  explicit Storage Object User role (`roles/storage.objectUser`) on all associated
  Cloud Storage buckets. By default, it has no access whatsoever.
  Without this role, vended credentials won't have sufficient scope to perform
  storage writes. If you use tools like `gcloud` or Terraform, you must grant this
  role manually.

  Creation of the auto-provisioned catalog service account is
  [eventually consistent](https://wikipedia.org/wiki/Eventual_consistency).
  This means that it takes time for the service account to propagate through the
  system. If you try to grant roles to the service account immediately after
  creation, the request might fail. To learn how long it takes, on average, for
  access changes to propagate, see [Access change propagation](https://docs.cloud.google.com/iam/docs/access-change-propagation.md.txt).

  If your associated Cloud Storage buckets are in a project protected by a
  [VPC Service Controls](https://docs.cloud.google.com/vpc-service-controls/docs/overview.md.txt) service perimeter that
  restricts `storage.googleapis.com`, then you must also configure an ingress
  rule to allow the catalog endpoint's auto-provisioned service account to
  access the bucket project. For more information, see
  [Use VPC Service Controls with credential vending](https://docs.cloud.google.com/lakehouse/docs/credential-vending.md.txt#vpc-sc).

#### Location

Familiarize yourself with the [location requirements](https://docs.cloud.google.com/lakehouse/docs/about-iceberg-rest-catalog-endpoint.md.txt#bucket_and_catalog_regions) before creating a
catalog.

- When you create a namespace, it automatically uses the same region as your
  catalog.

- If your catalog uses a multi-region bucket and you want to use it with the
  BigQuery multi-regions (`US` or `EU`), you must delete and
  recreate the catalog to specify the primary location.

- To route catalog requests directly to a specific region for data residency
  requirements, you can configure your tools and query engines to use
  [regional endpoints](https://docs.cloud.google.com/lakehouse/docs/regional-endpoints.md.txt)
  (`https://biglake.LOCATION.rep.googleapis.com/iceberg/v1/restcatalog`)
  instead of the global endpoint (`https://biglake.googleapis.com/iceberg/v1/restcatalog`).

## Set up the Iceberg REST catalog endpoint

Before you set up your catalog, we recommend reading the [Apache Iceberg REST
catalog endpoint overview](https://docs.cloud.google.com/lakehouse/docs/about-iceberg-rest-catalog-endpoint) to understand its resource hierarchy, catalog
types, and naming structure.

The following are the general steps to follow when using the Apache Iceberg REST
catalog endpoint in the Lakehouse runtime catalog:

1. Choose your catalog type---a [multiple-bucket](https://docs.cloud.google.com/lakehouse/docs/about-iceberg-rest-catalog-endpoint.md.txt#storage-multiple-buckets) catalog (recommended) or a [single-bucket](https://docs.cloud.google.com/lakehouse/docs/about-iceberg-rest-catalog-endpoint.md.txt#storage-single-bucket) catalog.
2. Create a catalog that points to your warehouse location.
3. Configure your client application to use the Apache Iceberg REST catalog endpoint.
4. Create a namespace or schema to organize your tables.
5. Create and query tables using your configured client.

### Create a catalog

Follow these steps to create a catalog based on your preferred credential mode
and bucket type.

### Console

**Create a multiple-bucket catalog (recommended)**

This configuration lets your catalog associate multiple buckets and
lets you name your catalog independently of any bucket name.

1. Open the **Lakehouse** page in the Google Cloud console.

   [Go to Lakehouse](https://console.cloud.google.com/biglake)
2. Click **Create catalog** , and then select **Lakehouse runtime catalog**.

3. For **Catalog type** , select **Iceberg Rest Catalog**.

4. For **Lakehouse catalog bucket options** , select
   **Multiple bucket catalog**.

5. For **Default Catalog Cloud Storage path**, enter or browse for the
   Cloud Storage path to use with your catalog.

6. For **Catalog ID**, enter a custom name for your catalog.

7. For **Primary location** , select a location. The location must be
   near the regions of the main bucket. The **Jurisdiction** field is
   automatically populated based on the bucket.

8. Click **Continue**.

9. In the **Data paths** step, if you want to add additional
   Cloud Storage paths, click **Add path** and enter or browse for
   a path in the **Select a Cloud Storage path** field.

10. Click **Continue**.

11. For **Authentication method** , select **End-user credentials** or
    **Credential vending mode**.

12. Click **Create**.

    Your catalog is created, and the **Catalog details** page opens.
13. If you selected **Credential vending mode**, perform the following
    additional steps:

    1. Under **Authentication method** , click **Set bucket permissions** (or **Set permissions for buckets** if your catalog uses more than one bucket).
    2. In the dialog, click **Confirm**.

#### Create a single-bucket catalog

> [!CAUTION]
> **Caution:** Avoid using the single-bucket configuration for new projects. This configuration restricts your catalog to a single bucket and locks the catalog name to the bucket name.

1. On the **Lakehouse** page, click **Create catalog** , and then select **Lakehouse runtime catalog**.
2. For **Catalog type** , select **Iceberg Rest Catalog**.
3. For **Lakehouse catalog bucket options** , select **Single bucket catalog**.
4. For **Default Catalog Cloud Storage bucket**, enter or browse for the Cloud Storage bucket to use with your catalog. For a single-bucket catalog, you can only have one catalog per bucket, and the catalog name matches the bucket name.
5. Click **Continue**.
6. For **Authentication method** , select **End-user credentials** or **Credential vending mode**.
7. Click **Create**.
8. If you selected **Credential vending mode** , perform the following additional steps:
   1. Under **Authentication method** , click **Set bucket permissions**.
   2. In the dialog, click **Confirm**.

### gcloud

**Create a multiple-bucket catalog (recommended)**

This configuration lets your catalog associate multiple buckets and
lets you name your catalog independently of any bucket name.

To create a multiple-bucket catalog (recommended), run
the [`gcloud biglake iceberg catalogs create`](https://docs.cloud.google.com/sdk/gcloud/reference/biglake/iceberg/catalogs/create.md.txt) command.

```bash
gcloud biglake iceberg catalogs create \
    CATALOG_NAME \
    --project PROJECT_ID \
    --catalog-type biglake \
    --default-location DEFAULT_LOCATION \
    --credential-mode CREDENTIAL_MODE \
    [--restricted-locations RESTRICTED_LOCATIONS] \
    [--primary-location LOCATION]
```

Replace the following:

- `CATALOG_NAME`: a name for your catalog. For multiple-bucket catalogs (recommended), this is your custom catalog name.
- `PROJECT_ID`: your Google Cloud project ID.
- `DEFAULT_LOCATION`: Specify the default storage location for the catalog. You can specify a bucket (`gs://my-bucket`) or a subpath (`gs://my-bucket/path`). All namespaces and tables in the catalog must reside under the specified path. For example, if you specify `gs://my-bucket/path`, you cannot create namespaces or tables under `gs://my-bucket/another/path`.
- `CREDENTIAL_MODE`: the authentication method. Use
  `end-user` for *End-user credentials* or `vended-credentials` for
  *Credential vending mode*.

  **Note** : If you use *Credential vending mode* ,
  you must explicitly grant the
  *Storage Object User* role (`roles/storage.objectUser`) on all associated
  storage buckets to your catalog's auto-provisioned Lakehouse runtime catalog
  service account.
- `RESTRICTED_LOCATIONS`: (Optional) Comma-separated
  list of additional allowed storage locations, in the format
  `gs://my-bucket-1/...,gs://my-bucket-2/...`. If you specify a path
  (such as `gs://my-bucket/path`), any namespaces or tables within that
  bucket must be under that path. All configured cloud storage locations
  across the default location and restricted locations must be in the same
  geographic region group or jurisdiction (such as the United States,
  Europe, Canada, or Asia). For example, you cannot mix a bucket in the US
  with a bucket in Europe. For a list of supported locations,
  see [Lakehouse locations](https://docs.cloud.google.com/lakehouse/docs/locations.md.txt).

  **Warning:** Avoid configuring overlapping paths with other
  catalogs to prevent unauthorized credential exposure.
  For more information, see [Multiple-bucket catalog](https://docs.cloud.google.com/lakehouse/docs/about-iceberg-rest-catalog-endpoint.md.txt#storage-multiple-buckets).
- `LOCATION`: (Optional) The primary region for the
  catalog to ensure interoperability with BigQuery. For
  Cloud Storage buckets in the US region (for example, `US` or
  `us-central1`) or the EU region (for example, `EU` or `europe-west4`),
  specify `US` or `EU` respectively to ensure the catalog is accessible and
  available for querying from the corresponding BigQuery
  multi-regions. For more information, see [Bucket and catalog regions](https://docs.cloud.google.com/lakehouse/docs/about-iceberg-rest-catalog-endpoint.md.txt#bucket_and_catalog_regions).

#### Create a single-bucket catalog

> [!CAUTION]
> **Caution:** Using the single-bucket configuration is highly discouraged for new projects. This configuration restricts your catalog to a single bucket and locks the catalog name to the bucket name.

To create a single-bucket catalog, run the following command:

```bash
gcloud biglake iceberg catalogs create \
    CATALOG_NAME \
    --project PROJECT_ID \
    --catalog-type gcs-bucket \
    --credential-mode CREDENTIAL_MODE
```

Replace the following:

- `CATALOG_NAME`: a name for your catalog. For single-bucket catalogs, this matches the Cloud Storage bucket ID used with the REST catalog.
- `PROJECT_ID`: your Google Cloud project ID.
- `CREDENTIAL_MODE`: the authentication method. Use `end-user` for *End-user credentials* or `vended-credentials` for *Credential vending mode*.

### Configure client application

After you create a catalog, configure your client application to use it. These
examples show you how to configure with or without credential vending.

### Cluster

Create a Managed Service for Apache Spark on Compute Engine cluster using
simplified configuration properties (recommended), or by specifying properties
manually.

#### Simplified configuration using properties (recommended)

Create a cluster with the catalog property:

```bash
gcloud dataproc clusters create CLUSTER_NAME \
  --enable-component-gateway \
  --project=PROJECT_ID \
  --region=REGION \
  --optional-components=ICEBERG \
  --image-version=DATAPROC_VERSION \
  --properties="dataproc:dataproc.lakehouse.catalog.CATALOG_NAME=projects/PROJECT_ID/catalogs/CATALOG_ID"
```

Replace the following:

- `CLUSTER_NAME`: a name for your cluster.
- `PROJECT_ID`: your Google Cloud project ID.
- `REGION`: the Managed Service for Apache Spark cluster region.
- `DATAPROC_VERSION`: the Managed Service for Apache Spark image version---for example, `2.3`.
- `CATALOG_NAME`: a name for the local Spark catalog (for example, `my_catalog`). It can be the same as `CATALOG_ID`.
- `CATALOG_ID`: the ID of the catalog you created.

> [!NOTE]
> **Note:** You can configure multiple catalogs separated by commas within the `--properties` flag. For example: `--properties="dataproc:dataproc.lakehouse.catalog.CATALOG_NAME_1=projects/PROJECT_ID_1/catalogs/CATALOG_ID_1,dataproc:dataproc.lakehouse.catalog.CATALOG_NAME_2=projects/PROJECT_ID_2/catalogs/CATALOG_ID_2"`

In the PySpark application file, create the `SparkSession` without specifying catalog configs:

```python
from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("APP_NAME").getOrCreate()
```

#### Manual configuration

If you do not use the simplified configuration property, create a cluster as
described above but without the `--properties` flag. Then, configure your
`SparkSession` manually:

```python
import pyspark
from pyspark.context import SparkContext
from pyspark.sql import SparkSession

catalog_name = "CATALOG_NAME"
spark = SparkSession.builder.appName("APP_NAME") \
  .config('spark.sql.defaultCatalog', 'CATALOG_NAME') \
  .config(f'spark.sql.catalog.{catalog_name}', 'org.apache.iceberg.spark.SparkCatalog') \
  .config(f'spark.sql.catalog.{catalog_name}.type', 'rest') \
  .config(f'spark.sql.catalog.{catalog_name}.uri', 'https://biglake.googleapis.com/iceberg/REST_API_VERSION/restcatalog') \
  .config(f'spark.sql.catalog.{catalog_name}.warehouse', 'WAREHOUSE_PATH') \
  .config(f'spark.sql.catalog.{catalog_name}.header.x-goog-user-project', 'PROJECT_ID') \
  .config(f'spark.sql.catalog.{catalog_name}.rest.auth.type', 'org.apache.iceberg.gcp.auth.GoogleAuthManager') \
  .config(f'spark.sql.catalog.{catalog_name}.io-impl', 'org.apache.iceberg.gcp.gcs.GCSFileIO') \
  .config('spark.sql.extensions', 'org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions') \
  .getOrCreate()
```

Replace the following:

- `CATALOG_NAME`: a name for the local Spark catalog (for example, `my_catalog`).
- `APP_NAME`: a name for your Spark session.
- `REST_API_VERSION`: set to `v1` for the stable version of the API.
- `WAREHOUSE_PATH`: The path to your warehouse. For multiple-bucket catalogs, use `bl://projects/PROJECT_ID/catalogs/CATALOG_ID`. For single-bucket catalogs, use `gs://CLOUD_STORAGE_BUCKET_NAME`.
- `PROJECT_ID`: the project that is billed for using the Apache Iceberg REST catalog endpoint, which might be different from the project that owns the Cloud Storage bucket. For details about project configuration when using a REST API, see [System parameters](https://docs.cloud.google.com/apis/docs/system-parameters.md.txt).

### Configure with credential vending

To use credential vending, you
must use a [catalog in credential vending mode](https://docs.cloud.google.com/lakehouse/docs/set-up-lakehouse-iceberg-rest-catalog.md.txt#create_a_catalog) and add
the `X-Iceberg-Access-Delegation` header to Iceberg
REST catalog requests with a value of `vended-credentials` by adding the
following line to the `SparkSession` builder:

```python
.config(f'spark.sql.catalog.{catalog_name}.header.X-Iceberg-Access-Delegation','vended-credentials')
```

<br />

#### Example with credential vending

The following example configures the query engine with credential vending:

```python
import pyspark
from pyspark.context import SparkContext
from pyspark.sql import SparkSession

catalog_name = "CATALOG_NAME"
spark = SparkSession.builder.appName("APP_NAME") \
  .config('spark.sql.defaultCatalog', 'CATALOG_NAME') \
  .config(f'spark.sql.catalog.{catalog_name}', 'org.apache.iceberg.spark.SparkCatalog') \
  .config(f'spark.sql.catalog.{catalog_name}.type', 'rest') \
  .config(f'spark.sql.catalog.{catalog_name}.uri', 'https://biglake.googleapis.com/iceberg/REST_API_VERSION/restcatalog') \
  .config(f'spark.sql.catalog.{catalog_name}.warehouse', 'WAREHOUSE_PATH') \
  .config(f'spark.sql.catalog.{catalog_name}.header.x-goog-user-project', 'PROJECT_ID') \
  .config(f'spark.sql.catalog.{catalog_name}.rest.auth.type', 'org.apache.iceberg.gcp.auth.GoogleAuthManager') \
  .config(f'spark.sql.catalog.{catalog_name}.io-impl', 'org.apache.iceberg.gcp.gcs.GCSFileIO') \
  .config(f'spark.sql.catalog.{catalog_name}.header.X-Iceberg-Access-Delegation','vended-credentials') \
  .config('spark.sql.extensions', 'org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions') \
  .getOrCreate()
```

For more information, see the [Headers in the `RESTCatalog`](https://py.iceberg.apache.org/configuration/#headers-in-rest-catalog) section of
the Apache Iceberg documentation.

Managed Service for Apache Spark clusters support Google authorization flows for
Apache Iceberg in the following releases:

- Managed Service for Apache Spark on Compute Engine 2.2 image versions 2.2.65 and later.
- Managed Service for Apache Spark on Compute Engine 2.3 image versions 2.3.11 and later.

### Serverless

Submit a [PySpark batch workload](https://docs.cloud.google.com/dataproc-serverless/docs/quickstarts/spark-batch#submit_a_spark_batch_workload) to Managed Service for Apache Spark
using simplified configuration properties (recommended), or by specifying
properties manually.

#### Simplified configuration using properties (recommended)

Submit a batch job with the catalog property:

```bash
gcloud dataproc batches submit pyspark PYSPARK_FILE \
    --project=PROJECT_ID \
    --region=REGION \
    --version=RUNTIME_VERSION \
    --properties="dataproc.lakehouse.catalog.CATALOG_NAME=projects/PROJECT_ID/catalogs/CATALOG_ID"
```

Replace the following:

- `PYSPARK_FILE`: the `gs://` Cloud Storage path to your PySpark application file.
- `PROJECT_ID`: your Google Cloud project ID.
- `REGION`: the region for the Managed Service for Apache Spark batch workload.
- `RUNTIME_VERSION`: the Managed Service for Apache Spark runtime version, for example `2.3`.
- `CATALOG_NAME`: a name for the local Spark catalog (for example, `my_catalog`). It can be the same as `CATALOG_ID`.
- `CATALOG_ID`: the ID of the catalog you created.

> [!NOTE]
> **Note:** You can configure multiple catalogs separated by commas within the `--properties` flag. For example: `--properties="dataproc.lakehouse.catalog.CATALOG_NAME_1=projects/PROJECT_ID_1/catalogs/CATALOG_ID_1,dataproc.lakehouse.catalog.CATALOG_NAME_2=projects/PROJECT_ID_2/catalogs/CATALOG_ID_2"`

In the PySpark application file, create the `SparkSession` without specifying catalog configs:

```python
from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("APP_NAME").getOrCreate()
```

#### Manual configuration

If you do not use the simplified configuration property, you must manually
specify catalog configs:

```bash
gcloud dataproc batches submit pyspark PYSPARK_FILE \
    --project=PROJECT_ID \
    --region=REGION \
    --version=RUNTIME_VERSION \
    --properties="\
    spark.sql.defaultCatalog=CATALOG_NAME,\
    spark.sql.catalog.CATALOG_NAME=org.apache.iceberg.spark.SparkCatalog,\
    spark.sql.catalog.CATALOG_NAME.type=rest,\
    spark.sql.catalog.CATALOG_NAME.uri=https://biglake.googleapis.com/iceberg/REST_API_VERSION/restcatalog,\
    spark.sql.catalog.CATALOG_NAME.warehouse=WAREHOUSE_PATH,\
    spark.sql.catalog.CATALOG_NAME.header.x-goog-user-project=PROJECT_ID,\
    spark.sql.catalog.CATALOG_NAME.rest.auth.type=org.apache.iceberg.gcp.auth.GoogleAuthManager,\
    spark.sql.catalog.CATALOG_NAME.io-impl=org.apache.iceberg.gcp.gcs.GCSFileIO,\
    spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions"
```

Replace the following:

- `PYSPARK_FILE`: the `gs://` Cloud Storage path to your PySpark application file.
- `REGION`: the region for the Managed Service for Apache Spark batch workload.
- `RUNTIME_VERSION`: the Managed Service for Apache Spark runtime version, for example `2.3`.
- `CATALOG_NAME`: a name for the local Spark catalog (for example, `my_catalog`).
- `REST_API_VERSION`: set to `v1` for the stable version of the API.
- `WAREHOUSE_PATH`: The path to your warehouse. For multiple-bucket catalogs, use `bl://projects/PROJECT_ID/catalogs/CATALOG_ID`. For single-bucket catalogs, use `gs://CLOUD_STORAGE_BUCKET_NAME`.
- `PROJECT_ID`: the project that is billed for using the Apache Iceberg REST catalog endpoint, which might be different from the project that owns the Cloud Storage bucket. For details about project configuration when using a REST API, see [System parameters](https://docs.cloud.google.com/apis/docs/system-parameters.md.txt).

### Configure with credential vending

To use credential vending, you must use a
[catalog in credential vending mode](https://docs.cloud.google.com/lakehouse/docs/set-up-lakehouse-iceberg-rest-catalog.md.txt#create_a_catalog) and add the
`X-Iceberg-Access-Delegation` header to the Apache Iceberg REST catalog endpoint requests with a value of `vended-credentials` by adding the following
line to the Managed Service for Apache Spark configurations:

```python
.config(f'spark.sql.catalog.{catalog_name}.header.X-Iceberg-Access-Delegation','vended-credentials')
```

<br />

#### Example with credential vending

The following example configures the query engine with credential vending:

```bash
gcloud dataproc batches submit pyspark PYSPARK_FILE \
    --project=PROJECT_ID \
    --region=REGION \
    --version=RUNTIME_VERSION \
    --properties="\
    spark.sql.defaultCatalog=CATALOG_NAME,\
    spark.sql.catalog.CATALOG_NAME=org.apache.iceberg.spark.SparkCatalog,\
    spark.sql.catalog.CATALOG_NAME.type=rest,\
    spark.sql.catalog.CATALOG_NAME.uri=https://biglake.googleapis.com/iceberg/REST_API_VERSION/restcatalog,\
    spark.sql.catalog.CATALOG_NAME.warehouse=WAREHOUSE_PATH,\
    spark.sql.catalog.CATALOG_NAME.header.x-goog-user-project=PROJECT_ID,\
    spark.sql.catalog.CATALOG_NAME.rest.auth.type=org.apache.iceberg.gcp.auth.GoogleAuthManager,\
    spark.sql.catalog.CATALOG_NAME.io-impl=org.apache.iceberg.gcp.gcs.GCSFileIO,\
    spark.sql.catalog.CATALOG_NAME.header.X-Iceberg-Access-Delegation=vended-credentials,\\
    spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions"
```

For more information, see the
[Headers in the `RESTCatalog`](https://py.iceberg.apache.org/configuration/#headers-in-rest-catalog)
section of the Apache Iceberg documentation.

Managed Service for Apache Spark supports Google authorization flows for
Apache Iceberg in the following runtime versions:

- Managed Service for Apache Spark 2.2 runtimes 2.2.60 and later
- Managed Service for Apache Spark 2.3 runtimes 2.3.10 and later

### Trino

To use Trino with the Apache Iceberg REST catalog endpoint, create a
Managed Service for Apache Spark cluster with the Trino component and configure catalog
properties using the `gcloud dataproc clusters create --properties` flag.
The following example creates a
Trino catalog named `CATALOG_NAME`:

```bash
gcloud dataproc clusters create CLUSTER_NAME \
    --enable-component-gateway \
    --region=REGION \
    --image-version=DATAPROC_VERSION \
    --network=NETWORK_ID \
    --optional-components=TRINO \
    --properties="\
    trino-catalog:CATALOG_NAME.connector.name=iceberg,\
    trino-catalog:CATALOG_NAME.iceberg.catalog.type=rest,\
    trino-catalog:CATALOG_NAME.iceberg.rest-catalog.uri=https://biglake.googleapis.com/iceberg/REST_API_VERSION/restcatalog,\
    trino-catalog:CATALOG_NAME.iceberg.rest-catalog.warehouse=WAREHOUSE_PATH,\
    trino-catalog:CATALOG_NAME.iceberg.rest-catalog.biglake.project-id=PROJECT_ID,\
    trino-catalog:CATALOG_NAME.iceberg.rest-catalog.rest.auth.type=org.apache.iceberg.gcp.auth.GoogleAuthManager"
```

Replace the following:

- `CLUSTER_NAME`: a name for your cluster.
- `REGION`: the Managed Service for Apache Spark cluster region.
- `DATAPROC_VERSION`: Managed Service for Apache Spark image version, for example `2.2`.
- `NETWORK_ID`: cluster network ID. For more information, see [Managed Service for Apache Spark cluster network configuration](https://docs.cloud.google.com/dataproc/docs/concepts/configuring-clusters/network).
- `CATALOG_NAME`: the name of your Trino catalog using the Apache Iceberg REST catalog endpoint.
- `REST_API_VERSION`: set to `v1` for the stable version of the API.
- `WAREHOUSE_PATH`: The path to your warehouse. For BigLake catalogs, use `bl://projects/PROJECT_ID/catalogs/CATALOG_ID`. For Cloud Storage bucket catalogs, use `gs://CLOUD_STORAGE_BUCKET_NAME`.
- `PROJECT_ID`: your Google Cloud project ID to use for Lakehouse runtime catalog.

After cluster creation, connect to the main VM instance, and use the Trino CLI:

```bash
trino --catalog=CATALOG_NAME
```

Managed Service for Apache Spark Trino supports Google authorization flows for
Apache Iceberg in the following releases:

- Managed Service for Apache Spark on Compute Engine 2.2 runtime versions 2.2.65 and later
- Managed Service for Apache Spark on Compute Engine 2.3 runtime versions 2.3.11 and later
- Managed Service for Apache Spark on Compute Engine 3.0 is not supported.

### Configure with credential vending

Credential vending is only supported on Trino version 481 and later.

### Apache Iceberg 1.10 or later

Open source Apache Iceberg 1.10 and later releases have built-in
support for Google authorization flows in `GoogleAuthManager`.
The following is an example of how to configure Spark
to use the Apache Iceberg REST catalog endpoint.

```python
import pyspark
from pyspark.context import SparkContext
from pyspark.sql import SparkSession

catalog_name = "CATALOG_NAME"
spark = SparkSession.builder.appName("APP_NAME") \
  .config(f'spark.sql.catalog.{catalog_name}', 'org.apache.iceberg.spark.SparkCatalog') \
  .config(f'spark.sql.catalog.{catalog_name}.type', 'rest') \
  .config(f'spark.sql.catalog.{catalog_name}.uri', 'https://biglake.googleapis.com/iceberg/REST_API_VERSION/restcatalog') \
  .config(f'spark.sql.catalog.{catalog_name}.warehouse', 'WAREHOUSE_PATH') \
  .config(f'spark.sql.catalog.{catalog_name}.header.x-goog-user-project', 'PROJECT_ID') \
  .config(f'spark.sql.catalog.{catalog_name}.rest.auth.type', 'org.apache.iceberg.gcp.auth.GoogleAuthManager') \
  .config(f'spark.sql.catalog.{catalog_name}.io-impl', 'org.apache.iceberg.gcp.gcs.GCSFileIO') \
  .config('spark.sql.extensions', 'org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions') \
  .config('spark.sql.defaultCatalog', 'CATALOG_NAME') \
  .getOrCreate()
```

Replace the following:

- `CATALOG_NAME`: a name for the local Spark catalog (for example, `my_catalog`).
- `APP_NAME`: a name for your Spark session.
- `REST_API_VERSION`: set to `v1` for the stable version of the API.
- `WAREHOUSE_PATH`: The path to your warehouse. For BigLake catalogs, use `bl://projects/PROJECT_ID/catalogs/CATALOG_ID`. For Cloud Storage bucket catalogs, use `gs://CLOUD_STORAGE_BUCKET_NAME`.
- `PROJECT_ID`: the project that is billed for using the Apache Iceberg REST catalog endpoint, which might be different from the project that owns the Cloud Storage bucket. For details about project configuration when using a REST API, see [System parameters](https://docs.cloud.google.com/apis/docs/system-parameters.md.txt).

### Configure with credential vending

The preceding example doesn't use credential vending. To use credential
vending, you must use a
[catalog in credential vending mode](https://docs.cloud.google.com/lakehouse/docs/set-up-lakehouse-iceberg-rest-catalog.md.txt#create_a_catalog) and add the
`X-Iceberg-Access-Delegation` header to the Apache Iceberg REST catalog endpoint requests with a value of `vended-credentials` by adding the following
line to the `SparkSession` builder:

```python
.config(f'spark.sql.catalog.{catalog_name}.header.X-Iceberg-Access-Delegation','vended-credentials')
```

<br />

#### Example with credential vending

The following example configures the query engine with credential vending:

```python
import pyspark
from pyspark.context import SparkContext
from pyspark.sql import SparkSession

catalog_name = "CATALOG_NAME"
spark = SparkSession.builder.appName("APP_NAME") \
  .config(f'spark.sql.catalog.{catalog_name}', 'org.apache.iceberg.spark.SparkCatalog') \
  .config(f'spark.sql.catalog.{catalog_name}.type', 'rest') \
  .config(f'spark.sql.catalog.{catalog_name}.uri', 'https://biglake.googleapis.com/iceberg/REST_API_VERSION/restcatalog') \
  .config(f'spark.sql.catalog.{catalog_name}.warehouse', 'WAREHOUSE_PATH') \
  .config(f'spark.sql.catalog.{catalog_name}.header.x-goog-user-project', 'PROJECT_ID') \
  .config(f'spark.sql.catalog.{catalog_name}.rest.auth.type', 'org.apache.iceberg.gcp.auth.GoogleAuthManager') \
  .config(f'spark.sql.catalog.{catalog_name}.io-impl', 'org.apache.iceberg.gcp.gcs.GCSFileIO') \
  .config(f'spark.sql.catalog.{catalog_name}.header.X-Iceberg-Access-Delegation','vended-credentials') \
  .config('spark.sql.extensions', 'org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions') \
  .config('spark.sql.defaultCatalog', 'CATALOG_NAME') \
  .getOrCreate()
```

For more information, see the
[Headers in the `RESTCatalog`](https://py.iceberg.apache.org/configuration/#headers-in-rest-catalog)
section of the Apache Iceberg documentation.

### Prior Apache Iceberg releases

For open source Apache Iceberg releases prior to 1.10, you can
configure standard OAuth authentication by configuring a session with the
following:

```python
import pyspark
from pyspark.context import SparkContext
from pyspark.sql import SparkSession

catalog_name = "CATALOG_NAME"
spark = SparkSession.builder.appName("APP_NAME") \
  .config('spark.jars.packages', 'org.apache.iceberg:iceberg-spark-runtime-3.5_2.12:1.9.1,org.apache.iceberg:iceberg-gcp-bundle:1.9.1') \
  .config(f'spark.sql.catalog.{catalog_name}', 'org.apache.iceberg.spark.SparkCatalog') \
  .config(f'spark.sql.catalog.{catalog_name}.type', 'rest') \
  .config(f'spark.sql.catalog.{catalog_name}.uri', 'https://biglake.googleapis.com/iceberg/REST_API_VERSION/restcatalog') \
  .config(f'spark.sql.catalog.{catalog_name}.warehouse', 'WAREHOUSE_PATH') \
  .config(f'spark.sql.catalog.{catalog_name}.header.x-goog-user-project', 'PROJECT_ID') \
  .config(f"spark.sql.catalog.{catalog_name}.token", "TOKEN") \
  .config(f"spark.sql.catalog.{catalog_name}.oauth2-server-uri", "https://oauth2.googleapis.com/token") \
  .config(f'spark.sql.catalog.{catalog_name}.io-impl', 'org.apache.iceberg.gcp.gcs.GCSFileIO') \
  .config('spark.sql.extensions', 'org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions') \
  .config('spark.sql.defaultCatalog', 'CATALOG_NAME') \
  .getOrCreate()
```

Replace the following:

- `CATALOG_NAME`: a name for the local Spark catalog (for example, `my_catalog`).
- `APP_NAME`: a name for your Spark session.
- `REST_API_VERSION`: set to `v1` for the stable version of the API.
- `WAREHOUSE_PATH`: The path to your warehouse. For BigLake catalogs, use `bl://projects/PROJECT_ID/catalogs/CATALOG_ID`. For Cloud Storage bucket catalogs, use `gs://CLOUD_STORAGE_BUCKET_NAME`.
- `PROJECT_ID`: the project that is billed for using the Apache Iceberg REST catalog endpoint, which might be different from the project that owns the Cloud Storage bucket. For details about project configuration when using a REST API, see [System parameters](https://docs.cloud.google.com/apis/docs/system-parameters.md.txt).
- `TOKEN`: your authentication token, which is valid for one hour---for example, a token generated using `gcloud auth application-default print-access-token`.

### Configure with credential vending

The preceding example doesn't use credential vending. To use credential
vending, you must use a
[catalog in credential vending mode](https://docs.cloud.google.com/lakehouse/docs/set-up-lakehouse-iceberg-rest-catalog.md.txt#create_a_catalog) and add the
`X-Iceberg-Access-Delegation` header to the Apache Iceberg REST catalog endpoint requests with a value of `vended-credentials` by adding the following
line to the `SparkSession` builder:

```python
.config(f'spark.sql.catalog.{catalog_name}.header.X-Iceberg-Access-Delegation','vended-credentials')
```

<br />

#### Example with credential vending

The following example configures the query engine with credential vending:

```python
import pyspark
from pyspark.context import SparkContext
from pyspark.sql import SparkSession

catalog_name = "CATALOG_NAME"
spark = SparkSession.builder.appName("APP_NAME") \
  .config(f'spark.sql.catalog.{catalog_name}', 'org.apache.iceberg.spark.SparkCatalog') \
  .config(f'spark.sql.catalog.{catalog_name}.type', 'rest') \
  .config(f'spark.sql.catalog.{catalog_name}.uri', 'https://biglake.googleapis.com/iceberg/REST_API_VERSION/restcatalog') \
  .config(f'spark.sql.catalog.{catalog_name}.warehouse', 'WAREHOUSE_PATH') \
  .config(f'spark.sql.catalog.{catalog_name}.header.x-goog-user-project', 'PROJECT_ID') \
  .config(f"spark.sql.catalog.{catalog_name}.token", "TOKEN") \
  .config(f"spark.sql.catalog.{catalog_name}.oauth2-server-uri", "https://oauth2.googleapis.com/token") \
  .config(f'spark.sql.catalog.{catalog_name}.io-impl', 'org.apache.iceberg.gcp.gcs.GCSFileIO') \
  .config(f'spark.sql.catalog.{catalog_name}.header.X-Iceberg-Access-Delegation','vended-credentials') \
  .config('spark.sql.extensions', 'org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions') \
  .config('spark.sql.defaultCatalog', 'CATALOG_NAME') \
  .getOrCreate()
```

For more information, see the
[Headers in the `RESTCatalog`](https://py.iceberg.apache.org/configuration/#headers-in-rest-catalog)
section of the Apache Iceberg documentation.

### Create a namespace or schema

After you configure your client, create a namespace or schema to organize your
tables. The syntax for creating a namespace or schema varies depending on your
query engine. The following examples show how to create them using Spark and
Trino.

> [!CAUTION]
> **Caution:** If you're using credential vending, don't create namespaces or schemas within other namespaces or schemas, because doing so can grant users access to unintended resources.

### Console

1. In the Google Cloud console, go to the **Lakehouse** page.

   [Go to Lakehouse](https://console.cloud.google.com/biglake)
2. Select an existing catalog, or create one if you don't have one.

3. On the **Catalog details** page, click **Create namespace**.

4. For **Namespace name**, enter a unique name for your namespace.

5. For **Location**, specify the Cloud Storage path to associate
   with your namespace:

   - **Multiple-bucket** (recommended): You can set any custom location as long as it is under a location allowed by the catalog (`default_location` or `restricted_locations`). If you don't specify a location, the namespace is created under the catalog's default location (for example, `gs://{path-to-default-location}/{namespace_name}`).
   - **Single-bucket**: The namespace location is automatically inherited from the catalog's single bucket.
6. Optional: For **Owner**, enter or update the namespace owner. By
   default, this field is populated with your username.

7. Optional: Under **Properties** , click **Add property** and enter a
   **Key** and **Value** to add custom metadata properties to the
   namespace.

8. Click **Create**.

### Spark

#### Cloud Storage warehouse

```python
spark.sql("CREATE NAMESPACE IF NOT EXISTS NAMESPACE_NAME;")
spark.sql("USE NAMESPACE_NAME;")
```

Replace `NAMESPACE_NAME` with a name for your namespace.

### Trino

#### Cloud Storage warehouse

```sql
CREATE SCHEMA IF NOT EXISTS  CATALOG_NAME.SCHEMA_NAME;
USE CATALOG_NAME.SCHEMA_NAME;
```

Replace the following:

- `CATALOG_NAME`: the name of your Trino catalog using the Apache Iceberg REST catalog endpoint.
- `SCHEMA_NAME`: a name for your schema.

## Upgrade a catalog

If you have an existing single-bucket catalog, you can upgrade
it to a multiple-bucket catalog type (recommended). Upgrading lets you
associate multiple buckets and configure restricted locations while retaining
the original catalog name.

To upgrade your catalog, see [Update a catalog](https://docs.cloud.google.com/lakehouse/docs/update-catalog.md.txt).

## What's next

- Learn how to [manage catalogs in the Google Cloud console](https://docs.cloud.google.com/lakehouse/docs/manage-catalogs.md.txt).

- Learn about [Apache Iceberg tables managed by Lakehouse](https://docs.cloud.google.com/lakehouse/docs/lakehouse-tables.md.txt).