Create a catalog

Creating a catalog endpoint (such as an Apache Iceberg REST catalog endpoint or Apache Hive catalog endpoint) establishes a management endpoint within the Lakehouse runtime catalog.

This endpoint points to an underlying Cloud Storage bucket, providing a metadata layer that lets query engines and open-source workloads interact directly with your tables.

When creating your catalog endpoint for Lakehouse for Apache Iceberg, you can choose between end-user credentials or credential vending mode for storage access delegation.

Before you begin

1. Read About the Lakehouse runtime catalog to understand how the Lakehouse runtime catalog works and the limitations for the service.
Sign in to your Google Cloud account. If you're new to Google Cloud, create an account to evaluate how our products perform in real-world scenarios. New customers also get $300 in free credits to run, test, and deploy workloads.

In the Google Cloud console, on the project selector page, select or create a Google Cloud project.

Roles required to select or create a project

• Select a project: Selecting a project doesn't require a specific IAM role—you can select any project that you've been granted a role on.
• Create a project: To create a project, you need the Project Creator role (roles/resourcemanager.projectCreator), which contains the resourcemanager.projects.create permission. Learn how to grant roles.

Note: If you don't plan to keep the resources that you create in this procedure, create a project instead of selecting an existing project. After you finish these steps, you can delete the project, removing all resources associated with the project.

Go to project selector

Verify that billing is enabled for your Google Cloud project.

Enable the BigLake API.

Roles required to enable APIs

To enable APIs, you need the serviceusage.services.enable permission. If you created the project, then you likely already have this permission through the Owner role (roles/owner). Otherwise, you can get this permission through the Service Usage Admin role (roles/serviceusage.serviceUsageAdmin). Learn how to grant roles.

Enable the API

In the Google Cloud console, on the project selector page, select or create a Google Cloud project.

Roles required to select or create a project

• Select a project: Selecting a project doesn't require a specific IAM role—you can select any project that you've been granted a role on.
• Create a project: To create a project, you need the Project Creator role (roles/resourcemanager.projectCreator), which contains the resourcemanager.projects.create permission. Learn how to grant roles.

Note: If you don't plan to keep the resources that you create in this procedure, create a project instead of selecting an existing project. After you finish these steps, you can delete the project, removing all resources associated with the project.

Go to project selector

Verify that billing is enabled for your Google Cloud project.

Enable the BigLake API.

Roles required to enable APIs

To enable APIs, you need the serviceusage.services.enable permission. If you created the project, then you likely already have this permission through the Owner role (roles/owner). Otherwise, you can get this permission through the Service Usage Admin role (roles/serviceusage.serviceUsageAdmin). Learn how to grant roles.

Enable the API

Required roles

To get the permissions that you need to create a catalog and register tables, ask your administrator to grant you the following IAM roles:

• Create a catalog:
  • BigLake Admin (roles/biglake.admin) on your project
  • Storage Admin (roles/storage.admin) on your project
• Register tables: BigLake Admin (roles/biglake.admin) on your project. Registering tables in a multiple-bucket catalog requires the specific table registration permission (biglake.tables.register), which is included in the BigLake Admin role.
• Use the auto-provisioned catalog service account in credential vending mode: Storage Object User (roles/storage.objectUser) on the target Cloud Storage buckets. After creating the catalog, explicitly grant the Storage Object User role (roles/storage.objectUser) on your storage buckets to the auto-provisioned service account.

For more information about granting roles, see Manage access to projects, folders, and organizations.

You might also be able to get the required permissions through custom roles or other predefined roles.

Note: Creation of the auto-provisioned catalog service account is eventually consistent. This means that it takes time for the service account to propagate through the system. If you try to grant roles to the service account immediately after creation, the request might fail. To learn how long it takes, on average, for access changes to propagate, see Access change propagation.

Create a catalog

Create a catalog.

Create a catalog

Follow these steps to create a catalog based on your preferred credential mode and bucket type. For more information about the different catalog types, see About the Apache Iceberg REST catalog endpoint.

Console

Create a multiple-bucket (bl://) catalog (recommended)

This configuration lets your catalog to associate multiple buckets and lets you name your catalog independently of any bucket name.

1. Open the Lakehouse page in the Google Cloud console.

   Go to Lakehouse

2. Click Create catalog.

3. For Catalog type, select Iceberg Rest Catalog.

4. For Lakehouse catalog bucket options, select Multiple bucket catalog.

5. For Default Catalog Cloud Storage path, enter or browse for the Cloud Storage path to use with your catalog.

6. For Catalog ID, enter a custom name for your catalog.

7. For Primary location, select a location. The location must be near the regions of the main bucket.

8. Click Continue.

9. In the Data paths step, add additional Cloud Storage paths if needed.

10. Click Continue.

11. For Authentication method, select End-user credentials or Credential vending mode.

12. Click Create.

    Your catalog is created and the Catalog details page opens.

13. If you selected Credential vending mode, perform the following additional steps:

    1. Under Authentication method, click Set bucket permissions.
    2. In the dialog, click Confirm.

Create a single-bucket (gs://) catalog

Caution: Using the single-bucket configuration is highly discouraged for new projects. This configuration restricts your catalog to a single bucket and locks the catalog name to the bucket name.

1. For Catalog type, select Iceberg Rest Catalog or Hive Metastore.
2. For Lakehouse catalog bucket options, select Single bucket catalog.
3. For Default Catalog Cloud Storage path, enter or browse for the Cloud Storage path to use with your catalog. (For a single-bucket (gs://) catalog, you can only have one catalog per bucket, and the catalog name matches the bucket name).
4. Click Continue.
5. For Authentication method, select End-user credentials or Credential vending mode.
6. Click Create.
7. If you selected Credential vending mode, perform the following additional steps:
   1. Under Authentication method, click Set bucket permissions.
   2. In the dialog, click Confirm.

gcloud

Create a multiple-bucket (bl://) catalog (recommended)

This configuration lets your catalog to associate multiple buckets and lets you name your catalog independently of any bucket name.

To create a multiple-bucket (bl://) catalog (recommended), run the gcloud biglake iceberg catalogs create command.

gcloud biglake iceberg catalogs create \
    CATALOG_NAME \
    --project PROJECT_ID \
    --catalog-type biglake \
    --default-location DEFAULT_LOCATION \
    --credential-mode CREDENTIAL_MODE \
    [--restricted-locations RESTRICTED_LOCATIONS] \
    [--primary-location LOCATION]

Replace the following:

• CATALOG_NAME: a name for your catalog. For multiple-bucket (bl://) catalogs (recommended), this is your custom catalog name. For single-bucket (gs://) catalogs, this matches the Cloud Storage bucket ID used with the REST catalog.
• PROJECT_ID: your Google Cloud project ID.
• DEFAULT_LOCATION: Specify the default storage location for the catalog. You can specify a bucket (gs://my-bucket) or a subpath (gs://my-bucket/path). All namespaces and tables in the catalog must reside under the specified path. For example, if you specify gs://my-bucket/path, you cannot create namespaces or tables under gs://my-bucket/another/path.

• CREDENTIAL_MODE: the authentication method. Use end-user for End-user credentials or vended-credentials for Credential vending mode.

  Note: If you use Credential vending mode, you must explicitly grant the Storage Object User role (roles/storage.objectUser) on all associated storage buckets to your catalog's auto-provisioned Lakehouse runtime catalog service account.

• RESTRICTED_LOCATIONS: (Optional) Comma-separated list of additional allowed storage locations, in the format gs://my-bucket-1/...,gs://my-bucket-2/.... If you specify a path (such as gs://my-bucket/path), any namespaces or tables within that bucket must be under that path. All configured cloud storage locations across the default location and restricted locations must be in the same geographic region group or jurisdiction (such as the United States, Europe, Canada, or Asia). For example, you cannot mix a bucket in the US with a bucket in Europe. For a list of supported locations, see Lakehouse locations.

  Warning: Avoid configuring overlapping paths with other catalogs to prevent unauthorized credential exposure. or more information, see Storage across Multiple buckets.

• LOCATION: (Optional) The primary region for the catalog to ensure interoperability with BigQuery. For Cloud Storage buckets in the US region (for example, US or us-central1) or the EU region (for example, EU or europe-west4), specify US or EU respectively to ensure the catalog is accessible and available for querying from the corresponding BigQuery multi-regions. For more information, see Bucket and catalog regions.

Create a single-bucket (gs://) catalog

Caution: Using the single-bucket configuration is highly discouraged for new projects. This configuration restricts your catalog to a single bucket and locks the catalog name to the bucket name.

To create a single-bucket (gs://) catalog, run the following command:

gcloud biglake iceberg catalogs create \
    CATALOG_NAME \
    --project PROJECT_ID \
    --catalog-type gcs-bucket \
    --credential-mode CREDENTIAL_MODE

Replace the following:

• CATALOG_NAME: a name for your catalog. For multiple-bucket (bl://) catalogs (recommended), this is your custom catalog name. For single-bucket (gs://) catalogs, this matches the Cloud Storage bucket ID used with the REST catalog.
• PROJECT_ID: your Google Cloud project ID.
• CREDENTIAL_MODE: the authentication method. Use end-user for End-user credentials or vended-credentials for Credential vending mode.

REST

Create an Iceberg REST catalog

To create a catalog management endpoint using the REST API, make a POST request to the CreateIcebergCatalog endpoint:

POST /iceberg/v1/restcatalog/extensions/projects/PROJECT_ID/catalogs?icebergCatalogId=CATALOG_ID

The request body must contain an IcebergCatalog JSON payload defining the catalog configuration, such as the underlying Cloud Storage bucket warehouse and authentication mode.

Replace the following:

• PROJECT_ID: your Google Cloud project ID.
• CATALOG_ID: the ID of your Lakehouse runtime catalog.