Import external Iceberg tables into Lakehouse runtime catalog using Dataflow

Your use case might require you to connect an external Iceberg REST Catalog (IRC) table to an existing Google Cloud Lakehouse table. Dataflow's job builder UI lets you build a pipeline that migrates your external open source Iceberg catalog tables into Lakehouse in a low-code or no-code way. This process lets you consolidate data into a unified Lakehouse-managed Iceberg format for cross-engine analytics.

Use the following connection details to import data from external Iceberg catalogs.

Before you begin

To import data, you need the following:

  1. Connection information for the external Iceberg REST Catalog. For example: catalog name, namespace, table name, account URI, and role to access the catalog.
  2. A Lakehouse Iceberg catalog, namespace, and table to import the data into.

Support and limitations

Importing data from external Iceberg catalogs to Google Cloud Lakehouse using Dataflow has the following limitations:

  • This feature supports reading from externally available Iceberg providers that support IRC (Iceberg Rest Catalog) into Lakehouse. Other Iceberg catalog types aren't supported.
  • This feature supports batch and streaming pipelines.

Import an external Iceberg catalog table

To import an external Iceberg catalog table into Google Cloud Lakehouse, complete the following steps:

  1. In the Google Cloud console, go to the Google Cloud Lakehouse Metastore page.

    Go to Google Cloud Lakehouse Metastore

  2. Select the catalog, namespace, and table you want to import data into.

  3. On the Table details page, click Import table.

  4. In the Import configuration dialog, select Import a table from an Apache Iceberg REST Catalog into Lakehouse (Batch).

    The Dataflow Job builder page opens.

  5. In the Sources section:

    1. To expand the Iceberg table source panel, click the expander arrow.

    2. In the Iceberg table field, enter the identifier of the Apache Iceberg table.

    3. In the Catalog name field, enter the name of the catalog.

    4. In the Filter field, enter the Iceberg filter to use. For example, id > 5.

    5. Optional: To specify source table column changes, use the Keep columns or Drop columns sections.

    6. In the Catalog type list of the Catalog properties section, select the type of catalog.

    7. In the Catalog URI field, enter the URI of the catalog. For example, http://localhost:8181.

    8. In the Warehouse name field, enter the catalog name.

      For some external Iceberg REST Catalog providers, the warehouse is abstracted, and the catalog name is provided as the warehouse name.

    9. In the Authentication type list, select the authentication type. For example, OAUTH2.

  6. Optional: In the Transforms section, add any transforms to the source data.

  7. In the Sink section:

    1. Optional: Review the Lakehouse table sink panel. The information in this panel, such as the Lakehouse table, catalog name, and warehouse location, is typically prepopulated.

  8. In the Dataflow options section, click Run job.

What's next