Import Delta Lake tables into Lakehouse using Dataflow

Migrating Delta Lake tables into Lakehouse for Apache Iceberg can be complex and expensive if you have to rewrite or move large volumes of data. To make your Delta Lake data available in Lakehouse without moving or rewriting the underlying files, you can use the Dataflow job builder. The job builder provides a low-code or no-code interface to import your Cloud Storage Delta Lake tables directly into Lakehouse.

For new tables, Dataflow automatically creates the schema. For existing tables, the schema isn't modified, so the destination table schema must correctly map to the source Delta Lake table for the job to succeed.

Use the following connection details to import data from a Delta Lake table stored in Cloud Storage.

Before you begin

To import Delta Lake table data, you need the following:

  1. Enable the Dataflow, BigQuery, and Lakehouse APIs.

    Roles required to enable APIs

    To enable APIs, you need the serviceusage.services.enable permission. If you created the project, then you likely already have this permission through the Owner role (roles/owner). Otherwise, you can get this permission through the Service Usage Admin role (roles/serviceusage.serviceUsageAdmin). Learn how to grant roles.

    Enable the APIs

  2. To get the permissions that you need to create the resources, ask your administrator to grant you the required Identity and Access Management (IAM) roles on your project.

  3. An existing Delta Lake table stored in a Cloud Storage bucket. The table directory must be a valid Delta Lake table root containing your Parquet data files and the _delta_log/ transaction log directory.

  4. A Lakehouse Iceberg catalog, namespace, and table to import the data into.

Support and limitations

Importing Delta Lake table data into Lakehouse for Apache Iceberg using Dataflow has the following limitations:

  • You must use a batch pipeline job to use this feature.
  • For existing destination tables, the schema isn't modified. The destination table schema must correctly map to the source Delta Lake table schema.
  • The source data must be a valid Delta Lake table stored in Cloud Storage. The table root directory must contain the Parquet data files and the _delta_log/ transaction log directory (containing JSON or Parquet log files) as created by Delta Lake.
  • Amazon S3 isn't supported for Delta Lake table sources.

Import a Delta Lake table

To import a Delta Lake table into Lakehouse for Apache Iceberg, complete the following steps:

  1. In the Google Cloud console, go to the Lakehouse Runtime catalog page.

    Go to Lakehouse runtime catalog

  2. Select the catalog, namespace, and table you want to import data into.

  3. On the Table details page, click Import table, and then select From Delta Lake (Batch).

    The Dataflow Job builder page opens with the Delta Lake to Lakehouse blueprint loaded.

  4. In the Sources section:

    1. To expand the ReadFromDeltaLake Delta Lake table source panel, click the expander arrow.

    2. In the Table path field, enter the Cloud Storage URI of the root directory of the Delta Lake table (the directory containing the data files and the _delta_log/ directory). For example, gs://BUCKET_NAME/tables/TABLE_NAME.

    3. Optional: In the Hadoop configuration properties field, configure any additional Hadoop configuration properties required to read from Cloud Storage. For example: fs.gs.impl=com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystem.

    4. Click Done.

  5. In the Sink section:

    1. Optional: Review the WriteToIceberg Lakehouse table sink panel. The information in this panel, such as the Lakehouse table, catalog name, and warehouse location, is typically prepopulated.

    2. Click Done.

  6. In the Dataflow options section, click Run job.

If you need to further customize the Dataflow pipeline used to import Delta Lake tables, you can do that using the job builder form or the YAML editor.

Examine the job output

After the job completes, you can verify that the data was registered with the Iceberg table by querying it in BigQuery.

  1. In the Dataflow job list, check that the job status is Succeeded.

    Go to Jobs

  2. If the job fails or has errors, check the job logs or worker logs for details.

  3. In the Google Cloud console, go to the BigQuery Studio page.

    Go to BigQuery

  4. In the query editor, enter a SQL query to inspect the table. You can use the PROJECT_ID.CATALOG.NAMESPACE.TABLE_NAME convention to query:

    SELECT * FROM `PROJECT_ID`.`CATALOG`.`NAMESPACE`.`TABLE_NAME` LIMIT 10;

  5. Click Run.

  6. Review the Query results to ensure the data was processed correctly.

What's next