Import Delta Lake tables into Lakehouse using Dataflow

The Dataflow job builder lets you import Cloud Storage Delta Lake tables directly into borderless Lakehouse through a low-code or no-code interface, reducing the complexity of large-scale data migrations. An autoscaling Dataflow batch job copies your data to the target Apache Iceberg table in Lakehouse and automatically creates the schema for new tables, while existing tables require a matching schema.

After you import your data, you can optionally enable continuous migration.

Before you begin

To import Delta Lake table data, you need the following:

  1. Enable the Dataflow, BigQuery, and Lakehouse APIs, if any are not already enabled.

    Roles required to enable APIs

    To enable APIs, you need the serviceusage.services.enable permission. If you created the project, then you likely already have this permission through the Owner role (roles/owner). Otherwise, you can get this permission through the Service Usage Admin role (roles/serviceusage.serviceUsageAdmin). Learn how to grant roles.

    Enable the APIs

  2. To get the permissions that you need to create the resources, ask your administrator to grant you the required Identity and Access Management (IAM) roles on your project.

  3. An existing Delta Lake table stored in a Cloud Storage bucket. The table directory must be a valid Delta Lake table root containing your Parquet data files and the _delta_log/ transaction log directory.

  4. A Lakehouse Iceberg catalog, namespace, and table to import the data into.

Support and limitations

Importing Delta Lake table data into Lakehouse using Dataflow has the following limitations:

  • You must use a batch pipeline job to use this feature.
  • For existing destination tables, the schema isn't modified. The destination table schema must correctly map to the source Delta Lake table schema.
  • The source data must be a valid Delta Lake table stored in Cloud Storage. The table root directory must contain the Parquet data files and the _delta_log/ transaction log directory (containing JSON or Parquet log files) as created by Delta Lake.
  • Amazon S3 isn't supported for Delta Lake table sources.

Import a Delta Lake table

To import a Delta Lake table into Lakehouse, complete the following steps:

  1. In the Google Cloud console, go to the Lakehouse runtime catalog page.

    Go to Lakehouse runtime catalog

  2. Select the catalog, namespace, and table you want to import data into.

  3. On the Table details page, click Import table, and then select From Delta Lake (Batch).

    The Dataflow Job builder page opens with the Delta Lake to Lakehouse blueprint loaded.

  4. In the Sources section:

    1. To expand the ReadFromDeltaLake Delta Lake table source panel, click the expander arrow.

    2. In the Table path field, enter the Cloud Storage URI of the root directory of the Delta Lake table (the directory containing the data files and the _delta_log/ directory). For example, gs://BUCKET_NAME/tables/TABLE_NAME.

    3. Optional: In the Hadoop configuration properties field, configure any additional Hadoop configuration properties required to read from Cloud Storage. For example: fs.gs.impl=com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystem.

    4. Click Done.

  5. In the Sink section:

    1. Optional: Review the WriteToIceberg Lakehouse table sink panel. The information in this panel, such as the Lakehouse table, catalog name, and warehouse location, is typically prepopulated.

    2. Click Done.

  6. In the Dataflow options section, click Run job.

If you need to further customize the Dataflow pipeline used to import Delta Lake tables, you can do that using the job builder form or the YAML editor.

Examine the job output

After the job completes, you can verify that the data was copied to the Lakehouse table by querying it in BigQuery.

  1. In the Dataflow job list, check that the job status is Succeeded.

    Go to Jobs

  2. If the job fails or has errors, check the job logs or worker logs for details.

  3. In the Google Cloud console, go to the BigQuery Studio page.

    Go to BigQuery

  4. In the query editor, enter a SQL query to inspect the table. You can use the PROJECT_ID.CATALOG.NAMESPACE.TABLE_NAME convention to query:

    SELECT * FROM `PROJECT_ID`.`CATALOG`.`NAMESPACE`.`TABLE_NAME` LIMIT 10;

  5. Click Run.

  6. Review the Query results to ensure the data was processed correctly.

Optional. Enable continuous migration

To set up end-to-end continuous migration from Delta Lake to Lakehouse, complete the following sequence of operations:

  1. Enable the delta.enableChangeDataFeed property on the input Delta Lake table.
  2. Perform an initial table migration to a commit ID or timestamp after setting the delta.enableChangeDataFeed property.
  3. Run scheduled CDC jobs for subsequent data using a frequency (commit ID or timestamp range) chosen for your workload requirements.

For more information, see Migrate Delta Lake change data capture (CDC) data into Lakehouse using Dataflow.

What's next