The Dataflow job builder lets you import Cloud Storage
Delta Lake tables directly into borderless Lakehouse through a
low-code or no-code interface, reducing the complexity of large-scale data
migrations. An autoscaling Dataflow batch job copies your data to
the target Apache Iceberg table in Lakehouse and automatically
creates the schema for new tables, while existing tables require a matching
schema.

After you import your data, you can optionally
[enable continuous migration](https://docs.cloud.google.com/dataflow/docs/guides/delta-lake-df-lakehouse-integration#continuous-migration).

## Before you begin

To import Delta Lake table data, you need the following:

1.


   Enable the Dataflow, BigQuery, and Lakehouse APIs, if any are not already enabled.


   **Roles required to enable APIs**


   To enable APIs, you need the `serviceusage.services.enable` permission. If you
   created the project, then you likely already have this permission through the
   Owner role (`roles/owner`). Otherwise, you can get this permission through the
   Service Usage Admin role (`roles/serviceusage.serviceUsageAdmin`).
   [Learn how to grant roles](https://docs.cloud.google.com/iam/docs/granting-changing-revoking-access).

   [Enable the APIs](https://console.cloud.google.com/apis/enableflow?apiid=dataflow.googleapis.com,bigquery.googleapis.com,biglake.googleapis.com)

   <br />

2. To get the permissions that you need to create the resources, ask your
   administrator to grant you the required Identity and Access Management (IAM) roles on
   your project.

3. An existing Delta Lake table stored in a Cloud Storage bucket. The table
   directory must be a valid Delta Lake table root containing your Parquet
   data files and the `_delta_log/` transaction log directory.

4. A Lakehouse Iceberg catalog, namespace, and
   [table](https://docs.cloud.google.com/biglake/docs/biglake-iceberg-tables) to import the data into.

## Support and limitations

Importing Delta Lake table data into Lakehouse using
Dataflow has the following limitations:

- You must use a batch pipeline job to use this feature.
- For existing destination tables, the schema isn't modified. The destination table schema must correctly map to the source Delta Lake table schema.
- The source data must be a valid Delta Lake table stored in Cloud Storage. The table root directory must contain the Parquet data files and the `_delta_log/` transaction log directory (containing JSON or Parquet log files) as created by Delta Lake.
- Amazon S3 isn't supported for Delta Lake table sources.

## Import a Delta Lake table

To import a Delta Lake table into Lakehouse, complete the following
steps:

1. In the Google Cloud console, go to the **Lakehouse runtime catalog** page.

   [Go to Lakehouse runtime catalog](https://console.cloud.google.com/biglake/metastore/catalogs)
2. Select the catalog, namespace, and table you want to import data into.

3. On the **Table details** page, click
   **Import table** , and then
   select **From Delta Lake (Batch)**.

   The Dataflow **Job builder** page opens with the
   **Delta Lake to Lakehouse** blueprint loaded.
4. In the **Sources** section:

   1. To expand the **ReadFromDeltaLake** **Delta Lake table** source panel,
      click the expander arrow.

   2. In the **Table path** field, enter the Cloud Storage URI of the
      root directory of the Delta Lake table (the directory containing the
      data files and the `_delta_log/` directory). For example,
      `gs://BUCKET_NAME/tables/TABLE_NAME`.

   3. Optional: In the **Hadoop configuration properties** field, configure
      any additional Hadoop configuration properties required to read from
      Cloud Storage. For example:
      `fs.gs.impl=com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystem`.

   4. Click **Done**.

5. In the **Sink** section:

   1. Optional: Review the **WriteToIceberg** **Lakehouse table** sink panel.
      The information in
      this panel, such as the Lakehouse table, catalog name, and
      warehouse location, is typically prepopulated.

   2. Click **Done**.

6. In the **Dataflow options** section, click **Run job**.

If you need to further customize the Dataflow pipeline used to
import Delta Lake tables, you can do that using the job builder form or the
YAML editor.

### Examine the job output

After the job completes, you can verify that the data was copied to the
Lakehouse table by querying it in BigQuery.

1. In the Dataflow job list, check that the job status is
   **Succeeded**.

   [Go to Jobs](https://console.cloud.google.com/dataflow)
2. If the job fails or has errors, check the
   [job logs or worker logs](https://docs.cloud.google.com/dataflow/docs/guides/logging) for details.

3. In the Google Cloud console, go to the BigQuery **Studio** page.

   [Go to BigQuery](https://console.cloud.google.com/bigquery)
4. In the query editor, enter a SQL query to inspect the table. You can use the
   `PROJECT_ID.CATALOG.NAMESPACE.TABLE_NAME`
   convention to query:

       SELECT * FROM `PROJECT_ID`.`CATALOG`.`NAMESPACE`.`TABLE_NAME` LIMIT 10;

5. Click **Run**.

6. Review the **Query results** to ensure the data was processed correctly.

## Optional. Enable continuous migration

To set up end-to-end continuous migration from Delta Lake to
Lakehouse, complete the following sequence of operations:

1. Enable the `delta.enableChangeDataFeed` property on the input Delta Lake table.
2. Perform an initial table migration to a commit ID or timestamp after setting the `delta.enableChangeDataFeed` property.
3. Run scheduled CDC jobs for subsequent data using a frequency (commit ID or timestamp range) chosen for your workload requirements.

For more information, see [Migrate Delta Lake change data capture (CDC) data
into Lakehouse using Dataflow](https://docs.cloud.google.com/dataflow/docs/guides/delta-lake-cdc-df-lakehouse-integration).

## What's next

- Learn more in the [Introduction to Lakehouse tables for Apache Iceberg in BigQuery](https://docs.cloud.google.com/biglake/docs/about-lakehouse-iceberg-rest-catalog-tables).
- Learn more about periodically [Migrating Delta Lake CDC data into Lakehouse](https://docs.cloud.google.com/dataflow/docs/guides/delta-lake-cdc-df-lakehouse-integration).
- Learn about [Change data capture ingestion in Lakehouse](https://docs.cloud.google.com/lakehouse/docs/change-data-capture).
- Learn more about how to [Create a custom job with the job builder UI](https://docs.cloud.google.com/dataflow/docs/guides/job-builder-custom-job).