Migrate metadata from Dataproc Metastore to Lakehouse

Preview

This feature is subject to the "Pre-GA Offerings Terms" in the General Service Terms section of the Service Specific Terms. Pre-GA features are available "as is" and might have limited support. For more information, see the launch stage descriptions.

This document explains how to migrate metadata from a Dataproc Metastore service to an Apache Iceberg REST catalog endpoint or a Hive catalog endpoint, built on borderless Lakehouse.

Use cases

• Serverless Modernization: Transition from a conventional Hive Metastore (HMS) to an automatically scaling, fully managed catalog, which eliminates the operational overhead of metastore management.
• Multi-Engine Collaboration: Enable data sharing across engines—including Apache Spark, Apache Flink, Apache Hive, and BigQuery—so data scientists and analysts can work on the same tables simultaneously without file duplication.
• Direct BigQuery Integration: Query open-source tables directly from BigQuery with high-performance execution.
• Unified Governance: Consolidate metadata into a single source of truth for simplified data discovery and consistent policy enforcement.
• Modern Table Formats: Seamlessly adopt advanced open formats like Apache Iceberg while maintaining full compatibility with your existing Hive workloads.

Before you begin

1. Make sure that an active Dataproc Metastore service exists as the migration source.
2. Make sure that the target Hive catalog or Iceberg catalog exists and includes the Cloud Storage buckets or paths where your source table data and metadata reside (for example, the Dataproc Metastore warehouse bucket, such as gs://gcs-your-project-name-0825d7b3-0627-4637-8fd0-cc6271d00eb4/hive-warehouse).

   If the destination catalog doesn't include the data location, table migration fails because the target catalog cannot register the tables. For Iceberg catalog creation, see Set up the Iceberg REST catalog endpoint.

   To create a Hive catalog, see Create a Lakehouse Hive catalog.
Sign in to your Google Cloud account. If you're new to Google Cloud, create an account to evaluate how our products perform in real-world scenarios. New customers also get $300 in free credits to run, test, and deploy workloads.

Verify that billing is enabled for your Google Cloud project.

Enable the Lakehouse for Apache Iceberg, Dataproc Metastore APIs.

Roles required to enable APIs

To enable APIs, you need the serviceusage.services.enable permission. If you created the project, then you likely already have this permission through the Owner role (roles/owner). Otherwise, you can get this permission through the Service Usage Admin role (roles/serviceusage.serviceUsageAdmin). Learn how to grant roles.

Enable the APIs

Verify that billing is enabled for your Google Cloud project.

Enable the Lakehouse for Apache Iceberg, Dataproc Metastore APIs.

Roles required to enable APIs

To enable APIs, you need the serviceusage.services.enable permission. If you created the project, then you likely already have this permission through the Owner role (roles/owner). Otherwise, you can get this permission through the Service Usage Admin role (roles/serviceusage.serviceUsageAdmin). Learn how to grant roles.

Enable the APIs

Required roles

To get the permissions that you need to trigger the migration, ask your administrator to grant you the following IAM roles on the Dataproc Metastore service:

• Start the migration: Dataproc Metastore Editor (roles/metastore.editor)
• Create Hive or Iceberg catalogs: BigLake Admin (roles/biglake.admin)
• Migrate metadata to destination catalogs using a target project: BigLake Admin (roles/biglake.admin) on the Dataproc Metastore service agent ([e-mail]).
• Write migration reports for the report bucket (if not using the service artifacts bucket): Storage Object Admin (roles/storage.objectAdmin) on the Dataproc Metastore service agent ([e-mail])

For more information about granting roles, see Manage access to projects, folders, and organizations.

You might also be able to get the required permissions through custom roles or other predefined roles.

How a migration works

The migration process works as follows:

1. Choose your target catalog: Select the destination Hive catalog endpoint or Apache Iceberg REST catalog endpoint for your migration.
2. Trigger migration: Run the gcloud beta metastore services migrations start command or call the startMigration method on your Dataproc Metastore service to initiate the migration.
3. Poll for status: Monitor the progress of the migration using the gcloud beta metastore services migrations describe command or by polling the target execution.
4. Review reports: Review the detailed JSON reports written to your specified Cloud Storage path to verify the results.

Run a migration

To run a migration, you trigger the migration process and then monitor its progress.

Start the migration

To trigger the metadata migration on a Dataproc Metastore service, use the gcloud CLI or the REST API.

gcloud

To start the migration using gcloud, run the gcloud beta metastore services migrations start command:

gcloud beta metastore services migrations start SERVICE_ID \
    --location=REGION \
    --hive-catalog="projects/PROJECT_ID/catalogs/HIVE_CATALOG_ID" \
    --hive-databases="HIVE_DB_1,HIVE_DB_2" \
    --iceberg-catalog="projects/PROJECT_ID/catalogs/ICEBERG_CATALOG_ID" \
    --iceberg-namespaces="ICEBERG_NAMESPACE_1,ICEBERG_NAMESPACE_2" \
    --async

Note: You can specify * for --hive-databases or --iceberg-namespaces to migrate all databases or namespaces.

Replace the following:

• SERVICE_ID: the ID of the Dataproc Metastore service
• REGION: the region of the Dataproc Metastore service
• PROJECT_ID: your Google Cloud project ID
• HIVE_CATALOG_ID: the destination Hive catalog ID
• HIVE_DB_1, HIVE_DB_2: the Hive databases to migrate.
• ICEBERG_CATALOG_ID: the destination Iceberg catalog ID
• ICEBERG_NAMESPACE_1, ICEBERG_NAMESPACE_2: the Iceberg namespaces to migrate.

REST

To trigger the metadata migration using the REST API, call the startMigration method with a BigLakeMetastoreMigrationConfig configuration:

Note: You can use ["*"] for databases or namespaces to migrate all databases or namespaces.

curl -X POST \
    -H "Authorization: Bearer $(gcloud auth print-access-token)" \
    -H "Content-Type: application/json" \
    -d '{
      "migrationExecution": {
        "biglakeMetastoreMigrationConfig": {
          "mode": "BACKFILL",
          "dryRun": false,
          "reportPath": "gs://BUCKET_NAME/PATH/",
          "conflictPolicy": "SKIP",
          "hiveConfig": {
            "catalog": "projects/PROJECT_ID/catalogs/HIVE_CATALOG_ID",
            "databases": ["HIVE_DB_1", "HIVE_DB_2"]
          },
          "icebergConfig": {
            "catalog": "projects/PROJECT_ID/catalogs/ICEBERG_CATALOG_ID",
            "namespaces": ["ICEBERG_NAMESPACE_1", "ICEBERG_NAMESPACE_2"]
          }
        }
      }
    }' \
    "https://metastore.googleapis.com/v1beta/projects/PROJECT_ID/locations/REGION/services/SERVICE_ID:startMigration"

Replace the following:

• BUCKET_NAME: the name of the Cloud Storage bucket for reports
• PATH: the path in the bucket for reports
• PROJECT_ID: your Google Cloud project ID
• HIVE_CATALOG_ID: the destination Hive catalog ID
• HIVE_DB_1, HIVE_DB_2: the Hive databases to migrate.
• ICEBERG_CATALOG_ID: the destination Iceberg catalog ID
• ICEBERG_NAMESPACE_1, ICEBERG_NAMESPACE_2: the Iceberg namespaces to migrate.
• REGION: the region of the Dataproc Metastore service
• SERVICE_ID: the ID of the Dataproc Metastore service

Poll the migration execution

The request starts a long-running operation (LRO) and returns a unique migration execution ID. You can monitor the progress of your run using the gcloud CLI or the REST API:

gcloud

To describe the migration execution using gcloud, run the gcloud beta metastore services migrations describe command:

gcloud beta metastore services migrations describe MIGRATION_EXECUTION_ID \
    --service=SERVICE_ID \
    --location=REGION

Replace the following:

• MIGRATION_EXECUTION_ID: the ID of the migration execution returned in the previous step
• SERVICE_ID: the ID of the Dataproc Metastore service
• REGION: the region of the Dataproc Metastore service

REST

To monitor the progress of your run using the REST API, call the get method on that execution path:

curl -X GET \
    -H "Authorization: Bearer $(gcloud auth print-access-token)" \
    "https://metastore.googleapis.com/v1beta/projects/PROJECT_ID/locations/REGION/services/SERVICE_ID/migrationExecutions/MIGRATION_EXECUTION_ID"

Replace the following:

• PROJECT_ID: your Google Cloud project ID
• REGION: the region of the Dataproc Metastore service
• SERVICE_ID: the ID of the Dataproc Metastore service
• MIGRATION_EXECUTION_ID: the ID of the migration execution returned in the previous step

Detailed migration report

After the migration (backfill or dry run) completes, the migration tool writes two detailed JSON report files based on the MigrationReport schema to the target Cloud Storage path specified in reportPath:

• summary.json: Contains the high-level aggregated MigrationSummary structure.
• full_report.json: Contains a detailed, more granular migration report. For more information, see CatalogReport.

Limitations

• The destination catalog must include the Cloud Storage buckets or paths where the source table data and metadata reside (such as the Dataproc Metastore warehouse bucket). If the target catalog is not configured with the data bucket location, the destination catalog cannot register the tables and table migration fails.
• The tool only supports a one-time backfill. Any metadata changes to your source Dataproc Metastore after the migration don't propagate automatically. You must rerun the migration to sync the target catalog with your source.
• The migration is bound by the limitations of the destination catalogs. If a Dataproc Metastore table contains a schema structure or property unsupported by the target catalog (such as complex types), the migration for that specific table fails.
• Dataproc Metastore permissions for tables or databases aren't migrated over to Lakehouse.

What's next

• Learn more about the Lakehouse runtime catalog.
• Learn how to Set up Spark and Hive with the Lakehouse runtime catalog.
• Learn how to Query Iceberg tables with Lakehouse runtime catalog, Spark and BigQuery.