REST Resource: projects.locations.services.migrationExecutions

Resource: MigrationExecution

The details of a migration execution resource.

JSON representation 
{
  "name": string,
  "createTime": string,
  "endTime": string,
  "state": enum (State),
  "phase": enum (Phase),
  "stateMessage": string,

  // Union field migration_config can be only one of the following:
  "cloudSqlMigrationConfig": {
    object (CloudSQLMigrationConfig)
  },
  "biglakeMetastoreMigrationConfig": {
    object (BigLakeMetastoreMigrationConfig)
  }
  // End of list of possible types for union field migration_config.
}
Fields
name

string

Output only. The relative resource name of the migration execution, in the following form: projects/{projectNumber}/locations/{locationId}/services/{serviceId}/migrationExecutions/{migration_execution_id}
createTime

string (Timestamp format)

Output only. The time when the migration execution was started.

Uses RFC 3339, where generated output will always be Z-normalized and use 0, 3, 6 or 9 fractional digits. Offsets other than "Z" are also accepted. Examples: "2014-10-02T15:01:23Z", "2014-10-02T15:01:23.045123456Z" or "2014-10-02T15:01:23+05:30".
endTime

string (Timestamp format)

Output only. The time when the migration execution finished.

Uses RFC 3339, where generated output will always be Z-normalized and use 0, 3, 6 or 9 fractional digits. Offsets other than "Z" are also accepted. Examples: "2014-10-02T15:01:23Z", "2014-10-02T15:01:23.045123456Z" or "2014-10-02T15:01:23+05:30".
state

enum (State)

Output only. The current state of the migration execution.
phase
(deprecated)

enum (Phase)

Output only. Deprecated: Phase was designed for incoming migrations to Dataproc Metastore, not applicable when migrating away from it. The current phase of the migration execution.
stateMessage

string

Output only. Additional information about the current state of the migration execution.
Union field migration_config. Configuration information required by Dataproc Metastore for starting the migration process migration_config can be only one of the following:
cloudSqlMigrationConfig
(deprecated)

object (CloudSQLMigrationConfig)

Deprecated: Migrations to Dataproc Metastore are no longer supported. Use BigLake Metastore migration instead. Configuration information specific to migrating from self-managed hive metastore on Google Cloud using Cloud SQL as the backend database to Dataproc Metastore.
biglakeMetastoreMigrationConfig

object (BigLakeMetastoreMigrationConfig)

Configuration information specific to migrating from Dataproc Metastore to BigLake Metastore.

CloudSQLMigrationConfig

Deprecated: Migrations to Dataproc Metastore are no longer supported. Use BigLake Metastore migration instead. Configuration information for migrating from self-managed hive metastore on Google Cloud using Cloud SQL as the backend database to Dataproc Metastore.

JSON representation 
{
  "cdcConfig": {
    object (CdcConfig)
  },
  "cloudSqlConnectionConfig": {
    object (CloudSQLConnectionConfig)
  }
}
Fields
cdcConfig

object (CdcConfig)

Required. Configuration information to start the Change Data Capture (CDC) streams from customer database to backend database of Dataproc Metastore. Dataproc Metastore switches to using its backend database after the cutover phase of migration.
cloudSqlConnectionConfig

object (CloudSQLConnectionConfig)

Required. Configuration information to establish customer database connection before the cutover phase of migration

CdcConfig

Configuration information to start the Change Data Capture (CDC) streams from customer database to backend database of Dataproc Metastore.

JSON representation 
{
  "vpcNetwork": string,
  "subnetIpRange": string,
  "username": string,
  "password": string,
  "reverseProxySubnet": string,
  "bucket": string,
  "rootPath": string
}
Fields
vpcNetwork

string

Required. Fully qualified name of the Cloud SQL instance's VPC network or the shared VPC network that Datastream will peer to, in the following format: projects/{projectId}/locations/global/networks/{network_id}. More context in https://cloud.google.com/datastream/docs/network-connectivity-options#privateconnectivity
subnetIpRange

string

Required. A /29 CIDR IP range for peering with datastream.
username

string

Required. The username that the Datastream service should use for the MySQL connection.
password

string

Required. Input only. The password for the user that Datastream service should use for the MySQL connection. This field is not returned on request.
reverseProxySubnet

string

Required. The URL of the subnetwork resource to create the VM instance hosting the reverse proxy in. More context in https://cloud.google.com/datastream/docs/private-connectivity#reverse-csql-proxy The subnetwork should reside in the network provided in the request that Datastream will peer to and should be in the same region as Datastream, in the following format. projects/{projectId}/regions/{region_id}/subnetworks/{subnetwork_id}
bucket

string

Optional. The bucket to write the intermediate stream event data in. The bucket name must be without any prefix like "gs://". See the bucket naming requirements. This field is optional. If not set, the Artifacts Cloud Storage bucket will be used.
rootPath

string

Optional. The root path inside the Cloud Storage bucket. The stream event data will be written to this path. The default value is /migration.

CloudSQLConnectionConfig

Configuration information to establish customer database connection before the cutover phase of migration

JSON representation 
{
  "instanceConnectionName": string,
  "ipAddress": string,
  "port": integer,
  "hiveDatabaseName": string,
  "username": string,
  "password": string,
  "proxySubnet": string,
  "natSubnet": string
}
Fields
instanceConnectionName

string

Required. Cloud SQL database connection name (projectId:region:instance_name)
ipAddress

string

Required. The private IP address of the Cloud SQL instance.
port

integer

Required. The network port of the database.
hiveDatabaseName

string

Required. The hive database name.
username

string

Required. The username that Dataproc Metastore service will use to connect to the database.
password

string

Required. Input only. The password for the user that Dataproc Metastore service will be using to connect to the database. This field is not returned on request.
proxySubnet

string

Required. The relative resource name of the subnetwork to deploy the SOCKS5 proxy service in. The subnetwork should reside in a network through which the Cloud SQL instance is accessible. The resource name should be in the format, projects/{projectId}/regions/{region_id}/subnetworks/{subnetwork_id}
natSubnet

string

Required. The relative resource name of the subnetwork to be used for Private Service Connect. Note that this cannot be a regular subnet and is used only for NAT. (https://cloud.google.com/vpc/docs/about-vpc-hosted-services#psc-subnets) This subnet is used to publish the SOCKS5 proxy service. The subnet size must be at least /29 and it should reside in a network through which the Cloud SQL instance is accessible. The resource name should be in the format, projects/{projectId}/regions/{region_id}/subnetworks/{subnetwork_id}

BigLakeMetastoreMigrationConfig

Defines the configuration required to migrate metadata from a Dataproc Metastore service to BigLake Metastore.

JSON representation 
{
  "mode": enum (MigrationMode),
  "dryRun": boolean,
  "reportPath": string,
  "hiveConfig": {
    object (HiveConfig)
  },
  "icebergConfig": {
    object (IcebergConfig)
  },
  "backfillStatus": {
    object (BackfillStatus)
  },
  "conflictPolicy": enum (ConflictPolicy)
}
Fields
mode

enum (MigrationMode)

Required. Defines the behavior of the migration execution.
dryRun

boolean

Optional. If true, performs discovery of requested resources and analysis against the target catalog to come up with a plan for each resource (e.g. Create, Update, Skip, etc.). No metadata is actually migrated.
reportPath

string

Optional. The Cloud Storage path where the backfill / dry run report should be written. If not provided, the report will be generated in the service's artifacts bucket. Format: "gs://path/to/folder"
hiveConfig

object (HiveConfig)

Optional. At least one of hiveConfig or icebergConfig must be provided, otherwise, a validation error will be thrown. If only one is provided, the service only migrates tables of that specific type. If both are provided, both Hive and Iceberg tables will be migrated.

Configuration for migrating Hive tables to a BigLake Hive catalog.
icebergConfig

object (IcebergConfig)

Optional. Configuration for migrating Iceberg tables to a BigLake Iceberg REST catalog.
backfillStatus

object (BackfillStatus)

Output only.

conflictPolicy

enum (ConflictPolicy)

Optional. The policy to handle conflicts when migrating resources, defaults to SKIP if not specified.

MigrationMode

The execution mode of the migration.

Enums
MIGRATION_MODE_UNSPECIFIED The migration mode is unspecified.
BACKFILL Performs the metadata migration of requested resources. The migration completes once the backfill is finished.

HiveConfig

Configuration for migrating Hive metadata.

JSON representation 
{
  "catalog": string,
  "databases": [
    string
  ]
}
Fields
catalog

string

Required. The target catalog for migrated databases and tables. Format: "projects/{project_id_or_number}/catalogs/{catalog_id}"
databases[]

string

Required. The list of databases to migrate to the Hive catalog. Use "*" to migrate all databases. Note: If Iceberg tables exist in these databases, they will only be migrated if icebergConfig is also specified.

IcebergConfig

Configuration for migrating Iceberg metadata.

JSON representation 
{
  "catalog": string,
  "namespaces": [
    string
  ]
}
Fields
catalog

string

Required. The target catalog for migrated Iceberg metadata. Format: "projects/{project_id_or_number}/catalogs/{catalog_id}"
namespaces[]

string

Required. The list of namespaces to migrate to the Iceberg REST catalog. Use "*" to migrate all namespaces. Note: If Hive tables exist in these namespaces, they will only be migrated if hiveConfig is also specified.

BackfillStatus

Backfill status for the migration execution.

JSON representation 
{
  "state": enum (State),
  "reportPath": string,
  "migrationSummary": {
    object (MigrationSummary)
  }
}
Fields
state

enum (State)

Output only. The current state of the backfill (or dry run).
reportPath

string

Output only. The Cloud Storage path where the backfill or dry run report is written. Format: "gs://path-to-report".
migrationSummary

object (MigrationSummary)

Output only. Summary of the migration results. This is populated after the backfill or dry run is finished.

State

The backfill state.

Enums
STATE_UNSPECIFIED The backfill state is unspecified.
PENDING Waiting to start.
RUNNING Backfill in progress.
SUCCEEDED Backfill complete, report is available
FAILED Backfill failed; check report for details

MigrationSummary

Summary of the migration results.

JSON representation 
{
  "createTime": string,
  "service": string,
  "dryRun": boolean,
  "catalogSummaries": [
    {
      object (CatalogSummary)
    }
  ]
}
Fields
createTime

string (Timestamp format)

Output only. The UTC time when this report was finalized.

Uses RFC 3339, where generated output will always be Z-normalized and use 0, 3, 6 or 9 fractional digits. Offsets other than "Z" are also accepted. Examples: "2014-10-02T15:01:23Z", "2014-10-02T15:01:23.045123456Z" or "2014-10-02T15:01:23+05:30".
service

string

Output only. The Dataproc Metastore service name (format: projects/*/locations/*/services/*) on which the migration was executed.
dryRun

boolean

Output only. Whether the migration was a dry run.
catalogSummaries[]

object (CatalogSummary)

Output only. Summary of results for each catalog involved in the migration.

CatalogSummary

Summary of results for a specific destination catalog.

JSON representation 
{
  "catalog": string,
  "catalogType": enum (CatalogType),
  "databaseSummaries": [
    {
      object (DatabaseSummary)
    }
  ]
}
Fields
catalog

string

Output only. The catalog resource name (format: projects/*/catalogs/*).
catalogType

enum (CatalogType)

Output only. The type of the catalog.
databaseSummaries[]

object (DatabaseSummary)

Output only. Summary of results for each database in the catalog.

CatalogType

The type of catalog.

Enums
CATALOG_TYPE_UNSPECIFIED The catalog type is unspecified.
HIVE BigLake Metastore Hive catalog.
ICEBERG BigLake Metastore Iceberg REST catalog.

DatabaseSummary

Summary of results for a specific database in a catalog.

JSON representation 
{
  "database": string,
  "planAction": enum (Action),
  "resultStatus": enum (State),
  "tableSummary": {
    object (TableSummary)
  }
}
Fields
database

string

Output only. The name of the database.
planAction

enum (Action)

Output only. The migration plan action for the database.
resultStatus

enum (State)

Output only. The migration result status for the database. This is only set if the migration is not a dry run.
tableSummary

object (TableSummary)

Output only. Aggregated summary of results for all tables in the database.

Action

The action that will be taken for a resource during migration.

Enums
ACTION_UNSPECIFIED The action is unspecified.
CREATE Resource missing; will be created.
UPDATE Resource exists at the target, but differs from the source; will be updated.
SKIP Resource exists at the target; no changes will be made.
DEPENDENCY_FAILURE Resource cannot be migrated due to a dependency failure (e.g., parent resource missing).
ERROR Resource cannot be migrated due to an error during discovery.

State

The state of the migration for a resource.

Enums
STATE_UNSPECIFIED The state is unspecified.
SUCCEEDED The resource was migrated successfully.
FAILED The resource failed to migrate.
SKIPPED The resource was skipped and will not be migrated.

TableSummary

Aggregated summary of results for all tables in a database.

JSON representation 
{
  "planCounts": {
    string: string,
    ...
  },
  "resultCounts": {
    string: string,
    ...
  },
  "partitionDiscoveredCount": string,
  "partitionSuccessCount": string,
  "partitionFailedCount": string
}
Fields
planCounts

map (key: string, value: string (int64 format))

Output only. Number of tables with a specific migration plan action. The key is the action name (e.g. CREATE, UPDATE, SKIP, etc.).

An object containing a list of "key": value pairs. Example: { "name": "wrench", "mass": "1.3kg", "count": "3" }.
resultCounts

map (key: string, value: string (int64 format))

Output only. Number of tables with a specific migration result status. The key is the status name (e.g. SUCCEEDED, FAILED, SKIPPED, etc.). This is only set if the migration is not a dry run.

An object containing a list of "key": value pairs. Example: { "name": "wrench", "mass": "1.3kg", "count": "3" }.
partitionDiscoveredCount

string (int64 format)

Output only. Partition migration summary across all Hive tables in the database.

The total number of partitions discovered at the source.
partitionSuccessCount

string (int64 format)

Output only. The total number of partitions successfully migrated at the target.
partitionFailedCount

string (int64 format)

Output only. The total number of partitions that failed to migrate at the target.

ConflictPolicy

The policy to handle conflicts when migrating resources to Lakehouse.

Enums
CONFLICT_POLICY_UNSPECIFIED The conflict policy is unspecified.
SKIP Skip migrating resources that already exist in the target catalog.
OVERWRITE Update resources that already exist in the target catalog.

State

The current state of the migration execution.

Enums
STATE_UNSPECIFIED The state of the migration execution is unknown.
STARTING The migration execution is starting.
RUNNING The migration execution is running.
CANCELLING The migration execution is in the process of being cancelled.
AWAITING_USER_ACTION The migration execution is awaiting user action.
SUCCEEDED The migration execution has completed successfully.
FAILED The migration execution has failed.
CANCELLED The migration execution is cancelled.
DELETING The migration execution is being deleted.
ROLLED_BACK The migration execution has been rolled back. This occurs when CancelMigration is invoked after the migration has completed and the metastore service is in the PROXY state, returning the service to the ACTIVE state. This enables rollback support when the customer wants to resume using DPMS after a successful migration.

Phase

Deprecated: Phase was designed for incoming migrations to Dataproc Metastore, not applicable when migrating away from it. The current phase of the migration execution.

Enums
PHASE_UNSPECIFIED The phase of the migration execution is unknown.
REPLICATION Replication phase refers to the migration phase when Dataproc Metastore is running a pipeline to replicate changes in the customer database to its backend database. During this phase, Dataproc Metastore uses the customer database as the hive metastore backend database.
CUTOVER Cutover phase refers to the migration phase when Dataproc Metastore switches to using its own backend database. Migration enters this phase when customer is done migrating all their clusters/workloads to Dataproc Metastore and triggers CompleteMigration.

Methods

delete

 Deletes a single migration execution.

get

 Gets details of a single migration execution.

list

 Lists migration executions on a service.