A common task you might complete is to move a Dataflow job from one Google Cloud project to another. For example, you might want to move jobs from a testing environment to a production environment. Dataflow jobs are bound to the project they were created in and can't be directly migrated to a different project. To move a job, you must stop the job in the old project and recreate it in the new project.
To move a job, you need to do the following:
- Stop or drain the job in the old project.
- Recreate the job in the new project. For example, use the same pipeline code, template, and configuration.
- Ensure that all necessary permissions, service accounts, and resources are
correctly set up in the new project for the job to access. This includes:
- Permissions: Ensure the user or service account running the pipeline
has the necessary Identity and Access Management (IAM) roles in the new project,
such as
roles/dataflow.adminandroles/dataflow.worker. - Service Accounts: The Dataflow worker service account will be different in the new project. Ensure this new service account has access to all the resources the pipeline needs, such as Cloud Storage buckets, BigQuery datasets, and Pub/Sub topics.
- Resources: If your pipeline uses resources like Cloud Storage buckets, BigQuery datasets, or Pub/Sub topics, you need to ensure they exist in the new project or are accessible from the new project. You might need to create new resources or grant cross-project permissions.
- VPC Service Controls: If you are using VPC Service Controls, ensure that the new project is part of the same service perimeter or that you have configured the necessary bridges to allow access to the required resources.
- Permissions: Ensure the user or service account running the pipeline
has the necessary Identity and Access Management (IAM) roles in the new project,
such as
When you recreate the job, it starts as a new job and does not preserve the state of the old job.
To avoid hardcoding project-specific parameters, such as the project ID, use pipeline options to pass these values into your pipeline.
For more information about the permissions and security of your Dataflow jobs, see Dataflow security and permissions.
What's next
- For best practices for developing, deploying, and managing your Dataflow pipelines, see Pipeline best practices.
- To review the process of deploying your pipeline after configuring your new project, see Deploy pipelines.