This page shows you how to transfer data between two POSIX file systems. Common
use cases include:

- **Burst to cloud and Hybrid HPC:** Quickly transfer large data sets from on-premises to the cloud for processing.
- **Migration and sync to Filestore:** Migrate or sync data from an on-premises file system to [Filestore](https://docs.cloud.google.com/filestore).
- **Managed file transfer:** Securely and reliably transfer data between data centers or between two in-cloud file systems.

## Transfer performance guidelines

The following guidelines can help to maximize performance during file system
to file system transfers.

### Agent deployment

In general, we recommend using three agents in each of the source and
destination agent pools. Monitor the transfer and add more agents if required.
Each agent needs 4 vCPU and 8 GiB RAM.

If you are migrating to a Filestore instance, Filestore
recommends using the `n2-standard-8` instance type for each agent. Specify
`nconnect=2` when you mount the instance to a Compute Engine VM. Refer to
[Filestore's performance guidelines](https://docs.cloud.google.com/filestore/docs/performance.md.txt) for
more information on how to optimize and test instance performance.

### Transferring large numbers of small files

For better performance when transferring a large number of small files, we
recommend splitting the files into multiple directories and avoiding a single
directory with millions of files.

## Before you begin

Before you can perform the tasks described on this page, complete the
[prerequisite steps](https://docs.cloud.google.com/storage-transfer/docs/on-prem-set-up.md.txt) and configure the
required permissions:

- [Required user permissions](https://docs.cloud.google.com/storage-transfer/docs/iam-user.md.txt)
- [Agent-based project-level permissions](https://docs.cloud.google.com/storage-transfer/docs/iam-agent-based.md.txt)
- [Configure access to a source: file system](https://docs.cloud.google.com/storage-transfer/docs/source-filesystem.md.txt)
- [Configure access to a sink: file system](https://docs.cloud.google.com/storage-transfer/docs/sink-filesystem.md.txt#intermediate-bucket-permissions)

## Create agent pools and install agents

For file system to file system transfers, you need to create
[agent pools](https://docs.cloud.google.com/storage-transfer/docs/on-prem-agent-pools.md.txt)
and [agents](https://docs.cloud.google.com/storage-transfer/docs/managing-on-prem-agents.md.txt)
for both the source and destination file systems. Agents for the source agent
pool need to be installed on machines or VMs that have access to the source file
system. Agents for the destination agent pool need to be installed on machines
or VMs that have access to the destination file system.


Don't include sensitive information such as personally identifiable information
(PII) or security data in your agent ID prefix or agent pool name. Resource names may be
propagated to the names of other Google Cloud resources and may be exposed
to Google-internal systems outside of your project.

### Create a source agent pool

Create a source agent pool using one of the following methods:

### gcloud CLI

Create a source agent pool by running:

```bash
gcloud transfer agent-pools create SOURCE_AGENT_POOL
```

Replace <var translate="no">SOURCE_AGENT_POOL</var> with the name that you want to give to
the source agent pool.

### Google Cloud console

1. In the Google Cloud console, go to the **Agent pools** page.

   [Go to Agent pools](https://console.cloud.google.com/transfer/agent-pools)

   The **Agent pools** page is displayed, listing your existing agent pools.
2. Click **Create another pool**.

3. Enter a name for the pool.

4. Click **Create**.

### Install agents for the source agent pool

Install agents for the source agent pool on a machine or VM that has access to
the source file system:

### gcloud CLI

Install agents for the source agent pool by running:

```bash
gcloud transfer agents install --pool=SOURCE_AGENT_POOL --count=NUMBER_OF_AGENTS \
  --mount-directories=MOUNT_DIRECTORIES
```

Replace the following:

- <var translate="no">SOURCE_AGENT_POOL</var> with the name of the source agent pool.
- <var translate="no">NUMBER_OF_AGENTS</var> with the number of agents that you want to install for the source agent pool. To determine the optimal number of agents for your environment, see [Agent requirements and best practices](https://docs.cloud.google.com/storage-transfer/docs/on-prem-agent-best-practices.md.txt).
- <var translate="no">MOUNT_DIRECTORIES</var> with a comma-separated list of directories on the source file system from which to copy. Omitting this flag mounts the entire file system, which could [present a security risk](https://docs.cloud.google.com/storage-transfer/docs/on-prem-agent-details.md.txt#restrict-directory-access).

### Google Cloud console

1. In the Google Cloud console, go to the **Agent pools** page.

   [Go to Agent pools](https://console.cloud.google.com/transfer/agent-pools)

   The **Agent pools** page is displayed, listing your existing agent pools.
2. Click the name of the source agent pool that you just created.

3. Under the **Agents** tab, click **Install agent**.

4. Follow the instructions in Google Cloud console to install Docker and start the agent.

### Create a destination agent pool and install agents

Repeat the preceding steps to create a destination agent pool and install
agents.

## Create a Cloud Storage bucket as an intermediary

File system to file system transfers require a Cloud Storage bucket
as an intermediary for the data transfer.

1. [Create](https://docs.cloud.google.com/storage/docs/creating-buckets.md.txt) a Cloud Storage
   [Standard](https://docs.cloud.google.com/storage/docs/storage-classes.md.txt#standard) class bucket with the
   following settings:

   - **Encryption** : You can specify a [customer-managed encryption key](https://docs.cloud.google.com/kms/docs/cmek.md.txt) (CMEK). Otherwise, a Google-owned and Google-managed encryption key is used.
   - **Object Versioning** , **Bucket Lock** , and **default object holds**: Keep these features disabled.
2. Grant the required roles on the intermediate bucket to both the
   Storage Transfer Service service agent and your source and destination transfer agent
   accounts. Follow the instructions in
   [Intermediate bucket permissions](https://docs.cloud.google.com/storage-transfer/docs/sink-filesystem.md.txt#intermediate-bucket-permissions).

## Create a transfer job

### gcloud CLI


To create a transfer from the source file system to the destination file system,
run

```bash
gcloud transfer jobs create posix://SOURCE_DIRECTORY posix://DESTINATION_DIRECTORY \
    --source-agent-pool=SOURCE_AGENT_POOL \
    --destination-agent-pool=DESTINATION_AGENT_POOL \
    --intermediate-storage-path=gs://STORAGE_BUCKET/FOLDER/
```

Replace the following variables:

- <var translate="no">SOURCE_DIRECTORY</var> with the path of the source directory.
- <var translate="no">DESTINATION_DIRECTORY</var> with the path of the destination directory.
- <var translate="no">SOURCE_AGENT_POOL</var> with the name of the source agent pool.
- <var translate="no">DESTINATION_AGENT_POOL</var> with the name of the destination agent pool.
- <var translate="no">STORAGE_BUCKET</var> with the name of the Cloud Storage bucket.
- <var translate="no">FOLDER</var> with the name of the folder you would like the data to be transferred into.

<br />

When you start a transfer job, the system first computes the data in the source
and destination to determine the source data that's new or updated since the
previous transfer. Only the new data is transferred.

### Client Libraries


### Go

```go

import (
	"context"
	"fmt"
	"io"

	storagetransfer "cloud.google.com/go/storagetransfer/apiv1"
	"cloud.google.com/go/storagetransfer/apiv1/storagetransferpb"
)

func transferFromPosix(w io.Writer, projectID string, sourceAgentPoolName string, rootDirectory string, gcsSinkBucket string) (*storagetransferpb.TransferJob, error) {
	// Your project id
	// projectId := "myproject-id"

	// The agent pool associated with the POSIX data source. If not provided, defaults to the default agent
	// sourceAgentPoolName := "projects/my-project/agentPools/transfer_service_default"

	// The root directory path on the source filesystem
	// rootDirectory := "/directory/to/transfer/source"

	// The ID of the GCS bucket to transfer data to
	// gcsSinkBucket := "my-sink-bucket"

	ctx := context.Background()
	client, err := storagetransfer.NewClient(ctx)
	if err != nil {
		return nil, fmt.Errorf("storagetransfer.NewClient: %w", err)
	}
	defer client.Close()

	req := &storagetransferpb.CreateTransferJobRequest{
		TransferJob: &storagetransferpb.TransferJob{
			ProjectId: projectID,
			TransferSpec: &storagetransferpb.TransferSpec{
				SourceAgentPoolName: sourceAgentPoolName,
				DataSource: &storagetransferpb.TransferSpec_PosixDataSource{
					PosixDataSource: &storagetransferpb.PosixFilesystem{RootDirectory: rootDirectory},
				},
				DataSink: &storagetransferpb.TransferSpec_GcsDataSink{
					GcsDataSink: &storagetransferpb.GcsData{BucketName: gcsSinkBucket},
				},
			},
			Status: storagetransferpb.TransferJob_ENABLED,
		},
	}

	resp, err := client.CreateTransferJob(ctx, req)
	if err != nil {
		return nil, fmt.Errorf("failed to create transfer job: %w", err)
	}
	if _, err = client.RunTransferJob(ctx, &storagetransferpb.RunTransferJobRequest{
		ProjectId: projectID,
		JobName:   resp.Name,
	}); err != nil {
		return nil, fmt.Errorf("failed to run transfer job: %w", err)
	}
	fmt.Fprintf(w, "Created and ran transfer job from %v to %v with name %v", rootDirectory, gcsSinkBucket, resp.Name)
	return resp, nil
}
```

### Java

```java
import com.google.storagetransfer.v1.proto.StorageTransferServiceClient;
import com.google.storagetransfer.v1.proto.TransferProto;
import com.google.storagetransfer.v1.proto.TransferTypes.GcsData;
import com.google.storagetransfer.v1.proto.TransferTypes.PosixFilesystem;
import com.google.storagetransfer.v1.proto.TransferTypes.TransferJob;
import com.google.storagetransfer.v1.proto.TransferTypes.TransferSpec;
import java.io.IOException;

public class TransferBetweenPosix {

  public static void main(String[] args) throws IOException {
    // TODO(developer): Replace these variables before running the sample.

    // Your project id
    String projectId = "my-project-id";

    // The agent pool associated with the POSIX data source. If not provided, defaults to the
    // default agent
    String sourceAgentPoolName = "projects/my-project-id/agentPools/transfer_service_default";

    // The agent pool associated with the POSIX data sink. If not provided, defaults to the default
    // agent
    String sinkAgentPoolName = "projects/my-project-id/agentPools/transfer_service_default";

    // The root directory path on the source filesystem
    String rootDirectory = "/directory/to/transfer/source";

    // The root directory path on the sink filesystem
    String destinationDirectory = "/directory/to/transfer/sink";

    // The ID of the GCS bucket for intermediate storage
    String bucketName = "my-intermediate-bucket";

    transferBetweenPosix(
        projectId,
        sourceAgentPoolName,
        sinkAgentPoolName,
        rootDirectory,
        destinationDirectory,
        bucketName);
  }

  public static void transferBetweenPosix(
      String projectId,
      String sourceAgentPoolName,
      String sinkAgentPoolName,
      String rootDirectory,
      String destinationDirectory,
      String bucketName)
      throws IOException {

    TransferJob transferJob =
        TransferJob.newBuilder()
            .setProjectId(projectId)
            .setTransferSpec(
                TransferSpec.newBuilder()
                    .setSinkAgentPoolName(sinkAgentPoolName)
                    .setSourceAgentPoolName(sourceAgentPoolName)
                    .setPosixDataSource(
                        PosixFilesystem.newBuilder().setRootDirectory(rootDirectory).build())
                    .setPosixDataSink(
                        PosixFilesystem.newBuilder().setRootDirectory(destinationDirectory).build())
                    .setGcsIntermediateDataLocation(
                        GcsData.newBuilder().setBucketName(bucketName).build())
                    .build())
            .setStatus(TransferJob.Status.ENABLED)
            .build();

    // Initialize client that will be used to send requests. This client only needs to be created
    // once, and can be reused for multiple requests. After completing all of your requests, call
    // the "close" method on the client to safely clean up any remaining background resources,
    // or use "try-with-close" statement to do this automatically.
    try (StorageTransferServiceClient storageTransfer = StorageTransferServiceClient.create()) {

      // Create the transfer job
      TransferJob response =
          storageTransfer.createTransferJob(
              TransferProto.CreateTransferJobRequest.newBuilder()
                  .setTransferJob(transferJob)
                  .build());

      System.out.println(
          "Created and ran a transfer job from "
              + rootDirectory
              + " to "
              + destinationDirectory
              + " with name "
              + response.getName());
    }
  }
}
```

### Node.js

```javascript

// Imports the Google Cloud client library
const {
  StorageTransferServiceClient,
} = require('@google-cloud/storage-transfer');

/**
 * TODO(developer): Uncomment the following lines before running the sample.
 */
// Your project id
// const projectId = 'my-project'

// The agent pool associated with the POSIX data source. Defaults to the default agent
// const sourceAgentPoolName = 'projects/my-project/agentPools/transfer_service_default'

// The agent pool associated with the POSIX data sink. Defaults to the default agent
// const sinkAgentPoolName = 'projects/my-project/agentPools/transfer_service_default'

// The root directory path on the source filesystem
// const rootDirectory = '/directory/to/transfer/source'

// The root directory path on the sink filesystem
// const destinationDirectory = '/directory/to/transfer/sink'

// The ID of the GCS bucket for intermediate storage
// const bucketName = 'my-intermediate-bucket'

// Creates a client
const client = new StorageTransferServiceClient();

/**
 * Creates a request to transfer from the local file system to the sink bucket
 */
async function transferDirectory() {
  const createRequest = {
    transferJob: {
      projectId,
      transferSpec: {
        sourceAgentPoolName,
        sinkAgentPoolName,
        posixDataSource: {
          rootDirectory,
        },
        posixDataSink: {
          rootDirectory: destinationDirectory,
        },
        gcsIntermediateDataLocation: {
          bucketName,
        },
      },
      status: 'ENABLED',
    },
  };

  // Runs the request and creates the job
  const [transferJob] = await client.createTransferJob(createRequest);

  const runRequest = {
    jobName: transferJob.name,
    projectId: projectId,
  };

  await client.runTransferJob(runRequest);

  console.log(
    `Created and ran a transfer job from '${rootDirectory}' to '${destinationDirectory}' with name ${transferJob.name}`
  );
}

transferDirectory();
```

### Python

```python
from google.cloud import storage_transfer


def transfer_between_posix(
    project_id: str,
    description: str,
    source_agent_pool_name: str,
    sink_agent_pool_name: str,
    root_directory: str,
    destination_directory: str,
    intermediate_bucket: str,
):
    """Creates a transfer between POSIX file systems."""

    client = storage_transfer.StorageTransferServiceClient()

    # The ID of the Google Cloud Platform Project that owns the job
    # project_id = 'my-project-id'

    # A useful description for your transfer job
    # description = 'My transfer job'

    # The agent pool associated with the POSIX data source.
    # Defaults to 'projects/{project_id}/agentPools/transfer_service_default'
    # source_agent_pool_name = 'projects/my-project/agentPools/my-agent'

    # The agent pool associated with the POSIX data sink.
    # Defaults to 'projects/{project_id}/agentPools/transfer_service_default'
    # sink_agent_pool_name = 'projects/my-project/agentPools/my-agent'

    # The root directory path on the source filesystem
    # root_directory = '/directory/to/transfer/source'

    # The root directory path on the destination filesystem
    # destination_directory = '/directory/to/transfer/sink'

    # The Google Cloud Storage bucket for intermediate storage
    # intermediate_bucket = 'my-intermediate-bucket'

    transfer_job_request = storage_transfer.CreateTransferJobRequest(
        {
            "transfer_job": {
                "project_id": project_id,
                "description": description,
                "status": storage_transfer.TransferJob.Status.ENABLED,
                "transfer_spec": {
                    "source_agent_pool_name": source_agent_pool_name,
                    "sink_agent_pool_name": sink_agent_pool_name,
                    "posix_data_source": {
                        "root_directory": root_directory,
                    },
                    "posix_data_sink": {
                        "root_directory": destination_directory,
                    },
                    "gcs_intermediate_data_location": {
                        "bucket_name": intermediate_bucket
                    },
                },
            }
        }
    )

    result = client.create_transfer_job(transfer_job_request)
    print(f"Created transferJob: {result.name}")

```

<br />

## Manage intermediary buckets

After a transfer job completes, Storage Transfer Service saves
[transfer logs](https://docs.cloud.google.com/storage-transfer/docs/on-prem-transfer-log-format.md.txt#viewing-logs)
listing transferred and failed-to-transfer data in the bucket. After the
transfer, clean-up tasks initiate automatically to delete the intermediate data.
In some instances, clean-up tasks fail to delete all of the data in the bucket.
To delete data that isn't erased during clean-up, use the instructions below to
either delete the data manually, or to set a lifecycle rule to delete the data
automatically.

### Manual clean-up

Delete data from the intermediary bucket by running the following
commands based on the type of data you want to delete.

- To erase data in the intermediary bucket that wasn't deleted during
  clean-up, run the following command:

  ```bash
  gcloud storage rm gs://STORAGE_BUCKET/PREFIX**
  ```
- To delete all data, including transfer logs, specify the root of the
  bucket using the matches-all (\*) wildcard.

  ```bash
  gcloud storage rm gs://STORAGE_BUCKET/*
  ```
- To delete the bucket, run the following command:

  ```bash
  gcloud storage rm gs://STORAGE_BUCKET
  ```

Replace the following variables:

- <var translate="no">STORAGE_BUCKET</var> with the name of the intermediary bucket.

- <var translate="no">PREFIX</var> with the name of the folder the data was transferred
  into within the intermediary bucket.

### Set a lifecycle rule

To delete data that isn't erased by the automatic clean-up cycle, set a
lifecycle rule for the Cloud Storage bucket. Use the
[`age` condition](https://docs.cloud.google.com/storage/docs/lifecycle.md.txt#age) to erase intermediate data in
the bucket by specifying a period longer than the longest transfer job that
uses the bucket as an intermediary. If the specified age
condition is shorter than the time required to download the file from the
intermediary bucket to the destination, the file transfer fails.

Optionally, use the [`matchesPrefix` condition](https://docs.cloud.google.com/storage/docs/lifecycle.md.txt#matchesprefix-suffix)
to erase data in the folder you specified for the intermediate bucket. To
delete transfer logs along with data in the bucket, the `matchesPrefix`
condition is not required.

## Preserving file metadata

To preserve file metadata, including numeric UID, GID, MODE, and symbolic links:

### gcloud CLI

Use the `--preserve-metadata` field to specify the preservation behavior for
this transfer. Options that apply to file system transfers are: `gid`,
`mode`, `symlink`, `uid`.

### REST API

Specify the appropriate options in a
[`metadataOptions`](https://docs.cloud.google.com/storage-transfer/docs/reference/rest/v1/TransferSpec.md.txt#MetadataOptions)
object.

See
[Preserving optional POSIX attributes](https://docs.cloud.google.com/storage-transfer/docs/metadata-preservation.md.txt#posix-metadata)
for more information.

## Example transfer using the gcloud CLI

In this example, we transfer data from the `/tmp/datasource` directory on VM1 to
`/tmp/destination` on VM2.

1. Set up the source of the transfer.

   1. Create the source agent pool:

      ```bash
      gcloud transfer agent-pools create source_agent_pool
      ```
   2. On VM1, install agents for `source_agent_pool` by running:

      ```bash
      gcloud transfer agents install --pool=source_agent_pool \
          --count=1 \
          --mount-directories="/tmp/datasource"
      ```
2. Set up the destination of the transfer.

   1. Create the destination agent pool:

      ```bash
      gcloud transfer agent-pools create destination_agent_pool
      ```
   2. On VM2, install agents for `destination_agent_pool` by running:

      ```bash
      gcloud transfer agents install --pool=destination_agent_pool \
          --count=3 \
          --mount-directories="/tmp/destination"
      ```
3. Create an intermediary Cloud Storage bucket.

   1. Create a bucket named `my-intermediary-bucket`:

          gcloud storage buckets create gs://my-intermediary-bucket

   2. Grant the required roles on `gs://my-intermediary-bucket` to the
      service agent and transfer agent accounts as described in
      [Intermediate bucket permissions](https://docs.cloud.google.com/storage-transfer/docs/sink-filesystem.md.txt#intermediate-bucket-permissions).

4. Create a transfer job by running:

   ```bash
   gcloud transfer jobs create posix:///tmp/datasource posix:///tmp/destination \
       --source-agent-pool=source_agent_pool \
       --destination-agent-pool=destination_agent_pool \
       --intermediate-storage-path=gs://my-intermediary-bucket
   ```

## What's next

- Learn more about [managing jobs](https://docs.cloud.google.com/storage-transfer/docs/manage-transfers.md.txt).