Data transfers can occur between the following:

1. Persistent Volume Claim (PVC) and object storage
2. Object storage and object storage (within GDC)

Object storage on GDC is S3-compatible and referred to as `s3` type in Kubernetes yamls.

### Types of data sources and destinations

1. Object storage (referred to as 's3'): Object storage present on GDC
2. Local storage (referred to as 'local'): Storage on attached PVCs

## Copying from object storage to object storage

Ensure you have the following prerequisites:

- An S3 Endpoint with read permissions for the source, and an s3 endpoint with write permissions for the destination.
- If you don't have bucket creation permission with the credentials, the transfer fails if the destination bucket doesn't exist. Ensure the destination bucket exists if that is the case.
- Privileges to create Jobs and create or read Secrets inside your cluster or namespace. See the following example for permissions.

### Set up the data transfer

To set up the data transfer, complete the following steps:

### Create the transfer namespace

Create the namespace where the transfer job will run:

```yaml
apiVersion: v1
kind: Namespace
metadata:
  name: NAMESPACE
```

### Configure credentials

Prepare your credentials for the transfer. Follow the option that best matches your transfer scenario.

#### GDC to GDC transfer

For a transfer occurring completely within the **same GDC universe**, the secrets already exist. Note that this method is strictly for transfers within a single universe, not for transfers between different universes.

1. To obtain the S3 credential Secrets, follow the instructions to [grant bucket access](https://docs.cloud.google.com/distributed-cloud/hosted/docs/latest/gdcag/platform/pa-user/grant-obtain-storage-access.md.txt#grant_bucket_access).

2. Create a service account (SA) in the destination namespace for your transfer job. Then, add permissions to allow this account to read Secrets in both the source and destination namespaces using cross-namespace RoleBindings.

   > [!NOTE]
   > **Note:** This step is only required if the source and destination buckets are in different namespaces. For transfers within the same GDC project, this step can be skipped.

   ```yaml
   ---
   apiVersion: v1
   kind: ServiceAccount
   metadata:
     name: transfer-service-account
     namespace: DESTINATION_NAMESPACE
   ---
   apiVersion: rbac.authorization.k8s.io/v1
   kind: Role
   metadata:
     name: read-secrets-role
     namespace: SOURCE_NAMESPACE
   rules:
   - apiGroups: [""]
     resources: ["secrets"]
     verbs: ["get", "watch", "list"]
   ---
   apiVersion: rbac.authorization.k8s.io/v1
   kind: RoleBinding
   metadata:
     name: read-source-secrets-rolebinding
     namespace: SOURCE_NAMESPACE
   subjects:
   - kind: ServiceAccount
     name: transfer-service-account
     namespace: DESTINATION_NAMESPACE
   roleRef:
     kind: Role
     name: read-secrets-role
     apiGroup: rbac.authorization.k8s.io
   ---
   apiVersion: rbac.authorization.k8s.io/v1
   kind: Role
   metadata:
     name: read-secrets-role
     namespace: DESTINATION_NAMESPACE
   rules:
   - apiGroups: [""]
     resources: ["secrets"]
     verbs: ["get", "watch", "list"]
   ---
   apiVersion: rbac.authorization.k8s.io/v1
   kind: RoleBinding
   metadata:
     name: read-dest-secrets-rolebinding
     namespace: DESTINATION_NAMESPACE
   subjects:
   - kind: ServiceAccount
     name: transfer-service-account
     namespace: DESTINATION_NAMESPACE
   roleRef:
     kind: Role
     name: read-secrets-role
     apiGroup: rbac.authorization.k8s.io
   ---
   ```

#### GDC to non-GDC transfer

For a transfer involving an external system, you must specify the access credentials explicitly.

1. Create credentials in the destination namespace:

   ```yaml
   ---
   apiVersion: v1
   kind: Secret
   metadata:
     name: src-secret
     namespace: NAMESPACE
   data:
     access-key-id: NkFDTUg3WDBCVDlQMVpZMU5MWjU= # base 64 encoded version of key
     access-key: VkRkeWJsbFgzb2FZanMvOVpnSi83SU5YUjk3Y0Q2TUdxZ2d4Q3dpdw== # base 64 encoded version of secret key
   ---
   apiVersion: v1
   kind: Secret
   metadata:
     name: dst-secret
     namespace: NAMESPACE
   data:
     access-key-id: NkFDTUg3WDBCVDlQMVpZMU5MWjU= # base 64 encoded version of key
     access-key: VkRkeWJsbFgzb2FZanMvOVpnSi83SU5YUjk3Y0Q2TUdxZ2d4Q3dpdw== # base 64 encoded version of secret key
   ---
   ```
2. Create a service account (SA) that is used by your transfer, and then add permissions to the account to read and write secrets using roles and role bindings. You don't need to add permissions if your default namespace SA or custom SA already has these permissions.

   ```yaml
   ---
   apiVersion: v1
   kind: ServiceAccount
   metadata:
     name: transfer-service-account
     namespace: NAMESPACE
   ---
   apiVersion: rbac.authorization.k8s.io/v1
   kind: Role
   metadata:
     name: read-secrets-role
     namespace: NAMESPACE
   rules:
   - apiGroups: [""]
     resources: ["secrets"]
     verbs: ["get", "watch", "list"]
   ---
   apiVersion: rbac.authorization.k8s.io/v1
   kind: RoleBinding
   metadata:
     name: read-secrets-rolebinding
     namespace: NAMESPACE
   subjects:
   - kind: ServiceAccount
     name: transfer-service-account
     namespace: NAMESPACE
   roleRef:
     kind: Role
     name: read-secrets-role
     apiGroup: rbac.authorization.k8s.io
   ---
   ```

### Obtain CA certificates

Obtain the CA certificates for your object storage systems. You can obtain the same certificates from your AO or PA by following the instructions to [fetch trust bundles](https://docs.cloud.google.com/distributed-cloud/hosted/docs/latest/gdcag/platform/pa-user/pki/fetch-trust-bundles.md.txt).

```yaml
---

apiVersion: v1
kind: Secret
metadata:
  name: src-cert
  namespace: NAMESPACE
data:
  ca.crt: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURBekNDQWV1Z0F3SUJBZ0lSQUpHM2psOFZhTU85a1FteGdXUFl3N3d3RFFZSktvWklodmNOQVFFTEJRQXcKR3pFWk1CY0dBMVVFQXhNUVltOXZkSE4wY21Gd0xYZGxZaTFqWVRBZUZ3MHlNekF5TVRVd01USXlNakZhRncweQpNekExTVRZd01USXlNakZhTUJzeEdUQVhCZ05WQkFNVEVHSnZiM1J6ZEhKaGNDMTNaV0l0WTJFd2dnRWlNQTBHCkNTcUdTSWI== # base 64 encoded version of certificate

---

apiVersion: v1
kind: Secret
metadata:
  name: dst-cert
  namespace: NAMESPACE
data:
  ca.crt: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURBekNDQWV1Z0F3SUJBZ0lSQUtoaEJXWWo3VGZlUUZWUWo0U0RpckV3RFFZSktvWklodmNOQVFFTEJRQXcKR3pFWk1CY0dBMVVFQXhNUVltOXZkSE4wY21Gd0xYZGxZaTFqWVRBZUZ3MHlNekF6TURZeU16TTROVEJhRncweQpNekEyTURReU16TTROVEJhTUJzeEdUQVhCZ05WQkFNVEVHSnZiM1J6ZEhKaGNDMTNaV0l0WTJFd2dnRWlNQTBHCkNTcUdTSWIzRFFF== # base 64 encoded version of certificate. Can be same OR different than source certificate.

---

```

### (Optional) Create a LoggingTarget

Create a LoggingTarget to see transfer-service logs in Loki.

```yaml
apiVersion: logging.gdc.goog/v1
kind: LoggingTarget
metadata:
  namespace: NAMESPACE # Same namespace as your transfer job
  name: logtarg1
spec:
  # Choose matching pattern that identifies pods for this job
  # Optional
  # Relationship between different selectors: AND
  selector:

    # Choose pod name prefix(es) to consider for this job
    # Observability platform will scrape all pods
    # where names start with specified prefix(es)
    # Should contain [a-z0-9-] characters only
    # Relationship between different list elements: OR
    matchPodNames:
      - transfer-job # Choose the prefix here that matches your transfer job name
  serviceName: transfer-service
```

### Create the transfer workload

You can run the transfer as a one-time operation using a `Job`, or schedule it to run repeatedly using a `CronJob`.

Choose the method that best fits your use case:

#### Create a one-time Job

Use this configuration for a single, manual data transfer.

```yaml
---
apiVersion: batch/v1
kind: Job
metadata:
  name: transfer-job
  namespace: NAMESPACE
spec:
  template:
    spec:
      serviceAccountName: transfer-service-account # The service account created in the previous step
      containers:
        - name: storage-transfer-pod
          image: gcr.io/private-cloud-staging/storage-transfer:latest
          imagePullPolicy: Always
          command:
            - /storage-transfer
          args:
            # The S3 endpoints for your source and destination
            - '--src_endpoint=SRC_ENDPOINT'
            - '--dst_endpoint=DST_ENDPOINT'
            # The Fully Qualified Names (FQN) of the buckets
            - '--src_path=SRC_BUCKET_FQN/'
            - '--dst_path=DST_BUCKET_FQN/'
            # Cross-namespace mapping: point directly to the live credentials using NAMESPACE/SECRET_NAME
            - '--src_credentials=NAMESPACE/SRC_SECRET_NAME'
            - '--dst_credentials=NAMESPACE/DST_SECRET_NAME'
            # Point to the CA certificate Secret created in the destination namespace
            - '--src_ca_certificate_reference=NAMESPACE/CA_CERT_SECRET'
            - '--dst_ca_certificate_reference=NAMESPACE/CA_CERT_SECRET'
            - '--src_type=s3'
            - '--dst_type=s3'
            - '--bandwidth_limit=BANDWIDTH_LIMIT' # Optional. Examples: '10K', '100M', '1G'
      restartPolicy: OnFailure
---
```

#### Create a scheduled CronJob

Use this configuration to keep your destination bucket continuously synchronized with your source bucket on a fixed schedule.

```yaml
---
apiVersion: batch/v1
kind: CronJob
metadata:
  name: transfer-cronjob
  namespace: NAMESPACE
spec:
  schedule: "0 * * * *"  # Runs at the top of every hour. Adjust the cron schedule as needed.
  concurrencyPolicy: Forbid
  successfulJobsHistoryLimit: 3
  failedJobsHistoryLimit: 5
  jobTemplate:
    spec:
      template:
        spec:
          serviceAccountName: transfer-service-account # The service account created in the previous step
          containers:
            - name: storage-transfer-pod
              image: gcr.io/private-cloud-staging/storage-transfer:latest
              imagePullPolicy: Always
              command:
                - /storage-transfer
              args:
                # The S3 endpoints for your source and destination
                - '--src_endpoint=SRC_ENDPOINT'
                - '--dst_endpoint=DST_ENDPOINT'
                # The Fully Qualified Names (FQN) of the buckets
                - '--src_path=SRC_BUCKET_FQN/'
                - '--dst_path=DST_BUCKET_FQN/'
                # Cross-namespace mapping: point directly to the live credentials using NAMESPACE/SECRET_NAME
                - '--src_credentials=NAMESPACE/SRC_SECRET_NAME'
                - '--dst_credentials=NAMESPACE/DST_SECRET_NAME'
                # Point to the CA certificate Secret created in the destination namespace
                - '--src_ca_certificate_reference=NAMESPACE/CA_CERT_SECRET'
                - '--dst_ca_certificate_reference=NAMESPACE/CA_CERT_SECRET'
                - '--src_type=s3'
                - '--dst_type=s3'
                - '--bandwidth_limit=BANDWIDTH_LIMIT' # Optional. Examples: '10K', '100M', '1G'
          restartPolicy: OnFailure
---
```

### Monitor your data transfer

After you instantiate the Job, you can monitor its status using `kubectl`
commands, such as `kubectl describe`. To verify the transfer, list the
objects inside of your destination bucket to validate that your data
transferred. The data transfer tool is independent of the location of the endpoints
involved in the transfer.

Run the following command to check the status of your workload:

```bash
# If you created a one-time Job:
kubectl describe job transfer-job -n NAMESPACE

# If you created a scheduled CronJob:
kubectl describe cronjob transfer-cronjob -n NAMESPACE
```

The preceding command tells you the status of the job.

The job prompts a pod to transfer the data. You can get the name of the pod and look at logs to see if there are any errors during the transfer.

To view pod logs, run the following:

```bash
# 1. First, find the exact pod name generated by your Job
kubectl get pods -n NAMESPACE | grep transfer

# 2. Then, view the logs for that specific pod
kubectl logs POD_NAME -n NAMESPACE
```

Successful job logs:

    DEBUG : Starting main for transfer
    I0607 21:34:39.183106       1 transfer.go:103]  "msg"="Starting transfer "  "destination"="sample-bucket" "source"="/data"
    2023/06/07 21:34:39 NOTICE: Bandwidth limit set to {100Mi 100Mi}
    I0607 21:34:49.238901       1 transfer.go:305]  "msg"="Job finished polling "  "Finished"=true "Number of Attempts"=2 "Success"=true
    I0607 21:34:49.239675       1 transfer.go:153]  "msg"="Transfer completed."  "AvgSpeed"="10 KB/s" "Bytes Moved"="10.0 kB" "Errors"=0 "Files Moved"=10 "FilesComparedAtSourceAndDest"=3 "Time since beginning of transfer"="1.0s"

Viewing logs lets you see the data transfer speed, which is not the same as bandwidth used, bytes moved, number of errored files, and files moved.

## Copy block storage to object storage

Ensure that you meet the following prerequisites:

- An S3 endpoint with a S3 key ID and secret access key with at least WRITE permissions to the dedicated bucket that you want to transfer data to.
- A working cluster with connectivity to the S3 endpoint.
- Privileges to create Jobs and Secrets inside your cluster.
- For replication of block storage, a Pod with an attached `PersistentVolumeClaim` (PVC) that you want to backup to object storage, and privileges to inspect running Jobs and PVCs.
- For replication of the block storage, a window during which no writes take place to the `PersistentVolume` (PV).
- For the restoration of block storage from an object storage endpoint, privileges to allocate a PV with sufficient capacity.

To replicate a PV to object storage, you must attach a volume to an existing
Pod. During the window of the transfer, the Pod must not perform any writes. To
avoid detaching the mounted PV from the Job, the data transfer process works by
running the transfer Job on the same machine as the Pod, and using a hostPath
mount to expose the volume on the disk. In preparation for the transfer, you must
first find the node on which the Pod is running, and additional metadata such as
the Pod UID and PVC type to reference the appropriate path on the Node. You must
substitute this metadata into the sample YAML file outlined in the
following section.

### Collect metadata

To collect the metadata required to create the data transfer Job, work through
these steps:

1. Find the Node that has the scheduled Pod:

   ```bash
   kubectl get pod POD_NAME -o jsonpath='{.spec.nodeName}'
   ```

   Record the output of this command as the <var translate="no">NODE_NAME</var> to
   use in the data transfer Job YAML file.
2. Find the Pod UID:

   ```bash
   kubectl get pod POD_NAME -o 'jsonpath={.metadata.uid}'
   ```

   Record the output of this command as the <var translate="no">POD_UID</var> to
   use in the data transfer Job YAML file.
3. Find the PVC name:

   ```bash
   kubectl get pvc www-web-0 -o 'jsonpath={.spec.volumeName}'
   ```

   Record the output of this command as the <var translate="no">PVC_NAME</var> to
   use in the data transfer Job YAML file.
4. Find the PVC storage provisioner:

   ```bash
   kubectl get pvc www-web-0 -o jsonpath='{.metadata.annotations.volume\.v1\.kubernetes\.io\/storage-provisioner}'
   ```

   Record the output of this command as the
   <var translate="no">PROVISIONER_TYPE</var> to use in the data transfer Job YAML
   file.

### Create secrets

To replicate file to object storage across clusters, you must first instantiate
the secrets inside your Kubernetes cluster. You must use matching keys for the
Secret data for the tool to pull the credentials.

To perform the transfer in an existing namespace, see the following example of
creating Secrets in a `transfer` namespace:

```yaml
apiVersion: v1
kind: Secret
metadata:
  name: src-secret
  namespace: transfer
data:
  access-key-id: c3JjLWtleQ== # echo -n src-key| base64 -w0
  access-key: c3JjLXNlY3JldA== # echo -n src-secret| base64 -w0
---
apiVersion: v1
kind: Secret
metadata:
  name: dst-secret
  namespace: transfer
data:
  access-key-id: ZHN0LWtleQ== # echo -n dst-key| base64 -w0
  access-key: ZHN0LXNlY3JldA== # echo -n dst-secret| base64 -w0
```

### Create the Job

With the data that you collected in the previous section, create a Job with the
data transfer tool. The data transfer Job has a `hostPath` mount referencing the
path for the PV of interest, and a `nodeSelector` for the relevant node.

The following is an example of a data transfer Job:

```yaml
apiVersion: batch/v1
kind: Job
metadata:
  name: transfer-job
  namespace: transfer
spec:
  template:
    spec:
      nodeSelector: NODE_NAME
      serviceAccountName: data-transfer-sa
      containers:
      - name: storage-transfer-pod
        image: storage-transfer
        command:
        - /storage-transfer
        args:
        - --dst_endpoint=https://your-dst-endpoint.com
        - --src_path=/pvc-data
        - --dst_path=transfer-dst-bucket
        - --dst_credentials=transfer/dst-secret
        - --src_type=local
        - --dst_type=s3
      volumeMounts:
      - mountPath: /pvc-data
        name: pvc-volume
      volumes:
      - name: pvc-volume
      hostPath:
        path: /var/lib/kubelet/pods/POD_UID/volumes/PROVISIONER_TYPE/PVC_NAME
      restartPolicy: Never
```

As with the [S3 data transfer](https://docs.cloud.google.com/distributed-cloud/hosted/docs/latest/gdcag/platform/pa-user/transfer-data.md.txt#file_object), you must create a Secret
containing the access keys for the destination endpoint in the Kubernetes
cluster, and the data transfer Job must run with a service account with
adequate privileges to read the Secret from the API server. Monitor the status
of the transfer with standard `kubectl` commands operating on the Job.

Consider the following details when transferring block storage to object storage:

- By default, symbolic links follow and replicate to the object storage, but a deep rather than shallow copy performs. Upon restoration, it destroys symlinks.
- As with object storage replication, cloning into a subdirectory of the bucket is destructive. Ensure that the bucket is available exclusively for your volume.

### Restore from object storage to block storage

#### Allocate a PV

To restore block storage from an object storage endpoint, follow these steps:

1. Allocate a persistent volume to target in the restore. Use a PVC to allocate
   the volume, as shown in the following example:

   ```yaml
   apiVersion: v1
   kind: PersistentVolumeClaim
   metadata:
     name: restore-pvc
     namespace: restore-ns
   spec:
     storageClassName: "default"
     accessModes:
     - ReadWriteOnce
     resources:
       requests:
         storage: 1Gi # Need sufficient capacity for full restoration.
   ```
2. Check the status of the PVC:

   ```bash
   kubectl get pvc restore-pvc -n restore-ns
   ```

   After the PVC is in a `Bound` state, it is ready to consume inside the Pod
   that rehydrates it.
3. If a Stateful set eventually consumes the PV, you must match
   the rendered StatefulSet PVCs. The Pods that StatefulSet produces
   consumes the hydrated volumes. The following example shows volume claim
   templates in a StatefulSet named `ss`.

   ```yaml
     volumeClaimTemplates:
     - metadata:
         name: pvc-name
       spec:
         accessModes: [ "ReadWriteOnce" ]
         storageClassName: "default"
         resources:
           requests:
             storage: 1Gi
   ```
4. Pre-allocate PVCs with names such as `ss-pvc-name-0` and `ss-pvc-name-1` to
   ensure that the resultant Pods consume the pre-allocated volumes.

#### Hydrate the PV

After the PVC is bound to a PV, start the Job to populate the PV:

```yaml
apiVersion: batch/v1
kind: Job
metadata:
  name: transfer-job
  namespace: transfer
spec:
  template:
    spec:
      serviceAccountName: data-transfer-sa
      volumes:
      - name: data-transfer-restore-volume
        persistentVolumeClaim:
          claimName: restore-pvc
      containers:
      - name: storage-transfer-pod
        image: storage-transfer
        command:
        - /storage-transfer
        args:
        - --src_endpoint=https://your-src-endpoint.com
        - --src_path=/your-src-bucket
        - --src_credentials=transfer/src-secret
        - --dst_path=/restore-pv-mnt-path
        - --src_type=s3
        - --dst_type=local
      volumeMounts:
      - mountPath: /restore-pv-mnt-path
        name: data-transfer-restore-volume
```

After the Job has finished running, the data from the object storage bucket
populates the volume. A separate Pod can consume the data by using
the same standard mechanisms for mounting a volume.