This page shows how to back up the etcd data store for your
GKE on AWS installation for recovery from events that may
damage your cluster's etcd data.

## Limitations

- Using a backup file to restore your etcd data is a last resort. We do
  not recommend restoring from a backup file unless the cluster is completely
  broken. Contact Google support for help in deciding the best course of action.

- This procedure does not back up data from your workloads, including
  PersistentVolumes.

- This backup cannot be used to restore a cluster from a different version of
  GKE on AWS.

## Backing up a user cluster

A user cluster backup is a snapshot of the user cluster's etcd store.
The etcd store contains all of the Kubernetes objects and custom objects
that represent the cluster's state. The snapshot contains the
data required to recreate the cluster's stateless workloads.

To create a snapshot of the etcd data store, perform the following steps:

1. Open a shell on the management service instance running etcd for your cluster.

   1. Find the IP address of your cluster's management service instance.

      ```bash
      export CLUSTER_ID=$(terraform output cluster_id)
      export MANAGEMENT_IP=$(aws ec2 describe-instances \
        --filters "Name=tag:Name,Values=$CLUSTER_ID-management-0" \
        --query "Reservations[*].Instances[*].PrivateIpAddress" \
        --output text)
      ```
   2. Use the `ssh` tool to open a connection to the management service instance.

      ### Direct connection

          ssh -i ~/.ssh/anthos-gke ubuntu@$MANAGEMENT_IP

      ### Bastion host

          export BASTION_DNS=$(terraform output bastion_dns_name)
          ssh -i ~/.ssh/anthos-gke -J ubuntu@$BASTION_DNS ubuntu@$MANAGEMENT_IP

   > [!NOTE]
   > **Note:** If the \`ssh\` command fails with a public key permission error, you might need to refresh your ssh authentication key with the following command:
   >
   > ```
   > ssh-add -k ~/.ssh/anthos-gke
   > ```
   > Replace <var translate="no">anthos-gke</var> with the name of the file containing your authentication key.

2. Create a directory to store the etcd backup data.

   ```
   mkdir ./etcd-backups
   ```
3. Use the `ps` command-line tool to find the process ID of the etcd
   process on that instance.

   ```
   ps -e | grep etcd
   ```

   The output shows details of your etcd process. The first element is etcd's
   process ID. In the following steps, replace <var translate="no">ETCD_PID</var> with this
   process ID.
4. Create a script within the etcd container's filesystem to take a snapshot.
   This script runs etcdctl to connect to the etcd daemon and perform a snapshot to
   back up the etcd database.

   ```
   cat << EOT > /tmp/etcdbackup.sh
   # Extract a snapshot of the anthos-gke etcd state database

   export ETCDCTL_API=3

   etcdctl \
    --endpoints=https://127.0.0.1:2379 \
    --cacert=/secrets/server-ca.crt \
    --cert=/secrets/server.crt \
    --key=/secrets/server.key \
    snapshot save /tmp/snapshot.db
   EOT

   chmod a+x /tmp/etcdbackup.sh
   sudo mv /tmp/etcdbackup.sh /proc/ETCD_PID/root/tmp/etcdbackup.sh
   ```
5. Use the `nsenter` command to run the script within the etcd container to
   create the snapshot.

   ```
   sudo nsenter --all --target ETCD_PID /tmp/etcdbackup.sh
   ```
6. Copy the snapshot file out of the etcd container.

   ```
   sudo cp /proc/ETCD_PID/root/tmp/snapshot.db ./etcd-backups
   ```
7. Copy all files in the /secrets directory of the etcd container to your
   backup directory. These files contain the certificates that encrypt and validate
   communication between etcd and other processes in the cluster. Together, the
   snapshot file and the certificates files are a full backup of your
   etcd cluster status.

   ```
   sudo cp -r /proc/ETCD_PID/root/secrets ./etcd-backups
   ```
8. Use the `tar` tool to bundle the etc-backup files into a convenient tar file.

   ```
   tar -cvf etcd-backup.tar etcd-backup
   ```
9. Exit to your local machine and use the `scp` tool to copy the
   etcd-backup.tar file from the management service instance. This example uses
   the BASTION_DNS and MANAGEMENT_IP environment variables defined earlier.

   ```
   scp -i ~/.ssh/anthos-gke -J ubuntu@$BASTION_DNS \
    ubuntu@$MANAGEMENT_IP:~/etcd-backup/backup.tar
   ```

### For More Information

- [scp man page](http://manpages.org/scp)
- [etcdctl man page](http://manpages.org/etcdctl)
- [tar man page](http://manpages.org/tar)