# Attach durable block storage to a TPU VM

A TPU VM includes a 10 GB boot disk. Some scenarios require additional storage
for training or preprocessing. Add a [Google Cloud Hyperdisk](https://docs.cloud.google.com/compute/docs/disks/hyperdisks.md.txt)
or a [Persistent Disk](https://docs.cloud.google.com/compute/docs/disks/persistent-disks.md.txt) (PD) volume to expand
local disk capacity.

For the highest performance and advanced features, use
Hyperdisk if it is available for your TPU version.
Otherwise, use Persistent Disk. For more information about block storage options in
Compute Engine, see [Choose a disk type](https://docs.cloud.google.com/compute/docs/disks.md.txt).

## TPU support for Hyperdisk and Persistent Disk

The following table shows the supported disk types for each TPU version:

| TPU version | Supported disk types | Maximum disks per VM (includes the boot disk) |
|---|---|---|
| TPU7x | Hyperdisk Balanced Hyperdisk ML | 128 |
| v6e | Hyperdisk Balanced Hyperdisk ML | 32 |
| v5p | Hyperdisk ML Balanced Persistent Disk | 128 |

## Access modes

Configure a disk attached to a single TPU VM (also called a single-host TPU slice),
for example, `ct6e-standard-4t`, in read-write (`rw`) or read-only (`ro`) mode.

When you attach a disk to a multi-host TPU slice, the disk attaches to each
VM in the slice. To prevent multiple TPU VMs from writing to a disk simultaneously,
you must configure all disks attached to a multi-host TPU slice
as read-only (`ro`). Read-only disks are useful for storing a dataset for
processing on a TPU slice.

> [!NOTE]
> **Note:** Because Hyperdisk Balanced doesn't support read-only mode, you can only attach a Hyperdisk Balanced volume to a single TPU VM.

## Prerequisites

Before using these procedures, set up a Google Cloud account and project. For more
information, see [Set up the Cloud TPU environment](https://docs.cloud.google.com/tpu/docs/setup-gcp-account-compute.md.txt).

## Create a disk

To create a disk, use the following command:

```bash
gcloud compute disks create DISK_NAME \
    --size DISK_SIZE  \
    --zone ZONE \
    --type DISK_TYPE
```

Replace the following placeholders:

- <var translate="no">DISK_NAME</var>: The name of the new disk.
- <var translate="no">DISK_SIZE</var>: The size of the new disk. The value must be a whole number followed by a size unit of GB for gibibyte, or TB for tebibyte. If you don't specify a size unit, the system assumes GB.
- <var translate="no">ZONE</var>: The name of the [zone](https://docs.cloud.google.com/tpu/docs/types-zones) in which to create the new disk. This must be the same zone where you create the TPU.
- <var translate="no">DISK_TYPE</var>: The [type of disk](https://docs.cloud.google.com/compute/docs/disks.md.txt#disk-types). Use one of these values: `hyperdisk-balanced`, `hyperdisk-ml`, or `pd-balanced`.

For Hyperdisk, you can optionally specify the `--access-mode`
flag with one of these values:

- <var translate="no">READ_WRITE_SINGLE</var>: Read-write access from one instance (default).
- <var translate="no">READ_ONLY_MANY</var>: (Hyperdisk ML only) Concurrent read-only access from multiple instances.

For more information about creating disks, see [Create a new Hyperdisk volume](https://docs.cloud.google.com/compute/docs/disks/add-hyperdisk.md.txt)
and [Create a new Persistent Disk volume](https://docs.cloud.google.com/compute/docs/disks/add-persistent-disk.md.txt).

## Attach a disk

Attach a disk volume to your TPU VM or slice when you create it, or attach one after creation.

### Attach a disk when you create a TPU VM

When you create a TPU VM or instance template, use the `--disk` flag to attach a disk volume.

#### Attach a disk when creating a single TPU VM

The following example shows how to attach a disk volume when you create a single
TPU VM:

```bash
  gcloud compute instances create TPU_NAME \
    --machine-type=MACHINE_TYPE \
    --image-family=IMAGE_FAMILY \
    --image-project=IMAGE_PROJECT \
    --zone=ZONE \
    --maintenance-policy=TERMINATE \
    --disk=name=DISK_NAME,device-name=DEVICE_NAME,mode=MODE
```

Replace the following placeholders:

- <var translate="no">TPU_NAME</var>: A name for your TPU VM.
- <var translate="no">MACHINE_TYPE</var>: The [machine type](https://docs.cloud.google.com/compute/docs/tpus/tpu-machines.md.txt#tpu_machine_types) for the TPU VM (for example `ct6e-standard-8t`).
- <var translate="no">IMAGE_FAMILY</var>: The [OS image family](https://docs.cloud.google.com/compute/docs/images.md.txt#image_families) for the TPU VM. If you want to install a specific OS version, use the `--image` flag. For more information about OS images, see [OS images](https://docs.cloud.google.com/compute/docs/images.md.txt).
- <var translate="no">IMAGE_PROJECT</var>: The project that contains the OS image. For TPU images, this is `ubuntu-os-accelerator-images`.
- <var translate="no">ZONE</var>: The [zone](https://docs.cloud.google.com/tpu/docs/types-zones) for the TPU VM.
- <var translate="no">DEVICE_NAME</var>: The name of the device to use for the disk. This name identifies the disk in the OS.
- <var translate="no">MODE</var>: The mode for the disk. This can be `rw` (read-write) or `ro` (read-only). For more information, see [Access modes](https://docs.cloud.google.com/tpu/docs/attach-durable-block-storage-compute.md.txt#access-modes).

#### Attach a disk when creating a multi-host TPU slice

When you create a multi-host TPU slice, you must specify `mode=read-only` (
Hyperdisk ML and Balanced Persistent Disk only). For more information, see
[Access modes](https://docs.cloud.google.com/tpu/docs/attach-durable-block-storage-compute.md.txt#access-modes).

To attach a disk across a multi-host TPU slice, create an instance template with
the attached disk, create a workload policy, and then create a MIG:

1. Create an instance template

   ```bash
   gcloud compute instance-templates create INSTANCE_TEMPLATE_NAME
   --machine-type=MACHINE_TYPE \
   --maintenance-policy=TERMINATE \
   --image-family=IMAGE_FAMILY \
   --image-project=IMAGE_PROJECT \
   --disk=name=DISK_NAME,mode=MODE
   ```

   Replace the following placeholders:
   - <var translate="no">INSTANCE_TEMPLATE_NAME</var>: The name for your instance template.
   - <var translate="no">MACHINE_TYPE</var>: The [machine type](https://docs.cloud.google.com/compute/docs/tpus/tpu-machines.md.txt#tpu_machine_types) for the TPU VM, for example, `ct6e-standard-8t`.
   - <var translate="no">IMAGE_FAMILY</var>: The [OS image family](https://docs.cloud.google.com/compute/docs/images.md.txt#image_families) for the TPU VM. If you want to install a specific OS version, use the `--image` flag. For more information about OS images, see [OS images](https://docs.cloud.google.com/compute/docs/images.md.txt).
   - <var translate="no">IMAGE_PROJECT</var>: The project that contains the OS image. For TPU images, this is `ubuntu-os-accelerator-images`.
   - <var translate="no">DISK_NAME</var>: The name of the disk to attach to the TPU VM.
   - <var translate="no">MODE</var>: The mode for the disk. The mode must be `ro` (read-only) for multi-host TPU slices.
2. Create a workload policy

   ```bash
   gcloud compute resource-policies create workload WORKLOAD_POLICY_NAME \
   --type=high-throughput \
   --accelerator-topology=TOPOLOGY
   ```

   Replace the following placeholders:
   - <var translate="no">WORKLOAD_POLICY_NAME</var>: The name of your workload policy.
   - <var translate="no">TOPOLOGY</var>: The topology of the TPU VMs, for example, `4x4x8`. For more information about topology for each version of TPU, see [System architecture](https://docs.cloud.google.com/tpu/docs/system-architecture-tpu-vm.md.txt#versions).
3. Create a MIG:

   ```bash
   gcloud compute instance-groups managed create MIG_NAME \
       --project=PROJECT_ID \
       --zone=ZONE \
       --template=TEMPLATE_NAME \
       --size=SIZE \
       --workload-policy=WORKLOAD_POLICY_NAME
   ```

   Replace the following placeholders:
   - <var translate="no">MIG_NAME</var>: The name for your MIG.
   - <var translate="no">PROJECT_ID</var>: The project ID.
   - <var translate="no">ZONE</var>: The [zone](https://docs.cloud.google.com/tpu/docs/types-zones) where the Cloud TPU is located.
   - <var translate="no">TEMPLATE_NAME</var>: The name for your instance template.
   - <var translate="no">SIZE</var>: The number of VMs for your multi-host TPU slice.
   - <var translate="no">WORKLOAD_POLICY_NAME</var>: The name of your workload policy.

#### Attach a disk when creating a MIG with multiple single-host TPU slices

If you create a single-host TPU slice, you can specify `mode=read-only`
(Hyperdisk ML and Balanced Persistent Disk only) or `mode=read-write`.

To attach durable storage to a MIG with independent single-host TPU slices, configure the instance template to create a new disk for each instance using
the `--create-disk` flag:

```bash
gcloud compute instance-templates create INSTANCE_TEMPLATE_NAME \
    --machine-type=MACHINE_TYPE \
    --maintenance-policy=TERMINATE \
    --image-family=IMAGE_FAMILY \
    --image-project=IMAGE_PROJECT \
    --disk=name=DISK_NAME,mode=MODE
```

Replace the following placeholders:

- <var translate="no">INSTANCE_TEMPLATE_NAME</var>: The name for your instance template.
- <var translate="no">MACHINE_TYPE</var>: The machine type for the TPU VM (for example, `ct6e-standard-8t`).
- <var translate="no">IMAGE_FAMILY</var>: The OS image family for the TPU VM.
- <var translate="no">IMAGE_PROJECT</var>: The project that contains the OS image (`ubuntu-os-accelerator-images`).
- <var translate="no">DISK_NAME</var>: The base name of the disk to create and attach to each TPU VM.
- <var translate="no">DISK_SIZE</var>: The size of the disk in GB.
- <var translate="no">DISK_TYPE</var>: The disk type (for example, `pd-balanced`, `hyperdisk-balanced`).

Then, create the MIG as shown in the [previous section](https://docs.cloud.google.com/tpu/docs/attach-durable-block-storage-compute.md.txt#attach-a-disk-when-creating-a-multi-host-tpu-slice),
setting `--size` to the your chosen number of TPU VMs.

### Attach a disk to an existing TPU VM

To attach a disk to an existing TPU VM, use the [`gcloud compute instances attach-disk`](https://docs.cloud.google.com/sdk/gcloud/reference/compute/instances/attach-disk.md.txt)
command.

```bash
gcloud compute instances attach-disk VM_NAME \
    --zone=ZONE \
    --disk=DISK_NAME \
    --mode=MODE
```

Replace the following placeholders:

- <var translate="no">VM_NAME</var>: The name of the TPU VM.
- <var translate="no">ZONE</var>: The [zone](https://docs.cloud.google.com/tpu/docs/types-zones) where the Cloud TPU is located.
- <var translate="no">DISK_NAME</var>: The name of the disk to attach to the TPU VM.
- <var translate="no">MODE</var>: The mode for the disk. For more information, see [Access modes](https://docs.cloud.google.com/tpu/docs/attach-durable-block-storage-compute.md.txt#access-modes).

If your VM shuts down for any reason, you might need to mount the disk after you
restart the VM. For information about enabling your disk to automatically mount
on VM restart, see [Configure automatic mounting on system restart](https://docs.cloud.google.com/compute/docs/disks/format-mount-disk-linux.md.txt#configure_automatic_mounting_on_vm_restart).

For more information about automatically deleting a disk, see
[Modify a disk's auto-delete setting](https://docs.cloud.google.com/compute/docs/disks/modify-persistent-disk.md.txt#updateautodelete).

## Format and mount a disk

If you attach a new, blank disk to your TPU VM, you must format and mount the disk before you can use it. If you attach a disk that already contains data,
you must mount it before you can use it.

For more information about formatting and mounting a non-boot disk, see
[Format and mount a non-boot disk on a Linux VM](https://docs.cloud.google.com/compute/docs/disks/format-mount-disk-linux.md.txt).

## Detach a disk

To detach a disk from your TPU VM, run the following command:

```bash
gcloud compute instances detach-disk VM_NAME \
    --zone=ZONE \
    --disk=DISK_NAME
```

Replace the following placeholders:

- <var translate="no">VM_NAME</var>: The name of the TPU VM.
- <var translate="no">ZONE</var>: The [zone](https://docs.cloud.google.com/tpu/docs/types-zones) where the Cloud TPU is located.
- <var translate="no">DISK_NAME</var>: The name of the disk to detach from the TPU VM.

For more information about detaching a disk, see [Detach a disk](https://docs.cloud.google.com/sdk/gcloud/reference/compute/instances/detach-disk.md.txt).

## Clean up

Delete your Cloud TPU and Compute Engine resources when you finish using them.

1. Disconnect from the Cloud TPU, if you have not already
   done so:

   ```bash
   exit
   ```
2. Delete your TPU VM:

   ```bash
   gcloud compute instances delete VM_NAME \
       --zone=ZONE
   ```

   Replace the following placeholders:
   - <var translate="no">VM_NAME</var>: The name of the TPU VM.
   - <var translate="no">ZONE</var>: The [zone](https://docs.cloud.google.com/tpu/docs/types-zones) where the Cloud TPU is located.

   If you created a multi-host TPU slice using a MIG, delete the instance group instead:

   ```bash
   gcloud compute instance-groups managed delete MIG_NAME \
       --zone=ZONE
   ```

   Replace the following placeholders:
   - <var translate="no">MIG_NAME</var>: The name of the MIG.
   - <var translate="no">ZONE</var>: The [zone](https://docs.cloud.google.com/tpu/docs/types-zones) where the Cloud TPU is located.
3. Verify the Cloud TPU deletion. Deletion might take
   several minutes.

   ```bash
   gcloud compute instances list --zone=ZONE
   ```
4. Verify that the disk automatically deletes when the TPU VM deletes
   by listing all disks in the zone where you created the disk:

   ```bash
   gcloud compute disks list --filter="zone:( ZONE )"
   ```

   Replace the following placeholders:
   - <var translate="no">ZONE</var>: The [zone](https://docs.cloud.google.com/tpu/docs/types-zones) where the Cloud TPU is located.

   If the disk does not delete when the TPU VM deletes,
   use the following command to delete it:

   ```bash
   gcloud compute disks delete DISK_NAME \
       --zone=ZONE
   ```

   Replace the following placeholders:
   - <var translate="no">DISK_NAME</var>: The name of the disk to delete.
   - <var translate="no">ZONE</var>: The [zone](https://docs.cloud.google.com/tpu/docs/types-zones) where the Cloud TPU is located.