Preemptible TPUs

Important: TPU Spot VMs are the latest version of preemptible TPUs. New and existing preemptible TPUs continue to be supported, and TPU Spot VMs use the same pricing model as preemptible TPUs. However, TPU Spot VMs provide new features that preemptible TPUs don't support. For example, TPU Spot VMs don't have the 24 hour run time limit of preemptible TPUs. For more information, see Manage TPU Spot VMs.

Caution: This page applies to the Cloud TPU API, which is no longer under active development. The Cloud TPU API will receive bug fixes and security updates only. For the latest features and support for the latest TPU versions, we recommend using Compute Engine or Google Kubernetes Engine to manage TPU resources. For more information, see TPU resources in Compute Engine and About TPUs in GKE. The Cloud TPU API is not supported on TPU7x and later TPU versions.

Preemptible TPUs cost much less than non-preemptible TPUs. The Cloud TPU service might preempt (shut down) these TPUs at any time, if it requires additional TPU resources.

Creating a preemptible TPU VM

gcloud

$ gcloud compute tpus tpu-vm create demo-tpu \
  --zone=europe-west4-a \
  --accelerator-type=v3-8 \
  --version=tpu-vm-tf-2.17.1-pjrt \
  --preemptible
  

where:

• demo-tpu is a name for the TPU.
• accelerator-type is the type of TPU. For more information about supported accelerator types for each TPU version, see TPU versions.
• version is the Cloud TPU software version to install.
• preemptible allows Cloud TPU to preempt the TPU.

Pricing and quota for preemptible TPUs

Pricing for preemptible TPUs is significantly lower than for normal TPUs. For details, see the pricing page. You are not charged for TPUs if they are preempted in the first minute after you create them.

Quota for preemptible TPUs is generally higher, and is separate from the quota for normal TPUs. See the quota page.

Detecting if a TPU has been preempted

You use the following gcloud command to check whether the Cloud TPU service has preempted your TPU:

List your available TPUs:

gcloud compute tpus tpu-vm list
  --zone=us-central1-b
  

The output of the command displays the details of the TPUs created in your project. If a TPU has been preempted, the status changes from READY to PREEMPTED.

For example:

NAME       ZONE           ACCELERATOR_TYPE  NETWORK_ENDPOINT   NETWORK  RANGE          STATUS
demo-tpu   us-central1-b  v2-8              10.240.1.2:8470    default  10.240.1.0/29  PREEMPTED

Note: If your Cloud TPU is preempted, you must delete it and create a new one as described in Managing TPUs.

Designing your machine learning application to run on preemptible TPUs

Make sure your application is resilient to restarts of the VM and TPU, by saving model checkpoints regularly and by configuring your application to restore the most recent checkpoint on restart.