# Troubleshoot your Cloud TPU workflow

> [!CAUTION]
> **Caution:** This page applies to the Cloud TPU API, which is no longer under active development. The Cloud TPU API will receive bug fixes and security updates only. For the latest features and support for the latest TPU versions, we recommend using Compute Engine or Google Kubernetes Engine to manage TPU resources. For more information, see [TPU resources in Compute Engine](https://docs.cloud.google.com/tpu/docs/tpus-in-compute-engine.md.txt) and [About TPUs in GKE](https://docs.cloud.google.com/kubernetes-engine/docs/concepts/tpus.md.txt). The Cloud TPU API is not supported on TPU7x and later TPU versions.

Once you have your training or inference workload running on TPUs, the next step
is to ensure your workload is working as expected. Cloud TPU generates
metrics and logs that enable you to look for and debug any TPU VMs that are not
behaving as expected. We refer to such VMs as *outliers* throughout this
documentation.

The general troubleshooting workflow is:

1. View Cloud TPU metrics to check for outlier TPU VMs
2. View Cloud TPU logs for the outlier TPU VMs
3. Profile your workload

You can view metrics and logs in the [Metrics
Explorer](https://docs.cloud.google.com/monitoring/charts/metrics-explorer) and the [Logs
Explorer](https://docs.cloud.google.com/logging/docs/view/logs-explorer-interface) in the Google Cloud
console. You can also use monitoring and logging dashboards to collect all
Cloud TPU related metrics and logs in individual dashboards.


## Cloud TPU VM metrics

Cloud Monitoring automatically collects metrics from your TPUs and their host
Compute Engine VMs. Metrics track numerical quantities over time, for example,
CPU utilization, network usage, or TensorCore idle duration. For more
information on Cloud TPU metrics, see [Monitoring TPU
VMs](https://docs.cloud.google.com/tpu/docs/troubleshooting/tpu-vm-monitoring).

## Cloud TPU logs

Cloud Logging automatically collects logs from your TPUs and their host
Compute Engine VMs. Cloud Logging tracks events generated by Cloud TPU.
You can also instrument your code to generate logs. Two types of logs are
generated by Cloud TPU:

- TPU worker logs
- Audited resource logs

TPU worker logs contain information about a specific TPU worker in a specific
zone, for example the amount of memory available on the TPU worker
(system_available_memory_GiB).

Audited Resource logs contain information about when a specific Cloud TPU API
was called and who made the call. For example `CreateNode`, `UpdateNode`, and
`DeleteNode`.

You can also use the `cloud-tpu-diagnostics` PyPi package to write stack traces
to logs. For more information, see [Debugging TPU
VMs](https://docs.cloud.google.com/tpu/docs/troubleshooting/debugging).

For more information about logs, see
[Logging](https://docs.cloud.google.com/tpu/docs/troubleshooting/tpu-vm-monitoring.md.txt#locate_logs).

## Cloud TPU health metrics

Cloud TPU health monitoring provides real-time information on the health
status of VMs and slices. Using either the Cloud Monitoring console or
[LibTPU Monitoring Library](https://docs.cloud.google.com/tpu/docs/tpu-monitoring-library.md.txt), you can identify
hardware failures as they occur. This lets you reallocate resources to avoid
complete job failures and minimize performance degradation.

For more information, see [TPU health
monitoring](https://docs.cloud.google.com/tpu/docs/troubleshooting/tpu-health).

## Monitoring and logging dashboards

Having a single page in the Google Cloud console can make viewing and interpreting
Cloud TPU-related metrics and logs easier. The
[cloud-tpu-monitoring-debugging](https://github.com/google/cloud-tpu-monitoring-debugging)
GitHub repository contains a set of scripts and configuration files that use
[Terraform](https://developer.hashicorp.com/terraform) to automatically deploy
dashboards that contain all Cloud TPU related metrics and logs in dashboards.
To set up these dashboards in your Google Cloud project, see [Monitoring and
Logging Dashboards](https://docs.cloud.google.com/tpu/docs/troubleshooting/dashboards).

## Profiling your workloads on TPU VMs

Profiling lets you optimize your model's training performance on TPU VMs. You
use [TensorBoard](https://www.tensorflow.org/tensorboard) and the [TPU
TensorBoard plug-in](https://docs.cloud.google.com/tpu/docs/profile-tpu-vm#install-plugin) to profile your
model. For more information about how to profile your workload, see [Profile
your model on TPU VMs](https://docs.cloud.google.com/tpu/docs/profile-tpu-vm).

For more information about using TensorBoard with one of the supported
frameworks, see the following documents:

- [JAX performance guide](https://jax.readthedocs.io/en/latest/profiling.html)
- [PyTorch performance guide](https://torchtpu.readthedocs.io/en/latest/performance_profiling_with_xprof/)

## VM boot failures with custom images

If a VM fails to boot after customization, review the serial console output
using the the Google Cloud console or `gcloud compute instances
get-serial-port-output`. This is the primary source of information for
diagnosis. Because you cannot repair the disk offline, you must recreate the TPU
resource after correcting the issue in your startup script or custom image
source.

For more information about customizing Cloud TPU VM boot disks, see [TPU VM
boot disk](https://docs.cloud.google.com/tpu/docs/storage-options#vmboot-disk).