# TPU v6e

This document describes the architecture and supported configurations of
Cloud TPU v6e (Trillium). On all technical
surfaces, such as the API and logs, and throughout this document, Trillium will
be referred to as v6e.

With a 256-chip footprint per Pod, v6e shares many similarities with
[v5e](https://docs.cloud.google.com/tpu/docs/v5e). This system is optimized for transformer, text-to-image,
and convolutional neural network (CNN) training, fine-tuning, and serving.

## System architecture

Each v6e chip contains one TensorCore and connects to the host through a PCIe
Gen5 interface. Each TensorCore contains two 256x256 matrix-multiply units
(MXU), a 128x8 vector processing unit (VPU) with 128 lanes x 8 sublanes, two
cross-lane units (XLUs), and a scalar processing unit (SPU). The following
table shows the key specifications and their values for TPU v6e.

| Specification | Values |
|---|---|
| Performance/total cost of ownership (TCO) (expected) | 1 |
| Peak compute per chip (bf16) | 918 TFLOPs |
| Peak compute per chip (Int8) | 1836 TOPs |
| HBM capacity per chip | 32 GB |
| HBM bandwidth per chip | 1638 GBps |
| Bidirectional inter-chip interconnect (ICI) bandwidth (per chip) | 800 GBps |
| ICI ports per chip | 4 |
| DRAM per host | 1536 GiB |
| Chips per host | 8 |
| TPU Pod size | 256 chips |
| Interconnect topology | 2D torus |
| BF16 peak compute per Pod | 234.9 PFLOPs |
| All-reduce bandwidth per Pod | 102.4 TB/s |
| Bisection bandwidth per Pod | 3.2 TB/s |
| Per-host NIC configuration | 4 x 200 Gbps NIC |
| Data center network bandwidth per Pod | 25.6 Tbps |
| Host interface | PCIe Gen5 x8 per chip |
| Special features | [SparseCore](https://docs.cloud.google.com/tpu/docs/system-architecture-tpu-vm#sparsecore) |

> [!NOTE]
> **Note:** The TPU system and `libtpu` runtime reserve part of physical high-bandwidth memory (HBM) for reliability and runtime overhead. Tools and frameworks such as `tpu-info`, JAX, and PyTorch report 31.25 GiB of usable HBM per chip, which is lower than nominal capacity.

## Supported configurations

> [!NOTE]
> **Note:** Use the [TPU topology
> visualizer](https://tpu-visualizer.uc.r.appspot.com/) to view 3D renderings of different TPU configurations.

The following table shows the 2D slice shapes that are supported for v6e:

| Topology | TPU chips | Hosts | VMs | Machine type | Scope |
|---|---|---|---|---|---|
| 1x1 | 1 | 1/8 | 1 | `ct6e-standard-1t` | Sub-host |
| 2x2 | 4 | 1/2 | 1 | `ct6e-standard-4t` | Sub-host |
| 2x4 | 8 | 1 | 1 | `ct6e-standard-8t` | Single-host |
| 2x4 | 8 | 1 | 2 | `ct6e-standard-4t` | Multi-host |
| 4x4 | 16 | 2 | 4 | `ct6e-standard-4t` | Multi-host |
| 4x8 | 32 | 4 | 8 | `ct6e-standard-4t` | Multi-host |
| 8x8 | 64 | 8 | 16 | `ct6e-standard-4t` | Multi-host |
| 8x16 | 128 | 16 | 32 | `ct6e-standard-4t` | Multi-host |
| 16x16 | 256 | 32 | 64 | `ct6e-standard-4t` | Multi-host |

> [!NOTE]
> **Note:** The 8-chip (`2x4`) multi-host configuration attached to two `ct6e-standard-4t` VMs is supported when you provision TPUs with GKE or Compute Engine, and isn't supported when you use the Cloud TPU API.

Slices with 8 chips (`v6e-8`) attached to a single VM are optimized for
inference, allowing all 8 chips to be used in a single serving workload.

For information about the number of VMs for each topology, see
[VM Types](https://docs.cloud.google.com/tpu/docs/v6e#vm-types).

### VM types

Each TPU v6e VM can contain 1, 4, or 8 chips. 4-chip and smaller
slices have the same non-uniform memory access (NUMA) node. For more information
about NUMA nodes, see [Non-uniform memory
access](https://en.wikipedia.org/wiki/Non-uniform_memory_access) on Wikipedia.

![Diagram of a v6e host](https://docs.cloud.google.com/static/tpu/docs/images/v6e-tpu-host.png)

v6e slices are created using half-host VMs, each with 4 TPU chips. There are two
exceptions to this rule:

- `v6e-1`: A VM with only a single chip, primarily intended for testing
- `v6e-8`: A full-host VM that has been optimized for an inference use case with all 8 chips attached to a single VM.

The following table shows a comparison of TPU v6e VM types:

| VM type | Number of vCPUs per VM | RAM (GB) per VM | Number of NUMA nodes per VM |
|---|---|---|---|
| 1-chip VM | 44 | 176 | 1 |
| 4-chip VM | 180 | 720 | 1 |
| 8-chip VM | 360 | 1440 | 2 |

> [!NOTE]
> **Note:** We don't recommend using a full-host VM (`v6e-8` with one VM) for dual networks due to performance impacts.

## What's next

- Run [training and inference using TPU v6e](https://docs.cloud.google.com/tpu/docs/v6e-intro)