This document explains the core concepts and advanced configurations available
when you use Cluster Toolkit to deploy Google Kubernetes Engine (GKE)
clusters with dedicated Cloud TPU node pools.

When you use Cluster Toolkit to provision
GKE clusters, you can deploy
dedicated Cloud TPU node pools designed for artificial intelligence (AI) and
machine learning (ML) workloads.
Cluster Toolkit provides blueprints to automate the deployment of
supported Cloud TPU architectures.

This document explains the core concepts and advanced configurations that are
available when you deploy Cloud TPU node pools with
Cluster Toolkit, including hardware topologies, multi-slice
configurations, storage integration, and compute consumption models.

## Key Cloud TPU concepts

To deploy a Cloud TPU cluster, you must understand how
Cloud TPU hardware is grouped and allocated. You create
Cloud TPU slices as a single, fixed hardware unit with physical,
high-speed interconnects between all nodes.

### Cloud TPU in GKE

TPUs are Google's custom-developed application-specific integrated circuits
(ASICs) that are used to accelerate machine learning workloads.
GKE provides full support for lifecycle management of
Cloud TPU nodes and node pools, including node
creation, configuration, and automatic upgrades.

For more information about how Cloud TPU works with GKE,
see [About TPUs in GKE](https://docs.cloud.google.com/kubernetes-engine/docs/concepts/tpus.md.txt).

### TPU slices

A Cloud TPU slice is a collection of Cloud TPU chips that are
physically connected by a dedicated, ultra-low-latency network called the
Inter-Chip Interconnect (ICI). This interconnect lets the nodes function as a
single supercomputer.

In Cluster Toolkit, the `num_slices` parameter specifies the number of
identical, independent Cloud TPU slices to create. Each slice
corresponds to an individual node pool.

### TPU topologies

The `tpu_topology` parameter defines the shape and total size of your
Cloud TPU slice. Topologies describe the number and physical
arrangement of the
Cloud TPU chips in a slice.

- **TPU v4** chips have a direct connection to the nearest neighboring chips in three dimensions, forming a 3D torus interconnect topology, such as `2x2x2`.
- **TPU v5e** chips are arranged in a 2D topology, such as `2x2` or `2x4`.
- **TPU v5p** chips have a direct connection to the nearest neighboring chips in three dimensions, forming a 3D torus interconnect topology, such as `2x2x1` or `2x2x2`.
- **TPU v6e** chips are arranged in a 2D torus topology, such as `2x4`.
- **Ironwood (TPU7x)** chips have a direct connection to the nearest neighboring chips in 3 dimensions, forming a 3D torus interconnect topology, such as `2x2x1`.

The numbers in the topology represent the number of chips along each dimension
of the TPU slice.

For more information about the architecture and configurations of these TPUs,
see [TPU v4](https://docs.cloud.google.com/tpu/docs/v4.md.txt#tpu-v4-config), [TPU v5e](https://docs.cloud.google.com/tpu/docs/v5e.md.txt#tpu-v5e-config),
[TPU v5p](https://docs.cloud.google.com/tpu/docs/v5p.md.txt#tpu-v5p-config), [TPU v6e](https://docs.cloud.google.com/tpu/docs/v6e.md.txt#configurations),
and [Ironwood (TPU7x)](https://docs.cloud.google.com/tpu/docs/tpu7x.md.txt#configurations) in the Cloud TPU
documentation.

## Comparison of blueprint options

Cluster Toolkit provides both standard and advanced blueprints for
deploying TPU v6e and Ironwood (TPU7x) clusters, and standard blueprints for
deploying TPU v4, TPU v5e, and TPU v5p clusters.

Both [standard](https://docs.cloud.google.com/cluster-toolkit/docs/deploy/gke/gke-tpu-overview.md.txt#standard-blueprints) and [advanced](https://docs.cloud.google.com/cluster-toolkit/docs/deploy/gke/gke-tpu-overview.md.txt#advanced-blueprints)
blueprints support the following features:

- **Multi-Virtual Private Cloud (VPC) network architecture**: provides the network architecture required for optimized high-throughput inter-chip communication.
- **Networking optimizations** : blueprints configure optimized networking for high-throughput inter-chip communication. Both Cloud TPU v6e and Ironwood (TPU7x) blueprints configure GKE-managed [Dynamic Resource Allocation for Networking
  (DRANET)](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/allocate-network-resources-dra).
- **Dedicated service accounts**: provides dedicated service accounts for the deployment, which helps improve security isolation.
- **Preconfigured workload orchestration and queuing** : Installs and configures [Kueue](https://kueue.sigs.k8s.io/), [JobSet](https://jobset.sigs.k8s.io/), and [Pathways on
  Cloud](https://docs.cloud.google.com/ai-hypercomputer/docs/workloads/pathways-on-cloud/pathways-intro) by default to manage distributed workloads and resource quotas.

For more information about blueprints, see the [Cluster blueprint
overview](https://docs.cloud.google.com/cluster-toolkit/docs/setup/cluster-blueprint).

### Standard blueprints

Standard blueprints provide a functional cluster with a dedicated
Cloud TPU node pool, standard networking configuration, and
preconfigured support for [Kueue](https://kueue.sigs.k8s.io/),
[JobSet](https://jobset.sigs.k8s.io/), and [Pathways on
Cloud](https://docs.cloud.google.com/ai-hypercomputer/docs/workloads/pathways-on-cloud/pathways-intro).

You can view the following standard blueprints on GitHub:

- Cloud TPU v4: [`examples/gke-tpu-v4/gke-tpu-v4.yaml`](https://github.com/GoogleCloudPlatform/cluster-toolkit/blob/main/examples/gke-tpu-v4/gke-tpu-v4.yaml)
- Cloud TPU v5e: [`examples/gke-tpu-v5e/gke-tpu-v5e.yaml`](https://github.com/GoogleCloudPlatform/cluster-toolkit/blob/main/examples/gke-tpu-v5e/gke-tpu-v5e.yaml)
- Cloud TPU v5p: [`examples/gke-tpu-v5p/gke-tpu-v5p.yaml`](https://github.com/GoogleCloudPlatform/cluster-toolkit/blob/main/examples/gke-tpu-v5p/gke-tpu-v5p.yaml)
- Cloud TPU v6e: [`examples/gke-tpu-v6e/gke-tpu-v6e.yaml`](https://github.com/GoogleCloudPlatform/cluster-toolkit/blob/main/examples/gke-tpu-v6e/gke-tpu-v6e.yaml)
- Ironwood (TPU7x): [`examples/gke-tpu-7x/gke-tpu-7x.yaml`](https://github.com/GoogleCloudPlatform/cluster-toolkit/blob/main/examples/gke-tpu-7x/gke-tpu-7x.yaml)

### Advanced blueprints

To optimize high-throughput inter-chip communication and data pipelines,
advanced blueprints add the following features to the standard blueprint:

- **Automatic bucket creation**: creates two Cloud Storage buckets to store training data and checkpoints.
- **Performance-tuned Cloud Storage FUSE mounts** : pre-configures Cloud Storage FUSE mounts in the cluster as Persistent Volumes for optimized high-throughput performance. Cloud Storage FUSE uses [Workload
  Identity](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/workload-identity) access. To learn how to mount Cloud Storage buckets in GKE, see [Access
  Cloud Storage buckets with the Cloud Storage FUSE CSI
  driver](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/persistent-volumes/cloud-storage-fuse-csi-driver).
- **Hyperdisk Balanced support** : Provides highly available and consistent performance across GKE nodes. For more information, see [About
  Hyperdisk for
  GKE](https://docs.cloud.google.com/kubernetes-engine/docs/concepts/hyperdisk).
- **Google Cloud Managed Lustre support** : provides a high-performance, fully managed parallel file system optimized for AI and high performance computing (HPC) workloads. For more information, see the [Google Cloud Managed Lustre
  overview](https://docs.cloud.google.com/managed-lustre/docs/overview).
- **Filestore support** : provides managed Network File System (NFS) capabilities that let multiple Cloud TPU hosts share logs, code, or datasets. For more information, see the [Filestore
  overview](https://docs.cloud.google.com/filestore/docs/overview).

You can view the following advanced blueprints on GitHub:

- Cloud TPU v6e: [`examples/gke-tpu-v6e/gke-tpu-v6e-advanced.yaml`](https://github.com/GoogleCloudPlatform/cluster-toolkit/blob/main/examples/gke-tpu-v6e/gke-tpu-v6e-advanced.yaml)
- Ironwood (TPU7x): [`examples/gke-tpu-7x/gke-tpu-7x-advanced.yaml`](https://github.com/GoogleCloudPlatform/cluster-toolkit/blob/main/examples/gke-tpu-7x/gke-tpu-7x-advanced.yaml)

## Workload orchestration, scheduling, and consumption

Cluster Toolkit supports the following orchestration, scheduling, and
consumption models to help you run distributed workloads and manage resource
availability as follows:

- **[Pathways on
  Cloud](https://docs.cloud.google.com/ai-hypercomputer/docs/workloads/pathways-on-cloud/pathways-intro)** : A distributed execution framework that lets a single JAX client orchestrate multi-slice workloads across remote Cloud TPU workers that are coordinated by a CPU-based head node. Standard and advanced Cloud TPU blueprints enable the `enable_pathways_for_tpus` variable by default. When this variable is enabled, Cluster Toolkit provisions a dedicated CPU node pool (`cpu-np`) to host the coordinator Pod and configures unified quotas for Kueue resource groups (`google.com/tpu`, `cpu`, and `memory`). When you submit a workload with the [`gcluster job
  submit --pathways`
  flag](https://docs.cloud.google.com/cluster-toolkit/docs/reference/gcluster-job-submit#flags), Cluster Toolkit automatically generates a multi-role manifest for a Kubernetes [JobSet](https://jobset.sigs.k8s.io/) that coordinates the Pathways ResourceManager, Interim Framework Runtime (IFRT) proxy, and Cloud TPU workers, with support for headless mode by using the `--pathways-headless` flag.
- **[Kueue](https://kueue.sigs.k8s.io/) and
  [JobSet](https://jobset.sigs.k8s.io/)** : Kubernetes-native systems that manage distributed workloads and resource quotas. Standard and advanced Cloud TPU blueprints install Kueue and the JobSet API by default through the [`kubectl-apply`
  module](https://docs.cloud.google.com/cluster-toolkit/docs/modules/kubectl-apply). JobSet manages distributed, multi-slice Cloud TPU workloads as a unified group of Kubernetes Jobs, while Kueue manages queuing for batch jobs, quota allocation, and [Topology-Aware Scheduling
  (TAS)](https://docs.cloud.google.com/ai-hypercomputer/docs/workloads/schedule-gke-workloads-tas). When you submit a workload by using the [`gcluster job submit`
  command](https://docs.cloud.google.com/cluster-toolkit/docs/reference/gcluster-job-submit), Cluster Toolkit deploys the workload as a Kubernetes JobSet integrated with Kueue. Cluster Toolkit dynamically applies an embedded Kueue configuration based on your Pathways and dynamic slicing settings, calculates and sets a Cloud TPU quota in the `ClusterQueue` resource in Kueue to match the capacity of your cluster, and configures `WorkloadPriorityClass` resources that you can select with the [`--priority`
  flag](https://docs.cloud.google.com/cluster-toolkit/docs/reference/gcluster-job-submit#flags) for workload preemption.
- **[Dynamic slicing](https://docs.cloud.google.com/kubernetes-engine/docs/concepts/dynamic-slicing.md.txt)** : On Ironwood (TPU7x) clusters, setting the `enable_dynamic_slicing_for_tpus` variable to `true` in your blueprint lets you dynamically group or partition physical cubes on Ironwood (TPU7x) at runtime with [Topology-Aware Scheduling
  (TAS)](https://docs.cloud.google.com/ai-hypercomputer/docs/workloads/schedule-gke-workloads-tas).
- **Compute consumption options** : You can configure standard or future reservations, Dynamic Workload Scheduler flex-start, Dynamic Workload Scheduler flex-start with queued provisioning, Spot VMs, or on-demand capacity in your deployment file to balance capacity availability, cost, and provisioning speed. For configuration steps, see [GKE consumption
  options](https://docs.cloud.google.com/cluster-toolkit/docs/deploy/gke/gke-consumption-options) and [About GPU, TPU, and H4D consumption with
  flex-start provisioning mode](https://docs.cloud.google.com/kubernetes-engine/docs/concepts/dws).

## What's next

- To learn about all supported compute consumption models and placement options, see [GKE consumption
  options](https://docs.cloud.google.com/cluster-toolkit/docs/deploy/gke/gke-consumption-options).
- To learn how to deploy a cluster with a dedicated Cloud TPU v4 node pool, see [Create a TPU v4 GKE
  cluster](https://docs.cloud.google.com/cluster-toolkit/docs/deploy/gke/gke-tpu-v4).
- To learn how to deploy a cluster with a dedicated Cloud TPU v5e node pool, see [Create a TPU v5e GKE
  cluster](https://docs.cloud.google.com/cluster-toolkit/docs/deploy/gke/gke-tpu-v5e).
- To learn how to deploy a cluster with a dedicated Cloud TPU v5p node pool, see [Create a TPU v5p GKE
  cluster](https://docs.cloud.google.com/cluster-toolkit/docs/deploy/gke/gke-tpu-v5p).
- To learn how to deploy a cluster with a dedicated Cloud TPU v6e node pool, see [Create a TPU v6e GKE
  cluster](https://docs.cloud.google.com/cluster-toolkit/docs/deploy/gke/gke-tpu-v6e).
- To learn how to deploy a cluster with a dedicated Ironwood (TPU7x) node pool, see [Create an Ironwood (TPU7x) GKE
  cluster](https://docs.cloud.google.com/cluster-toolkit/docs/deploy/gke/gke-tpu-7x).