Cloud TPU deployments overview

This document explains the core concepts and advanced configurations available when you use Cluster Toolkit to deploy Google Kubernetes Engine (GKE) clusters with dedicated Cloud TPU node pools.

When you use Cluster Toolkit to provision GKE clusters, you can deploy dedicated Cloud TPU node pools designed for artificial intelligence (AI) and machine learning (ML) workloads. Cluster Toolkit provides blueprints to automate the deployment of supported Cloud TPU architectures.

This document explains the core concepts and advanced configurations that are available when you deploy Cloud TPU node pools with Cluster Toolkit, including hardware topologies, multi-slice configurations, storage integration, and compute consumption models.

Key Cloud TPU concepts

To deploy a Cloud TPU cluster, you must understand how Cloud TPU hardware is grouped and allocated. You create Cloud TPU slices as a single, fixed hardware unit with physical, high-speed interconnects between all nodes.

Cloud TPU in GKE

TPUs are Google's custom-developed application-specific integrated circuits (ASICs) that are used to accelerate machine learning workloads. GKE provides full support for lifecycle management of Cloud TPU nodes and node pools, including node creation, configuration, and automatic upgrades.

For more information about how Cloud TPU works with GKE, see About TPUs in GKE.

TPU slices

A Cloud TPU slice is a collection of Cloud TPU chips that are physically connected by a dedicated, ultra-low-latency network called the Inter-Chip Interconnect (ICI). This interconnect lets the nodes function as a single supercomputer.

In Cluster Toolkit, the num_slices parameter specifies the number of identical, independent Cloud TPU slices to create. Each slice corresponds to an individual node pool.

TPU topologies

The tpu_topology parameter defines the shape and total size of your Cloud TPU slice. Topologies describe the number and physical arrangement of the Cloud TPU chips in a slice.

  • TPU v4 chips have a direct connection to the nearest neighboring chips in three dimensions, forming a 3D torus interconnect topology, such as 2x2x2.
  • TPU v5e chips are arranged in a 2D topology, such as 2x2 or 2x4.
  • TPU v5p chips have a direct connection to the nearest neighboring chips in three dimensions, forming a 3D torus interconnect topology, such as 2x2x1 or 2x2x2.
  • TPU v6e chips are arranged in a 2D torus topology, such as 2x4.
  • Ironwood (TPU7x) chips have a direct connection to the nearest neighboring chips in 3 dimensions, forming a 3D torus interconnect topology, such as 2x2x1.

The numbers in the topology represent the number of chips along each dimension of the TPU slice.

For more information about the architecture and configurations of these TPUs, see TPU v4, TPU v5e, TPU v5p, TPU v6e, and Ironwood (TPU7x) in the Cloud TPU documentation.

Comparison of blueprint options

Cluster Toolkit provides both standard and advanced blueprints for deploying TPU v6e and Ironwood (TPU7x) clusters, and standard blueprints for deploying TPU v4, TPU v5e, and TPU v5p clusters.

Both standard and advanced blueprints support the following features:

  • Multi-Virtual Private Cloud (VPC) network architecture: provides the network architecture required for optimized high-throughput inter-chip communication.
  • Networking optimizations: blueprints configure optimized networking for high-throughput inter-chip communication. Both Cloud TPU v6e and Ironwood (TPU7x) blueprints configure GKE-managed Dynamic Resource Allocation for Networking (DRANET).
  • Dedicated service accounts: provides dedicated service accounts for the deployment, which helps improve security isolation.
  • Preconfigured workload orchestration and queuing: Installs and configures Kueue, JobSet, and Pathways on Cloud by default to manage distributed workloads and resource quotas.

For more information about blueprints, see the Cluster blueprint overview.

Standard blueprints

Standard blueprints provide a functional cluster with a dedicated Cloud TPU node pool, standard networking configuration, and preconfigured support for Kueue, JobSet, and Pathways on Cloud.

You can view the following standard blueprints on GitHub:

Advanced blueprints

To optimize high-throughput inter-chip communication and data pipelines, advanced blueprints add the following features to the standard blueprint:

  • Automatic bucket creation: creates two Cloud Storage buckets to store training data and checkpoints.
  • Performance-tuned Cloud Storage FUSE mounts: pre-configures Cloud Storage FUSE mounts in the cluster as Persistent Volumes for optimized high-throughput performance. Cloud Storage FUSE uses Workload Identity access. To learn how to mount Cloud Storage buckets in GKE, see Access Cloud Storage buckets with the Cloud Storage FUSE CSI driver.
  • Hyperdisk Balanced support: Provides highly available and consistent performance across GKE nodes. For more information, see About Hyperdisk for GKE.
  • Google Cloud Managed Lustre support: provides a high-performance, fully managed parallel file system optimized for AI and high performance computing (HPC) workloads. For more information, see the Google Cloud Managed Lustre overview.
  • Filestore support: provides managed Network File System (NFS) capabilities that let multiple Cloud TPU hosts share logs, code, or datasets. For more information, see the Filestore overview.

You can view the following advanced blueprints on GitHub:

Workload orchestration, scheduling, and consumption

Cluster Toolkit supports the following orchestration, scheduling, and consumption models to help you run distributed workloads and manage resource availability as follows:

  • Pathways on Cloud: A distributed execution framework that lets a single JAX client orchestrate multi-slice workloads across remote Cloud TPU workers that are coordinated by a CPU-based head node. Standard and advanced Cloud TPU blueprints enable the enable_pathways_for_tpus variable by default. When this variable is enabled, Cluster Toolkit provisions a dedicated CPU node pool (cpu-np) to host the coordinator Pod and configures unified quotas for Kueue resource groups (google.com/tpu, cpu, and memory). When you submit a workload with the gcluster job submit --pathways flag, Cluster Toolkit automatically generates a multi-role manifest for a Kubernetes JobSet that coordinates the Pathways ResourceManager, Interim Framework Runtime (IFRT) proxy, and Cloud TPU workers, with support for headless mode by using the --pathways-headless flag.
  • Kueue and JobSet: Kubernetes-native systems that manage distributed workloads and resource quotas. Standard and advanced Cloud TPU blueprints install Kueue and the JobSet API by default through the kubectl-apply module. JobSet manages distributed, multi-slice Cloud TPU workloads as a unified group of Kubernetes Jobs, while Kueue manages queuing for batch jobs, quota allocation, and Topology-Aware Scheduling (TAS). When you submit a workload by using the gcluster job submit command, Cluster Toolkit deploys the workload as a Kubernetes JobSet integrated with Kueue. Cluster Toolkit dynamically applies an embedded Kueue configuration based on your Pathways and dynamic slicing settings, calculates and sets a Cloud TPU quota in the ClusterQueue resource in Kueue to match the capacity of your cluster, and configures WorkloadPriorityClass resources that you can select with the --priority flag for workload preemption.
  • Dynamic slicing: On Ironwood (TPU7x) clusters, setting the enable_dynamic_slicing_for_tpus variable to true in your blueprint lets you dynamically group or partition physical cubes on Ironwood (TPU7x) at runtime with Topology-Aware Scheduling (TAS).
  • Compute consumption options: You can configure standard or future reservations, Dynamic Workload Scheduler flex-start, Dynamic Workload Scheduler flex-start with queued provisioning, Spot VMs, or on-demand capacity in your deployment file to balance capacity availability, cost, and provisioning speed. For configuration steps, see GKE consumption options and About GPU, TPU, and H4D consumption with flex-start provisioning mode.

What's next