This document explains the core concepts and advanced configurations available when you use Cluster Toolkit to deploy Google Kubernetes Engine (GKE) clusters with dedicated Cloud TPU node pools.
When you use Cluster Toolkit to provision GKE clusters, you can deploy dedicated Cloud TPU node pools designed for artificial intelligence (AI) and machine learning (ML) workloads. Cluster Toolkit provides blueprints to automate the deployment of supported Cloud TPU architectures.
This document explains the core concepts and advanced configurations that are available when you deploy Cloud TPU node pools with Cluster Toolkit, including hardware topologies, multi-slice configurations, storage integration, and compute consumption models.
Key Cloud TPU concepts
To deploy a Cloud TPU cluster, you must understand how Cloud TPU hardware is grouped and allocated. You create Cloud TPU slices as a single, fixed hardware unit with physical, high-speed interconnects between all nodes.
Cloud TPU in GKE
TPUs are Google's custom-developed application-specific integrated circuits (ASICs) that are used to accelerate machine learning workloads. GKE provides full support for lifecycle management of Cloud TPU nodes and node pools, including node creation, configuration, and automatic upgrades.
For more information about how Cloud TPU works with GKE, see About TPUs in GKE.
TPU slices
A Cloud TPU slice is a collection of Cloud TPU chips that are physically connected by a dedicated, ultra-low-latency network called the Inter-Chip Interconnect (ICI). This interconnect lets the nodes function as a single supercomputer.
In Cluster Toolkit, the num_slices parameter specifies the number of
identical, independent Cloud TPU slices to create. Each slice
corresponds to an individual node pool.
TPU topologies
The tpu_topology parameter defines the shape and total size of your
Cloud TPU slice. Topologies describe the number and physical
arrangement of the
Cloud TPU chips in a slice.
- TPU v4 chips have a direct connection to the nearest
neighboring chips in three dimensions, forming a 3D torus interconnect topology,
such as
2x2x2. - TPU v5e chips are arranged in a 2D topology, such as
2x2or2x4. - TPU v5p chips have a direct connection to the nearest
neighboring chips in three dimensions, forming a 3D torus interconnect topology,
such as
2x2x1or2x2x2. - TPU v6e chips are arranged in a 2D torus topology, such
as
2x4. - Ironwood (TPU7x) chips have a direct connection to the nearest
neighboring chips in 3 dimensions, forming a 3D torus interconnect topology,
such as
2x2x1.
The numbers in the topology represent the number of chips along each dimension of the TPU slice.
For more information about the architecture and configurations of these TPUs, see TPU v4, TPU v5e, TPU v5p, TPU v6e, and Ironwood (TPU7x) in the Cloud TPU documentation.
Comparison of blueprint options
Cluster Toolkit provides both standard and advanced blueprints for deploying TPU v6e and Ironwood (TPU7x) clusters, and standard blueprints for deploying TPU v4, TPU v5e, and TPU v5p clusters.
Both standard and advanced blueprints support the following features:
- Multi-Virtual Private Cloud (VPC) network architecture: provides the network architecture required for optimized high-throughput inter-chip communication.
- Networking optimizations: blueprints configure optimized networking for high-throughput inter-chip communication. Both Cloud TPU v6e and Ironwood (TPU7x) blueprints configure GKE-managed Dynamic Resource Allocation for Networking (DRANET).
- Dedicated service accounts: provides dedicated service accounts for the deployment, which helps improve security isolation.
- Preconfigured workload orchestration and queuing: Installs and configures Kueue, JobSet, and Pathways on Cloud by default to manage distributed workloads and resource quotas.
For more information about blueprints, see the Cluster blueprint overview.
Standard blueprints
Standard blueprints provide a functional cluster with a dedicated Cloud TPU node pool, standard networking configuration, and preconfigured support for Kueue, JobSet, and Pathways on Cloud.
You can view the following standard blueprints on GitHub:
- Cloud TPU v4:
examples/gke-tpu-v4/gke-tpu-v4.yaml - Cloud TPU v5e:
examples/gke-tpu-v5e/gke-tpu-v5e.yaml - Cloud TPU v5p:
examples/gke-tpu-v5p/gke-tpu-v5p.yaml - Cloud TPU v6e:
examples/gke-tpu-v6e/gke-tpu-v6e.yaml - Ironwood (TPU7x):
examples/gke-tpu-7x/gke-tpu-7x.yaml
Advanced blueprints
To optimize high-throughput inter-chip communication and data pipelines, advanced blueprints add the following features to the standard blueprint:
- Automatic bucket creation: creates two Cloud Storage buckets to store training data and checkpoints.
- Performance-tuned Cloud Storage FUSE mounts: pre-configures Cloud Storage FUSE mounts in the cluster as Persistent Volumes for optimized high-throughput performance. Cloud Storage FUSE uses Workload Identity access. To learn how to mount Cloud Storage buckets in GKE, see Access Cloud Storage buckets with the Cloud Storage FUSE CSI driver.
- Hyperdisk Balanced support: Provides highly available and consistent performance across GKE nodes. For more information, see About Hyperdisk for GKE.
- Google Cloud Managed Lustre support: provides a high-performance, fully managed parallel file system optimized for AI and high performance computing (HPC) workloads. For more information, see the Google Cloud Managed Lustre overview.
- Filestore support: provides managed Network File System (NFS) capabilities that let multiple Cloud TPU hosts share logs, code, or datasets. For more information, see the Filestore overview.
You can view the following advanced blueprints on GitHub:
- Cloud TPU v6e:
examples/gke-tpu-v6e/gke-tpu-v6e-advanced.yaml - Ironwood (TPU7x):
examples/gke-tpu-7x/gke-tpu-7x-advanced.yaml
Workload orchestration, scheduling, and consumption
Cluster Toolkit supports the following orchestration, scheduling, and consumption models to help you run distributed workloads and manage resource availability as follows:
- Pathways on
Cloud:
A distributed execution framework that lets a single JAX client orchestrate
multi-slice workloads across remote Cloud TPU workers that are
coordinated by a CPU-based head node. Standard and advanced
Cloud TPU blueprints enable the
enable_pathways_for_tpusvariable by default. When this variable is enabled, Cluster Toolkit provisions a dedicated CPU node pool (cpu-np) to host the coordinator Pod and configures unified quotas for Kueue resource groups (google.com/tpu,cpu, andmemory). When you submit a workload with thegcluster job submit --pathwaysflag, Cluster Toolkit automatically generates a multi-role manifest for a Kubernetes JobSet that coordinates the Pathways ResourceManager, Interim Framework Runtime (IFRT) proxy, and Cloud TPU workers, with support for headless mode by using the--pathways-headlessflag. - Kueue and
JobSet: Kubernetes-native systems that
manage distributed workloads and resource quotas. Standard and advanced
Cloud TPU blueprints install Kueue and the JobSet API by default
through the
kubectl-applymodule. JobSet manages distributed, multi-slice Cloud TPU workloads as a unified group of Kubernetes Jobs, while Kueue manages queuing for batch jobs, quota allocation, and Topology-Aware Scheduling (TAS). When you submit a workload by using thegcluster job submitcommand, Cluster Toolkit deploys the workload as a Kubernetes JobSet integrated with Kueue. Cluster Toolkit dynamically applies an embedded Kueue configuration based on your Pathways and dynamic slicing settings, calculates and sets a Cloud TPU quota in theClusterQueueresource in Kueue to match the capacity of your cluster, and configuresWorkloadPriorityClassresources that you can select with the--priorityflag for workload preemption. - Dynamic slicing: On
Ironwood (TPU7x) clusters, setting the
enable_dynamic_slicing_for_tpusvariable totruein your blueprint lets you dynamically group or partition physical cubes on Ironwood (TPU7x) at runtime with Topology-Aware Scheduling (TAS). - Compute consumption options: You can configure standard or future reservations, Dynamic Workload Scheduler flex-start, Dynamic Workload Scheduler flex-start with queued provisioning, Spot VMs, or on-demand capacity in your deployment file to balance capacity availability, cost, and provisioning speed. For configuration steps, see GKE consumption options and About GPU, TPU, and H4D consumption with flex-start provisioning mode.
What's next
- To learn about all supported compute consumption models and placement options, see GKE consumption options.
- To learn how to deploy a cluster with a dedicated Cloud TPU v4 node pool, see Create a TPU v4 GKE cluster.
- To learn how to deploy a cluster with a dedicated Cloud TPU v5e node pool, see Create a TPU v5e GKE cluster.
- To learn how to deploy a cluster with a dedicated Cloud TPU v5p node pool, see Create a TPU v5p GKE cluster.
- To learn how to deploy a cluster with a dedicated Cloud TPU v6e node pool, see Create a TPU v6e GKE cluster.
- To learn how to deploy a cluster with a dedicated Ironwood (TPU7x) node pool, see Create an Ironwood (TPU7x) GKE cluster.