Create a GKE Cluster with Pathways

You can use Cluster Toolkit to create pre-configured Google Kubernetes Engine (GKE) clusters for Pathways-based workloads. You can also use gcloud to manually create GKE clusters for Pathways-based workloads.

Before you begin

Make sure you have:

Set up your local environment

gcloud auth application-default login

Define the following environment variables with values appropriate to your workload.

Create a GKE cluster

In the following example, you create a cluster with TPU v6e 2x4 node pools. You can create a cluster by using Cluster Toolkit or the gcloud CLI.

Cluster Toolkit

Set up Cluster Toolkit.
Deploy a GKE TPU cluster by following the instructions in Deploy a GKE TPU v6e cluster (for an overview of all TPU deployment options, see Cloud TPU deployments overview).

When configuring your deployment blueprint, do the following:
- Ensure that enable_pathways_for_tpus is set to true (enabled by default in Cluster Toolkit TPU blueprints). This automatically provisions the required cpu-np CPU node pool (n4-standard-64) and Kueue configurations for the Pathways controller.
- Set num_slices to the number of TPU slice node pools required for your workload.
- Set tpu_topology to your selected TPU slice topology (for example, 2x4 or 4x4).
- If you're using reserved capacity, set reservation to your TPU reservation name.
After the cluster is created, you can submit Pathways workloads by using the gcluster job submit command with the --pathways flag.

gcloud

Set some environment variables
```
CLUSTER=GKE_CLUSTER_NAME
PROJECT=PROJECT_ID
ZONE=ZONE
REGION=REGION
CLUSTER_VERSION=GKE_CLUSTER_VERSION
PW_CPU_MACHINE_TYPE="n2-standard-64"
NETWORK=NETWORK
SUBNETWORK=SUB_NETWORK
CLUSTER_NODEPOOL_COUNT=3
TPU_MACHINE_TYPE="ct6e-standard-4t"
WORKERS_PER_SLICE=2
TOPOLOGY="2x4"
NUM_CPU_NODES=1
```
Replace the following:
- CLUSTER: the GKE cluster name
- PROJECT_ID: your Google Cloud project name
- ZONE: the zone where you are creating resources
- REGION: the region where you are creating resources
- CLUSTER_VERSION: [Optional] the GKE cluster version, use 1.32.2-gke.1475000 or later
- NETWORK: [Optional] set a Virtual Private Cloud name, this must be created before creating your cluster
- SUBNETWORK: [Optional] set a subnetwork name, this must be created before creating your cluster

The following steps explain how to create a GKE cluster and set it up for running Pathways workloads.

Create a GKE cluster:

gcloud beta container clusters create ${CLUSTER} \
--project=${PROJECT} \
--zone=${ZONE} \
--cluster-version=${CLUSTER_VERSION} \
--scopes=storage-full,gke-default,cloud-platform \
--machine-type ${PW_CPU_MACHINE_TYPE} \
--network=${NETWORK} \
--subnetwork=${SUBNETWORK}

Create TPU node pools:

for i in $(seq 1 ${CLUSTER_NODEPOOL_COUNT}); do
gcloud container node-pools create "tpu-np-${i}" \
--project=${PROJECT} \
--zone=${ZONE} \
--cluster=${CLUSTER} \
--machine-type=${TPU_MACHINE_TYPE} \
--num-nodes=${WORKERS_PER_SLICE} \
--placement-type=COMPACT \
--tpu-topology=${TOPOLOGY} \
--scopes=storage-full,gke-default,cloud-platform \
--workload-metadata=GCE_METADATA
done

Create a CPU node pool:

gcloud container node-pools create "cpu-pathways-np" \
--project ${PROJECT} \
--zone ${ZONE} \
--cluster ${CLUSTER} \
--machine-type ${PW_CPU_MACHINE_TYPE} \
--num-nodes ${NUM_CPU_NODES} \
--scopes=storage-full,gke-default,cloud-platform \
--workload-metadata=GCE_METADATA

Install the JobSet API

Get credentials for the cluster and add them to your local kubectl context.

gcloud container clusters get-credentials ${CLUSTER} \
    [--zone=${ZONE} | --region=${REGION}] \
    --project=${PROJECT} \
    && kubectl config set-context --current --namespace=default

To use the Pathways architecture on your GKE cluster, you need to install the JobSet API.

kubectl apply --server-side -f https://github.com/kubernetes-sigs/jobset/releases/download/v0.8.0/manifests.yaml