Deploy a Slurm TPU v6e cluster

This document describes how to deploy a Slurm cluster that uses TPU v6e machine types and Google Cloud Managed Lustre storage by using Cluster Toolkit. For information about the cluster architecture, partitions, and slice topologies, see Slurm Cloud TPU deployments overview.

Before you begin

Before you create a Slurm cluster, complete the following prerequisites:

  • Choose a consumption option for how you obtain and use Cloud TPU resources.
  • Verify that you have ct6e-standard-4t capacity in your target zone for your chosen consumption option.
  • Verify that you have enough Managed Lustre quota in your target region. By default, the blueprint provisions 36 TiB of capacity.

  • Set up Cluster Toolkit to provision Slurm clusters.

Set up your command-line environment

In the Google Cloud console, activate Cloud Shell.

Activate Cloud Shell

At the bottom of the Google Cloud console, a Cloud Shell session starts and displays a command-line prompt. Cloud Shell is a shell environment with the Google Cloud CLI already installed and with values already set for your current project. It can take a few seconds for the session to initialize.

Required roles

To get the permissions that you need to deploy a Slurm cluster, ask your administrator to grant you the following IAM roles:

For more information about granting roles, see Manage access to projects, folders, and organizations.

You might also be able to get the required permissions through custom roles or other predefined roles.

Set up a storage bucket

Cluster blueprints use Terraform modules to provision Cloud infrastructure. A best practice when working with Terraform is to store the state remotely in a version enabled file. On Google Cloud, you can create a Cloud Storage bucket that has versioning enabled.

To create this bucket and enable versioning from the CLI, run the following commands:

gcloud storage buckets create gs://BUCKET_NAME \
    --project=PROJECT_ID \
    --default-storage-class=STANDARD --location=BUCKET_REGION \
    --uniform-bucket-level-access
gcloud storage buckets update gs://BUCKET_NAME --versioning

Replace the following:

Configure the deployment file

Configure the deployment file that specifies your Cloud Storage bucket, project and location settings, TPU partition sizes, and consumption option.

To configure the examples/machine-learning/tpu-v5e-v5p-v6e/tpu-slurm-deployment.yaml file, select your consumption option and update the file with the following configuration:

Reservation-bound

In this configuration, you provision TPU nodes by using capacity from a specific local or shared reservation:

---
terraform_backend_defaults:
  type: gcs
  configuration:
    bucket: BUCKET_NAME

vars:
  deployment_name: DEPLOYMENT_NAME
  project_id: PROJECT_ID
  region: REGION
  zone: ZONE
  tpu_machine_type: ct6e-standard-4t
  tpu_static_nodes: TPU_STATIC_NODES
  tpu_dynamic_max_nodes: TPU_DYNAMIC_MAX_NODES
  tpu_accelerator_topology: TPU_ACCELERATOR_TOPOLOGY
  tpu_reservation_name: RESERVATION_NAME

DWS Flex-start

In this configuration, you use Dynamic Workload Scheduler with Flex-start mode to request TPU capacity on demand for up to seven days. For more information about how to use Dynamic Workload Scheduler with Flex-start mode in Slurm, see Dynamic Workload Scheduler with Flex-start mode in Slurm on GitHub.

---
terraform_backend_defaults:
  type: gcs
  configuration:
    bucket: BUCKET_NAME

vars:
  deployment_name: DEPLOYMENT_NAME
  project_id: PROJECT_ID
  region: REGION
  zone: ZONE
  tpu_machine_type: ct6e-standard-4t
  tpu_static_nodes: TPU_STATIC_NODES
  tpu_dynamic_max_nodes: TPU_DYNAMIC_MAX_NODES
  tpu_accelerator_topology: TPU_ACCELERATOR_TOPOLOGY
  tpu_dws_flex_enabled: true

Spot VMs

In this configuration, you provision TPU nodes as preemptible Spot VMs at a discounted price for fault-tolerant workloads:

---
terraform_backend_defaults:
  type: gcs
  configuration:
    bucket: BUCKET_NAME

vars:
  deployment_name: DEPLOYMENT_NAME
  project_id: PROJECT_ID
  region: REGION
  zone: ZONE
  tpu_machine_type: ct6e-standard-4t
  tpu_static_nodes: TPU_STATIC_NODES
  tpu_dynamic_max_nodes: TPU_DYNAMIC_MAX_NODES
  tpu_accelerator_topology: TPU_ACCELERATOR_TOPOLOGY
  tpu_enable_spot_vm: true

Replace the following:

  • BUCKET_NAME: the name of the Cloud Storage bucket that you created to store Terraform state.
  • DEPLOYMENT_NAME: a unique name for your deployment, such as tpu-v6e-slurm.
  • PROJECT_ID: the ID of your Google Cloud project.
  • REGION: the region where your cluster is deployed, such as us-central1.
  • ZONE: the zone where your cluster is deployed, such as us-central1-b.
  • TPU_STATIC_NODES: the number of static TPU nodes in the default tpu partition, such as 8. This value must be a multiple of the number of compute instances per slice for the TPU_ACCELERATOR_TOPOLOGY value. Because each ct6e-standard-4t instance contains 4 TPU chips, you can calculate the number of instances per slice by using the following formula:

    Instances per slice = total chips in topology / 4
    

    For example, when you set the tpu_static_nodes field to 8:

    • A 4x4 topology (16 chips) provisions two 4-instance slices.
    • A 4x8 topology (32 chips) provisions one 8-instance slice.
  • TPU_DYNAMIC_MAX_NODES: the maximum number of on-demand dynamic TPU nodes in the tpu-dyn partition, such as 4.

  • TPU_ACCELERATOR_TOPOLOGY: the 2D topology per slice for the static tpu partition, such as the 4x8 or 4x4 topology. For supported topologies and their compute instance counts, see TPU topologies.

  • RESERVATION_NAME: the name of a reservation in the same project, or the full path of a shared reservation: projects/HOST_PROJECT_ID/reservations/SHARED_RESERVATION_NAME.

By default, the blueprint provisions on-demand capacity, and it consumes a reservation only when you specify a value for the tpu_reservation_name field in the examples/machine-learning/tpu-v5e-v5p-v6e/tpu-slurm-deployment.yaml file. If you want to provision on-demand capacity without a reservation, then omit the tpu_reservation_name field or set it to "". For more information about these provisioning models, see Consumption options.

Provision a Slurm cluster

To provision the cluster, run the ./gcluster deploy command from the cluster-toolkit directory. This step takes approximately 20 to 30 minutes:

./gcluster deploy \
    -d examples/machine-learning/tpu-v5e-v5p-v6e/tpu-slurm-deployment.yaml \
    examples/machine-learning/tpu-v5e-v5p-v6e/tpu-slurm-blueprint.yaml

Connect to the Slurm cluster

To access your cluster, sign in to the login node for Slurm by using either the Google Cloud console or the Google Cloud CLI:

Console

  1. In the Google Cloud console, go to the Compute Engine > VM instances page.

    Go to VM instances

  2. Locate the login node, which uses the TRUNCATED_DEPLOYMENT_NAME-slurm-login-001 naming pattern, such as tpuv6eslur-slurm-login-001.

  3. In the Connect column for the login node, click SSH.

gcloud

  1. Look up the name of the login node by using the gcloud compute instances list command:

    gcloud compute instances list --zones=ZONE --filter="name ~ login" --format="value(name)"
    

    Replace ZONE with the zone where your cluster is deployed.

  2. Connect to the login node by using the gcloud compute ssh command:

    gcloud compute ssh LOGIN_NODE_NAME \
        --project=PROJECT_ID \
        --zone=ZONE \
        --tunnel-through-iap
    

    Replace the following:

    • LOGIN_NODE_NAME: the name of the login node from the previous step, such as tpuv6eslur-slurm-login-001.
    • PROJECT_ID: the ID of your Google Cloud project.
    • ZONE: the zone where your cluster is deployed.

Verify the cluster

After you connect to the login node, complete the following steps to verify your partitions and run a test job with JAX:

  1. Verify that the tpu and tpu-dyn partitions are available:

    sinfo
    
  2. Create a shared virtual environment for Python in the /home directory and install JAX with TPU support:

    sudo mkdir -p "$HOME" && sudo chown "$(id -u):$(id -g)" "$HOME"
    python3 -m venv ~/jax-env
    ~/jax-env/bin/pip install -U pip "jax[tpu]" \
        -f https://storage.googleapis.com/jax-releases/libtpu_releases.html
    
  3. Run a verification job on either the tpu static partition or the tpu-dyn dynamic partition by using the --layout flag:

    • Static partition: set the --layout flag to match the tpu_accelerator_topology variable in your deployment file, and set the -N flag to the number of compute instances in that slice. For example, run the job on 8 nodes for a 4x8 topology:

      srun -N 8 -p tpu --layout=v6e=4x8 ~/jax-env/bin/python -c \
          "import jax; jax.distributed.initialize(); print(f'Host {jax.process_index()}: total={jax.device_count()}, local={jax.local_device_count()}')"
      
    • Dynamic partition: set the --layout flag to the slice topology that you want Slurm to provision on demand, and set the -N flag to the number of compute instances in that slice. For example, run the job on 2 nodes for a 2x4 topology:

      srun -N 2 -p tpu-dyn --layout=v6e=2x4 ~/jax-env/bin/python -c \
          "import jax; jax.distributed.initialize(); print(f'Host {jax.process_index()}: total={jax.device_count()}, local={jax.local_device_count()}')"
      

Redeploy the Slurm cluster

To update your Slurm cluster configuration and keep the existing VPC network and Managed Lustre instance in the cluster-env group, specify the --only cluster flag when you redeploy:

./gcluster deploy \
    -d examples/machine-learning/tpu-v5e-v5p-v6e/tpu-slurm-deployment.yaml \
    examples/machine-learning/tpu-v5e-v5p-v6e/tpu-slurm-blueprint.yaml \
    --only cluster -w

To change the tpu_accelerator_topology variable on a deployed static nodeset and keep your cluster-env network and Managed Lustre instance, destroy and redeploy the cluster group:

./gcluster destroy DEPLOYMENT_FOLDER --only cluster
./gcluster deploy \
    -d examples/machine-learning/tpu-v5e-v5p-v6e/tpu-slurm-deployment.yaml \
    examples/machine-learning/tpu-v5e-v5p-v6e/tpu-slurm-blueprint.yaml \
    --only cluster -w

Replace DEPLOYMENT_FOLDER with the name of your deployment folder, which is typically the same as the DEPLOYMENT_NAME value.

Clean up

To avoid incurring charges to your Google Cloud account for the resources used in this document, destroy the Slurm cluster and delete the state bucket if you no longer need it.

Destroy the Slurm cluster

  1. If you are connected to the login node, then disconnect from it.
  2. Navigate to the root of the cluster-toolkit directory.
  3. Destroy the cluster by using the ./gcluster destroy command:

    ./gcluster destroy DEPLOYMENT_FOLDER
    

    Replace DEPLOYMENT_FOLDER with the name of your deployment folder, which is typically the same as the DEPLOYMENT_NAME value.

Optional: Delete the storage bucket

Because the ./gcluster destroy command preserves your Terraform state bucket, delete the Cloud Storage bucket when you no longer need the deployment state.

What's next