This document describes how to deploy an A4X Max Slurm cluster. These clusters
provide direct access to the host server's CPU and memory, optimizing your
environment for compute-intensive, network-bound machine learning (ML) training
and high performance computing (HPC) workloads.

The A4X Max machine series runs on an exascale platform based on NVIDIA's
rack-scale architecture and is optimized for compute and memory-intensive,
network-bound ML training and HPC workloads. A4X Max offers bare metal
instances, which provide direct access to the host server's CPU and memory,
without the Compute Engine hypervisor layer.

To learn more about the A4X machine series, see the [A4X Max and A4X machine
series](https://docs.cloud.google.com/compute/docs/accelerator-optimized-machines#a4x-vms) section in the
Compute Engine documentation.

## A4X Max limitations

The following limitations apply to A4X Max bare metal instances in your Slurm
cluster:

> [!CAUTION]
> **Caution:** The [Compute Engine Service Level Agreement (SLA)](https://cloud.google.com/compute/sla) doesn't apply to the A4X Max machine series.

- You don't receive [sustained
  use discounts](https://docs.cloud.google.com/compute/docs/sustained-use-discounts) or [flexible committed use](https://docs.cloud.google.com/compute/docs/instances/committed-use-discounts-overview.md.txt#spend_based) discounts for instances that use this machine type.
- You can only create instances in certain [regions and zones](https://docs.cloud.google.com/compute/docs/regions-zones/gpu-regions-zones.md.txt#view-using-table).
- You can't use Persistent Disk (regional or zonal). You can only use [Google Cloud Hyperdisk](https://docs.cloud.google.com/compute/docs/disks/hyperdisks.md.txt).
- This machine type is only available on the [NVIDIA Grace platform](https://docs.cloud.google.com/compute/docs/cpu-platforms.md.txt#arm_processors).
- Machine type changes aren't supported for A4X Max. To switch to or from this machine type, you must create a new instance.
- You can't run Windows operating systems on this machine type. For a list of supported Linux operating systems, review the [supported operating systems for GPU instances](https://docs.cloud.google.com/compute/docs/images/os-details.md.txt#gpu-support).
- A4X Max instances don't support the following:
  - [Sole-tenancy](https://docs.cloud.google.com/compute/docs/nodes/sole-tenant-nodes.md.txt)
  - [Spot VMs](https://docs.cloud.google.com/compute/docs/instances/spot.md.txt)
  - [Flex-start VMs](https://docs.cloud.google.com/compute/docs/instances/about-flex-start-vms.md.txt)
- You can't attach Hyperdisk ML disks created before February 4, 2026 to A4X Max machine types.

## Before you begin

Before creating a Slurm cluster, if you haven't already done so, complete the following
steps:

1. **Choose a consumption option** : your choice of consumption option determines how you get and use GPU resources. To learn more, see [Choose a consumption option](https://docs.cloud.google.com/ai-hypercomputer/docs/consumption-models.md.txt).
2. **Obtain capacity** : the process to obtain capacity differs for each consumption option. To learn about the process to obtain capacity for your chosen consumption option, see [Capacity overview](https://docs.cloud.google.com/ai-hypercomputer/docs/obtain-capacity.md.txt).

   > [!NOTE]
   > **Note:** When you request A4X Max or A4X capacity, you obtain it in the *all capacity mode*. This mode is the only supported reservation operational mode for A4X Max or A4X machine types. For more information about all capacity mode, see [Reservation operational mode](https://docs.cloud.google.com/ai-hypercomputer/docs/create/review-configurations.md.txt#reservation-operational-mode).

3. **Enable the Google Cloud Managed Lustre API and verify regional quota and availability** : the default Cluster Toolkit blueprint for A4X Max provisions a [Google Cloud Managed Lustre](https://docs.cloud.google.com/managed-lustre/docs/overview.md.txt) (`lustre.googleapis.com`) instance (default `18,000 GiB` capacity at `500 MBps/TiB` throughput). Before deploying, ensure that the `lustre.googleapis.com` and `servicenetworking.googleapis.com` APIs are enabled and that your target zone supports Managed Lustre (see [Managed Lustre locations](https://docs.cloud.google.com/managed-lustre/docs/locations.md.txt)). Also verify that you have enough Managed Lustre capacity and throughput quota in the target region (see [Managed Lustre quotas](https://docs.cloud.google.com/managed-lustre/docs/quotas.md.txt)).
4. **Install Cluster Toolkit** : to provision Slurm clusters, you must use [Cluster Toolkit](https://docs.cloud.google.com/cluster-toolkit/docs/overview.md.txt) version `v1.104.0` or later.

   To install Cluster Toolkit, see
   [Set up Cluster Toolkit](https://docs.cloud.google.com/cluster-toolkit/docs/setup/configure-environment.md.txt).

In the Google Cloud console, activate Cloud Shell.

[Activate Cloud Shell](https://console.cloud.google.com/?cloudshell=true)


At the bottom of the Google Cloud console, a
[Cloud Shell](https://docs.cloud.google.com/shell/docs/how-cloud-shell-works.md.txt)
session starts and displays a command-line prompt. Cloud Shell is a shell environment
with the Google Cloud CLI
already installed and with values already set for
your current project. It can take a few seconds for the session to initialize.

### Required roles


To get the permissions that
you need to deploy a Slurm cluster,

ask your administrator to grant you the
following IAM roles:

- [Storage Admin](https://docs.cloud.google.com/iam/docs/roles-permissions/storage.md.txt#storage.admin) (`roles/storage.admin`) on your project
- [Compute Instance Admin (v1)](https://docs.cloud.google.com/iam/docs/roles-permissions/compute.md.txt#compute.instanceAdmin.v1) (`roles/compute.instanceAdmin.v1`) on your project
- [Service Account User](https://docs.cloud.google.com/iam/docs/roles-permissions/iam.md.txt#iam.serviceAccountUser) (`roles/iam.serviceAccountUser`) on the service account itself
- To configure private services access and Managed Lustre:
  - [Managed Lustre Admin](https://docs.cloud.google.com/iam/docs/roles-permissions/lustre.md.txt#lustre.admin) (`roles/lustre.admin`) on your project
  - [Compute Network Admin](https://docs.cloud.google.com/iam/docs/roles-permissions/compute.md.txt#compute.networkAdmin) (`roles/compute.networkAdmin`) on your project
- To consume a single-project reservation: [Compute Instance Admin (v1)](https://docs.cloud.google.com/iam/docs/roles-permissions/compute.md.txt#compute.instanceAdmin.v1) (`roles/compute.instanceAdmin.v1`) on your project
- To consume a shared reservation: [Compute Instance Admin (v1)](https://docs.cloud.google.com/iam/docs/roles-permissions/compute.md.txt#compute.instanceAdmin.v1) (`roles/compute.instanceAdmin.v1`) on both the owner project and any consumer project where you want to consume the reservation


For more information about granting roles, see [Manage access to projects, folders, and organizations](https://docs.cloud.google.com/iam/docs/granting-changing-revoking-access.md.txt).


You might also be able to get
the required permissions through [custom
roles](https://docs.cloud.google.com/iam/docs/creating-custom-roles) or other [predefined
roles](https://docs.cloud.google.com/iam/docs/roles-overview#predefined).

## Set up a storage bucket

Cluster blueprints use Terraform modules to provision Cloud infrastructure. A
best practice when working with Terraform is to
[store the state remotely](https://developer.hashicorp.com/terraform/language/state/remote)
in a version enabled file. On Google Cloud, you can create a Cloud Storage
bucket that has versioning enabled.

To create this bucket and enable versioning from the CLI, run the following commands:

```
gcloud storage buckets create gs://BUCKET_NAME \
    --project=PROJECT_ID \
    --default-storage-class=STANDARD --location=BUCKET_REGION \
    --uniform-bucket-level-access
gcloud storage buckets update gs://BUCKET_NAME --versioning
```

Replace the following:

- `BUCKET_NAME`: a name for your Cloud Storage bucket that meets the [bucket naming requirements](https://docs.cloud.google.com/storage/docs/buckets.md.txt#naming).
- `PROJECT_ID`: your project ID.
- `BUCKET_REGION`: any [available location](https://docs.cloud.google.com/storage/docs/locations.md.txt#available-locations).

## Open the Cluster Toolkit directory

To use Slurm with Google Cloud, you must [install
Cluster Toolkit](https://docs.cloud.google.com/cluster-toolkit/docs/setup/configure-environment).
In your terminal, navigate to the Cluster Toolkit directory:

```
cd cluster-toolkit
```

If you cloned the Cluster Toolkit GitHub repository instead of
downloading a prebuilt release, then run the `make` command first to build the
`gcluster` binary.

This cluster deployment requires Cluster Toolkit version
`v1.104.0` or later. Verify your installed version:

```
./gcluster --version
```

## Create a deployment file

Create a deployment file that you can use to specify the
Cloud Storage bucket, set names for your network and subnetwork, and set
deployment variables such as project ID, region, and zone.

To create your deployment file, use a text editor to create a YAML file named
`a4xmax-bm-slurm-deployment.yaml` and add the following content.

> [!TIP]
> **Tip:** For a more detailed example, copy and paste the contents of [`examples/machine-learning/a4x-maxgpu-4g-metal/a4xmax-bm-slurm-deployment.yaml`](https://github.com/GoogleCloudPlatform/cluster-toolkit/blob/main/examples/machine-learning/a4x-maxgpu-4g-metal/a4xmax-bm-slurm-deployment.yaml) into a file in your workspace and edit it.

```
terraform_backend_defaults:
  type: gcs
  configuration:
    bucket: BUCKET_NAME
vars:
  deployment_name: DEPLOYMENT_NAME
  project_id: PROJECT_ID
  region: REGION
  zone: ZONE
  a4x_max_cluster_size: NUMBER_OF_INSTANCES
  a4x_max_reservation_name: RESERVATION_NAME
```

Replace the following:

- `BUCKET_NAME`: the name of your Cloud Storage bucket, which you created in the previous section.
- `DEPLOYMENT_NAME`: a name for your deployment. If creating multiple clusters, ensure that you select a unique name for each one.
- `PROJECT_ID`: your project ID.
- `REGION`: the region that has the reserved machines.
- `ZONE`: the zone where you want to provision the cluster. If you're using a reservation-based consumption option, the region and zone information was provided by your account team when the [capacity was delivered](https://docs.cloud.google.com/ai-hypercomputer/docs/view-reserved-capacity.md.txt).
- `NUMBER_OF_INSTANCES`: the number of A4X Max bare metal instances in your cluster. You can specify any number of instances. However, A4X Max instances are physically interconnected by a [multi-node NVLink system](https://docs.nvidia.com/multi-node-nvlink-systems/mnnvl-user-guide/overview.html) in groups of 18 instances (72 GPUs) to form an NVLink domain.

  For optimal network performance, we recommend that you specify a value that is a multiple
  of 18 instances (for example, 18, 36, or 54). When you create an A4X Max cluster, the A4X Max blueprint automatically
  creates and applies a compact placement policy with a GPU topology of `1x72` for
  each group of 18 instances.
- `RESERVATION_NAME`: the name of your [reservation](https://docs.cloud.google.com/ai-hypercomputer/docs/view-reserved-capacity.md.txt#view-reservation).

## Provision a Slurm cluster

Cluster Toolkit provisions the cluster based on the deployment file
that you created in the previous step and the default cluster blueprint. For
more information about the software that is installed by the blueprint,
including NVIDIA drivers and CUDA, see [Slurm custom
images](https://docs.cloud.google.com/ai-hypercomputer/docs/images#slurm-images).

The A4X Max blueprint integrates Managed Lustre for
high-performance home directory storage (`/home`). The blueprint automatically
configures private service access (PSA) to peer your VPC network
with Managed Lustre. The blueprint also configures Lustre
networking (LNet) on the `eth0` primary network interface. For more information,
see the [Managed Lustre overview](https://docs.cloud.google.com/managed-lustre/docs/overview.md.txt).

Deploy the Slurm cluster. This process can take about 20 to 30 minutes:
To provision the cluster, use the [`./gcluster deploy`
command](https://docs.cloud.google.com/cluster-toolkit/docs/reference/gcluster-deploy) from the
Cluster Toolkit directory. This process can take about 20 to 30
minutes.

```
./gcluster deploy -d a4xmax-bm-slurm-deployment.yaml examples/machine-learning/a4x-maxgpu-4g-metal/a4xmax-bm-slurm-blueprint.yaml
```

> [!NOTE]
> **Note:** A4X Max uses the Arm64 architecture and runs on [bare metal
> instances](https://docs.cloud.google.com/compute/docs/instances/bare-metal-instances), so any software installed on these nodes must be compatible with that architecture.

## Connect to the Slurm cluster

To access your cluster, you must sign in to the Slurm login node. To sign in,
you can use either Google Cloud console or Google Cloud CLI.

### Console

1. Go to the **Compute Engine** \> **VM instances** page.

   [Go to the VM instances page](https://console.cloud.google.com/compute/instances)
2. Locate the login node. The node has a name with the pattern
   `DEPLOYMENT_NAME-login-001`.

3. In the **Connect** column of the login node, click **SSH**.

### gcloud

To connect to the login node, complete the following steps:

1. Identify the login node by using the
   [`gcloud compute instances list` command](https://docs.cloud.google.com/sdk/gcloud/reference/compute/instances/list).

   ```
   gcloud compute instances list \
     --zones=ZONE \
     --filter="name ~ login" \
     --format="value(name)"
   ```

   If the output lists multiple Slurm clusters, you can identify your login
   node by the `DEPLOYMENT_NAME` that you specified.
2. Use the [`gcloud compute ssh` command](https://docs.cloud.google.com/sdk/gcloud/reference/compute/ssh)
   to connect to the login node.

       gcloud compute ssh LOGIN_NODE \
         --zone=ZONE --tunnel-through-iap

   Replace the following:
   - `ZONE`: the zone where the VMs for your cluster are located.
   - `LOGIN_NODE`: the name of the login node, which you identified in the previous step.

## Test network performance on the Slurm cluster

We recommend that you validate the functionality of provisioned clusters. To
do so, use [NCCL/gIB tests](https://docs.cloud.google.com/ai-hypercomputer/docs/nccl/test-slurm.md.txt), which are
[NVIDIA Collective Communications Library (NCCL)
tests](https://developer.nvidia.com/nccl) that are optimized for the Google
environment.

## Redeploy the Slurm cluster

If you need to increase the number of compute nodes or add new partitions to
your cluster, you might need to update configurations for your Slurm cluster by
redeploying. You can speed up a redeployment by using an existing image from a
previous deployment. To avoid creating new images during a redeploy, specify
the `--only` flag.

To redeploy the cluster by using an existing image, do the following:

1. Connect to the cluster. For instructions, see
   [Connect to the Slurm cluster](https://docs.cloud.google.com/cluster-toolkit/docs/deploy/slurm/create-a4x-max-cluster.md.txt#connect-cluster).

2. Use the `./gcluster deploy` command for your required machine type:

   ```
   ./gcluster deploy \
      -d a4xmax-bm-slurm-deployment.yaml \
      examples/machine-learning/a4x-maxgpu-4g-metal/a4xmax-bm-slurm-blueprint.yaml \
      --only cluster-env,cluster -w
   ```

   This command applies only to redeployments when an image already exists; it
   redeploys only the cluster and its infrastructure.

## Destroy the Slurm cluster

When you destroy the Slurm cluster, all provisioned resources are deleted,
including the Managed Lustre instance that the blueprint mounts
at `/home` and all data stored on it. Before you destroy the cluster, back up
any data that you want to keep.

1. If you are connected to the cluster, disconnect from it.

2. In your terminal, navigate to the Cluster Toolkit directory.

3. To destroy the cluster, use the `./gcluster destroy` command:

   ```
   ./gcluster destroy DEPLOYMENT_FOLDER
   ```

   Replace the following:
   - `DEPLOYMENT_FOLDER`: the name of the deployment folder, typically the same as `DEPLOYMENT_NAME`.

   When the destruction completes, the output is similar to the following:

   ```
   Destroy complete! Resources: NUMBER destroyed.
   ```

To learn how to cleanly destroy infrastructure and for advanced manual
deployment instructions, see the deployment folder located at the root of
the Cluster Toolkit directory: `DEPLOYMENT_FOLDER`/instructions.txt

## What's next

- [Verify reservations
  consumption](https://docs.cloud.google.com/compute/docs/instances/reservations-consume#verify-consumption)
- [View topology of a compute
  instance](https://docs.cloud.google.com/ai-hypercomputer/docs/manage/instance-topology)
- Learn how to manage host events:
  - [Manage host events across compute
    instances](https://docs.cloud.google.com/ai-hypercomputer/docs/manage/host-events)
  - [Manage host events across
    reservations](https://docs.cloud.google.com/ai-hypercomputer/docs/manage/host-events-reservations)
- [Monitor Compute Engine instances and Slurm
  clusters](https://docs.cloud.google.com/ai-hypercomputer/docs/monitor)
- [Optimize cluster networking by using
  NCCL/gIB](https://docs.cloud.google.com/ai-hypercomputer/docs/nccl/overview)
- [Report faulty host](https://docs.cloud.google.com/ai-hypercomputer/docs/manage/report-faulty-host.md.txt)