This document describes how to deploy a GKE cluster that uses A4X
accelerator-optimized machine types by using Cluster Toolkit.

The A4X machine series runs on an exascale platform based on NVIDIA's rack-scale
architecture using NVIDIA GB200 Grace Blackwell Superchips and is optimized for
compute and memory-intensive, network-bound ML training and HPC workloads.

To learn more about the A4X machine series, see the [A4X Max and A4X machine
series](https://docs.cloud.google.com/compute/docs/accelerator-optimized-machines#a4x-vms) section in the
Compute Engine documentation.

## Before you begin


Before you start, make sure that you have performed the following tasks:

- Enable the Google Kubernetes Engine API.
[Enable Google Kubernetes Engine API](https://console.cloud.google.com/apis/enableflow?apiid=container.googleapis.com)
- To use the Google Cloud CLI for this task, [install](https://docs.cloud.google.com/sdk/docs/install) and then [initialize](https://docs.cloud.google.com/sdk/docs/initialize.md.txt) the gcloud CLI. If you previously installed the gcloud CLI, get the latest version by running the `gcloud components update` command. Earlier gcloud CLI versions might not support running the commands in this document.

  > [!NOTE]
  > **Note:** For existing gcloud CLI installations, make sure to set the `compute/region` [property](https://docs.cloud.google.com/sdk/docs/properties.md.txt#setting_properties). If you use primarily zonal clusters, set the `compute/zone` instead. By setting a default location, you can avoid errors in the gcloud CLI like the following: `One of [--zone, --region] must be supplied: Please specify location`. You might need to specify the location in certain commands if the location of your cluster differs from the default that you set.

### Required roles


To get the permissions that
you need to deploy the cluster,

ask your administrator to grant you the
following IAM roles:

- [Kubernetes Engine Admin](https://docs.cloud.google.com/iam/docs/roles-permissions/container.md.txt#container.admin) (`roles/container.admin`) on the project
- [Compute Admin](https://docs.cloud.google.com/iam/docs/roles-permissions/compute.md.txt#compute.admin) (`roles/compute.admin`) on the project
- [Storage Admin](https://docs.cloud.google.com/iam/docs/roles-permissions/storage.md.txt#storage.admin) (`roles/storage.admin`) on the project
- [Project IAM Admin](https://docs.cloud.google.com/iam/docs/roles-permissions/resourcemanager.md.txt#resourcemanager.projectIamAdmin) (`roles/resourcemanager.projectIamAdmin`) on the project
- [Service Account Admin](https://docs.cloud.google.com/iam/docs/roles-permissions/iam.md.txt#iam.serviceAccountAdmin) (`roles/iam.serviceAccountAdmin`) on the project
- [Service Account User](https://docs.cloud.google.com/iam/docs/roles-permissions/iam.md.txt#iam.serviceAccountUser) (`roles/iam.serviceAccountUser`) on the project
- [Service Usage Consumer](https://docs.cloud.google.com/iam/docs/roles-permissions/serviceusage.md.txt#serviceusage.serviceUsageConsumer) (`roles/serviceusage.serviceUsageConsumer`) on the project
- To consume a single-project reservation: [Compute Instance Admin (v1)](https://docs.cloud.google.com/iam/docs/roles-permissions/compute.md.txt#compute.instanceAdmin.v1) (`roles/compute.instanceAdmin.v1`) on the project
- To consume a shared reservation: [Compute Instance Admin (v1)](https://docs.cloud.google.com/iam/docs/roles-permissions/compute.md.txt#compute.instanceAdmin.v1) (`roles/compute.instanceAdmin.v1`) on both the owner project and any consumer project where you want to consume the reservation


For more information about granting roles, see [Manage access to projects, folders, and organizations](https://docs.cloud.google.com/iam/docs/granting-changing-revoking-access.md.txt).


You might also be able to get
the required permissions through [custom
roles](https://docs.cloud.google.com/iam/docs/creating-custom-roles) or other [predefined
roles](https://docs.cloud.google.com/iam/docs/roles-overview#predefined).

### Choose a consumption option and obtain capacity

To obtain capacity, complete the following steps:

1. **Choose a consumption option:** Make your choice based on how you want to
   get and use GPU resources. To learn more, see [Choose a consumption
   option](https://docs.cloud.google.com/ai-hypercomputer/docs/consumption-models).

   For GKE, consider the following additional information:
   - A4X compute instances cannot be provisioned by using flex-start.
   - You must use the [reservation-bound](https://docs.cloud.google.com/ai-hypercomputer/docs/create/review-configurations.md.txt#reservation-bound) provisioning model to create clusters with A4X. Other provisioning models are not supported.
2. **Obtain capacity:** The process to obtain capacity differs for each
   consumption option. To learn about the process for your chosen consumption
   option, see [Capacity overview](https://docs.cloud.google.com/ai-hypercomputer/docs/obtain-capacity.md.txt).

   > [!NOTE]
   > **Note:** When you request A4X capacity, you obtain it in *all capacity
   > mode* . This mode is the only supported reservation operational mode for A4X machine types. For more information about all capacity mode, see [Reservation operational
   > mode](https://docs.cloud.google.com/ai-hypercomputer/docs/create/review-configurations#reservation-operational-mode).

### Requirements

The following requirements apply to an AI-optimized GKE cluster
that uses A4X instances:

- Verify that you use GKE version 1.33.4-gke.1036000 or later
  (for 1.33), or version 1.32.8-gke.1108000 or later (for 1.32). These versions
  help ensure that A4X uses the following:

  - R580, which is the minimum GPU driver version for the GB200 GPUs in A4X instances. This driver is enabled by default.
  - Coherent Driver-based Memory Management (CDMM), which is enabled by default. NVIDIA recommends that Kubernetes clusters enable this mode to resolve memory over-reporting. CDMM lets you manage GPU memory through the driver instead of the operating system (OS). This approach helps you avoid OS onlining of GPU memory, and exposes the GPU memory as a Non-Uniform Memory Access (NUMA) node to the OS. Multi-instance GPUs are not supported when CDMM is enabled. For more information about CDMM, see [Hardware and
    Software
    Support](https://docs.nvidia.com/datacenter/tesla/tesla-release-notes-580-65-06/index.html#hardware-software-support).
  - GPUDirect RDMA, which is recommended to let A4X node pools use the networking capabilities of A4X.
- To use GPUDirect RDMA, the GKE nodes must use a
  Container-Optimized OS node image. Ubuntu and Windows node images are
  not supported.

## Customize the DRANET configuration

This GKE blueprint for A4X automatically configures
GKE-managed [Dynamic Resource Allocation for Networking
(DRANET)](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/allocate-network-resources-dra).

When you provision a node pool with DRANET enabled, Cluster Toolkit
automatically detects the machine type and requests the Multi-Host Remote
Direct Memory Access (MRDMA) interfaces (`mrdma.google.com`).
Cluster Toolkit also creates a `ResourceClaimTemplate` object
named `default-rdma` in your cluster. If you configure multiple node pools with
DRANET, then Cluster Toolkit automatically manages the deployment of
templates to prevent conflicts between duplicate resources.

To allocate network devices or change device classes, you can customize the
template by using the `dranet_allocation_mode`, `dranet_device_count`, and
`dranet_device_class_name` variables in your node pool module settings.

The following example shows how to configure DRANET in a node pool:

```yaml
  - id: a4x-pool
    source: modules/compute/gke-node-pool
    settings:
      enable_dranet: true
      # Optional: Override DRANET template defaults
      # dranet_allocation_mode: ExactCount
      # dranet_device_count: 4
```

To allocate DRANET resources, do one of the following depending on your
deployment method:

- If you use Cluster Toolkit workload templates, such as the [`gke-job-template`
  module](https://docs.cloud.google.com/cluster-toolkit/docs/modules/gke-job-template), then you don't need to configure the resources manually. The toolkit automatically injects the default resource claims into your workloads.
- If you deploy a custom workload by using a Kubernetes YAML manifest, you must
  explicitly configure the claims in your manifest. To configure the claims, do
  the following:

  - **Declare the claim at the Pod level:** In the `spec.template.spec.resourceClaims` section, add a claim that references the `default-rdma` template:

  ```yaml
  spec:
    template:
      spec:
        resourceClaims:
        - name: rdma
          resourceClaimTemplateName: default-rdma
  ```
  - **Bind the claim at the container level:** In the `resources.claims` section in your container definition, add a claim to bind the Pod-level claim that you declared:

  ```yaml
  resources:
    claims:
    - name: rdma
  ```

To view examples of custom workload manifests, see [Deploy a workload with RDMA
resources](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/allocate-network-resources-dra#deploy-workload-rdma)
or the [`gke-a4x/nccl-jobset-example.yaml`](https://github.com/GoogleCloudPlatform/cluster-toolkit/blob/main/examples/gke-a4x/nccl-jobset-example.yaml)
example on GitHub.

## Create an A4X GKE cluster

To create an A4X GKE cluster, do the following:

> [!NOTE]
> **Note:** If you create multiple clusters by using these same cluster blueprints, verify that all VPC network and subnet names are unique per project to prevent errors.

1. [Launch Cloud Shell](https://docs.cloud.google.com/shell/docs/launching-cloud-shell.md.txt). Although it's possible to use a different environment, we recommend that you use Cloud Shell because the dependencies are already pre-installed for Cluster Toolkit. If you don't want to use Cloud Shell, then follow the instructions to [install dependencies](https://docs.cloud.google.com/cluster-toolkit/docs/setup/install-dependencies.md.txt) to prepare a different environment.
2. Create a Cloud Storage bucket to store the state of the Terraform deployment:

   ```
       gcloud storage buckets create gs://BUCKET_NAME \
           --default-storage-class=STANDARD \
           --project=PROJECT_ID \
           --location=COMPUTE_REGION_TERRAFORM_STATE \
           --uniform-bucket-level-access
       gcloud storage buckets update gs://BUCKET_NAME --versioning
       
   ```

   Replace the following variables:
   - `BUCKET_NAME`: the name of the new Cloud Storage bucket.
   - `PROJECT_ID`: your Google Cloud project ID.
   - `COMPUTE_REGION_TERRAFORM_STATE`: the compute region where you want to store the state of the Terraform deployment.
3. Choose a consumption option for your GKE deployment:

   By default, the deployment file is configured to consume
   [specific
   reservations](https://docs.cloud.google.com/cluster-toolkit/docs/deploy/gke/gke-consumption-options#reservations). To use an alternative consumption model
   , update your deployment YAML file. For
   more information, see
   [GKE
   consumption options](https://docs.cloud.google.com/cluster-toolkit/docs/deploy/gke/gke-consumption-options).
4. In the
   [`examples/gke-a4x/gke-a4x-deployment.yaml`](https://github.com/GoogleCloudPlatform/cluster-toolkit/blob/main/examples/gke-a4x/gke-a4x-deployment.yaml)
   deployment file from the GitHub repository, fill in the following settings in the
   `terraform_backend_defaults` and `vars` sections to
   match the specific values for your deployment:

   - `BUCKET`: the name of the Cloud Storage bucket that you created in the previous step.
   - `PROJECT_ID`: your Google Cloud project ID.
   - `DEPLOYMENT_NAME`: a unique name for the deployment, which must be between 6 and 30 characters in length. If the deployment name isn't unique within a project, then cluster creation fails. The default value is `gke-a4x`.
   - `REGION`: the compute region for the cluster.
   - `ZONE`: the compute zone for the node pool of A4X machines. This zone must match the zone where machines are available in your reservation.
   - `AUTHORIZED_CIDR`: the IP address range that you want to let connect to the cluster. This CIDR block must include the IP address of the machine that calls Terraform. For more information, see [How authorized networks work](https://docs.cloud.google.com/kubernetes-engine/docs/concepts/network-isolation.md.txt#how_authorized_networks_work).
   - `STATIC_NODE_COUNT`: the number of A4X nodes in your cluster's node pool, which must be 18 nodes or less. We recommend that you use 18 nodes to obtain the GPU topology of `1x72` in one subblock by using an NVLink domain.
   - For the `reservation` field, use one of the following, depending on whether you want to target specific [blocks](https://docs.cloud.google.com/ai-hypercomputer/docs/terminology.md.txt#block) in a reservation when you provision the node pool:
     - To place the node pool anywhere in the reservation, provide the name of your reservation (`RESERVATION_NAME`).
     - To target a specific block within your reservation, use the reservation and block names in the following format:

       ```
       RESERVATION_NAME/reservationBlocks/BLOCK_NAME
       ```
       If you don't know which blocks are available in your reservation, then see [View
       the topology of a reservation](https://docs.cloud.google.com/ai-hypercomputer/docs/view-reserved-capacity#view-capacity-topology).
   - `RESERVATION_PROJECT_ID`: the Google Cloud project ID that has the reservation. Your reservation might exist in a different project from `PROJECT_ID`.
   - `NUM_NODE_POOLS`: the number of node pools to enable in your cluster. If you want a cluster larger than 18 nodes, then you can specify multiple node pools. The default value is `1`.

   > [!TIP]
   > You can run `./gcluster explain examples/gke-a4x/gke-a4x.yaml` from the Cluster Toolkit directory to inspect all customizable variables, deployment groups and modules, and required APIs for this blueprint.

   To modify advanced settings, edit the
   [`examples/gke-a4x/gke-a4x.yaml`](https://github.com/GoogleCloudPlatform/cluster-toolkit/blob/main/examples/gke-a4x/gke-a4x.yaml)
   file.
5. Optional: Configure [Google Cloud Managed Lustre](https://docs.cloud.google.com/managed-lustre/docs/overview.md.txt) to provision a managed parallel file system optimized for AI and HPC workloads. The blueprint enables the [Managed Lustre CSI driver](https://docs.cloud.google.com/managed-lustre/docs/lustre-csi-driver-overview.md.txt) by default. To configure Managed Lustre for your A4X GKE deployment, do the following:
   1. Open the [`examples/gke-a4x/gke-a4x.yaml`](https://github.com/GoogleCloudPlatform/cluster-toolkit/blob/main/examples/gke-a4x/gke-a4x.yaml) file.
   2. In the `vars` section, uncomment the Managed Lustre variables `lustre_instance_id`, `lustre_size_gib`, and `per_unit_storage_throughput`.
   3. Find the section commented `# --- MANAGED LUSTRE ADDITIONS ---` and uncomment the `private_service_access`, `lustre_firewall_rule`, `managed-lustre`, and `lustre-pv` modules.
6. [Generate Application Default Credentials (ADC)](https://docs.cloud.google.com/docs/authentication/provide-credentials-adc.md.txt#google-idp) to provide access to Terraform. If you're using Cloud Shell, use the `gcloud auth application-default login` command:

   ```
       gcloud auth application-default login
       
   ```
7. Deploy the blueprint by using the [`gcluster
   deploy` command](https://docs.cloud.google.com/cluster-toolkit/docs/reference/gcluster-deploy) to provision the GKE infrastructure with A4X machine types:

   ```
       cd ~/cluster-toolkit
       ./gcluster deploy -d \
           examples/gke-a4x/gke-a4x-deployment.yaml \
           examples/gke-a4x/gke-a4x.yaml
       
   ```

   > [!NOTE]
   > **Note:** If you encounter an error at the JobSet or Kueue installation stage when you deploy your blueprint, then deploy the blueprint again with the addition of the `-w` flag (`./gcluster deploy -w -d ...`) to complete the deployment.

8. When prompted, select **(A)pply** to deploy the blueprint.
   - The blueprint creates VPC networks, service accounts, a cluster, and a node pool. It also configures GKE-managed Dynamic Resource Allocation for Networking (DRANET).
   - To support the `fio-bench-job-template` job template, the blueprint creates Google Cloud buckets, network storage, and PersistentVolume resources.

## Test network performance

We recommend that you validate the functionality of provisioned clusters. To do
so, use [NCCL/gIB tests](https://docs.cloud.google.com/ai-hypercomputer/docs/nccl/test-gke.md.txt), which are
[NVIDIA Collective Communications Library (NCCL)
tests](https://developer.nvidia.com/nccl) that are optimized for the Google
environment.

## Clean up resources

To avoid recurring charges for the resources that you used on this page, use the
[`gcluster destroy` command](https://docs.cloud.google.com/cluster-toolkit/docs/reference/gcluster-destroy.md.txt)
to clean up the resources that are provisioned by Cluster Toolkit, including
the
VPC networks and the GKE cluster:

```bash
cd ~/cluster-toolkit
./gcluster destroy DEPLOYMENT_NAME
```

Replace `DEPLOYMENT_NAME` with the name of your deployment directory (which matches the `deployment_name` value set in your deployment file).

## What's next

- To learn about scheduling workloads on your GKE clusters by using TAS and Kueue, see [Schedule GKE workloads with Topology
  Aware
  Scheduling](https://docs.cloud.google.com/ai-hypercomputer/docs/workloads/schedule-gke-workloads-tas).
- To learn about managing common events that are relevant to GKE clusters and AI workloads, see [Manage AI-optimized GKE
  clusters](https://docs.cloud.google.com/ai-hypercomputer/docs/manage/manage-gke-clusters).
- For information about testing your environment for proper setup and optimization, see [Cluster networking optimization
  overview](https://docs.cloud.google.com/ai-hypercomputer/docs/nccl/overview).