This document summarizes how you create a cluster for your AI workloads on AI Hypercomputer. Specifically, this document guides you through the process and choices to make when you start a cluster. As an alternative to manual evaluation, you can prompt Gemini in the Google Cloud console to compare consumption options for your workload and budget. For more information, see [Design AI infrastructure with Compute Advisor](https://docs.cloud.google.com/ai-hypercomputer/docs/design-with-gemini.md.txt). This document assumes that you are familiar with commonly used terminology for AI and ML workloads, such as model training and inference. It also assumes that you have identified the specific AI workload for which you need to determine the optimal machine type and capacity needs for your deployment---for example, foundation model pre-training, fine-tuning, or inference. > [!TIP] > **Tip:** For a quick end-to-end tutorial that deploys an A4 machine type on Slurm, see instead [Create an AI-optimized Slurm cluster with an A4 machine > type](https://docs.cloud.google.com/ai-hypercomputer/docs/create/create-a-slurm-cluster-with-a4). ## Start a cluster Starting a cluster involves the following steps: 1. [Determine your workload and choose a machine type](https://docs.cloud.google.com/ai-hypercomputer/docs/process-overview.md.txt#determine-workload-and-choose-a-machine-type) 2. [Choose a consumption option and obtain capacity](https://docs.cloud.google.com/ai-hypercomputer/docs/process-overview.md.txt#choose-a-consumption-option-and-obtain-capacity) 3. [Choose a deployment option](https://docs.cloud.google.com/ai-hypercomputer/docs/process-overview.md.txt#choose-a-deployment-option) 4. [Choose an orchestrator](https://docs.cloud.google.com/ai-hypercomputer/docs/process-overview.md.txt#choose-an-orchestrator) 5. [Choose the operating system and cluster image](https://docs.cloud.google.com/ai-hypercomputer/docs/process-overview.md.txt#choose-the-operating-system-image) 6. [Create your cluster](https://docs.cloud.google.com/ai-hypercomputer/docs/process-overview.md.txt#create-your-cluster) 7. [Provision storage for your workload](https://docs.cloud.google.com/ai-hypercomputer/docs/process-overview.md.txt#storage) ### Determine your workload and choose a machine type Select a machine type for your AI workload. AI Hypercomputer supports cluster creation for both [clustered GPUs and general GPUs](https://docs.cloud.google.com/ai-hypercomputer/docs/choose-infrastructure). To help you choose, determine your workload requirements and match them with the recommended machine type and GPU type: - Clustered GPUs: best for large-scale, high-performance workloads such as pre-training foundation models, fine-tuning large models, and inference across multiple hosts. - General GPUs: best for mainstream inference and serving, retrieval-augmented generation (RAG), and cost-effective small-to-medium model training and fine-tuning. To match your workload to the recommended machine type, use this table: | GPU type | Workload or use case | Recommended machine types | |---|---|---| | **Clustered GPU** | Pre-training foundation models and inference across multiple hosts | **A5X** (NVIDIA Vera Rubin)^\*^, **A4X Max** (NVIDIA GB300)^\*^, **A4X** (NVIDIA GB200)^\*^ | | **Clustered GPU** | Large model training, fine-tuning, and inference | **A4** (NVIDIA B200), **A3 Ultra** (NVIDIA H200 141GB) | | **Clustered GPU** | Mainstream model inference and fine-tuning | **A3 Mega** (NVIDIA H100 80GB), **A3 High** (NVIDIA H100 80GB) | | **General GPU** | High-throughput edge serving and inference | **A3 Edge** (NVIDIA H100 80GB) | | **General GPU** | High-performance single-node serving and small-scale fine-tuning | **A2** (NVIDIA A100) | | **General GPU** | Cost-optimized entry-level inference | **G4** (NVIDIA RTX PRO 6000), **N1** (NVIDIA T4 or V100) | | **General GPU** | Mainstream inference, RAG, and small-to-medium model training | **G2** (NVIDIA L4) | > [!CAUTION] > **Caution:** ^\*^The [Compute Engine Service Level Agreement > (SLA)](https://cloud.google.com/compute/sla) doesn't apply to the A5X, A4X Max, and A4X machine series. For detailed information about each machine series, see [About GPU accelerators](https://docs.cloud.google.com/ai-hypercomputer/docs/understand-gpus). ### Choose a consumption option and obtain capacity Select a [consumption option](https://docs.cloud.google.com/ai-hypercomputer/docs/consumption-models.md.txt) for your GPU resources based on your chosen machine type and whether you use General GPUs or Clustered GPUs. #### Consumption options for general GPUs | Consumption option | Available for | Best for | How to request | |---|---|---|---| | **On-demand** | All [general GPUs](https://docs.cloud.google.com/ai-hypercomputer/docs/understand-gpus.md.txt). | Workloads that don't require assured capacity. | Create a compute instance or cluster and specify the standard provisioning model. For instructions, see [Create VM instances](https://docs.cloud.google.com/ai-hypercomputer/docs/create/create-overview.md.txt). **Tip:** To increase your chances of obtaining general GPU capacity, use Flex-start or Spot. | | **Standard reservations and standard future reservations** | All [general GPUs](https://docs.cloud.google.com/ai-hypercomputer/docs/understand-gpus.md.txt). | Workloads that require assured capacity either immediately (standard reservations) or for a specific future date (standard future reservations). | Create an on-demand reservation or a future reservation request. For instructions, see [Reserve capacity](https://docs.cloud.google.com/ai-hypercomputer/docs/reserve-capacity.md.txt). | | **Flex-start** | All [GPU machine types](https://docs.cloud.google.com/ai-hypercomputer/docs/understand-gpus.md.txt) except for A5X, A4X Max, and A4X. | Workloads that require short-lived, dense clusters that last up to seven days. Offers dense resource allocation and up to a 53% discount on supported machine types; otherwise, standard on-demand rates apply. | Create a request by using Compute Engine, Cluster Director, Cluster Toolkit, or GKE. Resources are provisioned as soon as they become available (start time isn't immediate). For instructions, see [Obtain capacity](https://docs.cloud.google.com/ai-hypercomputer/docs/obtain-capacity.md.txt). | | **Spot** | All [GPU machine types](https://docs.cloud.google.com/ai-hypercomputer/docs/understand-gpus.md.txt) except A5X, A4X Max, and A4X. | Workloads that are fault-tolerant, batch, or short-lived. Offers the deepest discount (between 61% and 90%), but compute resources can be preempted at any time. | Create instances or node pools immediately by using the Spot provisioning model. For instructions, see [Obtain capacity](https://docs.cloud.google.com/ai-hypercomputer/docs/obtain-capacity.md.txt). | #### Consumption options for clustered GPUs | Consumption option | Available for | Best for | How to request | |---|---|---|---| | **Future reservations for capacity blocks** | All [clustered GPUs](https://docs.cloud.google.com/ai-hypercomputer/docs/understand-gpus.md.txt) and A3 Edge machine types. | Workloads that require stability for an extended period of time, such as pre-training foundation models or foundation model inference across multiple hosts. This reservation method offers dense resource allocation and an optional Hyperdisk pool, as well as discounts for vCPUs, memory, GPUs, Local SSD disks, and Hyperdisk pools if you reserve resources for 365 days or more. | Request resources through your Google account team for a future date and time, and for a duration of 90 days or more. | | **Future reservations in calendar mode** | All [clustered GPUs](https://docs.cloud.google.com/ai-hypercomputer/docs/understand-gpus.md.txt) except for A5X, A4X Max, and A4X. | Workloads that run for up to 90 days and require stability, such as pre-training or fine-tuning models. Offers dense resource allocation and up to a 53% discount. | Create a self-service reservation request for a future date and time; Google Cloud must approve the request. For instructions, see [Reserve capacity](https://docs.cloud.google.com/ai-hypercomputer/docs/reserve-capacity.md.txt). | | **Flex-start** | All [GPU machine types](https://docs.cloud.google.com/ai-hypercomputer/docs/understand-gpus.md.txt) except for A5X, A4X Max, and A4X. | Workloads that require short-lived, dense clusters that last up to seven days. Offers dense resource allocation and up to a 53% discount on supported machine types; otherwise, standard on-demand rates apply. | Create a request by using Compute Engine, Cluster Director, Cluster Toolkit, or GKE. Resources are provisioned as soon as they become available (start time isn't immediate). For instructions, see [Obtain capacity](https://docs.cloud.google.com/ai-hypercomputer/docs/obtain-capacity.md.txt). | | **Spot** | All [GPU machine types](https://docs.cloud.google.com/ai-hypercomputer/docs/understand-gpus.md.txt) except A5X, A4X Max, and A4X. | Workloads that are fault-tolerant, batch, or short-lived. Offers the deepest discount (between 61% and 90%), but compute resources can be preempted at any time. | Create instances or node pools immediately by using the Spot provisioning model. For instructions, see [Obtain capacity](https://docs.cloud.google.com/ai-hypercomputer/docs/obtain-capacity.md.txt). | ### Choose a deployment option Depending on the level of control that you need over your cluster deployment, choose between the following options: - **[Highly managed deployment](https://docs.cloud.google.com/ai-hypercomputer/docs/process-overview.md.txt#highly-managed):** managed services automate the setup, orchestration, and configuration of your compute, networking, and storage resources. - **[Less managed, more control deployment](https://docs.cloud.google.com/ai-hypercomputer/docs/process-overview.md.txt#less-managed):** you manually create and manage your VMs, custom environments, and orchestration layers. #### Highly managed Highly managed deployment options automate the setup and orchestration of your compute, networking, and storage resources, which reduces the operational overhead of cluster management. To deploy and configure your infrastructure, use Cluster Director, Cluster Toolkit, or GKE. - [**Cluster Director**](https://docs.cloud.google.com/cluster-director/docs/overview.md.txt): a Google Cloud product that automates the complex setup and configuration of clusters, helping you configure compute, networking, and storage resources for your clusters to maximize performance and minimize downtimes. Cluster Director is designed for IT administrators and AI researchers who want to avoid the overhead of managing a cluster, and instead focus on running their workloads. - [**Cluster Toolkit**](https://docs.cloud.google.com/cluster-toolkit/docs/overview.md.txt): an open source tool offered by Google that simplifies cluster configuration and deployment for GKE or Compute Engine. You use predefined blueprints to deploy common configurations, such as A4 machine types with Slurm. You can modify blueprints to customize deployments and your software stack. - [**GKE**](https://docs.cloud.google.com/kubernetes-engine/docs/concepts/kubernetes-engine-overview.md.txt): a managed Kubernetes service and open source container orchestration platform. GKE offers features like autoscaling and high availability. It's also able to orchestrate containerized applications, support specialized hardware, and is compatible with the Google Cloud ecosystem, making it well-suited for deploying and managing AI or ML workloads. You can deploy GKE clusters by using GKE directly or by using Cluster Toolkit. #### Less managed, more control For more granular control over your clusters and the software installed on them, create a Compute Engine cluster by using managed Compute Engine instance groups (MIGs) or by creating instances in bulk. Then, manually install any key software you need on the instances. ### Choose an orchestrator An orchestrator automates the management of your clusters. With an orchestrator, you don't have to manage each compute instance in the cluster. An orchestrator, such as Slurm or GKE, handles tasks like job queuing, resource allocation, autoscaling (in the case of GKE), and other day-to-day cluster management tasks. - **Slurm**: Slurm is an open source orchestrator commonly used for HPC, AI, or ML workloads. For a managed Slurm experience, you can use Cluster Director. To use Slurm natively, you can use Cluster Toolkit (which offers cluster blueprints that automatically install Slurm on your clusters), or you can manually install Slurm on a Compute Engine cluster. - **GKE**: GKE is a managed service built on top of Kubernetes, an open-source container orchestration platform. GKE is ideal for deploying and managing AI or ML workloads, because of its ability to orchestrate containerized applications, its support of specialized hardware, and its place in the Google Cloud ecosystem. You can deploy GKE clusters by using GKE directly or by using Cluster Toolkit. - **Bring your own orchestrator**: to use other orchestrators, or to run workloads without a scheduler, choose one of the following options: - **Managed compute clusters in Cluster Director** : automate Virtual Private Cloud (VPC) and RDMA networking, shared storage mounts, and the lifecycle management of Compute Engine MIGs. For more information, see [Managed compute orchestration in Cluster Director](https://docs.cloud.google.com/cluster-director/docs/managed-compute-orchestration.md.txt). - **Compute Engine clusters**: provide the least managed option on Google Cloud. With this option, you're responsible for setting up, maintaining, and updating your instances, networking, and storage. ### Choose the operating system image The image that you should use depends on which GPU infrastructure you're using (clustered GPUs or general GPUs) and how you plan to deploy your clusters (GKE, Slurm, or Compute Engine). For GKE clusters, use Container-Optimized OS. For Compute Engine and Slurm clusters, select an operating system image optimized for accelerators, such as GPUs. In addition, you can select a Deep Learning Software Layer (DLSL) container image for your workload. For detailed information, review [AI Hypercomputer images](https://docs.cloud.google.com/ai-hypercomputer/docs/software-stack). #### Images for GKE clusters To create GKE clusters, use the default container OS images for both Standard and Autopilot modes. However, in Standard mode, you can also choose to use other available images, like Ubuntu. If you use Cluster Toolkit to deploy your cluster, then you can only use container OS images, as these are the images built into the cluster blueprints. For more information about each node image, see [Node images](https://docs.cloud.google.com/kubernetes-engine/docs/concepts/node-images) in the GKE documentation. GKE also offers Deep Learning Software Layer (DLSL) container images that install packages like NVIDIA CUDA, and NCCL, as well as ML frameworks like PyTorch, providing a ready-to-use environment for deep learning workloads. These prebuilt DLSL container images are tested and verified to work seamlessly on GKE clusters. #### OS images for Compute Engine clusters AI Hypercomputer offers images optimized for running AI and ML workloads by using Compute Engine. Choose the OS you are most familiar with: - Rocky Linux 9 accelerator - Rocky Linux 8 accelerator - Ubuntu 24.04 LTS accelerator - Ubuntu 22.04 LTS accelerator If you use Cluster Toolkit, then these accelerator images are already bundled into Cluster Toolkit blueprints, because Cluster Toolkit creates custom images that extend the Ubuntu LTS Accelerator OS images. For more information about each OS image, see [Operating system details](https://docs.cloud.google.com/compute/docs/images/os-details) in the Compute Engine documentation. ### Create your cluster After you review the cluster creation process and make preliminary decisions for your workload, create your cluster by using one of these options: - Create a GKE cluster: - [Create a GKE cluster by using Cluster Toolkit](https://docs.cloud.google.com/ai-hypercomputer/docs/create/gke-ai-hypercompute.md.txt) - [Create a custom GKE cluster](https://docs.cloud.google.com/ai-hypercomputer/docs/create/gke-create-a4-a3-ultra.md.txt) - Create a Slurm cluster: - [Create a fully managed cluster by using Cluster Director](https://docs.cloud.google.com/ai-hypercomputer/docs/create/create-fully-managed-slurm-cluster.md.txt) - [Create a self-managed cluster by using Cluster Toolkit](https://docs.cloud.google.com/ai-hypercomputer/docs/create/create-self-managed-slurm-cluster.md.txt) - Create a managed compute cluster or standalone VM instances: - [Create a custom cluster](https://docs.cloud.google.com/cluster-director/docs/create/custom-cluster.md.txt) in Cluster Director - [Create VM instances (single, bulk, or MIGs)](https://docs.cloud.google.com/ai-hypercomputer/docs/create/create-overview.md.txt) ### Provision storage for your workload Choose a [storage service](https://docs.cloud.google.com/ai-hypercomputer/docs/storage.md.txt) to provision, based on performance, cost, and storage architecture requirements.