To create the infrastructure for tightly-coupled applications that scale across
multiple nodes, you can create a cluster of virtual machine (VM) instances. This
guide provides a high-level overview of the key considerations and steps to
configure a cluster of virtual machine (VM) instances for high performance
computing (HPC) workloads using dense resource allocation.

With [H4D](https://docs.cloud.google.com/compute/docs/compute-optimized-machines#h4d_series),
Compute Engine adds support for running massive HPC workloads by
treating an entire cluster of VM instances as a single computer. Using
topology-aware placement of VMs lets you access many instances within a single
networking superblock and minimizes network latency. You can also configure
[Cloud RDMA](https://docs.cloud.google.com/compute/docs/networking/using-irdma) on these instances
to maximize inter-node communication performance, which is crucial for
tightly-coupled HPC workloads.

> [!NOTE]
> **Note:** This type of configuration relies on similar features and concepts as those documented in the [AI Hypercomputer documentation](https://docs.cloud.google.com/ai-hypercomputer/docs/overview) for accelerator-optimized VMs with GPUs.

You create these HPC VM clusters with H4D by reserving capacity blocks instead
of individual resources. Using capacity blocks for your cluster enables
[enhanced cluster management capabilities](https://docs.cloud.google.com/compute/docs/hpc/cluster-capabilities).

HPC clusters with H4D instances can be created either with or without
enhanced cluster management capabilities. If you don't require enhanced cluster management capabilities features with your H4D HPC
cluster, or if you want to create HPC clusters using a machine series other than
H4D, then use the following instructions for creating HPC instances or clusters:

- [Managed instance groups (MIGs)](https://docs.cloud.google.com/compute/docs/instance-groups/creating-groups-of-managed-instances)
- [Bulk API](https://docs.cloud.google.com/compute/docs/instances/multiple/about-bulk-creation)
- [Create an HPC-ready VM instance](https://docs.cloud.google.com/compute/docs/instances/create-hpc-vm)

## Cluster terminology

When working with capacity blocks, the following terms are used:

<br />

Block[](https://docs.cloud.google.com/compute/docs/hpc/overview-hpc-clusters#block) A collection of sub-blocks interconnected by a non-blocking, high-speed network that provides high bandwidth for all host machines in the block. Cluster[](https://docs.cloud.google.com/compute/docs/hpc/overview-hpc-clusters#cluster) A collection of blocks interconnected by a high-speed network fabric. A cluster can scale to thousands of CPUs for running large-scale HPC workloads. Each cluster is globally unique. Communication across different blocks adds only one additional hop, maintaining high performance and predictability, even at a massive scale. Cluster-level metadata is also available to orchestrators for intelligent, large-scale job placement. Cluster Toolkit[](https://docs.cloud.google.com/compute/docs/hpc/overview-hpc-clusters#cluster-toolkit) An open source tool offered by Google that simplifies the configuration and deployment for clusters that use either Slurm or Google Kubernetes Engine. You use predefined blueprints to build a deployment folder that is based on the blueprint. You can modify blueprints or the deployment folder to customize deployments and your software stack. You then use Terraform or Packer to run the commands generated by Cluster Toolkit to deploy the cluster. Dense deployment[](https://docs.cloud.google.com/compute/docs/hpc/overview-hpc-clusters#dense) A resource request that allocates your compute instance resources physically close to each other to minimize network hops and optimize for the lowest latency. Network fabric[](https://docs.cloud.google.com/compute/docs/hpc/overview-hpc-clusters#fabric) High-speed network infrastructure that provides high bandwidth and low latency across all blocks and Google Cloud services in a cluster. The fabric uses the architecture of the [Jupiter](https://cloud.google.com/blog/topics/systems/the-evolution-of-googles-jupiter-data-center-network) network, and it uses software-defined networking and optical circuit switches to optimize performance. Node or host[](https://docs.cloud.google.com/compute/docs/hpc/overview-hpc-clusters#node) A single physical server machine in the data center. Each host has its associated compute resources such as CPUs, memory, and network interfaces. The number and configuration of these compute resources depend on the machine type. Compute instances are provisioned on top of a physical host. Orchestrator[](https://docs.cloud.google.com/compute/docs/hpc/overview-hpc-clusters#orchestrator) An orchestrator automates the management of your clusters. With an orchestrator, you don't have to manage each VM instance in the cluster. An orchestrator, such as [Slurm](https://slurm.schedmd.com/overview.html) or Google Kubernetes Engine (GKE), handles tasks like job queueing, resource allocation, auto scaling (with GKE), and other day-to-day cluster management tasks. Sub-blocks[](https://docs.cloud.google.com/compute/docs/hpc/overview-hpc-clusters#subblock) A group of hosts and associated connectivity hardware that are located on a single physical rack. A top-of-rack (ToR) switch connects these hosts, enabling extremely efficient, single-hop communication between any two CPUs within the sub-block. Cloud RDMA facilitates this direct communication.

<br />

## Overview of cluster creation process with H4D VMs

To create HPC clusters after you reserve capacity blocks, you must complete the
following steps:

1. [Review available provisioning models](https://docs.cloud.google.com/compute/docs/hpc/overview-hpc-clusters#provisioning-overview)
2. [Choose a consumption option and obtain capacity](https://docs.cloud.google.com/compute/docs/hpc/overview-hpc-clusters#consumption-and-capacity)
3. [Choose a deployment option and orchestrator](https://docs.cloud.google.com/compute/docs/hpc/overview-hpc-clusters#deployment)
4. [Choose the operating system or cluster image](https://docs.cloud.google.com/compute/docs/hpc/overview-hpc-clusters#choose-os)
5. [Create your cluster](https://docs.cloud.google.com/compute/docs/hpc/overview-hpc-clusters#create-cluster)

## Provisioning models for VM and cluster creation

When creating VM instances, you can use the provisioning models described in
[Compute Engine instances provisioning models](https://docs.cloud.google.com/compute/docs/instances/provisioning-models).

To create a tightly-coupled H4D instances, you must use one of the following
provisioning models to obtain the necessary resources for creating compute
instances:

- **Reservation-bound**: you can reserve resources at a discounted price for a
  future date and duration. At the start of your reservation period, you can use
  the reserved resources to create VMs or clusters. You have exclusive access to
  your reserved resources for the reservation period.

- **Flex-start**: you can request discounted resources for up to seven days.
  Compute Engine makes best-effort attempts to schedule the provisioning
  of your requested resources as soon as they're available. You have exclusive
  access to your obtained resources for your requested period.

- **Spot**: based on availability, you can immediately obtain deeply discounted
  resources. However, Compute Engine might stop or delete the VM
  instances at any time to reclaim capacity.

### Reservation-bound provisioning model

The reservation-bound provisioning model links your created VM instances to the
capacity that you previously reserved. When you reserve capacity,
Compute Engine creates an empty reservation. Then, at the reservation
start time, the following occurs:

- Compute Engine adds your reserved resources to the reservation.
  You have exclusive access to the reserved capacity until the reservation end
  time.

- Google Cloud charges you for the reserved capacity until the end of your
  reservation period, whether you use the capacity or not.

You can then use the reserved resources to create VMs without additional
charges. You only pay for resources that aren't included in the reservation,
such as disks or IP addresses.

You can reserve resources for as many VMs as you like for as
long as you like for a future date. Then, you can use the reserved resources to
create and run VMs until the end of the reservation period. If you reserve
resources for one year or longer, then you must purchase and attach a
resource-based commitment.

To provision resources using the reservation-bound provisioning model, see:

- For long-running, large-scale distributed workloads with densely allocated
  resources:
  [Reserve capacity through your account team](https://docs.cloud.google.com/compute/docs/hpc/reserve-capacity-account-team)

- For short-running (up to 90 days) distributed workloads with densely allocated
  resources:
  [Future reservation requests in calendar mode](https://docs.cloud.google.com/compute/docs/instances/future-reservations-calendar-mode-overview)

You can use reservation-bound provisioning with H4D instances by specifying the
reservation-bound provisioning model when creating individual VMs, a HPC
cluster, or a group of VMs.

### Flex-start provisioning model

To run short-duration workloads that require densely allocated resources, you
can request compute resources for up to seven days by using Flex-start. Whenever
resources are available, Compute Engine creates your requested number of
VMs. You can stop standalone Flex-start VMs, but you can't stop
Flex-start VMs that a managed instance group (MIG) creates through
resize requests. The Flex-start VMs exist until you delete them, or
until Compute Engine deletes the VMs at the end of their run duration.

Flex-start is ideal for workloads that can start at any time. The flex-start
provisioning model provisions resources from a secure capacity pool, so the
allocated resources are densely allocated to minimize network latency.

When you add Flex-start VMs to a
managed instance group (MIG) by using resize requests, the MIG creates the VMs
all at once. This approach helps you avoid unnecessary charges for
partial capacity that Compute Engine might deliver while you wait for
the full capacity needed to start your workload.

You can use Flex-start provisioning with H4D instances, using any available
deployment model.

### Spot provisioning model

To run fault-tolerant workloads, you can obtain compute resources immediately
based on availability. You get resources at the lowest price possible. However,
Compute Engine might stop or delete the created Spot VMs at any
time to reclaim capacity. This process is called
[*preemption*](https://docs.cloud.google.com/compute/docs/instances/spot#preemption).

Spot VMs are ideal for workloads where interruptions are acceptable,
such as:

- Batch processing
- High performance computing (HPC)
- Data analytics
- Continuous integration and continuous deployment (CI/CD)
- Media encoding

You can use Spot VMs with any machine type, except A4X, X4, and bare
metal machine types. Dense allocation depends on resource availability. To help
ensure a closer allocation, you can apply a compact placement policy to the
Spot VMs.

> [!NOTE]
> **Note:** Spot VMs are not covered by any Service Level Agreement and are excluded from the [Compute Engine SLA](https://cloud.google.com/compute/sla).

You can use Spot VMs with the following dense deployment options:

- [Create a HPC Slurm cluster with H4D](https://docs.cloud.google.com/compute/docs/hpc/create-slurm-cluster-h4d)
- [Bulk create HPC-optimized instances with H4D](https://docs.cloud.google.com/compute/docs/hpc/bulk-create-hpc-vms)
- [Create a HPC MIG with H4D machine series](https://docs.cloud.google.com/compute/docs/hpc/create-mig-hpc-dense)

## Choose a consumption option and obtain capacity

Consumption options determine how resources are obtained for your cluster. To
create a cluster that uses enhanced cluster management capabilities, you must request capacity blocks for a
[dense deployment](https://docs.cloud.google.com/compute/docs/hpc/overview-hpc-clusters#dense).

The following table summarizes the key differences between the consumption
options for capacity blocks:

> [!NOTE]
> **Note:** You can also request a future reservation for more than 90 days. If you need to reserve this capacity, see [Reserve capacity through your account team](https://docs.cloud.google.com/compute/docs/hpc/reserve-capacity-account-team).

| Consumption option | Future reservations for more than 90 days (for capacity blocks) | Future reservations for up to 90 days (in calendar mode) | Flex-start | Spot |
| Workload characteristics | Long-running, large-scale distributed workloads that require densely allocated resources | Short-duration workloads that require densely allocated resources | Short-duration workloads that require densely allocated resources | Fault-tolerant workloads |
| Lifespan | Any time | Up to 90 days | Up to 7 days | Any time, but subject to [preemption](https://docs.cloud.google.com/compute/docs/instances/spot#preemption) |
| Preemptible | No | No | No | Yes |
| Capacity assurance | Very high | Very high | Best effort | Best effort |
| Quota | Check that you have enough quota before creating instances. | [CPU quota](https://docs.cloud.google.com/compute/resource-usage#cpu_quota) is consumed. | [Preemptible quota](https://docs.cloud.google.com/compute/resource-usage#preemptible-quotas) is consumed. | [Preemptible quota](https://docs.cloud.google.com/compute/resource-usage#preemptible-quotas) is consumed. |
| Pricing | - See [pricing for VMs](https://docs.cloud.google.com/compute/vm-instance-pricing#compute-optimized_machine_types). If you reserve resources for a year or longer, then you must purchase and attach a [resource-based commitment](https://docs.cloud.google.com/compute/docs/instances/signing-up-committed-use-discounts) to your reserved resources. - You're charged for the reservation period. See [reservations billing](https://docs.cloud.google.com/compute/docs/instances/reservations-overview#reservation-billing). | - Discounted (up to 25%). See [Dynamic Workload Scheduler pricing](https://cloud.google.com/products/dws/pricing). - You're charged for the reservation period. See [reservations billing](https://docs.cloud.google.com/compute/docs/instances/reservations-overview#reservation-billing). | - Discounted (up to 25%). See [Dynamic Workload Scheduler pricing](https://cloud.google.com/products/dws/pricing). - You pay as you go (PAYG). | - Deeply discounted (up to 91%). See [Spot VMs pricing](https://cloud.google.com/spot-vms/pricing) and [pricing for compute-optimized VMs](https://cloud.google.com/compute/vm-instance-pricing#compute-optimized-machine-type-family). - You pay as you go (PAYG). |
| Resource allocation | [Dense](https://docs.cloud.google.com/compute/docs/hpc/overview-hpc-clusters#dense) | [Dense](https://docs.cloud.google.com/compute/docs/hpc/overview-hpc-clusters#dense) | [Dense](https://docs.cloud.google.com/compute/docs/hpc/overview-hpc-clusters#dense) | Standard ([Compact placement policy](https://docs.cloud.google.com/compute/docs/instances/placement-policies-overview#about-compact-policies) optional) |
| Provisioning model | Reservation-bound | Reservation-bound | Flex-start | Spot |
| Creation method | To create HPC clusters and VMs, you must do the following: 1. [Reserve capacity through your account team](https://docs.cloud.google.com/compute/docs/hpc/reserve-capacity-account-team) 2. At your chosen date and time, you can use the reserved capacity to create HPC clusters. See [Choose a deployment option](https://docs.cloud.google.com/compute/docs/hpc/overview-hpc-clusters#deployment). | To create HPC clusters and VMs, you must do the following: 1. [Create a future reservation request in calendar mode](https://docs.cloud.google.com/compute/docs/instances/create-future-reservations-calendar-mode) 2. At your chosen date and time, you can use the reserved capacity to create HPC clusters. See [Choose a deployment option](https://docs.cloud.google.com/compute/docs/hpc/overview-hpc-clusters#deployment). | To create VMs, select one of the following options: - [Create a standalone Flex-start VMs](https://docs.cloud.google.com/compute/docs/instances/create-flex-start-vm). - [Create Flex-start VMs all at once by using MIG resize requests](https://docs.cloud.google.com/compute/docs/hpc/create-hpc-migs-flex-start). - Use GKE to [run high performance computing (HPC) workloads with H4D](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/run-hpc-workloads). When your requested capacity becomes available, Compute Engine provisions it. | You can immediately create VMs. See [Choose a deployment option](https://docs.cloud.google.com/compute/docs/hpc/overview-hpc-clusters#deployment). |
|---|---|---|---|---|

## Choose a deployment option

High performance computing (HPC) workloads aggregate computing resources to gain
performance greater than that of a single workstation, server, or computer. HPC
is used to solve problems in academic research, science, design, simulation, and
business intelligence.

For HPC clusters with enhanced cluster management capabilities, choose the H4D machine series. If you plan
to use a different machine series, follow the documentation at
[Create an HPC-ready VM instance](https://docs.cloud.google.com/compute/docs/instances/create-hpc-vm)
instead of using the deployment methods listed on this page.

Some of the available deployment options
include the installation and configuration of an [orchestrator](https://docs.cloud.google.com/compute/docs/hpc/overview-hpc-clusters#orchestrator)
for enhanced management of the HPC cluster.

For the most appropriate option to create your VMs or clusters for your use
case, choose one of the following:

| Option | Use case |
|---|---|
| Cluster Toolkit | You want to use open-source software that simplifies the process for you to deploy both Slurm and Google Kubernetes Engine (GKE) clusters. [Cluster Toolkit](https://docs.cloud.google.com/compute/docs/hpc/overview-hpc-clusters#cluster-toolkit) is designed to be highly customizable and extensible. To learn more, see the following: - [Create an H4D Slurm cluster with enhanced cluster management capabilities](https://docs.cloud.google.com/compute/docs/hpc/create-slurm-cluster-h4d) - [Quickstart: Create a Cloud RDMA-enabled HPC Slurm cluster](https://docs.cloud.google.com/cluster-toolkit/docs/quickstarts/create-a-slurm-cluster-h4d) |
| GKE | You want maximum flexibility in configuring your Google Kubernetes Engine cluster based on the needs of your workload. To learn more, see [Run HPC workloads with H4D](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/run-hpc-workloads). |
| Use Compute Engine | You want full control of the infrastructure layer so that you can set up your own orchestrator. To learn more, see the following: - Create an HPC-optimized instance (*non-dense deployments*) - [Create an HPC-ready VM instance](https://docs.cloud.google.com/compute/docs/instances/create-hpc-vm) - [Create an instance that uses Cloud RDMA](https://docs.cloud.google.com/compute/docs/instances/create-vm-with-rdma) - [Create H4D instances in bulk](https://docs.cloud.google.com/compute/docs/hpc/bulk-create-hpc-vms) - Create a managed instance group (MIG) with H4D instances - [Create a HPC MIG with H4D machine series](https://docs.cloud.google.com/compute/docs/hpc/create-mig-hpc-dense) - [Quickstart: Create a MIG with H4D machine types and flex-start](https://docs.cloud.google.com/compute/docs/hpc/create-hpc-migs-flex-start) - [Quickstart: Create a MIG for HPC workloads with reservation-bound consumption](https://docs.cloud.google.com/compute/docs/hpc/create-hpc-migs-gsc-reservations) |

## Choose the operating system image

The operating system (OS) image you choose depends on the service you use to
deploy your cluster.

- **For clusters on GKE** : Use a GKE node image,
  such as Container-Optimized OS. If you use Cluster Toolkit to
  deploy your GKE cluster, a Container-Optimized OS
  image is used by default. For more information about node images, see
  [Node images](https://docs.cloud.google.com/kubernetes-engine/docs/concepts/node-images) in the
  GKE documentation.

- **For clusters on Compute Engine**: You can use one of the following
  images:

  - **[HPC VM image](https://console.cloud.google.com/marketplace/product/click-to-deploy-images/hpc-vm-image-rocky-linux-8)**: A Rocky Linux 8 image that is optimized for tightly-coupled HPC workloads.
  - **[OS image provided by Google Cloud](https://docs.cloud.google.com/compute/docs/images/os-details)**: OS images that support H4D. You will need to configure these for your HPC workloads.
  - **[Custom images](https://docs.cloud.google.com/compute/docs/images/create-custom)** : You can create and use your own custom images. To include HPC-specific optimizations, we recommend that you [create a custom image using the HPC VM image](https://docs.cloud.google.com/compute/docs/instances/create-hpc-vm#create_a_custom_image_using_the_hpc_vm_image).
- **For Slurm Clusters**: Cluster Toolkit deploys the Slurm Cluster
  with a HPC VM image based on Rocky Linux 8 that is optimized for
  tightly-coupled HPC workloads.

## Create your HPC cluster

After you review the cluster creation process and make preliminary decisions
for your workload, create your cluster by using any of the
[deployment options](https://docs.cloud.google.com/compute/docs/hpc/overview-hpc-clusters#deployment).

## Enhanced cluster management capabilities for your HPC cluster

When you create H4D instances with densely allocated resources using the
deployment methods mentioned in [Choose a deployment option](https://docs.cloud.google.com/compute/docs/hpc/overview-hpc-clusters#deployment),
you can use enhanced HPC cluster management capabilities with your instances.

For more information about these capabilities, see
[Enhanced HPC cluster management with H4D instances](https://docs.cloud.google.com/compute/docs/hpc/cluster-capabilities).

## What's next

- Learn more about [Cluster Toolkit](https://docs.cloud.google.com/cluster-toolkit/docs/overview).
- Try the Quickstart tutorial [Deploy an HPC cluster with Slurm](https://docs.cloud.google.com/cluster-toolkit/docs/quickstarts/slurm-cluster).
- Review [best practices for running HPC workloads](https://docs.cloud.google.com/cluster-toolkit/docs/tutorials/best-practices)