Create a nodeset for a Slurm partition

Use the schedmd-slurm-gcp-v6-nodeset module to define a group of compute nodes, known as a nodeset. This module creates the data structure required by the schedmd-slurm-gcp-v6-partition module to build a partition.

Nodesets let you add different types of nodes to a partition. This configuration lets you run jobs that use a mix of hardware specifications.

For a complete list of inputs and outputs, see the schedmd-slurm-gcp-v6-nodeset module in the Cluster Toolkit GitHub repository.

Before you begin

Before you begin, verify that you meet the following requirements:

  • You have installed and configured Cluster Toolkit. For installation instructions, see Set up Cluster Toolkit.
  • You have an existing cluster blueprint. You can use and modify an existing blueprint or create one from scratch. For a working example of a blueprint configured for the slurm-nodeset module, see the examples/hpc-slurm.yaml file. For more information about creating and customizing blueprints, see Cluster blueprint.
  • To view a complete list of blueprints that support the slurm-nodeset module, go to the Cluster blueprint catalog page, click the Select scheduler list, and then select Slurm.

Required roles

To get the permissions that you need to deploy the nodeset, ask your administrator to grant you the following IAM roles on your project:

For more information about granting roles, see Manage access to projects, folders, and organizations.

You might also be able to get the required permissions through custom roles or other predefined roles.

Create a nodeset

To create a nodeset, add the schedmd-slurm-gcp-v6-nodeset module to your blueprint. You must specify the machine type and the maximum number of dynamic nodes.

The following example uses the schedmd-slurm-gcp-v6-nodeset module as input to create a partition module that does the following:

  • Uses the c2-standard-30 machine type.
  • Sets a maximum count of 200 dynamically created nodes for the nodeset.
  • Uses a nodeset name of ghpc.
  • Connects to the module with the network ID by using the use field.
  • Mounts nodes to the homefs network file system by using the use field.
- id: nodeset
  source: community/modules/compute/schedmd-slurm-gcp-v6-nodeset
  use:
  - network
  settings:
    node_count_dynamic_max: 200
    machine_type: c2-standard-30

- id: compute_partition
  source: community/modules/compute/schedmd-slurm-gcp-v6-partition
  use:
  - homefs
  - nodeset
  settings:
    partition_name: compute

Target specific nodesets

To specify nodes from a specific nodeset in a partition, use the --nodelist or -w flag. This example selects three nodes from cluster-compute-group-[0-2] in the compute partition:

srun -N 3 -p compute --nodelist cluster-compute-group-[0-2] hostname

Depending on how the nodes differ, you can also use the --constraint or -C flag or other flags, such as --mincpus, to specify nodes with the selected characteristics.

Configure reservations

The schedmd-slurm-gcp-v6-nodeset module lets you use Compute Engine reservations to get capacity for your Slurm nodes. You can configure both local and shared reservations by using the reservation_name setting. The reservation you use with this module must be a specifically targeted reservation of type SPECIFIC_RESERVATION.

Reservation types

You can configure the following types of specific reservations:

  • Local reservation: a reservation located within the same project as the cluster (var.project_id).
  • Shared reservation: a shared reservation from a different host project. Shared reservations let you manage reservations centrally across multiple projects.

Reservation name formats

The format that you specify for the reservation_name setting determines the type of reservation that is used:

  • Local reservation format: for a reservation in the same project as the cluster (var.project_id), specify the reservation name:

    reservation_name: RESERVATION_NAME

  • Shared reservation format: for a reservation shared from a different project, specify the full resource path:

    reservation_name: projects/HOST_PROJECT_ID/reservations/RESERVATION_NAME/reservationBlocks/BLOCK_ID/reservationSubBlocks/SUBBLOCK_ID

Replace the following:

  • HOST_PROJECT_ID: the project ID of the host project where the shared reservation was created.
  • RESERVATION_NAME: the name assigned to the specific reservation.
  • BLOCK_ID: the identifier for a specific reservation block, if the reservation contains multiple blocks. This field is optional.
  • SUBBLOCK_ID: the identifier for a specific reservation subblock within a block. This field is optional.

Deploy without a reservation

To deploy nodes without a specific reservation, or to use automatically consumed reservations, set the reservation_name setting to an empty string ("").

Configure zone policies

You can specify additional zones for your nodeset to create VM instances by using bulk creation. Configuring partitions with popular VM families lets you access more compute resources across zones.

If Google Cloud adds a new zone to the region while the cluster is active, then nodes in the partition might be created in that zone. In this case, you might need to redeploy the partition to help ensure that the partition denies the zone that was newly added.

vars:
  zone: us-central1-f

-   id: zonal-nodeset
  source: community/modules/compute/schedmd-slurm-gcp-v6-nodeset

The following example configuration lets you create VM instances in additional zones:

vars:
  zone: us-central1-f

-   id: multi-zonal-nodeset
  source: community/modules/compute/schedmd-slurm-gcp-v6-nodeset
  settings:
    zones:
    -   us-central1-a
    -   us-central1-b

Configure VM instance placement in the nodeset

Use the enable_placement setting to decide how Google Cloud places your VM instances in the nodeset.

By default, the enable_placement setting is set to a value of true. With this setting enabled, the module groups your VM instances close together to help them communicate faster by reducing network latency.

In certain cases, it might be helpful to disable the enable_placement setting, such as when you want to do any of the following:

  • Avoid resource shortages. When placement is enabled, Google Cloud tries to create all requested VMs in the same hardware group. If there isn't enough space in that group, the request might fail completely. Disabling placement lets Google Cloud create your VMs wherever space is available.
  • Improve reliability. To help ensure that a single hardware failure doesn't affect all of your VM instances at once, spread your VMs across different hardware.
  • Avoid extra costs. If you use specific types of reservations, such as non-dense reservations, then you can avoid the extra cost of placement policies.

To control the maximum distance between nodes in a placement group, use the placement_max_distance setting. Lower maximum distance values help ensure closer instance placement, but can result in fewer available machines for compute instance placement. Cluster Toolkit restricts which values are available for the placement_max_distance setting based on your chosen machine type.

For more information about supported machine types and maximum distance values, see About compact placement policies in the Compute Engine documentation.

Configure custom images

To create valid custom images for the node group virtual machine (VM) instances or for custom instance templates, see Slurm on Google Cloud custom images on GitHub.

Configure GPU support

For more information about configuring GPUs in Slurm on Google Cloud v6 and other Cluster Toolkit modules, see GPU support on GitHub.

Get support

The Cluster Toolkit team maintains the wrapper around the slurm-on-gcp Terraform modules. For support with the underlying modules, see the instructions in the Slurm on Google Cloud README on GitHub.

What's next