Managed compute orchestration in Cluster Director

This document explains the features, orchestrator integration patterns, and limitations of managed compute orchestration in Cluster Director. To compare managed compute orchestration with Slurm orchestration, instead see Choose an orchestrator.

With managed compute orchestration, Cluster Director provisions and manages compute nodes by using zonal Compute Engine managed instance groups (MIGs) without a Slurm controller or login nodes. Cluster Director sets up Virtual Private Cloud (VPC) and RDMA networks, mounts shared storage, and configures topology-aware placement. You can connect to the compute instances in each MIG by using SSH to run distributed workloads, or you can integrate your own orchestrator.

Features of a MIG in a managed compute cluster

When you create a managed compute cluster, Cluster Director provisions your compute, networking, and storage resources by creating one or more zonal MIGs. Each MIG references a single compute resource configuration with your specified machine type and consumption option, and a cluster can include multiple MIGs with different machine types.

The following sections describe the features that Cluster Director configures for each MIG.

Predictable instance naming

For each MIG that you define in your cluster, Cluster Director creates a zonal MIG and an instance template with a name that starts with MIG_NAME-it-.

Cluster Director names the compute instances in the MIG by using the name pattern MIG_NAME-###. This pattern produces compute instances with names starting with MIG_NAME-001, MIG_NAME-002, and continuing up to the number of compute instances that you add to the MIG.

Topology-aware placement

To minimize network latency for distributed workloads, Cluster Director places compute instances in a MIG close to each other as follows:

  • MIGs with GPU instances: Cluster Director creates a compact placement policy and attaches it to the MIG's instance template. For more information, see About compact placement policies.

  • MIGs with TPU instances: Cluster Director attaches a high-throughput workload policy with your specified TPU slice topology to each MIG with TPU instances. Each TPU MIG represents a single TPU slice, and the total number of TPU chips in the MIG must equal the total chip count of the slice topology. To run workloads across multiple TPU slices, you use a separate MIG for each slice. For more information, see About workload policies in MIGs.

Storage and additional disks

You can configure the following storage options for the compute instances in a MIG:

  • Boot disk: you can select one of the supported managed compute OS images, or a custom image, and configure the boot disk type and size.

  • Shared network storage: you can mount any combination of Filestore instances, Google Cloud Managed Lustre instances, or Cloud Storage buckets on your MIG. When you add or remove shared storage mounts on an existing MIG, Cluster Director updates the mounts on your running compute instances within 30 seconds and doesn't restart the compute instances.

  • Additional block disks: you can create new non-boot disks with a custom device name for each compute instance in a MIG. When you configure a non-boot disk, Cluster Director creates a separate disk for each compute instance in read-write mode. On MIGs with GPU or CPU instances that use any consumption option except Flex-start, Cluster Director configures each non-boot disk as a stateful disk so that replacement compute instances reattach the same disk across rolling updates or preemptions.

Compute instance access, startup scripts, and metadata

You can customize compute instance identity, access, and boot behavior on each MIG as follows:

  • Service account and scopes: by default, compute instances in a MIG use the Compute Engine default service account. You can specify a custom service account and access scopes when you create a MIG.

  • OS Login, external IP addresses, and network tags: you can enable or disable OS Login and external IP addresses, and attach network tags to the compute instances in each MIG.

  • Startup scripts: you can provide a startup script that runs when each compute instance boots and configure a startup script timeout.

  • Custom metadata and labels: you can attach custom metadata key-value pairs and resource labels to the compute instances in a MIG.

Integrate your own orchestrator

If you run a custom cluster scheduler—such as Ray, SkyPilot, or HTCondor—or launch distributed jobs with tools such as mpirun or torchrun, then you can integrate your orchestrator with a managed compute cluster by using the following patterns:

  • Separate head nodes and worker pools into distinct MIGs: define one or more MIGs in the same cluster to match your orchestrator's architecture. Each MIG uses a single machine type, and a cluster can include multiple MIGs with different machine types—for example, an N2 MIG to host your scheduler head node or coordinator, and one or more GPU or TPU MIGs for your worker pools.

  • Bootstrap worker agents with startup scripts and metadata: configure a startup script and custom metadata on each MIG so that every compute instance installs your orchestrator runtime (such as Ray, Docker, or a worker daemon) and registers with your head node when the compute instance boots. If installing a package takes longer than five minutes, then increase the startup script timeout on the MIG.

  • Discover and connect to worker nodes: your orchestrator or job launcher can address compute instances by their predictable internal hostnames (MIG_NAME-001 through MIG_NAME-TARGET_SIZE) over the cluster's VPC network.

  • Share environments and checkpoints over network storage: mount a shared Filestore or Managed Lustre instance—for example, in the /home directory—across all MIGs in your cluster. This shared file system lets you configure passwordless SSH, build shared Python virtual environments or container images once, and write checkpoints to a single location.

  • Scale worker pools: to scale a worker pool up or down, modify the MIG's target size in Cluster Director. Before you reduce the target size, drain active tasks on the target worker nodes in your orchestrator.

Limitations

When you create or modify a managed compute cluster, the following limitations apply to the MIGs and instance templates in your cluster:

  • Single machine type per MIG: each MIG in a managed compute cluster can use only one compute resource configuration and one machine type. To use multiple machine types in the same cluster, create a separate MIG for each machine type.

  • No built-in autoscaling or node draining: MIGs don't autoscale based on a job queue. If you reduce the target size of a MIG, Compute Engine deletes instances without checking for active workloads or draining tasks. To prevent workload interruptions when you scale down a MIG, drain active tasks on the target compute instances in your orchestrator before you modify your cluster.

  • Service-managed instance templates: Cluster Director generates and manages each MIG's instance template. You can't supply an existing instance template when you create or update a MIG, and you can't modify the generated instance template outside of Cluster Director.

  • Rolling VM replacement on template updates: on MIGs that don't use the Flex-start consumption option, Cluster Director creates a new instance template and triggers an immediate rolling replacement of all compute instances in the MIG if you modify any of the following properties:

    • Boot disks

    • Non-boot disks

    • Startup scripts

    • OS Login

    • External IP addresses

    • Custom metadata

    • TPU slice topology

  • MIGs with Flex-start VMs: MIGs that use the Flex-start consumption option don't support stateful disks, and Compute Engine doesn't replace Flex-start VMs if they encounter a host error or when they reach the end of their run duration.

  • MIGs with Spot VMs: MIGs that use Spot VMs have the following limitations:

    • If Compute Engine preempts a Spot VM, then any non-boot disks that are attached to the VM remain in your project until you delete the MIG. This configuration lets you create snapshots of the disks to back up your data, or reattach the disks to a replacement VM.

    • You can only configure Compute Engine to stop (TERMINATE) Spot VMs on preemption.

  • Single TPU slice per MIG: each MIG with TPU instances represents a single TPU slice, and the total number of TPU chips in the MIG must equal the total chip count of the slice topology. To provision multiple TPU slices in the same cluster, use a separate MIG for each slice.

What's next