Node lifecycle

This document explains the lifecycle of a cluster node in Cluster Director. A node in a cluster transitions through various states from its creation to its deletion. Understanding these states lets you do the following:

  • Manage resources and costs: understand when Cluster Director provisions and stops dynamic nodes to run queued jobs and reduce unnecessary charges.

  • Minimize disruptions: anticipate planned host maintenance events.

  • Troubleshoot errors: identify why a node fails to boot, restart, or resume.

TPU nodes in a slice share the same lifecycle. If a TPU node undergoes maintenance, encounters a host error, or is deleted, then all other TPU nodes in that slice undergo the same lifecycle event. This behavior doesn't apply to GPU or CPU nodes. To check the current state of a node, view the details of your cluster.

Node states

The following diagram shows the different states that Cluster Director can set a node to:

State diagram for Cluster Director nodes across the UNPROVISIONED, CREATING, ACTIVE, SUSPENDING, SUSPENDED, REPAIRING, FAILED, and DELETING states.

The states shown in the preceding diagram are as follows:

Unprovisioned state

When Cluster Director creates a dynamic node, the node starts in the UNPROVISIONED state. From this state, a dynamic node can transition to one of the following states:

  • CREATING: when you submit a job that requires more compute resources than the ones in your static nodes, Cluster Director attempts to allocate resources to create the dynamic node.

  • DELETING: when you reduce the dynamic node count by modifying or deleting the cluster, the delete operation for the node starts.

Creating state

When a node is in the CREATING state, Cluster Director attempts to allocate resources to create and boot your node.

From this state, the node can transition to one of the following states:

  • ACTIVE: when Cluster Director allocates resources for your node, the node boots up and starts running.

  • FAILED: the node encounters a host error.

Active state

When a node is in the ACTIVE state, the node is healthy, running, and ready to run Slurm jobs.

From this state, the node can transition to one of the following states:

  • SUSPENDING: you stop or suspend the node, or Cluster Director stops the node after it preempts a Spot VM that is configured to be stopped (TERMINATE) on preemption.

  • REPAIRING: planned or unplanned host maintenance for the node starts.

  • UNPROVISIONED: for a dynamic GPU or CPU node, Slurm deletes the compute instance after the node remains in the idle state for five minutes. For a dynamic TPU node, Slurm deletes the compute instance when the job finishes.

  • DELETING: a delete operation starts after one of the following events occurs:

    • A Flex-start VM reaches the end of its run duration, or the reservation that you used to create the node reaches its end time.

    • Cluster Director preempts a Spot VM that is configured to be deleted (DELETE) on preemption.

    • You manually delete the node.

    • You reduce the static node count (staticNodeCount) or maximum dynamic node count (maxDynamicNodeCount) in a Slurm nodeset, or you reduce the target size (targetSize) of a MIG in a managed compute cluster.

    • You delete the cluster.

  • FAILED: the node encounters a host error.

Suspending state

When a node is in the SUSPENDING state, Cluster Director stops or suspends the node. You can only stop GPU or CPU nodes, and you can only suspend CPU nodes. For more information, see Suspend, stop, or reset Compute Engine instances.

From this state, when the stop or suspend operation completes, the node transitions to the SUSPENDED state.

Suspended state

When a node is in the SUSPENDED state, the stop or suspend operation completes.

From this state, the node transitions to one of the following states:

  • ACTIVE: you resume or restart the node. You can only resume or restart a node if the zone has available resources or you have a reservation. Otherwise, the operation fails and the node remains stopped or suspended.

  • DELETING: you delete the node or the cluster.

Repairing state

When a node is in the REPAIRING state, Cluster Director stops the node to perform host maintenance on the node's underlying Compute Engine infrastructure. Only GPU or TPU nodes enter this state. Nodes that use N2 machine types keep running during maintenance because Cluster Director live migrates them.

From this state, the node transitions to one of the following states:

  • ACTIVE: when host maintenance completes, the node restarts. Cluster Director helps ensure that the resources needed to restart the node are available.

  • FAILED: the node encounters a host error.

Failed state

When a node is in the FAILED state, the node is unhealthy or encountered a host error. To help you troubleshoot the issue, Cluster Director provides details about the error in Logs Explorer, or in the stateMessage field when you use the Google Cloud CLI or REST API.

From this state, when you delete the node or cluster, the node transitions to the DELETING state.

Deleting state

When a node is in the DELETING state, Cluster Director deletes the node and releases its resources. A node enters the DELETING state after one of the following events occurs:

  • The reservation that you used to create the node reaches its end time.

  • You delete the node.

  • You reduce the static node count (staticNodeCount) or maximum dynamic node count (maxDynamicNodeCount) in a nodeset.

  • You delete the cluster.

When the delete operation finishes, Cluster Director deletes the node and all of its attached resources.

Pricing for clusters and nodes

When you create a cluster, you incur charges for the underlying Google Cloud resources that your cluster uses:

  • Nodes: when a node is in the ACTIVE state, you incur charges for vCPUs, memory, and any attached GPUs or TPUs. For any disks and external IP addresses that are attached to a node's compute instance, you incur charges as long as the compute instance exists, including in the SUSPENDED state. Dynamic nodes in the UNPROVISIONED state have no compute instance and don't incur charges. For more information, see the following:

  • Networking resources: you incur charges for data transfer (egress) and any additional networking features that your cluster uses. For more information, see All networking pricing.

  • Storage resources: you incur charges for the storage resources that your cluster uses, as long as those resources exist. For more information, see the following:

For more information, review the pricing for Cluster Director.

What's next