This document explains how to view the physical layout and real-time status of your cluster nodes in Cluster Director.
You can view the topology of the nodes in your cluster by using the Google Cloud console, including their health, usage, and maintenance schedules. This information can help you to manage high-performance workloads, troubleshoot networking issues, and help ensure optimal cluster configuration.
Before you begin
Before you view the topology of the nodes in your cluster, consider the following:
You can view topology information only for GPU or TPU instances.
When you access and use the Google Cloud console, you don't need to authenticate. You can automatically use Google Cloud services and APIs.
Required roles
To get the permissions that
you need to view clusters,
ask your administrator to grant you the
Hypercompute Cluster Viewer (roles/hypercomputecluster.viewer) IAM role on the project.
For more information about granting roles, see Manage access to projects, folders, and organizations.
This predefined role contains the permissions required to view clusters. To see the exact permissions that are required, expand the Required permissions section:
Required permissions
The following permissions are required to view clusters:
-
To view the details of a cluster:
hypercomputecluster.clusters.describe
You might also be able to get these permissions with custom roles or other predefined roles.
View a cluster topology
To view a cluster topology by using the Google Cloud console, complete the following steps:
In the Google Cloud console, go to the Cluster Director page.
In the navigation menu, click Clusters. The Clusters page appears.
In the Clusters table, in the Name column, click the name of the cluster that you want to view the details of. A page that gives the details of the cluster appears, and the Details tab is selected.
Click the Topology tab. Then, based on the topology information that you want to view, click one of the following tabs:
To view information about cluster health, click the Health tab.
To view information about cluster maintenance, click the Maintenance tab.
To view information about CPU, GPU, and TPU usage in your cluster, click the CPU utilization, GPU utilization, and TPU utilization tabs.
For more information, see Cluster topology tabs in this document.
Optional: For clusters with TPU nodes, you can switch between the following views:
Grid view: shows a summary across all TPU compute nodes in your cluster and a per-slice view for the selected tab (Maintenance, CPU utilization, TPU utilization, or Health).
Table view: lists each TPU node and its metrics in columns, with TPU nodes grouped by slice.
Cluster topology tabs
When you view a cluster topology, you can view the following tabs to monitor the status and performance of your cluster:
Cluster health information
When you view the Health tab on the page that gives the details of a cluster, you can see the health status for each compute node in your cluster. For clusters with TPU nodes, the grid view displays an overview of node health across your cluster and lets you inspect the health of individual TPU nodes within each slice. Each node is color-coded to indicate one of three operational states:
Healthy: the node is fully operational.
Suspected bad: the system has detected potential issues with your node, but the node remains operational. To run tests that verify the status of a GPU compute node, see Verify cluster health.
Unhealthy: the node encountered a hardware or system failure. Based on the type of compute node, the following applies:
GPU compute nodes: the node failed an automated health check and Slurm drained the node. To undrain the node, see Troubleshoot GPU health check failures.
TPU compute nodes: Slurm doesn't run automated health checks. To manually recover an unresponsive TPU node, see Reset unresponsive TPU nodes.
Cluster maintenance information
When you view the Maintenance tab on the page that gives the details of a cluster, you can view information about any upcoming maintenance events for the nodes in your cluster. For clusters with TPU nodes, viewing the maintenance information for each TPU slice helps you identify which slice contains a node that requires maintenance. This information lets you anticipate and plan for necessary downtime.
If a maintenance event is scheduled for GPU compute nodes in your cluster, then you can start maintenance before the scheduled time. You can't start maintenance on demand for TPU compute nodes. For instructions, see Manually start planned host maintenance events.
Cluster CPU, GPU, and TPU usage information
When you view the CPU utilization, GPU utilization, and TPU utilization tabs on the page that gives the details of a cluster, you can see a visual representation of resource usage across all nodes in your cluster in real time. For clusters with TPU nodes, the grid view shows the average utilization across all TPU compute nodes and utilization ranges for each TPU slice. These views help you understand how your workloads are distributed. Use this information to identify performance bottlenecks and optimize your workloads.