This document explains how to view the physical layout and real-time status of
your cluster nodes in Cluster Director.

You can view the topology of the nodes in your cluster by using the
Google Cloud console, including their health, usage, and maintenance schedules. This
information can help you to manage high-performance workloads, troubleshoot
networking issues, and help ensure optimal cluster configuration.

## Before you begin

Before you view the topology of the nodes in your cluster, consider the
following:

- You can view topology information only for GPU, H4D, or TPU instances.

- When you access and use the Google Cloud console, you don't need to
  authenticate. You can automatically use Google Cloud services and APIs.

### Required roles


To get the permissions that
you need to view clusters,

ask your administrator to grant you the
[Cluster Director Viewer](https://docs.cloud.google.com/iam/docs/roles-permissions/hypercomputecluster.md.txt#hypercomputecluster.viewer) (`roles/hypercomputecluster.viewer`) IAM role on the project.


For more information about granting roles, see [Manage access to projects, folders, and organizations](https://docs.cloud.google.com/iam/docs/granting-changing-revoking-access.md.txt).


This predefined role contains

the permissions required to view clusters. To see the exact permissions that are
required, expand the **Required permissions** section:


#### Required permissions

The following permissions are required to view clusters:

- To view the details of a cluster: `hypercomputecluster.clusters.get`


You might also be able to get
these permissions
with [custom roles](https://docs.cloud.google.com/iam/docs/creating-custom-roles.md.txt) or
other [predefined roles](https://docs.cloud.google.com/iam/docs/roles-overview.md.txt#predefined).

## View a cluster topology

To view a cluster topology by using the Google Cloud console, complete the
following steps:

1. In the Google Cloud console, go to the **Cluster Director** page.

   [Go to Cluster Director](https://console.cloud.google.com/cluster-director)
2. In the navigation menu, click **Clusters** . The **Clusters** page
   appears.

3. In the **Clusters** table, in the **Name** column, click the name of the
   cluster that you want to view the details of. A page that gives the
   details of the cluster appears, and the **Details** tab is selected.

4. Click the **Topology** tab. Then, based on the topology information that you
   want to view, click one of the following tabs:

   - To view information about cluster health, click the **Health** tab.

   - To view information about cluster maintenance, click the **Maintenance**
     tab.

   - To view information about CPU, GPU, and TPU usage in your cluster, click
     the **CPU utilization** , **GPU utilization** , and **TPU utilization**
     tabs.

   For more information, see [Cluster topology tabs](https://docs.cloud.google.com/cluster-director/docs/monitor/view-topology.md.txt#understand) in this
   document.
5. Optional: For clusters with TPU nodes, you can switch between the following
   views:

   - **Grid view** : shows a summary across all TPU compute nodes in your
     cluster and a per-slice view for the selected tab (**Maintenance** ,
     **CPU utilization** , **TPU utilization** , or **Health**).

   - **Table view**: lists each TPU node and its metrics in columns, with
     TPU nodes grouped by slice.

## Cluster topology tabs

When you view a cluster topology, you can view the following tabs to monitor the
status and performance of your cluster:

- [Cluster health information](https://docs.cloud.google.com/cluster-director/docs/monitor/view-topology.md.txt#cluster-health)

- [Cluster maintenance information](https://docs.cloud.google.com/cluster-director/docs/monitor/view-topology.md.txt#cluster-maintenance)

- [Cluster CPU, GPU, and TPU usage information](https://docs.cloud.google.com/cluster-director/docs/monitor/view-topology.md.txt#cluster-usage)

### Cluster health information

When you view the **Health** tab on the page that gives the details of a
cluster, you can see the health status for each compute node in your cluster.
For clusters with TPU nodes, the grid view displays an overview of node health
across your cluster and lets you inspect the health of individual TPU nodes
within each slice. Each node is color-coded to indicate one of three operational
states:

- **Healthy**: the node is fully operational.

- **Suspected bad** : the system has detected potential issues with your node,
  but the node remains operational. To run tests that verify the status of a
  GPU or H4D compute node, see
  [Verify cluster health](https://docs.cloud.google.com/cluster-director/docs/monitor/cluster-health.md.txt#verify-health).

- **Unhealthy**: the node encountered a hardware or system failure. Based on
  the type of compute node, the following applies:

  - **GPU compute node** : the node failed an automated health check and
    Slurm drained the node. To undrain the node, see
    [Troubleshoot GPU health check failures](https://docs.cloud.google.com/cluster-director/docs/troubleshoot/health-checks.md.txt).

  - **TPU compute node** : Slurm doesn't run automated health checks. To
    manually recover an unresponsive TPU node, see
    [Reset unresponsive TPU nodes](https://docs.cloud.google.com/cluster-director/docs/troubleshoot/cluster-errors.md.txt#reset-tpu-nodes).

  - **H4D compute node**: if an H4D node encounters a hardware or host
    error, then do one of the following:

    - For H4D nodes that you created by using reserved capacity, report
      the host as faulty to repair and restart the node. For instructions,
      see
      [Report a faulty host](https://docs.cloud.google.com/compute/docs/hpc/report-faulty-host-h4d.md.txt).

    - For H4D nodes that you created by using Flex-start, Spot, or
      on-demand capacity, delete the node so that Slurm can recreate it.
      For instructions, see
      [Delete nodes](https://docs.cloud.google.com/cluster-director/docs/workloads/configure-slurm.md.txt#power-down-nodes).

### Cluster maintenance information

When you view the **Maintenance** tab on the page that gives the details of a
cluster, you can view information about any upcoming
[maintenance events](https://docs.cloud.google.com/compute/docs/instances/host-maintenance-overview.md.txt#maintenanceevents)
for the nodes in your cluster. For clusters with TPU nodes, viewing the
maintenance information for each TPU slice helps you identify which slice
contains a node that requires maintenance. This information lets you anticipate
and plan for necessary downtime.

If a maintenance event is scheduled for GPU or H4D compute nodes in your
cluster, then you can start maintenance before the scheduled time. You can't
start maintenance for TPU compute nodes. For instructions, see
[Manually start planned host maintenance events](https://docs.cloud.google.com/cluster-director/docs/monitor/cluster-health.md.txt#start-maintenance).

### Cluster CPU, GPU, and TPU usage information

When you view the **CPU utilization** , **GPU utilization** , and **TPU
utilization** tabs on the page that gives the details of a cluster, you can see
a visual representation of resource usage across all nodes in your cluster in
real time. For clusters with TPU nodes, the grid view shows the average
utilization across all TPU compute nodes and utilization ranges for each TPU
slice. These views help you understand how your workloads are distributed. Use
this information to identify performance bottlenecks and optimize your
workloads.

## What's next

- [Monitor cluster performance with prebuilt dashboards](https://docs.cloud.google.com/cluster-director/docs/monitor/dashboards.md.txt)

- [Manage cluster health](https://docs.cloud.google.com/cluster-director/docs/monitor/cluster-health.md.txt)