Run health scans on GPUs and TPUs

The cluster health scanner is a self-service diagnostic tool that validates the readiness of your GPU and Cloud TPU capacity before you schedule a workload. The cluster health scanner runs benchmarks that simulate real workload traffic on your accelerator nodes to measure the performance of the accelerators, their memory, and the interconnects between them. You can use the results to identify and remove faulty nodes before they cause your training or inference jobs to stop responding, run slowly, or fail.

For TPUs, the cluster health scanner complements TPU health monitoring, which continuously reports hardware health signals. The cluster health scanner actively runs workloads on idle capacity to measure performance and expose faults that only appear under load.

The cluster health scanner is completely open source. To view the source code, contribute, or file issues, see the cluster health scanner GitHub repository.

This document describes how to install the cluster health scanner, run a health scan on GPU and TPU reservations, interpret the results, and remediate unhealthy nodes.

Supported environments

The cluster health scanner supports the following deployment environments:

Environment Supported accelerators Client environment Installation options
Google Kubernetes Engine (GKE) clusters GPUs and TPUs (node pools that use static or dynamic slicing) A Linux machine that has kubectl access to the target cluster Python package or container image
Slurm clusters GPUs only The Slurm controller or login node, where the sinfo and sbatch commands are available Python package only
Standalone TPU instances TPUs only The TPU instance itself Python package or container image (Container-Optimized OS)

The cluster health scanner supports the following GPU and TPU machine types:

  • GPU machine types:
    • A5X: a5x-highgpu-4g-metal (GKE only)
    • A4X Max: a4x-maxgpu-4g-metal
    • A4X: a4x-highgpu-4g
    • A4: a4-highgpu-8g
    • A3 Ultra: a3-ultragpu-8g
    • A3 Mega: a3-megagpu-8g
  • TPU machine types:

Diagnostic suites

You can install the cluster health scanner as a Python package or as a container image. Both options provide the chs command-line interface (CLI) and the diagnostic benchmarks. You run the CLI from a machine with access to your GKE cluster, on a Slurm controller or login node, or directly on a standalone TPU instance. The CLI then runs the diagnostic tests on the target GPU or TPU nodes.

The cluster health scanner provides two diagnostic suites for each accelerator type:

  • Lite: Runs a fast screening check. On GPUs and standalone TPU instances, the Lite suite checks individual hosts. On GKE TPU node pools, it also runs a quick collective check (tpu-collectives) across the TPU slice. A collective operation is a communication pattern in which all accelerators exchange data, such as AllReduce, which combines values from every accelerator and shares the result with all accelerators. Use the Lite suite for quick health checks or before you run single-host workloads.
  • Standard: Runs comprehensive validation across your target capacity. On GPUs, the Standard suite runs collective operations across multiple nodes and measures GPU memory bandwidth. On TPUs, it runs all Lite tests plus sustained compute, memory, and network stress tests. Use the Standard suite before you run multi-host training or inference workloads, where network performance and stability under sustained load directly affect job performance.
Suite Accelerator Tests Focus
lite GPU nvidia-smi, dcgm, gemm, nccl-single-node Fast validation of GPU and driver health, compute throughput, and intra-node GPU communication.
TPU tpu-info, tpu-gemm, tpu-hbm, tpu-host-device, tpu-collectives Fast validation of device health, compute throughput, high-bandwidth memory (HBM) bandwidth, host-to-device bandwidth, and Inter-Chip Interconnect (ICI) collective bandwidth.
standard GPU nvidia-smi, dcgm, gemm, nccl-cluster, nccl-pairwise, stream Multi-node network validation across the target nodes, plus memory bandwidth.
TPU All TPU lite tests, plus tpu-compute-stress, tpu-hbm-stress, and tpu-network-stress Sustained stress tests that expose instability, silent data corruption, and ICI bit errors.

The nccl-domain GPU test isn't part of a suite. To run nccl-domain, use the --tests flag.

GPU suites and tests

The cluster health scanner provides the following suites for GPU nodes:

  • Lite: Validates the GPU driver and hardware, measures compute throughput, and runs an NVIDIA Collective Communications Library (NCCL) test within each node.
  • Standard: Runs NCCL collectives across all target nodes and between pairs of nodes to measure the performance of the inter-node network, and measures memory bandwidth.

The following table describes each GPU test:

Test Description Suites
nccl-single-node Runs NCCL collective tests between the GPUs within a single node. lite
nvidia-smi Checks basic GPU health and the GPU driver by using the NVIDIA System Management Interface. lite, standard
dcgm Runs NVIDIA Data Center GPU Manager (DCGM) diagnostics to perform deep hardware checks on each GPU. lite, standard
gemm Runs general matrix multiplications (GEMM) to measure GPU compute throughput. lite, standard
nccl-cluster Runs NCCL collective tests across all target nodes to measure cluster-wide network bandwidth. standard
nccl-pairwise Runs NCCL tests between pairs of nodes to measure node-to-node latency and isolate faulty nodes or links. standard
stream Runs the STREAM benchmark to measure GPU memory bandwidth. standard
nccl-domain Runs NCCL collective tests within each multi-node NVLink domain. Supported only on machine types with multi-node NVLink domains. None (run with --tests)

TPU suites and tests

The cluster health scanner provides the following suites for TPU nodes:

  • Lite: Validates the performance of the TPU chips and HBM on each host, measures host-to-device bandwidth, and runs a collective operations test to measure ICI bandwidth.
  • Standard: Runs collective operations across the TPU chips in a slice, plus sustained stress tests on the TPU chips, HBM, and ICI links.

The following table describes each TPU test:

Test Description Suites
tpu-info Checks basic TPU health, including device count, LibTPU version, and TPU device type. lite, standard
tpu-gemm Runs general matrix multiplications (GEMM) to measure sustained peak TFLOPS. lite, standard
tpu-hbm Measures HBM read and write bandwidth. lite, standard
tpu-collectives Times collective operations, such as AllReduce and AllGather, to measure ICI bandwidth (GKE clusters only). lite, standard
tpu-host-device Measures host-to-device bandwidth. lite, standard
tpu-compute-stress Runs sustained matrix multiplications to verify computational stability and expose silent data corruption. standard
tpu-hbm-stress Stresses HBM to measure maximum bandwidth and validate data integrity. standard
tpu-network-stress Sends large random payloads across ICI links to measure sustained bandwidth and detect bit errors (GKE clusters only). standard

Before you begin

Before you use the cluster health scanner, install the required tools, configure permissions, and connect to your cluster or instance.

Installations

Before you run a health scan, complete the installations for your deployment environment.

GKE clusters

To run health scans on a GKE cluster, make sure that you have a target cluster and install the required tools on the machine where you plan to run scans:

Slurm clusters

To run health scans on a Slurm cluster, make sure that you have a target GPU cluster and install the required tools on the Slurm controller or login node, where the sinfo and sbatch commands are available:

For Slurm clusters, use the Python package installation option. Slurm doesn't support the container image option.

Standalone TPU instances

To run health scans on a standalone TPU instance, make sure that you have a target TPU instance and install the required tools directly on the instance:

  • gcloud CLI: Install and initialize the gcloud CLI.
  • Cluster health scanner prerequisites: Install the tools for the cluster health scanner installation option that you plan to use:
    • Python package: Python 3.12 (specifically Python 3.12 on Linux x86_64, not 3.13 or later) and Git. If the default python3 version on your instance is earlier or later than 3.12, you can use uv to create a Python 3.12 virtual environment (see Troubleshoot the cluster health scanner).
    • Container image: Docker. On Container-Optimized OS, the container image is the only supported installation option.
  • Standalone TPU instance: Make sure that you have a standalone TPU instance (TPU7x or TPU v6e), which exposes its TPU devices under /dev/vfio (you can verify this on the instance by running ls /dev/vfio). For more information, see Create Cloud TPU VMs.

Required roles

To get the permissions that you need to run a health scan, ask your administrator to grant you the roles/compute.viewer role on your project. This role lets the cluster health scanner read Compute Engine metadata and topology information for your nodes.

You also need the following Identity and Access Management (IAM) roles, depending on your deployment environment:

Environment Required roles
GKE clusters Kubernetes Engine Developer (roles/container.developer)
Slurm clusters
  • Compute OS Login (roles/compute.osLogin) or Compute OS Admin Login (roles/compute.osAdminLogin)
  • IAP-Secured Tunnel User (roles/iap.tunnelResourceAccessor), if connecting through Identity-Aware Proxy (IAP)
Standalone TPU instances
  • TPU Admin (roles/tpu.admin)
  • IAP-Secured Tunnel User (roles/iap.tunnelResourceAccessor), if connecting through IAP

If you configure a Cloud Storage bucket for reports, you also need the Storage Object Admin role (roles/storage.objectAdmin) on the bucket. This role lets the cluster health scanner upload result artifacts, such as the health report, to the bucket.

For more information about granting roles, see Manage access to projects, folders, and organizations.

Connect to your cluster or instance

Connect to the environment where you plan to run health scans.

GKE clusters

Set up Application Default Credentials (ADC) and configure kubectl to communicate with your GKE cluster:

gcloud auth application-default login
gcloud container clusters get-credentials CLUSTER_NAME \
    --location=LOCATION \
    --project=PROJECT_ID

Slurm clusters

Connect to your Slurm controller or login node by using SSH:

gcloud compute ssh SLURM_NODE_NAME \
    --zone=ZONE \
    --project=PROJECT_ID

Standalone TPU instances

Connect to your standalone TPU instance by using SSH:

gcloud compute tpus tpu-vm ssh TPU_NAME \
    --zone=ZONE \
    --project=PROJECT_ID

Install the cluster health scanner

Install the cluster health scanner on the machine where you run scans, which depends on your environment:

  • GKE clusters: Any machine that has kubectl access to your cluster.
  • Slurm clusters: The Slurm controller or login node.
  • Standalone TPU instances: The TPU instance itself.

On GKE and Slurm clusters, the cluster health scanner automatically pulls the diagnostic container images that match your GPU or TPU nodes and schedules them on the cluster, so you don't need to install any software on the individual compute nodes. On a standalone TPU instance, you install and run the cluster health scanner directly on the instance.

You can install the cluster health scanner as a Python package or as a container image. The chs commands in the rest of this document work with either installation option. To see detailed usage information for any command, append the --help flag.

Python package

Create a Python virtual environment and install the cluster health scanner package from GitHub. On GKE and Slurm clusters, use Python 3.12 or later. On standalone TPU instances (x86_64), you must use Python 3.12 (with Python 3.13 or later, pip install doesn't install the TPU benchmark dependencies, and chs run skips the instance):

python3 -m venv chs-venv
source chs-venv/bin/activate
pip install git+https://github.com/GoogleCloudPlatform/cluster-health-scanner.git

To install a specific release, append @VERSION to the repository URL:

pip install git+https://github.com/GoogleCloudPlatform/cluster-health-scanner.git@VERSION

For a list of releases, see cluster health scanner releases on GitHub.

Container image

The cluster health scanner is distributed as a container image in Artifact Registry. The image supports both GPUs and TPUs. You can use the container image for GKE clusters and standalone TPU instances, but not for Slurm clusters.

  1. Configure Docker to authenticate with Artifact Registry and pull the cluster health scanner container image:

    gcloud auth configure-docker us-docker.pkg.dev
    docker pull us-docker.pkg.dev/gce-ai-infra/chs/chs:latest
    

    To pull a specific release instead of latest, replace latest with the version number. For example, 0.3.0 for GitHub release v0.3.0. For a list of releases, see cluster health scanner releases on GitHub.

  2. Create a shell alias to run chs commands directly. The alias that you create depends on your environment:

    • GKE clusters: The alias mounts your kubeconfig file and Google Cloud credentials in read-only mode. The alias also stores the state of your runs in the chs-local-state Docker volume and writes output files to the chs-output directory in your current directory:

      alias chs='docker run --rm -it --network host \
          -v ~/.kube:/workspace/.kube:ro \
          -v ~/.config/gcloud:/workspace/.config/gcloud:ro \
          -v chs-local-state:/workspace/.local/share \
          -v "$(pwd)/chs-output":/workspace/chs-output \
          us-docker.pkg.dev/gce-ai-infra/chs/chs:latest'
      
    • Standalone TPU instances: On a TPU instance that runs Container-Optimized OS, the container image is the only supported installation option. The alias grants the container access to the TPU devices and host PID namespace of the instance, mounts the host root file system at /host, and mounts your Google Cloud credentials in read-only mode. The alias also stores the state of your runs in the chs-local-state Docker volume and writes output files to the chs-output directory in your current directory:

      alias chs='docker run --rm -it --privileged --pid=host \
          -v /:/host \
          -v ~/.config/gcloud:/workspace/.config/gcloud:ro \
          -v chs-local-state:/workspace/.local/share \
          -v "$(pwd)/chs-output":/workspace/chs-output \
          us-docker.pkg.dev/gce-ai-infra/chs/chs:latest'
      

Optional: Set up a Cloud Storage bucket for reports

By default, the cluster health scanner saves reports and diagnostic logs to a local output directory. You can configure a Cloud Storage bucket to also store them remotely.

  1. If you don't already have a bucket for health scan reports, create one:

    gcloud storage buckets create gs://BUCKET_NAME \
        --location=BUCKET_LOCATION \
        --project=PROJECT_ID
    
  2. Grant the Storage Object Admin role (roles/storage.objectAdmin) to your principal user account or service account:

    gcloud storage buckets add-iam-policy-binding gs://BUCKET_NAME \
        --member=PRINCIPAL \
        --role=roles/storage.objectAdmin
    

    The PRINCIPAL member is the principal that writes the report. For example, user:EMAIL or serviceAccount:SERVICE_ACCOUNT_EMAIL.

  3. Set the bucket where the cluster health scanner stores reports. The bucket must already exist because the cluster health scanner doesn't automatically create it. You only need to run this command once for each environment:

    chs configure storage gcs BUCKET_NAME
    

    To store reports under a folder prefix in the bucket, append the prefix to the bucket name:

    chs configure storage gcs BUCKET_NAME/PREFIX
    

Run a health scan

This section describes how to verify that your nodes are idle, start a scan, and monitor its progress.

Verify that the target nodes are idle

The cluster health scanner only runs on idle capacity, and doesn't automatically stop or evict your workloads. Before you start a scan, drain or stop any workloads that run on your target nodes or instance.

If the cluster health scanner detects active workloads on a target node or instance, it handles the capacity as follows:

  • GPU: The cluster health scanner skips the busy nodes and runs the scan on the remaining idle nodes.
  • TPU: How the cluster health scanner handles busy capacity depends on the environment:
    • GKE clusters: Multi-host TPU tests (tpu-collectives and tpu-network-stress) require a complete idle slice, while single-host TPU tests can run on individual idle nodes:
      • Static-slicing node pools (--nodepools or --nodes): If some nodes in a pool are busy when you run the lite or standard suite, the cluster health scanner logs a warning, omits the multi-host tests, and runs the single-host TPU tests on the remaining idle nodes.
      • Dynamic-slicing node pools (--nodepools or --nodes): Targeting a pool with --nodepools requires the entire node pool to be idle so that the cluster health scanner can provision a full-pool Slice resource. If you target specific nodes with --nodes (on GKE version 1.36.0-gke.3712000 or later), the cluster health scanner provisions an individual 2x2x1 sub-slice for each idle node, skips busy nodes, and runs single-host TPU tests on the idle nodes.
    • Standalone TPU instances: The cluster health scanner can't run if another process is using the TPU devices under /dev/vfio.

To make sure that your target capacity is idle, follow the steps for your environment.

GKE clusters

  1. Stop or delete any active workload Jobs or Pods on the target node pool, or wait for running workloads to finish.

    If you use kubectl drain to evict workloads from the node pool, uncordon the nodes before you start the scan so that the cluster health scanner can schedule diagnostic Pods on them (the cluster health scanner skips cordoned nodes):

    kubectl drain -l cloud.google.com/gke-nodepool=NODE_POOL_NAME \
        --ignore-daemonsets --delete-emptydir-data
    kubectl uncordon -l cloud.google.com/gke-nodepool=NODE_POOL_NAME
    

    Make sure that no new workloads are scheduled on the node pool while the scan runs.

  2. Verify that no workload Pods remain on the node pool:

    kubectl get pods --all-namespaces -o wide \
        --field-selector=status.phase=Running | grep NODE_POOL_NAME
    

    System Pods, such as those managed by DaemonSets in the kube-system namespace, can continue to run.

Slurm clusters

  1. Check for running jobs in the target partition:

    squeue --partition=PARTITION_NAME
    

    If jobs are running in the partition, wait for them to finish or cancel them with scancel JOB_ID.

  2. Verify that the nodes in the target partition are in the idle state:

    sinfo --partition=PARTITION_NAME
    

Standalone TPU instances

Verify that no other processes are using the TPU devices on the instance:

sudo fuser -v /dev/vfio/*

If any processes are listed, wait for the workloads to finish or stop the processes before you start a scan.

Start a scan

The flags that you pass to the chs run command depend on the environment where you run the cluster health scanner. The cluster health scanner automatically detects the scheduler and the accelerator type, so you don't need to specify them.

To list the available diagnostic suites and tests for an accelerator type, run the following commands:

  • For GPUs:

    chs list suites --gpu
    chs list tests --gpu
    
  • For TPUs:

    chs list suites --tpu
    chs list tests --tpu
    

GKE node pools

Run the cluster health scanner from any machine that has kubectl access to your cluster, such as your local machine or Cloud Shell.

To start a scan on one or more node pools, run the following command:

chs run \
    --nodepools=NODE_POOL_NAMES \
    --suite=SUITE

Replace the following:

  • NODE_POOL_NAMES: a comma-separated list of the GPU or TPU node pools to scan. For example: a4-pool-1,a4-pool-2 or tpu7x-pool-1,tpu7x-pool-2. All target node pools must use the same machine type.
  • SUITE: the diagnostic suite to run, either lite or standard. For more information about each suite, see Diagnostic suites.

To run specific tests instead of a full suite, replace the --suite flag with the --tests flag and specify a comma-separated list of test names:

chs run \
    --nodepools=NODE_POOL_NAMES \
    --tests=TEST_NAMES

To scan specific nodes instead of an entire node pool, replace the --nodepools flag with the --nodes flag and specify a comma-separated list of node names.

When you specify multiple node pools in the --nodepools flag, the cluster health scanner handles them based on the accelerator type:

  • GPU node pools: Combines all idle GPU nodes across the specified node pools into a single target set and runs multi-node tests (nccl-cluster and nccl-pairwise) across all of them together.
  • TPU node pools: Evaluates each node pool (TPU slice) independently and in parallel within a single run, without cross-slice (Multislice) testing.

Slurm partitions

Run the cluster health scanner on the Slurm controller or login node, where the sinfo and sbatch commands are available. Slurm requires the Python package installation option. For installation instructions, see Python package.

Before you start a scan, configure Enroot credentials on the cluster so that the nodes can pull benchmark images from us-docker.pkg.dev:

gcloud auth configure-docker us-docker.pkg.dev
mkdir -p ~/.enroot
cat <<EOF > "${HOME}/.enroot/.credentials"
machine us-docker.pkg.dev login oauth2accesstoken password $(gcloud auth application-default print-access-token)
EOF

Start a scan on one or more partitions:

chs run \
    --partitions=PARTITION_NAMES \
    --suite=SUITE

Replace the following:

  • PARTITION_NAMES: a comma-separated list of the Slurm partitions to scan.
  • SUITE: the diagnostic suite to run, either lite or standard. For more information about each suite, see Diagnostic suites.

To run specific GPU tests instead of a full suite, replace the --suite flag with the --tests flag and specify a comma-separated list of test names (for example, nccl-pairwise,gemm or nccl-domain):

chs run \
    --partitions=PARTITION_NAMES \
    --tests=TEST_NAMES

Standalone TPU instances

Run the cluster health scanner directly on the TPU instance. The TPU instance must expose its TPU devices under /dev/vfio (you can verify this on the instance by running ls /dev/vfio).

If you don't run the cluster health scanner as the root user, raise the memlock limit before you run TPU tests:

ulimit -l unlimited

To start a scan on the TPU instance, run the following command:

chs run --suite=SUITE

Replace SUITE with the diagnostic suite to run, either lite or standard. For more information about each suite, see Diagnostic suites.

To run specific TPU tests instead of a full suite, replace the --suite flag with the --tests flag and specify a comma-separated list of test names (for example, tpu-host-device,tpu-hbm-stress):

chs run --tests=TEST_NAMES

View runs and results

The cluster health scanner stores run history and configuration settings locally on the client machine where you run chs commands (in ~/.local/share/chs/, or in the chs-local-state Docker volume when you use the container image). As a result, the chs history and chs results commands only list runs that were started from that environment.

To list your previous runs and their run IDs, run the following command:

chs history

To display the results for a run and the locations of the interactive HTML report (bill_of_health.html) and JSON report (bill_of_health.json), run the following command:

chs results RUN_ID

If you omit RUN_ID, the cluster health scanner prompts you to select a run. The command prints local paths, or Cloud Storage URIs if you configured a bucket. It reads the stored results of a past run and doesn't start new jobs on your cluster or instance.

Cancel a scan

To cancel a scan, stop its jobs, and remove the resources that it created on your cluster or instance, run one of the following commands:

  • Cancel your most recent run:

    chs cancel
    
  • Cancel a specific run:

    chs cancel RUN_ID
    
  • Cancel every cluster health scanner run on the cluster or instance, including runs started by other users:

    chs cancel --all
    

Interpret the results

When a scan completes, the cluster health scanner saves the following artifacts to your local output directory, and uploads them to your Cloud Storage bucket if you configured storage:

  • Interactive HTML report (bill_of_health.html): shows each test, its measured metrics, and its status.
  • JSON report (bill_of_health.json): contains the parameter sweeps and measured metrics for each test.
  • Workspace directory: contains the diagnostic logs.

View the health report

To find the exact local paths or Cloud Storage URIs (gs://BUCKET/PREFIX/bills_of_health/RUN_ID.html and gs://BUCKET/PREFIX/bills_of_health/RUN_ID.json) for a scan's reports, run the following command:

chs results RUN_ID

If you configured a Cloud Storage bucket, you can view the JSON report directly from the URI returned by chs results:

gcloud storage cat gs://BUCKET_NAME/PREFIX/bills_of_health/RUN_ID.json

Report contents

The health report lists every node that the cluster health scanner tested, along with the benchmark results for each node. GPU and TPU scans use the same report format. Each node receives one of the following statuses:

Status Description
HEALTHY All tests met the expected performance for the machine type.
DEGRADED The node completed the tests, but one or more results were below the expected performance.
POTENTIALLY DEGRADED A screening test was below the expected performance, and its corresponding stress test wasn't run to confirm the result.
INCOMPLETE One or more tests failed to run, timed out, were canceled, or didn't report metrics on the node.

The bill_of_health.json file contains the following top-level fields:

  • report_id: the unique identifier for the scan run.
  • chs_version: the version of the cluster health scanner that generated the report.
  • created_at: the timestamp when the report was generated.
  • system_info: metadata about the system under test, such as the cluster type and machine type.
  • test_results: a list of test result objects. Each entry includes the following fields:
    • test_name: the name of the diagnostic test.
    • result_status: the outcome of the test, such as RESULT_STATUS_SUCCESS or RESULT_STATUS_FAILED.
    • target_resources: the nodes (node_name and instance_id) that the test ran on.
    • metrics: the measured metrics and validation criteria for the test.

The health report uses a consistent, structured JSON format. You can load reports into BigQuery or other analysis tools and compare reports from successive scans of the same capacity to detect gradual performance drift.

Remediate unhealthy nodes

If the health report identifies a node or instance with a DEGRADED, POTENTIALLY DEGRADED, or INCOMPLETE status, take the following steps:

  1. To rule out a transient issue, rerun the scan on the GKE node pool or node, Slurm partition, or standalone TPU instance before you cordon or drain the node (the cluster health scanner skips cordoned GKE nodes and drained Slurm nodes). For instructions, see Start a scan.

  2. If the issue persists, prevent new workloads from being scheduled on the affected node or instance:

    • GKE clusters: Cordon the unhealthy node:

      kubectl cordon NODE_NAME
      
    • Slurm clusters: Drain the unhealthy node:

      sudo scontrol update NodeName=NODE_NAME State=DRAIN Reason="CHS unhealthy"
      
    • Standalone TPU instances: Don't start new workloads on the TPU instance until you finish remediating the issue.

  3. Repair or replace the node or instance by using one of the following options, depending on the accelerator type and environment:

    When you contact support, attach the workspace directory from your local output directory or Cloud Storage bucket, which contains the diagnostic logs, and include the run ID.

  4. After the node or instance is repaired or replaced, make it schedulable again so that the cluster health scanner doesn't skip it, and then run a new scan to confirm that the capacity is healthy:

    • GKE clusters: Uncordon the node, and then rerun the scan before you schedule new workloads:

      kubectl uncordon NODE_NAME
      
    • Slurm clusters: Resume the node so that it returns to the idle state, and then rerun the scan before you schedule new jobs:

      sudo scontrol update NodeName=NODE_NAME State=RESUME
      
    • Standalone TPU instances: Rerun the scan on the instance, and then resume running your workloads once the scan confirms that the instance is healthy.

Telemetry

The cluster health scanner sends anonymous usage metrics to Google to help improve the tool, and doesn't collect personally identifiable information (PII). Telemetry is enabled by default. To opt out, run chs configure telemetry disable or set the CHS_TELEMETRY=0 environment variable.

What's next

To resolve common issues with the cluster health scanner, see Troubleshoot the cluster health scanner.