The cluster health scanner is a self-service diagnostic tool that validates the readiness of your GPU and Cloud TPU capacity before you schedule a workload. The cluster health scanner runs benchmarks that simulate real workload traffic on your accelerator nodes to measure the performance of the accelerators, their memory, and the interconnects between them. You can use the results to identify and remove faulty nodes before they cause your training or inference jobs to stop responding, run slowly, or fail.
For TPUs, the cluster health scanner complements TPU health monitoring, which continuously reports hardware health signals. The cluster health scanner actively runs workloads on idle capacity to measure performance and expose faults that only appear under load.
The cluster health scanner is completely open source. To view the source code, contribute, or file issues, see the cluster health scanner GitHub repository.
This document describes how to install the cluster health scanner, run a health scan on GPU and TPU reservations, interpret the results, and remediate unhealthy nodes.
Supported environments
The cluster health scanner supports the following deployment environments:
| Environment | Supported accelerators | Client environment | Installation options |
|---|---|---|---|
| Google Kubernetes Engine (GKE) clusters | GPUs and TPUs (node pools that use static or dynamic slicing) | A Linux machine that has kubectl access to the target cluster |
Python package or container image |
| Slurm clusters | GPUs only | The Slurm controller or login node, where the sinfo and sbatch commands are available |
Python package only |
| Standalone TPU instances | TPUs only | The TPU instance itself | Python package or container image (Container-Optimized OS) |
The cluster health scanner supports the following GPU and TPU machine types:
- GPU machine types:
- TPU machine types:
Diagnostic suites
You can install the cluster health scanner as a Python package or as a container
image. Both options provide the chs command-line interface (CLI) and the
diagnostic benchmarks. You run the CLI from a machine with access to your
GKE cluster, on a Slurm controller or login node, or directly on
a standalone TPU instance. The CLI then runs the diagnostic tests on the target
GPU or TPU nodes.
The cluster health scanner provides two diagnostic suites for each accelerator type:
- Lite: Runs a fast screening check. On GPUs and standalone TPU instances,
the Lite suite checks individual hosts. On GKE TPU node
pools, it also runs a quick collective check (
tpu-collectives) across the TPU slice. A collective operation is a communication pattern in which all accelerators exchange data, such asAllReduce, which combines values from every accelerator and shares the result with all accelerators. Use the Lite suite for quick health checks or before you run single-host workloads. - Standard: Runs comprehensive validation across your target capacity. On GPUs, the Standard suite runs collective operations across multiple nodes and measures GPU memory bandwidth. On TPUs, it runs all Lite tests plus sustained compute, memory, and network stress tests. Use the Standard suite before you run multi-host training or inference workloads, where network performance and stability under sustained load directly affect job performance.
| Suite | Accelerator | Tests | Focus |
|---|---|---|---|
lite |
GPU | nvidia-smi, dcgm, gemm, nccl-single-node |
Fast validation of GPU and driver health, compute throughput, and intra-node GPU communication. |
| TPU | tpu-info, tpu-gemm, tpu-hbm, tpu-host-device, tpu-collectives |
Fast validation of device health, compute throughput, high-bandwidth memory (HBM) bandwidth, host-to-device bandwidth, and Inter-Chip Interconnect (ICI) collective bandwidth. | |
standard |
GPU | nvidia-smi, dcgm, gemm, nccl-cluster, nccl-pairwise, stream |
Multi-node network validation across the target nodes, plus memory bandwidth. |
| TPU | All TPU lite tests, plus tpu-compute-stress, tpu-hbm-stress, and tpu-network-stress |
Sustained stress tests that expose instability, silent data corruption, and ICI bit errors. |
The nccl-domain GPU test isn't part of a suite. To run nccl-domain, use the
--tests flag.
GPU suites and tests
The cluster health scanner provides the following suites for GPU nodes:
- Lite: Validates the GPU driver and hardware, measures compute throughput, and runs an NVIDIA Collective Communications Library (NCCL) test within each node.
- Standard: Runs NCCL collectives across all target nodes and between pairs of nodes to measure the performance of the inter-node network, and measures memory bandwidth.
The following table describes each GPU test:
| Test | Description | Suites |
|---|---|---|
nccl-single-node |
Runs NCCL collective tests between the GPUs within a single node. | lite |
nvidia-smi |
Checks basic GPU health and the GPU driver by using the NVIDIA System Management Interface. | lite, standard |
dcgm |
Runs NVIDIA Data Center GPU Manager (DCGM) diagnostics to perform deep hardware checks on each GPU. | lite, standard |
gemm |
Runs general matrix multiplications (GEMM) to measure GPU compute throughput. | lite, standard |
nccl-cluster |
Runs NCCL collective tests across all target nodes to measure cluster-wide network bandwidth. | standard |
nccl-pairwise |
Runs NCCL tests between pairs of nodes to measure node-to-node latency and isolate faulty nodes or links. | standard |
stream |
Runs the STREAM benchmark to measure GPU memory bandwidth. | standard |
nccl-domain |
Runs NCCL collective tests within each multi-node NVLink domain. Supported only on machine types with multi-node NVLink domains. | None (run with --tests) |
TPU suites and tests
The cluster health scanner provides the following suites for TPU nodes:
- Lite: Validates the performance of the TPU chips and HBM on each host, measures host-to-device bandwidth, and runs a collective operations test to measure ICI bandwidth.
- Standard: Runs collective operations across the TPU chips in a slice, plus sustained stress tests on the TPU chips, HBM, and ICI links.
The following table describes each TPU test:
| Test | Description | Suites |
|---|---|---|
tpu-info |
Checks basic TPU health, including device count, LibTPU version, and TPU device type. | lite, standard |
tpu-gemm |
Runs general matrix multiplications (GEMM) to measure sustained peak TFLOPS. | lite, standard |
tpu-hbm |
Measures HBM read and write bandwidth. | lite, standard |
tpu-collectives |
Times collective operations, such as AllReduce and AllGather, to measure ICI bandwidth (GKE clusters only). | lite, standard |
tpu-host-device |
Measures host-to-device bandwidth. | lite, standard |
tpu-compute-stress |
Runs sustained matrix multiplications to verify computational stability and expose silent data corruption. | standard |
tpu-hbm-stress |
Stresses HBM to measure maximum bandwidth and validate data integrity. | standard |
tpu-network-stress |
Sends large random payloads across ICI links to measure sustained bandwidth and detect bit errors (GKE clusters only). | standard |
Before you begin
Before you use the cluster health scanner, install the required tools, configure permissions, and connect to your cluster or instance.
Installations
Before you run a health scan, complete the installations for your deployment environment.
GKE clusters
To run health scans on a GKE cluster, make sure that you have a target cluster and install the required tools on the machine where you plan to run scans:
- Google Cloud CLI: Install and initialize the gcloud CLI.
kubectlandgke-gcloud-auth-plugin: Installkubectland configure cluster access.- Cluster health scanner prerequisites:
- Python package: Python 3.12 or later and Git.
- Container image: Docker.
- GKE cluster: Make sure that you have a GKE cluster with one or more GPU or TPU node pools. For more information, see About GPUs in GKE and About TPUs in GKE.
Slurm clusters
To run health scans on a Slurm cluster, make sure that you have a target GPU
cluster and install the required tools on the Slurm controller or login node,
where the sinfo and sbatch commands are available:
- gcloud CLI: Install and initialize the gcloud CLI.
- Python package prerequisites: Install Python 3.12 or later and Git.
- Slurm cluster: Make sure that you have a Slurm cluster with GPU nodes. For more information, see Create a fully managed Slurm cluster for AI workloads and Create a self-managed Slurm cluster for AI workloads.
For Slurm clusters, use the Python package installation option. Slurm doesn't support the container image option.
Standalone TPU instances
To run health scans on a standalone TPU instance, make sure that you have a target TPU instance and install the required tools directly on the instance:
- gcloud CLI: Install and initialize the gcloud CLI.
- Cluster health scanner prerequisites: Install the tools for the cluster
health scanner installation option that you plan to use:
- Python package: Python 3.12
(specifically Python 3.12 on Linux
x86_64, not 3.13 or later) and Git. If the defaultpython3version on your instance is earlier or later than 3.12, you can useuvto create a Python 3.12 virtual environment (see Troubleshoot the cluster health scanner). - Container image: Docker. On Container-Optimized OS, the container image is the only supported installation option.
- Python package: Python 3.12
(specifically Python 3.12 on Linux
- Standalone TPU instance: Make sure that you have a standalone TPU
instance (TPU7x or TPU v6e), which exposes its TPU devices under
/dev/vfio(you can verify this on the instance by runningls /dev/vfio). For more information, see Create Cloud TPU VMs.
Required roles
To get the permissions that you need to run a health scan, ask your
administrator to grant you the roles/compute.viewer role on your project. This
role lets the cluster health scanner read Compute Engine metadata and topology
information for your nodes.
You also need the following Identity and Access Management (IAM) roles, depending on your deployment environment:
| Environment | Required roles |
|---|---|
| GKE clusters | Kubernetes Engine Developer (roles/container.developer) |
| Slurm clusters |
|
| Standalone TPU instances |
|
If you configure a Cloud Storage bucket for reports,
you also need the Storage Object Admin role (roles/storage.objectAdmin) on the
bucket. This role lets the cluster health scanner upload result artifacts, such
as the health report, to the bucket.
For more information about granting roles, see Manage access to projects, folders, and organizations.
Connect to your cluster or instance
Connect to the environment where you plan to run health scans.
GKE clusters
Set up Application Default Credentials (ADC) and configure kubectl to
communicate with your GKE cluster:
gcloud auth application-default login
gcloud container clusters get-credentials CLUSTER_NAME \
--location=LOCATION \
--project=PROJECT_ID
Slurm clusters
Connect to your Slurm controller or login node by using SSH:
gcloud compute ssh SLURM_NODE_NAME \
--zone=ZONE \
--project=PROJECT_ID
Standalone TPU instances
Connect to your standalone TPU instance by using SSH:
gcloud compute tpus tpu-vm ssh TPU_NAME \
--zone=ZONE \
--project=PROJECT_ID
Install the cluster health scanner
Install the cluster health scanner on the machine where you run scans, which depends on your environment:
- GKE clusters: Any machine that has
kubectlaccess to your cluster. - Slurm clusters: The Slurm controller or login node.
- Standalone TPU instances: The TPU instance itself.
On GKE and Slurm clusters, the cluster health scanner automatically pulls the diagnostic container images that match your GPU or TPU nodes and schedules them on the cluster, so you don't need to install any software on the individual compute nodes. On a standalone TPU instance, you install and run the cluster health scanner directly on the instance.
You can install the cluster health scanner as a Python package or as a container
image. The chs commands in the rest of this document work with either
installation option. To see detailed usage information for any command, append
the --help flag.
Python package
Create a Python virtual environment and install the cluster health scanner
package from GitHub. On GKE and Slurm clusters, use Python 3.12
or later. On standalone TPU instances (x86_64), you must use Python 3.12 (with
Python 3.13 or later, pip install doesn't install the TPU benchmark
dependencies, and chs run skips the instance):
python3 -m venv chs-venv
source chs-venv/bin/activate
pip install git+https://github.com/GoogleCloudPlatform/cluster-health-scanner.git
To install a specific release, append @VERSION to the repository URL:
pip install git+https://github.com/GoogleCloudPlatform/cluster-health-scanner.git@VERSION
For a list of releases, see cluster health scanner releases on GitHub.
Container image
The cluster health scanner is distributed as a container image in Artifact Registry. The image supports both GPUs and TPUs. You can use the container image for GKE clusters and standalone TPU instances, but not for Slurm clusters.
Configure Docker to authenticate with Artifact Registry and pull the cluster health scanner container image:
gcloud auth configure-docker us-docker.pkg.dev docker pull us-docker.pkg.dev/gce-ai-infra/chs/chs:latestTo pull a specific release instead of
latest, replacelatestwith the version number. For example,0.3.0for GitHub releasev0.3.0. For a list of releases, see cluster health scanner releases on GitHub.Create a shell alias to run
chscommands directly. The alias that you create depends on your environment:GKE clusters: The alias mounts your
kubeconfigfile and Google Cloud credentials in read-only mode. The alias also stores the state of your runs in thechs-local-stateDocker volume and writes output files to thechs-outputdirectory in your current directory:alias chs='docker run --rm -it --network host \ -v ~/.kube:/workspace/.kube:ro \ -v ~/.config/gcloud:/workspace/.config/gcloud:ro \ -v chs-local-state:/workspace/.local/share \ -v "$(pwd)/chs-output":/workspace/chs-output \ us-docker.pkg.dev/gce-ai-infra/chs/chs:latest'Standalone TPU instances: On a TPU instance that runs Container-Optimized OS, the container image is the only supported installation option. The alias grants the container access to the TPU devices and host PID namespace of the instance, mounts the host root file system at
/host, and mounts your Google Cloud credentials in read-only mode. The alias also stores the state of your runs in thechs-local-stateDocker volume and writes output files to thechs-outputdirectory in your current directory:alias chs='docker run --rm -it --privileged --pid=host \ -v /:/host \ -v ~/.config/gcloud:/workspace/.config/gcloud:ro \ -v chs-local-state:/workspace/.local/share \ -v "$(pwd)/chs-output":/workspace/chs-output \ us-docker.pkg.dev/gce-ai-infra/chs/chs:latest'
Optional: Set up a Cloud Storage bucket for reports
By default, the cluster health scanner saves reports and diagnostic logs to a local output directory. You can configure a Cloud Storage bucket to also store them remotely.
If you don't already have a bucket for health scan reports, create one:
gcloud storage buckets create gs://BUCKET_NAME \ --location=BUCKET_LOCATION \ --project=PROJECT_IDGrant the Storage Object Admin role (
roles/storage.objectAdmin) to your principal user account or service account:gcloud storage buckets add-iam-policy-binding gs://BUCKET_NAME \ --member=PRINCIPAL \ --role=roles/storage.objectAdminThe
PRINCIPALmember is the principal that writes the report. For example,user:EMAILorserviceAccount:SERVICE_ACCOUNT_EMAIL.Set the bucket where the cluster health scanner stores reports. The bucket must already exist because the cluster health scanner doesn't automatically create it. You only need to run this command once for each environment:
chs configure storage gcs BUCKET_NAMETo store reports under a folder prefix in the bucket, append the prefix to the bucket name:
chs configure storage gcs BUCKET_NAME/PREFIX
Run a health scan
This section describes how to verify that your nodes are idle, start a scan, and monitor its progress.
Verify that the target nodes are idle
The cluster health scanner only runs on idle capacity, and doesn't automatically stop or evict your workloads. Before you start a scan, drain or stop any workloads that run on your target nodes or instance.
If the cluster health scanner detects active workloads on a target node or instance, it handles the capacity as follows:
- GPU: The cluster health scanner skips the busy nodes and runs the scan on the remaining idle nodes.
- TPU: How the cluster health scanner handles busy capacity depends on the
environment:
- GKE clusters: Multi-host TPU tests (
tpu-collectivesandtpu-network-stress) require a complete idle slice, while single-host TPU tests can run on individual idle nodes:- Static-slicing node pools (
--nodepoolsor--nodes): If some nodes in a pool are busy when you run theliteorstandardsuite, the cluster health scanner logs a warning, omits the multi-host tests, and runs the single-host TPU tests on the remaining idle nodes. - Dynamic-slicing node pools (
--nodepoolsor--nodes): Targeting a pool with--nodepoolsrequires the entire node pool to be idle so that the cluster health scanner can provision a full-poolSliceresource. If you target specific nodes with--nodes(on GKE version1.36.0-gke.3712000or later), the cluster health scanner provisions an individual2x2x1sub-slice for each idle node, skips busy nodes, and runs single-host TPU tests on the idle nodes.
- Static-slicing node pools (
- Standalone TPU instances: The cluster health scanner can't run if
another process is using the TPU devices under
/dev/vfio.
- GKE clusters: Multi-host TPU tests (
To make sure that your target capacity is idle, follow the steps for your environment.
GKE clusters
Stop or delete any active workload Jobs or Pods on the target node pool, or wait for running workloads to finish.
If you use
kubectl drainto evict workloads from the node pool, uncordon the nodes before you start the scan so that the cluster health scanner can schedule diagnostic Pods on them (the cluster health scanner skips cordoned nodes):kubectl drain -l cloud.google.com/gke-nodepool=NODE_POOL_NAME \ --ignore-daemonsets --delete-emptydir-data kubectl uncordon -l cloud.google.com/gke-nodepool=NODE_POOL_NAMEMake sure that no new workloads are scheduled on the node pool while the scan runs.
Verify that no workload Pods remain on the node pool:
kubectl get pods --all-namespaces -o wide \ --field-selector=status.phase=Running | grep NODE_POOL_NAMESystem Pods, such as those managed by DaemonSets in the
kube-systemnamespace, can continue to run.
Slurm clusters
Check for running jobs in the target partition:
squeue --partition=PARTITION_NAMEIf jobs are running in the partition, wait for them to finish or cancel them with
scancel JOB_ID.Verify that the nodes in the target partition are in the
idlestate:sinfo --partition=PARTITION_NAME
Standalone TPU instances
Verify that no other processes are using the TPU devices on the instance:
sudo fuser -v /dev/vfio/*
If any processes are listed, wait for the workloads to finish or stop the processes before you start a scan.
Start a scan
The flags that you pass to the chs run command depend on the environment where
you run the cluster health scanner. The cluster health scanner automatically
detects the scheduler and the accelerator type, so you don't need to specify
them.
To list the available diagnostic suites and tests for an accelerator type, run the following commands:
For GPUs:
chs list suites --gpu chs list tests --gpuFor TPUs:
chs list suites --tpu chs list tests --tpu
GKE node pools
Run the cluster health scanner from any machine that has kubectl access to
your cluster, such as your local machine or Cloud Shell.
To start a scan on one or more node pools, run the following command:
chs run \
--nodepools=NODE_POOL_NAMES \
--suite=SUITE
Replace the following:
NODE_POOL_NAMES: a comma-separated list of the GPU or TPU node pools to scan. For example:a4-pool-1,a4-pool-2ortpu7x-pool-1,tpu7x-pool-2. All target node pools must use the same machine type.SUITE: the diagnostic suite to run, eitherliteorstandard. For more information about each suite, see Diagnostic suites.
To run specific tests instead of a full suite, replace the --suite flag with
the --tests flag and specify a comma-separated list of test names:
chs run \
--nodepools=NODE_POOL_NAMES \
--tests=TEST_NAMES
To scan specific nodes instead of an entire node pool, replace the --nodepools
flag with the --nodes flag and specify a comma-separated list of node names.
When you specify multiple node pools in the --nodepools flag, the cluster
health scanner handles them based on the accelerator type:
- GPU node pools: Combines all idle GPU nodes across the specified node
pools into a single target set and runs multi-node tests (
nccl-clusterandnccl-pairwise) across all of them together. - TPU node pools: Evaluates each node pool (TPU slice) independently and in parallel within a single run, without cross-slice (Multislice) testing.
Slurm partitions
Run the cluster health scanner on the Slurm controller or login node, where the
sinfo and sbatch commands are available. Slurm requires the Python package
installation option. For installation instructions, see Python
package.
Before you start a scan, configure Enroot credentials on the cluster so that the
nodes can pull benchmark images from us-docker.pkg.dev:
gcloud auth configure-docker us-docker.pkg.dev
mkdir -p ~/.enroot
cat <<EOF > "${HOME}/.enroot/.credentials"
machine us-docker.pkg.dev login oauth2accesstoken password $(gcloud auth application-default print-access-token)
EOF
Start a scan on one or more partitions:
chs run \
--partitions=PARTITION_NAMES \
--suite=SUITE
Replace the following:
PARTITION_NAMES: a comma-separated list of the Slurm partitions to scan.SUITE: the diagnostic suite to run, eitherliteorstandard. For more information about each suite, see Diagnostic suites.
To run specific GPU tests instead of a full suite, replace the --suite flag
with the --tests flag and specify a comma-separated list of test names (for
example, nccl-pairwise,gemm or nccl-domain):
chs run \
--partitions=PARTITION_NAMES \
--tests=TEST_NAMES
Standalone TPU instances
Run the cluster health scanner directly on the TPU instance. The TPU instance
must expose its TPU devices under /dev/vfio (you can verify this on the
instance by running ls /dev/vfio).
If you don't run the cluster health scanner as the root user, raise the
memlock limit before you run TPU tests:
ulimit -l unlimited
To start a scan on the TPU instance, run the following command:
chs run --suite=SUITE
Replace SUITE with the diagnostic suite to run, either lite or standard.
For more information about each suite, see Diagnostic
suites.
To run specific TPU tests instead of a full suite, replace the --suite flag
with the --tests flag and specify a comma-separated list of test names (for
example, tpu-host-device,tpu-hbm-stress):
chs run --tests=TEST_NAMES
View runs and results
The cluster health scanner stores run history and configuration settings locally
on the client machine where you run chs commands (in ~/.local/share/chs/, or
in the chs-local-state Docker volume when you use the container image). As a
result, the chs history and chs results commands only list runs that were
started from that environment.
To list your previous runs and their run IDs, run the following command:
chs history
To display the results for a run and the locations of the interactive HTML
report (bill_of_health.html) and JSON report (bill_of_health.json), run the
following command:
chs results RUN_ID
If you omit RUN_ID, the cluster health scanner prompts you to select a run.
The command prints local paths, or Cloud Storage URIs if you configured
a bucket. It reads the stored results of a past run and doesn't start new jobs
on your cluster or instance.
Cancel a scan
To cancel a scan, stop its jobs, and remove the resources that it created on your cluster or instance, run one of the following commands:
Cancel your most recent run:
chs cancelCancel a specific run:
chs cancel RUN_IDCancel every cluster health scanner run on the cluster or instance, including runs started by other users:
chs cancel --all
Interpret the results
When a scan completes, the cluster health scanner saves the following artifacts to your local output directory, and uploads them to your Cloud Storage bucket if you configured storage:
- Interactive HTML report (
bill_of_health.html): shows each test, its measured metrics, and its status. - JSON report (
bill_of_health.json): contains the parameter sweeps and measured metrics for each test. - Workspace directory: contains the diagnostic logs.
View the health report
To find the exact local paths or Cloud Storage URIs
(gs://BUCKET/PREFIX/bills_of_health/RUN_ID.html and
gs://BUCKET/PREFIX/bills_of_health/RUN_ID.json) for a scan's reports, run the
following command:
chs results RUN_ID
If you configured a Cloud Storage bucket, you can view the JSON report
directly from the URI returned by chs results:
gcloud storage cat gs://BUCKET_NAME/PREFIX/bills_of_health/RUN_ID.json
Report contents
The health report lists every node that the cluster health scanner tested, along with the benchmark results for each node. GPU and TPU scans use the same report format. Each node receives one of the following statuses:
| Status | Description |
|---|---|
HEALTHY |
All tests met the expected performance for the machine type. |
DEGRADED |
The node completed the tests, but one or more results were below the expected performance. |
POTENTIALLY DEGRADED |
A screening test was below the expected performance, and its corresponding stress test wasn't run to confirm the result. |
INCOMPLETE |
One or more tests failed to run, timed out, were canceled, or didn't report metrics on the node. |
The bill_of_health.json file contains the following top-level fields:
report_id: the unique identifier for the scan run.chs_version: the version of the cluster health scanner that generated the report.created_at: the timestamp when the report was generated.system_info: metadata about the system under test, such as the cluster type and machine type.test_results: a list of test result objects. Each entry includes the following fields:test_name: the name of the diagnostic test.result_status: the outcome of the test, such asRESULT_STATUS_SUCCESSorRESULT_STATUS_FAILED.target_resources: the nodes (node_nameandinstance_id) that the test ran on.metrics: the measured metrics and validation criteria for the test.
The health report uses a consistent, structured JSON format. You can load reports into BigQuery or other analysis tools and compare reports from successive scans of the same capacity to detect gradual performance drift.
Remediate unhealthy nodes
If the health report identifies a node or instance with a DEGRADED,
POTENTIALLY DEGRADED, or INCOMPLETE status, take the following steps:
To rule out a transient issue, rerun the scan on the GKE node pool or node, Slurm partition, or standalone TPU instance before you cordon or drain the node (the cluster health scanner skips cordoned GKE nodes and drained Slurm nodes). For instructions, see Start a scan.
If the issue persists, prevent new workloads from being scheduled on the affected node or instance:
GKE clusters: Cordon the unhealthy node:
kubectl cordon NODE_NAMESlurm clusters: Drain the unhealthy node:
sudo scontrol update NodeName=NODE_NAME State=DRAIN Reason="CHS unhealthy"Standalone TPU instances: Don't start new workloads on the TPU instance until you finish remediating the issue.
Repair or replace the node or instance by using one of the following options, depending on the accelerator type and environment:
- GPU:
- GKE clusters: Report the faulty host so that Compute Engine can repair it. For more information, see Report faulty hosts through GKE.
- Slurm clusters: Report the faulty host so that Compute Engine can repair it. For more information, see Report faulty host.
- To get help from Google, contact Cloud Customer Care.
- TPU (GKE clusters and standalone TPU instances):
- If you use TPUs in All Capacity mode, report the faulty host. For more information, see Report and repair faulty hosts in All Capacity mode.
- For other capacity modes, see Failure recovery of TPU instances.
- To get help from Google, contact Cloud TPU support.
When you contact support, attach the workspace directory from your local output directory or Cloud Storage bucket, which contains the diagnostic logs, and include the run ID.
- GPU:
After the node or instance is repaired or replaced, make it schedulable again so that the cluster health scanner doesn't skip it, and then run a new scan to confirm that the capacity is healthy:
GKE clusters: Uncordon the node, and then rerun the scan before you schedule new workloads:
kubectl uncordon NODE_NAMESlurm clusters: Resume the node so that it returns to the
idlestate, and then rerun the scan before you schedule new jobs:sudo scontrol update NodeName=NODE_NAME State=RESUMEStandalone TPU instances: Rerun the scan on the instance, and then resume running your workloads once the scan confirms that the instance is healthy.
Telemetry
The cluster health scanner sends anonymous usage metrics to Google to help
improve the tool, and doesn't collect personally identifiable information (PII).
Telemetry is enabled by default. To opt out, run chs configure telemetry
disable or set the CHS_TELEMETRY=0 environment variable.
What's next
To resolve common issues with the cluster health scanner, see Troubleshoot the
cluster health scanner.