This document explains how to generate a comprehensive GPU bug report for
Compute Engine VMs and Google Kubernetes Engine (GKE) nodes that use
[NVIDIA Blackwell GPUs](https://www.nvidia.com/en-us/data-center/technologies/blackwell-architecture/)
and later architectures. To identify which GPU model your machine type uses, see
[GPU models](https://docs.cloud.google.com/compute/docs/gpus.md.txt#gpu-models).

For NVIDIA Blackwell and later GPU architectures, the standard
[`nvidia-bug-report.sh`](https://docs.nvidia.com/deploy/pdf/RMA_Process.pdf)
script does not capture all of the low-level hardware and fabric data required
to troubleshoot issues such as
[GPU Xid errors](https://docs.nvidia.com/deploy/xid-errors/index.html#xid-error-listing)
or multi-node performance degradation. A comprehensive GPU bug report includes
the following diagnostic data:

- **Standard GPU and system diagnostics:** GPU state, driver logs, PCI bus information, and kernel logs from `nvidia-bug-report.sh`, along with GPU, NIC, and CPU interconnect topology, storage I/O statistics, and RDMA device and port status (which is useful for machine types with NVIDIA ConnectX NIC links).
- **NVIDIA Firmware Tools (MFT) data:** On Blackwell and later GPUs, `nvidia-bug-report.sh` uses NVIDIA MFT to collect the physical layer status of NVLink connections, internal GPU register values, and raw diagnostic segments from the firmware.
- **NVIDIA IMEX domain diagnostics:** Multi-node NVLink machine types, such as A4X and A4X Max, rely on the NVIDIA Internode Memory Exchange (IMEX) daemon to manage the multi-node NVLink domain. Diagnosing NVLink domain issues requires the IMEX domain status across nodes and the IMEX daemon logs.

## Before you begin

For the full prerequisites and setup instructions, see the GPU bug report
[README](https://github.com/GoogleCloudPlatform/cluster-toolkit/blob/main/community/goog-gpu-bug-report/README.md).
The following list summarizes the prerequisites:

- Ensure that the NVIDIA GPU driver is installed on your Compute Engine VM or GKE node.
- On Compute Engine VMs, configure Docker to authenticate to Artifact Registry so that Docker can pull the collector container image.
- Optional: On Compute Engine VMs that run a non-COS OS (for example, Ubuntu, Debian, or a RHEL-family OS), install and configure the NVIDIA Container Toolkit for Docker. If the toolkit isn't available, `run-goog-gpu-bug-report.sh` tries to mount the host's NVIDIA driver directories instead.
- Optional: To upload the bug report to a Cloud Storage bucket, verify that the service account of the VM or GKE node has write access to the bucket.

## Generate a comprehensive GPU bug report

Use the open-source
[Google Cloud GPU Bug Report Collector](https://github.com/GoogleCloudPlatform/cluster-toolkit/tree/main/community/goog-gpu-bug-report)
to collect all of the preceding diagnostic logs in a single command across
Container-Optimized OS (COS) and other Linux distributions, including
Ubuntu, Debian, and RHEL-family guest operating systems (such as Rocky Linux),
on both x86 (`linux/amd64`) and Arm (`linux/arm64`) instances.

### How MFT kernel modules are handled

For Blackwell and later GPUs, the collector automatically installs the MFT
userspace tools inside the container and handles the MFT kernel modules
(`mst_pci` and `mst_pciconf`) based on your guest operating system:

- **Container-Optimized OS (COS):** Downloads and loads the pre-compiled, COS-signed MFT kernel modules that match your running COS kernel version.
- **Other Linux distributions (including Ubuntu, Debian, and RHEL-family OSes):** Attempts to build and load the MFT kernel modules from source using DKMS against the host's running kernel headers.
- **Automatic driverless fallback:** On both COS and other Linux distributions, if loading or building the MFT kernel modules fails, the collector automatically falls back to driverless MFT mode, which still collects the GPU NVLink diagnostics through the NVIDIA driver.

### Run the collector

1. [Connect to your Linux GPU instance](https://docs.cloud.google.com/compute/docs/connect/standard-ssh.md.txt).

2. Run the collector for your environment:

   - **On a Compute Engine VM:** Download and run
     [`run-goog-gpu-bug-report.sh`](https://github.com/GoogleCloudPlatform/cluster-toolkit/blob/main/community/goog-gpu-bug-report/run-goog-gpu-bug-report.sh):

     ```bash
     bash run-goog-gpu-bug-report.sh
     ```

     To also upload the generated `.tar.gz` archive to a Cloud Storage bucket,
     pass `--gcs_bucket` after `--`:

     ```bash
     bash run-goog-gpu-bug-report.sh -- --gcs_bucket=BUCKET_NAME
     ```
   - **On a GKE cluster:** Update
     `GKE_NODE_NAME` and
     `NUMBER_OF_GPUS` in
     [`bug-report-pod.yaml`](https://github.com/GoogleCloudPlatform/cluster-toolkit/blob/main/community/goog-gpu-bug-report/bug-report-pod.yaml)
     and apply the Pod manifest (see the
     [README](https://github.com/GoogleCloudPlatform/cluster-toolkit/blob/main/community/goog-gpu-bug-report/README.md#to-run-in-a-gke-cluster)
     for details):

     ```bash
     kubectl apply -f bug-report-pod.yaml
     ```

### Collector flags and output location

By default, the collector writes the individual diagnostic files and bundles
them into a single archive at the following path on the host VM:

    /tmp/goog_gpu_bug_reports/utc_TIMESTAMP/vm_id_VM_ID/goog_gpu_bug_report_VM_ID_utc_TIMESTAMP.tar.gz

You can pass the following collector flags (after `--` when using
`run-goog-gpu-bug-report.sh`):

| Collector flag | Description |
|---|---|
| `--gcs_bucket=BUCKET_NAME` | Uploads the bundled `.tar.gz` bug report archive to the specified Cloud Storage bucket (`gs://BUCKET_NAME/bug_report/utc_TIMESTAMP/vm_id_VM_ID/`). |
| `--mft_kernel_modules=auto` | **Default.** Attempts to load MFT kernel modules (COS-signed modules on COS, or DKMS build on other Linux distributions) and automatically falls back to driverless MFT if module loading fails. |
| `--mft_kernel_modules=never` | Skips building or loading MFT kernel modules on the host (and unloads them if already loaded), running MFT in driverless mode. |

For example, to run MFT in driverless mode without loading kernel modules on the
host and upload the report to a Cloud Storage bucket:

```bash
bash run-goog-gpu-bug-report.sh -- --mft_kernel_modules=never --gcs_bucket=BUCKET_NAME
```

For direct `docker run` commands, custom image builds, and instructions on
verifying the MFT output inside `nvidia-bug-report.log.gz`, see the collector's
[README](https://github.com/GoogleCloudPlatform/cluster-toolkit/blob/main/community/goog-gpu-bug-report/README.md).