This document explains how to generate a comprehensive GPU bug report for Compute Engine VMs and Google Kubernetes Engine (GKE) nodes that use NVIDIA Blackwell GPUs and later architectures. To identify which GPU model your machine type uses, see GPU models.
For NVIDIA Blackwell and later GPU architectures, the standard
nvidia-bug-report.sh
script does not capture all of the low-level hardware and fabric data required
to troubleshoot issues such as
GPU Xid errors
or multi-node performance degradation. A comprehensive GPU bug report includes
the following diagnostic data:
- Standard GPU and system diagnostics: GPU state, driver logs, PCI bus
information, and kernel logs from
nvidia-bug-report.sh, along with GPU, NIC, and CPU interconnect topology, storage I/O statistics, and RDMA device and port status (which is useful for machine types with NVIDIA ConnectX NIC links). - NVIDIA Firmware Tools (MFT) data: On Blackwell and later GPUs,
nvidia-bug-report.shuses NVIDIA MFT to collect the physical layer status of NVLink connections, internal GPU register values, and raw diagnostic segments from the firmware. - NVIDIA IMEX domain diagnostics: Multi-node NVLink machine types, such as A4X and A4X Max, rely on the NVIDIA Internode Memory Exchange (IMEX) daemon to manage the multi-node NVLink domain. Diagnosing NVLink domain issues requires the IMEX domain status across nodes and the IMEX daemon logs.
Before you begin
For the full prerequisites and setup instructions, see the GPU bug report README. The following list summarizes the prerequisites:
- Ensure that the NVIDIA GPU driver is installed on your Compute Engine VM or GKE node.
- On Compute Engine VMs, configure Docker to authenticate to Artifact Registry so that Docker can pull the collector container image.
- Optional: On Compute Engine VMs that run a non-COS OS (for example,
Ubuntu, Debian, or a RHEL-family OS), install and configure the NVIDIA
Container Toolkit for Docker. If the toolkit isn't available,
run-goog-gpu-bug-report.shtries to mount the host's NVIDIA driver directories instead. - Optional: To upload the bug report to a Cloud Storage bucket, verify that the service account of the VM or GKE node has write access to the bucket.
Generate a comprehensive GPU bug report
Use the open-source
Google Cloud GPU Bug Report Collector
to collect all of the preceding diagnostic logs in a single command across
Container-Optimized OS (COS) and other Linux distributions, including
Ubuntu, Debian, and RHEL-family guest operating systems (such as Rocky Linux),
on both x86 (linux/amd64) and Arm (linux/arm64) instances.
How MFT kernel modules are handled
For Blackwell and later GPUs, the collector automatically installs the MFT
userspace tools inside the container and handles the MFT kernel modules
(mst_pci and mst_pciconf) based on your guest operating system:
- Container-Optimized OS (COS): Downloads and loads the pre-compiled, COS-signed MFT kernel modules that match your running COS kernel version.
- Other Linux distributions (including Ubuntu, Debian, and RHEL-family OSes): Attempts to build and load the MFT kernel modules from source using DKMS against the host's running kernel headers.
- Automatic driverless fallback: On both COS and other Linux distributions, if loading or building the MFT kernel modules fails, the collector automatically falls back to driverless MFT mode, which still collects the GPU NVLink diagnostics through the NVIDIA driver.
Run the collector
Run the collector for your environment:
On a Compute Engine VM: Download and run
run-goog-gpu-bug-report.sh:bash run-goog-gpu-bug-report.shTo also upload the generated
.tar.gzarchive to a Cloud Storage bucket, pass--gcs_bucketafter--:bash run-goog-gpu-bug-report.sh -- --gcs_bucket=BUCKET_NAMEOn a GKE cluster: Update
GKE_NODE_NAMEandNUMBER_OF_GPUSinbug-report-pod.yamland apply the Pod manifest (see the README for details):kubectl apply -f bug-report-pod.yaml
Collector flags and output location
By default, the collector writes the individual diagnostic files and bundles them into a single archive at the following path on the host VM:
/tmp/goog_gpu_bug_reports/utc_TIMESTAMP/vm_id_VM_ID/goog_gpu_bug_report_VM_ID_utc_TIMESTAMP.tar.gz
You can pass the following collector flags (after -- when using
run-goog-gpu-bug-report.sh):
| Collector flag | Description |
|---|---|
--gcs_bucket=BUCKET_NAME |
Uploads the bundled .tar.gz bug report archive to the specified Cloud Storage bucket (gs://BUCKET_NAME/bug_report/utc_TIMESTAMP/vm_id_VM_ID/). |
--mft_kernel_modules=auto |
Default. Attempts to load MFT kernel modules (COS-signed modules on COS, or DKMS build on other Linux distributions) and automatically falls back to driverless MFT if module loading fails. |
--mft_kernel_modules=never |
Skips building or loading MFT kernel modules on the host (and unloads them if already loaded), running MFT in driverless mode. |
For example, to run MFT in driverless mode without loading kernel modules on the host and upload the report to a Cloud Storage bucket:
bash run-goog-gpu-bug-report.sh -- --mft_kernel_modules=never --gcs_bucket=BUCKET_NAME
For direct docker run commands, custom image builds, and instructions on
verifying the MFT output inside nvidia-bug-report.log.gz, see the collector's
README.