Generate a bug report for Blackwell and later GPUs

This document explains how to generate a comprehensive GPU bug report for Compute Engine VMs and Google Kubernetes Engine (GKE) nodes that use NVIDIA Blackwell GPUs and later architectures. To identify which GPU model your machine type uses, see GPU models.

For NVIDIA Blackwell and later GPU architectures, the standard nvidia-bug-report.sh script does not capture all of the low-level hardware and fabric data required to troubleshoot issues such as GPU Xid errors or multi-node performance degradation. A comprehensive GPU bug report includes the following diagnostic data:

  • Standard GPU and system diagnostics: GPU state, driver logs, PCI bus information, and kernel logs from nvidia-bug-report.sh, along with GPU, NIC, and CPU interconnect topology, storage I/O statistics, and RDMA device and port status (which is useful for machine types with NVIDIA ConnectX NIC links).
  • NVIDIA Firmware Tools (MFT) data: On Blackwell and later GPUs, nvidia-bug-report.sh uses NVIDIA MFT to collect the physical layer status of NVLink connections, internal GPU register values, and raw diagnostic segments from the firmware.
  • NVIDIA IMEX domain diagnostics: Multi-node NVLink machine types, such as A4X and A4X Max, rely on the NVIDIA Internode Memory Exchange (IMEX) daemon to manage the multi-node NVLink domain. Diagnosing NVLink domain issues requires the IMEX domain status across nodes and the IMEX daemon logs.

Before you begin

For the full prerequisites and setup instructions, see the GPU bug report README. The following list summarizes the prerequisites:

  • Ensure that the NVIDIA GPU driver is installed on your Compute Engine VM or GKE node.
  • On Compute Engine VMs, configure Docker to authenticate to Artifact Registry so that Docker can pull the collector container image.
  • Optional: On Compute Engine VMs that run a non-COS OS (for example, Ubuntu, Debian, or a RHEL-family OS), install and configure the NVIDIA Container Toolkit for Docker. If the toolkit isn't available, run-goog-gpu-bug-report.sh tries to mount the host's NVIDIA driver directories instead.
  • Optional: To upload the bug report to a Cloud Storage bucket, verify that the service account of the VM or GKE node has write access to the bucket.

Generate a comprehensive GPU bug report

Use the open-source Google Cloud GPU Bug Report Collector to collect all of the preceding diagnostic logs in a single command across Container-Optimized OS (COS) and other Linux distributions, including Ubuntu, Debian, and RHEL-family guest operating systems (such as Rocky Linux), on both x86 (linux/amd64) and Arm (linux/arm64) instances.

How MFT kernel modules are handled

For Blackwell and later GPUs, the collector automatically installs the MFT userspace tools inside the container and handles the MFT kernel modules (mst_pci and mst_pciconf) based on your guest operating system:

  • Container-Optimized OS (COS): Downloads and loads the pre-compiled, COS-signed MFT kernel modules that match your running COS kernel version.
  • Other Linux distributions (including Ubuntu, Debian, and RHEL-family OSes): Attempts to build and load the MFT kernel modules from source using DKMS against the host's running kernel headers.
  • Automatic driverless fallback: On both COS and other Linux distributions, if loading or building the MFT kernel modules fails, the collector automatically falls back to driverless MFT mode, which still collects the GPU NVLink diagnostics through the NVIDIA driver.

Run the collector

  1. Connect to your Linux GPU instance.

  2. Run the collector for your environment:

    • On a Compute Engine VM: Download and run run-goog-gpu-bug-report.sh:

      bash run-goog-gpu-bug-report.sh
      

      To also upload the generated .tar.gz archive to a Cloud Storage bucket, pass --gcs_bucket after --:

      bash run-goog-gpu-bug-report.sh -- --gcs_bucket=BUCKET_NAME
      
    • On a GKE cluster: Update GKE_NODE_NAME and NUMBER_OF_GPUS in bug-report-pod.yaml and apply the Pod manifest (see the README for details):

      kubectl apply -f bug-report-pod.yaml
      

Collector flags and output location

By default, the collector writes the individual diagnostic files and bundles them into a single archive at the following path on the host VM:

/tmp/goog_gpu_bug_reports/utc_TIMESTAMP/vm_id_VM_ID/goog_gpu_bug_report_VM_ID_utc_TIMESTAMP.tar.gz

You can pass the following collector flags (after -- when using run-goog-gpu-bug-report.sh):

Collector flag Description
--gcs_bucket=BUCKET_NAME Uploads the bundled .tar.gz bug report archive to the specified Cloud Storage bucket (gs://BUCKET_NAME/bug_report/utc_TIMESTAMP/vm_id_VM_ID/).
--mft_kernel_modules=auto Default. Attempts to load MFT kernel modules (COS-signed modules on COS, or DKMS build on other Linux distributions) and automatically falls back to driverless MFT if module loading fails.
--mft_kernel_modules=never Skips building or loading MFT kernel modules on the host (and unloads them if already loaded), running MFT in driverless mode.

For example, to run MFT in driverless mode without loading kernel modules on the host and upload the report to a Cloud Storage bucket:

bash run-goog-gpu-bug-report.sh -- --mft_kernel_modules=never --gcs_bucket=BUCKET_NAME

For direct docker run commands, custom image builds, and instructions on verifying the MFT output inside nvidia-bug-report.log.gz, see the collector's README.