Known issues

This page describes known issues that you might run into while running your AI-optimized VMs or clusters. For issues with Compute Engine VMs, see Compute Engine known issues.

Issues

The following section lists known issues and workarounds for AI Hypercomputer.

Workload interruptions on A4 VMs due to firmware issues for NVIDIA B200 GPUs

NVIDIA has identified two firmware issues for B200 GPUs, which A4 VMs use. These issues can cause workload interruptions, such as the nvidia-smi command not responding. If you notice workload interruptions on A4 VMs, then confirm that one of the following conditions are true:

• The VM's uptime (lastStartTimestamp) exceeds 65 days.
• Logs show an Xid 149 error message that mentions 0x02a.

If one of these conditions is true, then you must reset the GPUs that are attached to your A4 VMs. To prevent future interruptions to your workload due to firmware issues, we recommend that you reset your GPUs at least once every 60 days. For more information, see Reset GPUs.

Note: If your A4 VMs are part of a GKE cluster, then you can use the GKE GPU node reset tool to resolve minor GPU issues, such as unresponsive nvidia-smi commands, without rebooting your VMs.

The metadata server might display old physicalHost VM metadata

After experiencing a host error or using the report faulty host API to move a compute instance to a new host, when you query the metadata server, it might display the physicalHost metadata of the compute instance's previous host.

To workaround this issue, do one of the following:

• Use the instances.get method or the gcloud compute instances describe command to retrieve the correct physicalHost information.
• Stop and then start your instance. This process updates the physicalHost information in the metadata server.
• Wait 24 hours for the impacted instance's physicalHost information to be updated.