About GPU accelerators

AI Hypercomputer provides vertically integrated infrastructure optimized for demanding artificial intelligence (AI) and machine learning (ML) workloads. High-performance NVIDIA GPUs are categorized by how the system handles their lifecycle events and maintenance: General GPUs and Clustered GPUs.

To identify the optimal infrastructure category for your workload, see Choose your accelerator infrastructure.

General GPUs

General GPUs are managed as independent compute units, by using standard Compute Engine and Google Kubernetes Engine (GKE) APIs to integrate directly with Google Cloud's control plane.

General GPUs include the following machine series:

  • G2 (NVIDIA L4): optimized for mainstream inference and RAG.
  • A2 (NVIDIA A100): optimized for high-performance single-node serving and small-scale fine-tuning.
  • A3 High (1, 2, or 4 GPUs): optimized for standard training and inference workloads that don't require an 8-GPU synchronized cluster.
  • A3 Edge (NVIDIA H100): optimized for inference that is distributed across multiple hosts with high throughput using standard VPC networking.
  • G4 (NVIDIA RTX PRO 6000): optimized for cost-effective entry-level inference and graphics workloads.
  • N1 (NVIDIA T4 or V100): optimized for entry-level inference and research tasks that prioritize cost over high-performance interconnects.

Clustered GPUs

Clustered GPUs are managed as a single, tightly coupled system of thousands of interconnected accelerators using Cluster Director.

Clustered GPUs include the following machine series:

  • A4X Max: designed for bare-metal exascale training and high performance computing, featuring NVIDIA GB300 Grace Blackwell Ultra Superchips.
  • A4X and A4: designed for foundation model training and exascale foundation model training.
  • A3 series (Ultra, Mega, High with 8 GPUs): optimized for large-scale distributed training and multi-host inference involving massive data tables.

GPU performance considerations

To evaluate and optimize your workload's performance, you can track hardware-level metrics such as CUDA core and Tensor Core utilization. To monitor your resources and evaluate performance across both General and Clustered environments, see the following:

  • Monitoring dashboards: track real-time GPU utilization and memory usage for individual standalone instances or orchestrated cluster fleets.
  • Goodput metrics recipes: measure the percentage of time your accelerators perform useful computation versus system overhead, which is highly recommended for large-scale distributed training on clustered hardware.

What's next