Use this document to identify the optimal accelerator infrastructure for your artificial intelligence (AI) and machine learning (ML) workloads based on your lifecycle stage, such as small-scale prototyping, real-time inference, and massive distributed training. AI Hypercomputer organizes accelerator offerings into two categories: general GPUs and clustered GPUs.
GPU infrastructure
AI Hypercomputer offers two distinct categories of GPUs. Each category has a distinct management model that determines how the system handles lifecycle events, manages maintenance, and optimizes networking for your workload:
General GPU management model
The general GPU management model is designed for workloads that prioritize resource independence and high availability. This model uses the standard Google Cloud management plane, providing a familiar experience for teams already using Compute Engine or GKE.
Maintenance: asynchronous. Individual instances are updated independently. In a multinode serving fleet, a maintenance event on one node doesn't impact the availability of others, allowing traffic to be redirected to healthy nodes without stalling the entire job.
Primary use cases: recommended for mainstream tasks like real-time inference, model serving, small-scale prototyping, and development environments where supercomputing scale is not required.
General GPU machine series
The following machine series are general GPUs:
- G2 (L4)
- G4 (RTX PRO 6000)
- A2 (A100)
- N1+T4
- A3 Edge
- A3 High (1, 2, or 4 GPUs)
For technical information about each of the general GPU machines, see GPU machine types.
Clustered GPU management model
The clustered GPU management model is a supercomputing-class environment engineered for large-scale distributed training. Resources are managed as a single, tightly coupled system by using a clustered GPU stack.
Maintenance: coordinated. This model coordinates maintenance events to support tightly coupled workloads. For distributed training, you can use synchronized maintenance, where updates are applied across all nodes simultaneously. This approach prevents node restarts from stalling jobs and maximizes goodput. This model also offers 90-day maintenance notifications.
Primary use cases: designed for training exascale foundation models with trillions of parameters or running complex high performance computing (HPC) simulations, such as drug discovery and protein folding.
Clustered GPU machine series
The following machine series are clustered GPUs:
- A4X
- A4 (Blackwell)
- A3 Ultra
- A3 Mega
- A3 High (8 GPUs)
For technical information about each of the clustered GPU machines, see GPU machine types.
Capability matrix
Compare the technical and operational characteristics of the general GPU and clustered GPU management models:
| Characteristic | General GPU management model | Clustered GPU management model |
|---|---|---|
| Primary use cases | Inference, model serving, prototyping, and development | Large-scale distributed training and HPC |
| Maintenance | Asynchronous: Independent node updates allow traffic to be redirected without stalling jobs | Coordinated: Supports cluster-wide coordinated (synchronized) updates to keep nodes in sync and maximize goodput |
| Target hardware | G2 (L4), G4 (RTX PRO 6000), A2 (A100), N1+T4, A3 Edge, and A3 High (1, 2, or 4 GPUs) | A4X, A4 (Blackwell), A3 Ultra, A3 Mega, and A3 High (8 GPUs) |
| Networking | Standard VPC networking over gVNIC interfaces (with the exception of A3 Edge, which uses GPUDirect-TCPX over multiple VPCs) | Specialized low-latency fabrics (RDMA, RoCE, TCPX, or NVIDIA NVLink) |
| Management stack | Standard Google Cloud plane (Compute Engine or GKE) | Specialized management stack (for example, Cluster Director) |
| Reliability | Standard node auto-repair and pod-level uptime | Specialized Resource and Recovery suite |
Decision matrix
Use this matrix to identify which hardware families align with your specific workload intent:
| If your workload involves... | Recommended GPU infrastructure | Recommended machine series |
|---|---|---|
| Prototyping and development | General GPUs | G2, G4, A2, N1+T4, A3 Edge |
| Real-time inference (< 100B parameters) | General GPUs | G2, G4, A2, N1+T4, A3 Edge |
| Inference across multiple GPUs | Clustered GPUs | A3, A4 |
| Large-scale training and inference | Clustered GPUs | A4, A3 series |
| Massive-scale training (> 500B parameters) and disaggregated inference | Clustered GPUs | A4X Max, A4X, A4 |
For a detailed decision tree mapping model architectures directly to hardware, see Choose an accelerator.
After establishing these primary criteria, choose an orchestration platform. This choice depends on your team's preference for automated management or granular control over the operating system and drivers.
What's next
- To select an orchestrator and deployment option, see Choose an orchestrator and deployment option.
- To deploy inference workloads or run large-scale training, see AI Hypercomputer tutorials.
- To optimize your system efficiency, see Goodput optimization recipes.