View GKE node topology

This document shows you how to view the physical topology of your Google Kubernetes Engine (GKE) nodes on A4X Max, A4X, A4, A3 Ultra, A3 Mega, and A3 High Compute Engine instances. Understanding the physical location of your nodes helps you optimize Pod placement for your AI workloads.

Before you begin

Before you start, make sure that you have performed the following tasks:

• Enable the Google Kubernetes Engine API.
Enable Google Kubernetes Engine API
• To use the Google Cloud CLI for this task, install and then initialize the gcloud CLI. If you previously installed the gcloud CLI, get the latest version by running the gcloud components update command. Earlier gcloud CLI versions might not support running the commands in this document.
  Note: For existing gcloud CLI installations, make sure to set the compute/region property. If you use primarily zonal clusters, set the compute/zone instead. By setting a default location, you can avoid errors in the gcloud CLI like the following: One of [--zone, --region] must be supplied: Please specify location. You might need to specify the location in certain commands if the location of your cluster differs from the default that you set.

Connect to your cluster

• Run the following command to connect to your cluster:

  gcloud container clusters get-credentials CLUSTER_NAME
  

  Replace CLUSTER_NAME with the name of your cluster.

Understand GKE node topology

You can understand the physical topology of GKE nodes on A4X Max, A4X, A4, and A3 Ultra Compute Engine instances by referring to the following node labels:

• cloud.google.com/gce-topology-block: the organization-specific ID of the reserved block where the VM is located. A block is a collection of sub-blocks connected by a layer of distributed network fabric.
• cloud.google.com/gce-topology-subblock: the organization-specific ID of the sub-block where the VM is located. A sub-block is a group of hosts and associated connectivity hardware:
  • For A4 and A3 Ultra VMs, the hosts connect through a large-scale distributed Jupiter network fabric.
  • A4X Max and A4X Compute Engine instances are organized into sub-blocks, each constituting an NVIDIA NVLink Domain (NVL72). Within each NVL72, GPUs are tightly interconnected through a dedicated high-bandwidth NVLink fabric. Across sub-blocks, GPUs can connect through RDMA NICs by using the RoCE fabric. Host communication across sub-blocks, as well as other network traffic such as storage, uses the Jupiter network fabric through high-speed frontend Ethernet NICs.
• cloud.google.com/gce-topology-host: the organization-specific ID of the host where the VM is located. A host or node is a single physical server machine in the data center. Each GKE node is provisioned on a VM instance that is provisioned on top of a physical host.
• kubernetes.io/hostname: the hostname of the Kubernetes node. This is typically also the GKE node name.

Note: For A4X Max, A4X, A4, and A3 Ultra nodes, GKE adds topology labels automatically because compact placement is enabled by default. For A3 Mega, A3 High, or other supported machine types, when you provision the nodes as Spot VMs, you must define compact placement on the GKE nodes before you can view their physical topology for Topology Aware Scheduling (TAS). Otherwise, the topology labels don't appear on the nodes. For more information about provisioning A3 Mega and A3 High nodes with a placement policy, see Create a GPU node pool.

View the physical topology of your GKE cluster nodes

Run the following command to get the node labels for your GKE cluster nodes with a specific accelerator:

kubectl get nodes -l cloud.google.com/gke-accelerator=ACCELERATOR \
    -ocustom-columns='0-NAME:.metadata.name,0-BLOCK:.metadata.labels.cloud\.google\.com/gce-topology-block,0-SUBBLOCK:.metadata.labels.cloud\.google\.com/gce-topology-subblock,0-HOST:.metadata.labels.cloud\.google\.com/gce-topology-host'| sort -k2,4

Replace ACCELERATOR with the name of the accelerator, such as nvidia-h200-141gb.

The output displays the block, sub-block, and host of each of the GKE nodes with the specified accelerator.

For more information, see View the topology of a reservation.

What's next

• Learn how to schedule GKE workloads with Topology Aware Scheduling.
• Learn how to manage host maintenance events, cluster upgrades, faulty host reporting for GKE clusters