High availability load balancing for VCF appliances

This document explains how regional internal load balancers and zonal network endpoint groups (NEGs) in VMware solution on Google Cloud provide Layer 2 high availability, virtual machine mobility (vMotion), and automated failover for VMware Cloud Foundation (VCF) management appliances deployed in the Virtual Private Cloud (VPC) underlay, as shown in the following diagram (click to enlarge):

Internal load balancer and regional backend service with GARP_RA steering traffic through a zonal NEG during vMotion. Internal load balancer and regional backend service with GARP_RA steering traffic through a zonal NEG during vMotion.

Why underlay appliances require a load balancer

In on-premises data centers, management virtual machines—such as vCenter Server, SDDC Manager, and clustered NSX Manager nodes—rely on physical Layer 2 broadcast domains. When an appliance fails over or migrates between ESXi hosts by using vMotion, the appliance broadcasts a Gratuitous ARP (GARP) packet to update the MAC address tables of physical top-of-rack (ToR) switches so that its IP address floats across physical servers.

A Google Cloud VPC network is a routed Layer 3 software-defined network (SDN) with the following behaviors:

  • Every IP address is programmatically bound to a specific Compute Engine virtual network interface (vNIC) in the SDN forwarding table.
  • The VPC routing plane doesn't rely on physical switch MAC learning to route Layer 3 traffic to floating IP addresses.
  • If an underlay management VM moves to a different bare-metal ESXi host, the VPC routing table requires an explicit mechanism to steer traffic for that appliance's IP address to the new host interface.

To support IP mobility without manual route updates or VM re-addressing, VMware solution provisions a regional internal passthrough Network Load Balancer configured with Fast IP Move (GARP_RA) for each floating underlay management appliance.

High-availability architecture components

Each network appliance in VMware solution uses a three-tier Google Cloud load balancing architecture to support high availability and vMotion mobility:

Zonal network endpoint groups (NEGs)

  • Construct type: Zonal NEGs of type GCE_VM_IP_DEDICATED_BACKEND configured in the target VPC subnet.
  • Function: A dedicated NEG in each zone groups the primary or dynamic network interfaces of all bare-metal ESXi instances that belong to the cluster's vMotion mobility domain.
  • Reusability: All underlay appliance load balancers within the same cluster and subnet share a single set of zonal NEGs.

Regional backend service with Fast IP Move

  • Construct type: Regional backend service (loadBalancingScheme: INTERNAL, protocol: UNSPECIFIED) that references the cluster's zonal NEGs.
  • Failover policy (haPolicy): Configured with fastIPMove: GARP_RA.
  • Function: Instructs the Google Cloud Andromeda data plane to monitor the broadcast-enabled subnet for GARP and Router Advertisement (RA) frames emitted by the management appliance VM and dynamically update the active backend host.

Regional internal passthrough Network Load Balancer

  • Construct type: Regional internal forwarding rule configured with the appliance's reserved virtual IP (VIP) address.
  • Function: Serves as the persistent IP endpoint for the management appliance or cluster VIP. Administrators, workloads, and other VCF components connect to this VIP address.

Underlay high availability with dedicated NEGs

When multiple network interfaces (such as the primary nic0 interface and VLAN-tagged Dynamic NICs) connect to the same VPC network, standard GCE_VM_IP NEGs cannot target arbitrary sub-interfaces. Instead, VMware solution uses GCE_VM_IP_DEDICATED_BACKEND NEGs, which operate as follows:

  • Automatic interface targeting: Automatically attaches the specific primary vNIC or Dynamic NIC that matches the NEG's subnet.
  • Single NEG per zone: A regional backend service configured with GCE_VM_IP_DEDICATED_BACKEND NEGs can contain only one NEG per zone.
  • Internal HA policy requirement: GCE_VM_IP_DEDICATED_BACKEND NEGs are supported only with regional backend services configured for internal passthrough Network Load Balancers (loadBalancingScheme: INTERNAL) that use haPolicy.

HA policy restrictions on broadcast subnets

When you deploy haPolicy backend services in subnets with ARP broadcasting and MAC learning enabled, the following rules apply:

  • Load balancing scheme: Only internal passthrough Network Load Balancers (INTERNAL) are supported.
  • Subnet alignment: All backend NEGs across all zones for a given backend service must belong to the same VPC subnet.
  • Forwarding rule mapping: A backend service configured with broadcast NEGs can be referenced by only one forwarding rule.
  • Immutable backend subnet type: You must attach backends in an ARP broadcasting subnet when you create the backend service; you can't replace them with backends in non-broadcast subnets later.

Prevent HA policy cross-claiming

In an ARP-broadcast-enabled subnet, a host interface that belongs to one HA policy backend service could temporarily claim leadership for an unrelated HA policy if it broadcasts a GARP frame for that policy's forwarding rule VIP.

To prevent unintended cross-claiming, include the exact same set of ESXi host backends (all hosts in the cluster's zonal NEG) across all HA policy backend services operating in that subnet.

What's next