Layer 7 load balancing reference architecture

This reference architecture provides a customer-managed Layer 7 load balancing solution in Google Distributed Cloud (GDC) air-gapped. By deploying popular open-source controllers like HAProxy Kubernetes Ingress or NGINX Gateway Fabric on a GDC standard cluster, customers can seamlessly route L7 traffic to hybrid environments. This architecture uses TLS Termination (Ingress / HTTPRoute) to route traffic based on Server Name Indication (SNI) to both built-in containerized pods and applications hosted on external virtual machines.

Features and capabilities

  • Advanced Layer 7 Routing: Supports host-based routing, path-based routing, and advanced traffic management such as header-based matching and traffic weighting.
  • Centralized TLS Management: Provides centralized TLS management and termination.
  • Hybrid Environment Support: Seamlessly routes L7 traffic to both built-in containerized pods and applications hosted on external virtual machines.
  • Flexible Controller Options: Offers a choice between modern NGINX Gateway Fabric (using Gateway API) or the well-established HAProxy Ingress Controller.
  • Air-Gapped Readiness: Integrated with Harbor Registry to store and serve proxy and application images within private, disconnected environments.
  • Complex Traffic Splitting: Enables advanced traffic distribution strategies for modern application delivery.

Architectural principles

  • Two-Tier Load Balancing Pattern: This architecture combines GDC's built-in L4 Load Balancer (as the entry point) with a customer-managed L7 Gateway deployed within Kubernetes.
  • Role-Oriented Separation: It follows a design that separates responsibilities between infrastructure providers, cluster operators, and developers through specific resources (like GatewayClass, Gateway, and HTTPRoute, or Ingress).
  • Transition to Modern Standards: The architecture prioritizes the Gateway API for modern routing capabilities while maintaining support for the legacy Ingress API where stability for specific controllers is required.
  • Customer-Managed Flexibility: By deploying self-managed controllers, customers gain control over L7 features and configuration that are not natively provided by the GDC managed L4 balancer.
  • Abstraction of Workload Location: The architecture treats containerized and VM-based workloads consistently through Kubernetes Service abstractions (standard or headless).

Conceptual architecture

Conceptual architecture diagram for Layer 7 load balancing on GDC air-gapped.

Logical architecture

The logical architecture consists of several functional components working in concert to route external traffic to internal or external backends:

  1. Ingress/Gateway Entry Point: A Client initiates HTTPS requests which first hit the GDC L4 Load Balancer. This built-in balancer serves as the entry point and distributes TCP/443 traffic to the L7 Load Balancer.
  2. Control Plane: Ingress or Gateway Controllers (HAProxy or NGINX operators) monitor Kubernetes resources such as Ingress, Gateway, and HTTPRoute
  3. Data Plane: Based on the observed resources, the controllers dynamically update the underlying Proxies which perform the actual traffic routing based on SNI-based host routing rules and TLS Termination.
  4. Backend Services:
    • Containerized Pods: Managed as standard Deployments and exposed with regular Kubernetes Services.
    • External VMs: Integrated into the cluster with a headless Kubernetes Service and a custom endpoint (or EndpointSlice) containing the VM's direct IP
  5. Artifact Management: All images for the proxies and applications are served from a local Harbor Registry to ensure functionality in an air-gapped environment

Core concepts and technologies

The architecture uses a two-tier load balancing pattern, combining GDC's built-in L4 Load Balancer with a customer-managed L7 Gateway deployed in Kubernetes. The key components include the following:

  • Client: An entity initiating HTTPS requests to interact with the applications.
  • GDC Standard Cluster: GDC provides a built-in way to create Kubernetes Vanilla Clusters. In this solution, the cluster will host the L7 LB and its controllers, along with the workloads and healess service for external VMs
  • GDC L4 Load Balancer: The built-in L4 load balancer serving as the entry point, distributing TCP/443 traffic directly to the Kubernetes Pods running the Ingress or Gateway controllers.
  • Ingress and Gateway Controllers: HAProxy or NGINX operators running in the Standard Cluster. For Nginx, they monitor Gateway API resources (like Gateway and HTTPRoute) and dynamically update the underlying proxies. For HAProxy, they monitor the Ingress resources and dynamically update the underlying proxies.
    • HAProxy Ingress Controller: An Ingress controller for Kubernetes that utilizes HAProxy as a reverse proxy and Layer 7 load balancer, also handling TLS termination.
    • Nginx Gateway Fabric: An implementation of the Gateway API using NGINX as the data plane.
  • Ingress and Gateway (with HTTPRoute): Standardized Kubernetes resources defining the physical listening port (443) and the SNI-based host routing rules with TLS Termination.
  • Containerized Workload (Pods): A standard Kubernetes Deployment exposed internally using a regular Kubernetes Service.
  • VM-based Workload (External): A workload hosted on an external VM on the project network, exposed to the proxy with a headless Kubernetes Service and a custom endpoint containing the VM's direct IP.
  • Harbor Registry: A private container registry used to store and serve the proxy and application images in the air-gapped environment.

Ingress versus Gateway APIs

The Kubernetes networking landscape is transitioning from the legacy Ingress API to the modern Gateway API. Understanding the differences between these two is critical for selecting the right architecture for your workloads.

  • Ingress API: Historically, Ingress has been the standard for managing external access to HTTP services. It provides host-based and path-based routing, but its design is relatively basic and lacks built-in support for advanced traffic management (for example, header-based matching or traffic weighting) without relying on vendor-specific annotations. Importantly, the Ingress API is now considered feature-frozen by the Kubernetes community (Ingress limitations and evolution of Kubernetes networking).

  • Gateway API: Often referred to as the V2 of Ingress, the Gateway API is a collection of Custom Resource Definitions (CRDs) that provide a more robust and extensible model for service networking. It is designed to be role-oriented, separating responsibilities between infrastructure providers (GatewayClass), cluster operators (Gateway), and developers (HTTPRoute/TLSRoute) (Gateway API in GKE). This separation allows for more granular control and better multi-tenancy support.

In this GDC air-gapped solution, we have purposefully chosen different APIs for the two featured controllers:

  • NGINX (Gateway API): For the NGINX implementation, we utilize NGINX Gateway Fabric, which is a mature and built-in implementation of the Gateway API. Leveraging the Gateway API for NGINX allows customers to benefit from the modern standards and advanced routing capabilities (like HTTPRoute) that are built in to the API.
  • HAProxy (Ingress API): Although the Gateway API is the strategic direction, we have selected the Ingress API for HAProxy in this guide. This is because HAProxy's Gateway API integration isn't considered mature. By using the well-established HAProxy Ingress Controller, we ensure a stable and reliable L7 solution for customers who prefer HAProxy, while still providing a path toward modern L7 capabilities.

Assumptions and constraints

Assumptions

  • Infrastructure Readiness: The deployment assumes the availability of GDC air-gapped 1.15.x Standard Clusters.
  • Local Registry: It is assumed that a local Harbor Registry is correctly configured to store and serve proxy and application images in the air-gapped environment.
  • Native Entry Point: Traffic is assumed to first hit the built-in GDC L4 Load Balancer, which then distributes traffic to the L7 proxies.

Constraints

  • Customer-Managed Solution: There is no managed, SLA-backed L7 service in GDC; this implementation is entirely self-managed by the customer.
  • Standard Clusters Only: Shared User Clusters are out of scope due to the high privileges required for Ingress and Gateway Controller deployment.
  • API Maturity: While NGINX uses the modern Gateway API, HAProxy is constrained to the Ingress API because its Gateway API integration is not yet considered mature.
  • Air-gapped Maintenance: All container images must be manually promoted and managed within the local Harbor registry.
  • Community Editions: The architecture is designed for open-source community editions of Nginx and HAProxy. Enterprise Editions may work but have not been explicitly tested.
  • Operational Ownership: Customers must manage their own scaling, high availability, and patching for the third-party controllers.
  • Support Boundary: Google support covers only the underlying GDC infrastructure; issues with HAProxy or NGINX software itself are the customer's or vendor's responsibility.
  • API Obsolescence: The Ingress API (used for HAProxy) is feature-frozen by the Kubernetes community, which may limit future routing capabilities.

Licensing

  • Community Editions: The architecture is designed for open-source community editions (typically Apache 2.0 or similar), which are free to use but customer-managed.
  • Enterprise Procurement: Procurement and management of commercial licenses, such as NGINX Plus or HAProxy Enterprise, are the sole responsibility of the customer.

Additional materials