MariaDB database reference architecture on GDC air-gapped

This reference architecture provides a conceptual framework for deploying and operating highly available, customer-managed MariaDB 12 databases on Google Distributed Cloud (GDC) air-gapped. It enables enterprise and early access customers to reliably maintain critical database workloads using a robust, multi-zone virtual machine (VM) setup.

Because GDC air-gapped currently lacks support for cross-zone Kubernetes stretch clusters, this architecture relies strictly on dedicated VMs deployed across three availability domains to ensure continuous operations and withstand a full zone failure without data loss.

Features and capabilities

  • Multi-zone Resiliency: A highly resilient 3-node configuration deployed across three distinct availability zones to protect against single infrastructure zone failures.
  • Automated High Availability and Consensus: Uses Galera for synchronous multi-master clustering, providing automatic failure detection, node consensus, and global data synchronization without split-brain scenarios.
  • Intelligent Traffic Routing: Co-located HAProxy instances manage connection pooling and strictly route traffic—directing write operations (e.g., port 3306) to an active primary node and load-balancing read operations (e.g., port 3307) across synced replicas.
  • Global Load Balancing: Integrates with the platform-native GDC Global L4 Load Balancer to provide a single, stable Virtual IP (VIP) for client applications, abstracting the underlying node topology.
  • Database-Aware Failover Validation: Utilizes a lightweight custom Python script or an OSS solution like clustercheck (e.g., exposed on port 9200) to serve health checks to the load balancer and HAProxy, ensuring traffic is only routed to healthy and fully synchronized nodes.

Architectural principles

  • Synchronous Majority-Based Consensus: Prioritizes strict data consistency over eventual consistency. Galera enforces a quorum-based model requiring majority agreement, eliminating the risk of data loss or split-brain during network partitions.
  • Separation of Concerns: Decouples the database replication layer (Galera) from the connection routing and failover detection layer (HAProxy and Python health check service).
  • Infrastructure Optimization: Specifically designed for air-gapped environments, utilizing robust VMs to bypass current Kubernetes networking limitations.

Architecture

Three VM architecture that runs a colocated stack of services.

Concepts and technologies

This section details the functional components and their specific responsibilities within the multi-zone architecture.

Infrastructure and platform

  • Virtual Machines (VMs): Three dedicated Compute instances, each deployed into a separate availability zone to form the failure domain boundaries.
  • GDC Global L4 Load Balancer: A platform-managed networking construct that exposes a stable internal VIP, automatically evaluating Python-driven HTTP health checks to redirect inbound traffic.

Services and logic

  • MariaDB 12: The core relational database engine.
  • Galera Plugin: The clustering framework responsible for synchronous multi-master replication and verifying node quorum.
  • HAProxy: Acts as the traffic router on each VM. Configured in an active/backup topology for writes to prevent deadlocks, and round-robin for distributed reads.
  • Python Health Check Service: A lightweight HTTP service (wrapped in systemd) that continuously polls the local database state and signals its health status back to HAProxy and the L4 Load Balancer via port 9200.

Data flow and interfaces

  1. Applications send database requests to the Global L4 Load Balancer VIP.
  2. The Load Balancer proxies the connection to a healthy HAProxy instance on one of the VMs.
  3. Based on the requested port, HAProxy dynamically forwards traffic: Port 3306 strictly targets the active node for writes, while Port 3307 cycles reads across the cluster.

Design considerations

General considerations

  • Performance vs. Consistency Trade-offs: Because Galera is natively synchronous, every transaction must be acknowledged by cluster peers. Performance directly correlates to the inter-zone network latency within the GDC environment.
  • Resource Management: Deploying HAProxy directly on the database VMs optimizes hardware utilization but requires careful resource tuning to prevent connection pooling overhead from starving the core MariaDB processes.

Health check considerations

While this solution uses a custom Python script for its simplicity and lack of external dependencies, customers may choose to implement a standard OSS health check using xinetd and clustercheck.

  • Custom Python Script: Leverages the pre-installed Python 3 and http.server module, which is ideal for air-gapped environments as it requires no additional package transfers. However, the customer must maintain the custom script code.
  • OSS Tools (xinetd / clustercheck): Shifts the maintenance burden to the open-source community but increases architectural complexity and dependency management during the initial air-gapped software preparation.

Performance considerations

When planning your deployment, consider the following concrete factors to optimize for performance and reliability:

  • Hardware Sizing: While requirements vary by workload, use these standard profiles as a starting point for each VM:
    • Development/Proof-of-Concept: 2 vCPU, 8GB RAM (Minimum for stable operation).
    • Small Production: 4 vCPU, 16GB RAM. Suitable for internal tools with moderate concurrency.
    • Standard Production: 8 vCPU, 32GB RAM. The recommended baseline for mission-critical applications.
    • High-Throughput: 16+ vCPU, 64GB+ RAM. Required for workloads with large working sets that benefit from extensive data caching in the InnoDB buffer pool.
  • Storage Performance: High-performance storage is vital for database durability and cluster stability. SSD disks are highly recommended to minimize write latency during the "commit" phase of synchronous replication. For optimal performance, aim for a p99 disk write latency of less than 10ms to ensure that the Galera gcache or MySQL binary logs do not become a bottleneck.
  • Network Latency and Flow Control: The latency between VMs directly impacts the transaction commit speed.
    • Synchronous Replication: In a Galera or Group Replication cluster, every write transaction must be acknowledged by a majority of nodes. Inter-zone latency in GDC air-gapped is typically less than 1ms, which is excellent for keeping write overhead minimal (typically 10-30%).
    • Flow Control: High network jitter or a single "slow" node can trigger Galera Flow Control, pausing writes across the entire cluster to allow the laggard to catch up.
  • The Role of HAProxy (or MySQL Router): a load balancer is still essential:
    • Traffic Splitting: HAProxy ensures that write traffic is strictly directed to a single active node (Port 3306) to prevent deadlocks, while balancing read-only queries across all synced replicas (Port 3307).
    • Connection Stability: It provides a layer of abstraction that prevents applications from needing to handle complex failover logic manually.
  • Sidecar Components: The custom Python Health Check service and the load balancer are lightweight but critical. Ensure that the VMs have consistent CPU availability; in high-load scenarios, CPU "stealing" at the hypervisor level can delay health check responses, causing the Global Load Balancer to prematurely evict a healthy node from the rotation.

Design decision

The architectural choices made in this SRA explicitly address GDC air-gapped limitations and optimize for high throughput.

  • Virtual Machines over Kubernetes: GDC air-gapped does not currently support Kubernetes clusters that span multiple physical zones. A VM-based approach was strictly chosen because deploying dedicated VMs across separate zones is the only viable method to achieve true multi-zone high availability and survive a total zone failure.
  • Galera vs. Primary/Secondary Manual Setup (with ProxySQL): A traditional primary/secondary architecture paired with ProxySQL was evaluated as a feasible alternative. However, handling manual failover logic alongside ProxySQL involves significant scripting and operational complexity. Galera was selected instead because its natively synchronous nature drastically reduces failover complexity and provides significantly better performance.
  • Rejection of Orchestrator: Orchestrator is highly regarded as an industry-standard topology manager for automated failover. However, it was evaluated and ultimately rejected because it only supports MariaDB 10; it lacks compatibility with MariaDB versions strictly greater than 11. This rendered it incompatible with our target MariaDB 12 requirement.
  • Platform-Native Global Load Balancing: Leveraging the native GDC Global L4 Load Balancer ensures that the VIP is controlled by the GDC control plane, keeping the entry point resilient and simplifying cross-zone traffic delivery.

Assumptions and limitations

Assumptions

  • Availability of Infrastructure: Customers have sufficient project quota to provision dedicated, properly sized VMs and Global Load Balancers distributed evenly across 3 availability zones.
  • Secure Networking: Key-based access and proper ProjectNetworkPolicies (PNPs) are established to permit intra-cluster Galera synchronization and health check polling.

Limitations

  • Kubernetes Unsupported: Customers strictly seeking containerized/Kubernetes-based solutions cannot achieve multi-zone HA until stretch clusters are fully supported by the platform.
  • Manual Upgrades Required: Unlike managed services, this solution places the responsibility of routine OS-level patching, database minor version upgrades, and Galera version management entirely on the customer.
  • Network Latency Sensitivity: Synchronous replication demands a high-quality, stable network. Network jitter or latency spikes between the air-gapped zones will proportionally delay write operations across the MariaDB cluster.

Additional materials