Design and configure GKE clusters for cost optimization

This document helps you navigate Google Kubernetes Engine (GKE) documentation to find guidelines and recommendations for cost optimization. GKE offers extensive autoscaling and scheduling capabilities that you can use to minimize cluster cost while maintaining application stability.

For a consolidated overview of all GKE best practices, see Best practices for GKE.

You should already be familiar with the following:

Overview

When you implement GKE, you must consider different technical aspects to align with application and business requirements. Beyond defining networking, security, storage, and other technical aspects, you must evaluate both cost and performance to meet business needs. Instead of treating cost and performance as distinct entities, you must integrate them from the initial stages of infrastructure planning to define a unified relationship that determines both reliability and cloud spending. Low cost and high reliability are expected, but as you scale, the complexity of managing this trade-off increases.

To achieve low cost and application stability, you can set or tune the following GKE features:

  • GKE configuration
  • Workload configuration
  • Cost baseline and visibility

Use GKE Autopilot

For small sandbox or development environments, select Autopilot clusters. In Autopilot, GKE dynamically manages nodes and you are billed only for requested Pod capacity, which helps you avoid VM, node operating system, and system overhead charges.

For more information, see GKE Autopilot overview.

Understand how autoscaling works

GKE autoscaling controllers dynamically adjust resources as traffic requests change.

Add and remove Pods based on utilization metrics

A HorizontalPodAutoscaler (HPA) adds and removes Pods based on CPU or custom metrics.

To understand and configure horizontal Pod autoscaling, see the following GKE documentation:

Configure a target utilization threshold (for example, 70% or 80%) to maintain a buffer that handles traffic surges while additional replica Pods start.

Scale your Pods based on utilization metrics

Use VerticalPodAutoscaler (VPA) to dynamically size container CPU and memory requests for workloads that don't employ horizontal Pod autoscaling, or when peak workloads are unknown.

To understand and configure vertical Pod autoscaling, see the following GKE documentation:

Keep VPA in Off (recommendation-only) mode for at least 24 hours (ideally one week) in production-like environments to capture representative traffic patterns. To prevent erratic sizing adjustments, specify explicit minimum and maximum bounds in your VerticalPodAutoscaler object before enabling Initial or Auto modes.

Automate infrastructure scaling using Cluster Autoscaler

To scale underlying compute nodes based on active scheduling simulation rather than metric loads, enable Cluster Autoscaler in GKE Standard node pools. Specify minimum node parameters to support baseline night capacity.

Always configure a PodDisruptionBudget (PDB) object for system and application Pods. This configuration helps ensure that Cluster Autoscaler does not inadvertently cause service disruption when consolidating or scaling down underutilized node pools.

To understand and configure Cluster Autoscaler, see the following GKE documentation:

Deploy dynamic node pools using node pool auto-creation

Enable node pool auto-creation to automatically generate custom GKE node pools whose shapes, CPU counts, or memory limits precisely fit the scheduling parameters of pending Pods. This feature minimizes resources left over on oversized nodes.

To understand and configure node pool auto-creation, see the following GKE documentation:

Autoscaling checklist

Infrastructure characteristics

Align cluster hardware, location, and node network rules with cost-optimization priorities.

Select the appropriate machine types

Select the appropriate machine types for your cluster based on the location of your users and the location of the data that your cluster needs to access.

For more information, see the Machine families resource and comparison guide.

Deploy fault-tolerant workloads on Spot VMs

Use Spot VMs to run stateless, fault-tolerant, or batch workloads at a discount of up to 91% compared to on-demand VM instances.

For more information, see the following GKE documentation:

Map efficient machine families and OS system settings

Customize node pool machine settings with cost-efficient instance profiles (for example, E2 VM architectures).

For more information about sizing nodes, configuring Spot VM preemption timings, and configuring kernel configurations, see About node pools.

Select the appropriate region

When latency does not affect your users, run cluster workloads in Compute Engine regions with lower operating costs.

For more information, see Best practices for Compute Engine region selection.

Sign up for CUD

Purchase committed-use discounts (CUDs) to secure deeply discounted pricing (up to 70%) for baseline compute resources over a one-year or three-year term.

For more information, see Resource-based committed use discounts.

Consider networking costs

Regional and multi-zonal GKE clusters improve application reliability but can generate internal cross-zone network egress costs.

To minimize and control networking costs, consider the following:

  • Cross-zone data transfers: although regional clusters increase availability by spreading workloads across zones, there are associated costs for data transferred between these zones.

For more information, see All networking pricing.

Deploy single-zone clusters for non-production environments

In non-production environments, to avoid cross-zone network charges and reduce VM overhead, deploy single-zone clusters instead of regional or multi-zonal clusters.

For more information, see About cluster configuration choices.

Optimize cluster DNS resolution paths and ingress traffic

To optimize cluster DNS resolution and ingress traffic, you can deploy NodeLocal DNSCache and network endpoint groups (NEGs).

When you run DNS-heavy workloads, NodeLocal DNSCache runs a local DNS daemon on each node. This configuration prevents high query loads from exhausting CoreDNS, avoiding the need to scale CoreDNS and reducing overall GKE costs.

For ingress traffic, container-native load balancing through NEGs routes traffic directly to Pod IP addresses instead of instance groups. This direct routing facilitates graceful traffic redirection during Pod scaling actions.

For more information, see:

Apply resource quotas per namespace

Deploy standard Kubernetes ResourceQuota objects per namespace in multi-tenant clusters to lock down CPU and memory shape thresholds and to prevent individual teams from scheduling non-compliant workloads that trigger unexpected compute charges.

For more information, see Namespaces in the Kubernetes documentation.

Implement Policy Controller audits

Deploy Policy Controller to dynamically audit and enforce cluster compliance with corporate standards. Policy Controller uses admission control to reject misconfigured resources.

For more information, see the following:

Gate noncompliant manifests in CI/CD pipelines

Validate cost policy compliance earlier in your development lifecycle. Integrate validation scripts (such as kpt parsing) into pre-commit or pull-request checks to audit and block noncompliant manifests before they reach the cluster.

For more information, see Validate apps against company policies in a CI pipeline.

Infrastructure checklist

Application and workload optimization

Configure your workloads to use resources efficiently and reduce operational overhead.

Specify matching memory requests and limits

Specify precise container CPU and memory requests before deployment. For CPU, configure requests to meet your service-level objectives (SLOs) but leave limits unrestricted. For memory, ensure that the requested allocation matches the memory limit.

For more information, see Resize CPU and Memory Resources assigned to Containers in the Kubernetes documentation.

Accelerate container start times

Make container images as small as possible to minimize image download times.

Configure PDBs

Specify a PodDisruptionBudget (PDB) object for application replicas to limit voluntary disruptions and ensure stability when GKE scales down or when node upgrades occur.

For more information, see Specifying a Disruption Budget for your Application.

Set meaningful readiness and liveness probes

Configure readiness and liveness probes for all containers to help ensure that GKE both routes traffic only to ready Pods and restarts failed instances, preventing traffic loss during autoscaling.

For more information, see Configure Liveness, Readiness, and Startup Probes.

Configure graceful application shutdown

Prepare containers for graceful termination by listening for the SIGTERM signal, finalizing in-flight requests before exiting, or by configuring preStop hooks.

For more information, see Termination and graceful shutdown of preemptible VMs.

Implement retries with exponential backoff

Implement exponential backoff retries at the application or service mesh level to handle transient failures or potential Spot VM preemptions.

For more information, see Retries in Istio documentation.

Application and workload optimization checklist

Cost baseline and visibility

To optimize costs, you first need visibility into your GKE spending and how it's allocated. This visibility helps you attribute costs to the teams and business units that incur them.

The following GKE documentation describes how to establish deep visibility into GKE billing, resource consumption, and baseline metrics.

Enable GKE cost allocation

Enable GKE cost allocation to get visibility into workload resource requests and associated costs. Cost allocation attributes cluster costs to namespaces and Kubernetes labels of your workloads.

Export these details to BigQuery to analyze the data in Cloud Billing. Use this analysis to identify which workloads cause billing peaks, perform chargebacks, and optimize resource requests.

For more information, see Get key spending insights for your GKE resource allocation and cluster costs.

Review log and metric ingestion volumes

Enabling Cloud Logging and Cloud Monitoring for your clusters incurs costs. High volumes of log and custom metric ingestion can result in unexpected charges. Centrally audit which log levels and custom metrics are ingested.

For more information about troubleshooting high logging API usage or log write limit timeouts, see:

Monitor the health of Metrics Server

Monitor the health of the Metrics Server Deployment, because GKE's built-in autoscaling controllers rely on it to retrieve CPU and memory metrics.

For more information, see Troubleshoot horizontal Pod autoscaling.

Foster a cost-saving culture

Provide developers with access to cloud spending dashboards and establish FinOps training to align architectural decisions with business cost budgets.

For more information about organizational cost-efficiency culture, see Spread the cost saving culture.

Cost baseline and visibility checklist

What's next

For more information about the architectural principles and organizational culture required for cost-efficiency, see Best practices for running cost-optimized Kubernetes applications on GKE.