This tutorial shows how to apply privately used public IP (PUPI) addresses to Google Kubernetes Engine (GKE) Pod address blocks. Service consumer organizations that are IPv4-address constrained can use PUPI addresses in service producer virtual private clouds (VPCs) as an address-management option.
This document is intended for network architects and GKE system administrators whose companies offer managed services over a GKE infrastructure on Google Cloud.
Introduction
Some companies want to deliver managed services to their customers over
Kubernetes or GKE clusters on Google Cloud. However,
Kubernetes can require many IP addresses for its various components. For some
organizations, meeting this requirement is difficult or impossible because
they're unable to assign appropriately sized 10.0.0.0/8, 172.16.0.0/12, and
192.168.0.0/16 (RFC 1918)
Classless Inter-domain Routing (CIDR) blocks.
One way to mitigate address exhaustion is to use privately used public IP (PUPI) addresses for the GKE Pod CIDR block. PUPIs are any public IP addresses not owned by Google that a customer can use privately on Google Cloud. The customer doesn't necessarily own these addresses.
The following diagram shows a company (producer) that offers a managed service to a customer (consumer).
This setup involves the following considerations:
- Primary CIDR block: A non-PUPI CIDR block that is used for nodes and ILB and must be non-overlapping across VPCs.
- Producer secondary CIDR block: A PUPI CIDR block that is used for
Pods (for example,
45.45.0.0/16). - Consumer secondary CIDR block: Any other PUPI CIDR block on the customer
side (for example,
5.5/16).
The company's managed service is in the producer VPC (vpc-producer) and is
built on a GKE deployment. The company's
GKE cluster uses the PUPI 45.0.0.0/8 CIDR block for Pod
addresses. The customer's applications are located in the consumer VPC
(vpc-consumer). The customer also has a GKE installation.
The GKE cluster in the consumer VPC uses the PUPI
5.0.0.0/8 CIDR block for Pod addresses. The two VPCs are peered with one
another. Both VPCs use the RFC 1918 address space for node, service, and load
balancing addresses.
By default, the consumer VPC (vpc-consumer) exports all RFC 1918 to the
producer VPC (vpc-producer). Unlike RFC 1918 private addresses and extended
private addresses (CGN, Class E), PUPIs aren't automatically advertised to VPC
peers by default. If the vpc-consumer Pods must communicate with
vpc-producer, the consumer must enable the VPC peering connection to export
PUPI addresses. Likewise, the producer must configure the producer VPC to import
PUPI routes over the VPC peering connection.
The vpc-consumer address space that is exported into vpc-producer must not
overlap with any RFC 1918 or PUPI address used in vpc-producer. The producer
must inform the consumer what PUPI CIDR blocks the managed service uses and
ensure that the consumer isn't using these blocks. The producer and consumer
must also agree and assign non-overlapping address space for internal load
balancing (ILB) and node addresses in vpc-producer.
In most cases, resources in vpc-consumer communicate with services in
vpc-producer through ILB addresses in the producer cluster. If the producer
Pods are required to initiate communication directly with resources in
vpc-consumer, and PUPI addressing doesn't overlap, then the producer must
configure the producer VPC to export the PUPI routes over the VPC peering
connection. Likewise, the consumer must configure the VPC peering connection to
import routes into vpc-consumer. If the consumer VPC already uses the PUPI
address, then the producer should instead configure the IP masquerade feature
and hide the Pod IP addresses behind the producer node IP addresses.
The following table shows the default import and export settings for each VPC.
You can modify default VPC peering settings by using the
gcloud compute networks peerings update
Google Cloud CLI command.
| PUPI flag | Import | Export |
|---|---|---|
| Producer side (flags controlled through service networking) | To turn on: Default behavior: Turned off |
To turn off: Default behavior: Turned on |
| Consumer side (owned by customer, not required to be modified through service networking) | Turned off (default) | Turned on (default) |
These settings result in the following:
- The producer VPC sees all the customer routes.
- The consumer VPC doesn't see the PUPI routes configured on the Pod subnet in the producer VPC.
- Traffic originating from the producer Pods to the
vpc-consumernetwork must be translated behind the node addresses in the producer cluster.
Qualifications
- The address range that you select for a PUPI can't be reachable through the internet or be an address that Google owns.
- The Node IPs and primary range need to be non-overlapping between the two VPCs.
- If direct Pod-to-Pod communication is required between the customer VPC and the managed service, then the producer Pod IP addresses must be translated behind their corresponding Node IPs.
Inspecting the supporting infrastructure
You now verify that Terraform successfully created the resources by checking whether the resources respond after you send a command.
Verify the projects
In Cloud Shell, list the project:
gcloud projects list | grep pupi-pidThe output is similar to the following:
... pupi-pid--1234567899 pupi-test 777999333555 ...
In this output,
pupi-testis the project name, andpupi-pid-is the prefix to your project ID.List the API status:
gcloud services list --project=$TF_VAR_pid \ | grep -E "compute|container"The output is similar to the following:
... compute.googleapis.com Compute Engine API container.googleapis.com Kubernetes Engine API containerregistry.googleapis.com Container Registry API ...
This output shows that the Compute Engine, GKE, and Container Registry APIs are enabled.
Verify the networks and subnetworks
In Cloud Shell, verify the producer network and subnetwork:
gcloud compute networks describe producer \ --project=$TF_VAR_pid gcloud compute networks subnets describe producer-nodes \ --project=$TF_VAR_pid \ --region=$TF_VAR_region1The output is similar to the following:
... kind: compute#network name: producer ... ipCidrRange: 10.128.0.0/24 kind: compute#isubnetwork name: producer-nodes ... secondaryIpRanges: - ipCidrRange: 45.45.45.0/24 rangeName: producer-pods - ipCidrRange: 172.16.45.0/24 rangeName: producer-cluster ...
This output shows the following:
- The network was created with the
10.128.0.0/24CIDR block. - The two subnets were created with the
45.45.45.0/24and172.16.45.0/24CIDR blocks.
- The network was created with the
Verify the consumer network and subnetwork:
gcloud compute networks describe consumer \ --project=$TF_VAR_pid gcloud compute networks subnets describe consumer-nodes \ --project=$TF_VAR_pid \ --region=$TF_VAR_region2The output is similar to the following:
... kind: compute#network name: consumer ... ipCidrRange: 10.129.0.0/24 kind: compute#isubnetwork name: consumer-nodes ... secondaryIpRanges: - ipCidrRange: 5.5.5.0/24 rangeName: producer-pods - ipCidrRange: 172.16.5.0/24 rangeName: consumer-cluster ...
This output shows the following:
- The network was created using the
10.129.0.0/24CIDR block. - The two subnets were created using the
5.5.5.0/24and172.16.5.0/24CIDR blocks.
- The network was created using the
Verify the GKE cluster and its resources
In Cloud Shell, get the cluster credentials:
gcloud container clusters get-credentials consumer-cluster \ --project=$TF_VAR_pid \ --zone=$TF_VAR_zone2The output is similar to the following:
... Fetching cluster endpoint and auth data. kubeconfig entry generated for consumer-cluster. ...
Verify the cluster:
gcloud container clusters list \ --project=$TF_VAR_pid \ --zone=$TF_VAR_zone2The output is similar to the following:
NAME LOCATION MASTER_VERSION MASTER_IP MACHINE_TYPE NODE_VERSION NUM_NODES STATUS consumer-cluster us-west2-b 1.14.10-gke.17 35.236.104.74 n1-standard-1 1.14.10-gke.17 3 RUNNING
This output shows a cluster that is named
consumer-cluster.Verify the Hello World app:
kubectl get deployment my-appThe output is similar to the following:
... NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE my-app 3 3 3 3 118m ...
This output shows a deployment that is named
my-app.Verify the internal load balancer service:
kubectl get service hello-serverThe output is similar to the following:
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE hello-server LoadBalancer 172.16.5.99 10.129.0.200 8080:31673/TCP 4d23h
This output shows a service that is named
hello-server.
Verifying the solution
In Cloud Shell, validate that you successfully created VPC peering:
gcloud alpha compute networks peerings listThe output is similar to the following:
NAME NETWORK PEER_PROJECT PEER_NETWORK IMPORT_CUSTOM_ROUTES EXPORT_CUSTOM_ROUTES STATE STATE_DETAILS consumer consumer pupi-pid--1324732197 producer False False ACTIVE [2020-02-26T11:33:16.886-08:00]: Connected. producer producer pupi-pid--1324732197 consumer False False ACTIVE [2020-02-26T11:33:16.886-08:00]: Connected.
This output shows peerings that are named
consumerandproducer.Validate that the consumer VPC exports PUPI routes:
gcloud alpha compute networks peerings list-routes consumer \ --direction=OUTGOING \ --network=consumer \ --region="$TF_VAR_region2"The output is similar to the following:
DEST_RANGE TYPE NEXT_HOP_REGION PRIORITY STATUS 10.129.0.0/24 SUBNET_PEERING_ROUTE us-west2 1000 accepted by peer 172.16.5.0/24 SUBNET_PEERING_ROUTE us-west2 1000 accepted by peer 5.5.5.0/24 SUBNET_PEERING_ROUTE us-west2 1000 accepted by peer
This output shows all three consumer CIDR blocks.
Validate the PUPI routes that the producer VCP imported:
gcloud alpha compute networks peerings list-routes producer \ --direction=INCOMING \ --network=producer \ --region="$TF_VAR_region1"The output is similar to the following:
DEST_RANGE TYPE NEXT_HOP_REGION PRIORITY STATUS 10.129.0.0/24 SUBNET_PEERING_ROUTE us-west2 1000 accepted 172.16.5.0/24 SUBNET_PEERING_ROUTE us-west2 1000 accepted 5.5.5.0/24 SUBNET_PEERING_ROUTE us-west2 1000 accepted
This output shows all three consumer CIDR blocks.
Validate that the GKE Pods have a PUPI address:
kubectl get pod -o wideThe output is similar to the following:
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES my-app-594b56d7bc-642d8 1/1 Running 0 4d23h 5.5.5.21 gke-consumer-cluster-default-pool-cd302b68-tccf <none> <none> my-app-594b56d7bc-chnw8 1/1 Running 0 4d23h 5.5.5.38 gke-consumer-cluster-default-pool-cd302b68-h8v9 <none> <none> my-app-594b56d7bc-fjvbz 1/1 Running 0 4d23h 5.5.5.20 gke-consumer-cluster-default-pool-cd302b68-tccf <none> <none>
The IP addresses of the Pods fall within the
5.5.5/24range.