Remotely access a private cluster using a bastion host

Warning: This page is archived and is not actively maintained. The commands on this page might not work and could cause disruptions to your cluster. We recommend that you create your cluster in version 1.29 and later for customizable and simplified access to the control plane and cluster network. To learn more see, Customize your network isolation in GKE.

This tutorial shows you how to access a private cluster in Google Kubernetes Engine (GKE) over the internet by using a bastion host.

You can create GKE private clusters with no client access to the public endpoint. This access option improves the cluster security by preventing all internet access to the control plane. However, disabling access to the public endpoint prevents you from interacting with your cluster remotely, unless you add the IP address of your remote client as an authorized network.

This tutorial shows you how to set up a bastion host, which is a special-purpose host machine designed to withstand attack. The bastion host uses Tinyproxy to forward client traffic to the cluster. You use Identity-Aware Proxy (IAP) to securely access the bastion host from your remote client.

Note: This tutorial provides instructions for working with this app: Tinyproxy. The instructions might not represent newer versions of the app. For more information, see the documentation: Tinyproxy.

Create a private cluster

Create a new private cluster with no client access to the public endpoint. Place the cluster in its own subnet. You can do this using the Google Cloud CLI or the Google Cloud console.

gcloud

Run the following command:

gcloud container clusters create-auto CLUSTER_NAME \
    --location=CONTROL_PLANE_LOCATION \
    --create-subnetwork=name=SUBNET_NAME \
    --enable-master-authorized-networks \
    --enable-private-nodes \
    --enable-private-endpoint

Replace the following:

• CLUSTER_NAME: the name of the new cluster.
• CONTROL_PLANE_LOCATION: the Compute Engine region of the control plane of your cluster.
• SUBNET_NAME: the name of the new subnetwork in which you want to place the cluster.

Console

Create a Virtual Private Cloud subnetwork

1. Go to the VPC networks page in the Google Cloud console.

   Go to VPC networks

2. Click the default network.

3. In the Subnets section, click Add subnet.

4. On the Add a subnet dialog, specify the following:

   1. Name: A name for the new subnet.
   2. Region: A region for the subnet. This must be the same as the cluster region.
   3. IP address range: Specify 10.2.204.0/22 or another range that doesn't conflict with other ranges in the VPC network.
   4. For Private Google Access, select the On option.

5. Click Add.

Create a private cluster

1. Go to the Google Kubernetes Engine page in the Google Cloud console.

   Go to Google Kubernetes Engine

2. Click Create.

3. Click Configure for GKE Autopilot.

4. Specify a Name and Region for the new cluster. The region must be the same as the subnet.

5. In the Networking section, select the Private cluster option.

6. Clear the Access control plane using its external IP address checkbox.

7. From the Node subnet drop-down list, select the subnet you created.

8. Optionally, configure other settings for the cluster.

9. Click Create.

You can also use a GKE Standard cluster with the --master-ipv4-cidr flag specified.

Create a bastion host VM

Create a Compute Engine VM within the private cluster internal network to act as a bastion host that can manage the cluster.

gcloud

Create a Compute Engine VM:

gcloud compute instances create INSTANCE_NAME \
    --zone=COMPUTE_ZONE \
    --machine-type=e2-micro \
    --network-interface=no-address,network-tier=PREMIUM,subnet=SUBNET_NAME

Replace the following:

• INSTANCE_NAME: the name of the VM.
• COMPUTE_ZONE: the Compute Engine zone for the VM. Place this in the same region as the cluster.

• SUBNET_NAME: the subnetwork in which you want to place the VM.

  Note: We recommend that you create the VM in the same VPC subnet as the cluster. If you create the VM in a different subnet, add the VM IP address range as an authorized network for the cluster.

Console

1. Go to the VM instances page in the Google Cloud console.

   Go to VM instances

2. Click Create instance.

3. Specify the following:

   1. Name: the name of your VM.
   2. Region and Zone: the region and zone of your VM. Use the same region as your cluster.
   3. Machine type: a machine type. Choose a small machine type, such as e2-micro.
   4. For Network interfaces, select the same VPC network and subnet as the cluster.
   5. Optionally, configure other settings for the instance.

4. Click Create.

Note: Create the VM in the same VPC subnet as the cluster. If you create the VM in a different subnet, add the VM IP address range as an authorized network for the cluster.

Note: You should automate bastion host setup and upgrades using a tool like Terraform.

Create firewall rule

To allow IAP to connect to your bastion host VM, create a firewall rule.

Deploy the proxy

Note: Some commands in this section require administrator privileges.

With the bastion host and the private cluster configured, you must deploy a proxy daemon in the host to forward traffic to the cluster control plane. For this tutorial, you install Tinyproxy.

1. Start a session into your VM:

   gcloud compute ssh INSTANCE_NAME --tunnel-through-iap --project=PROJECT_ID
   

2. Install Tinyproxy:

   sudo apt install tinyproxy
   

3. Open the Tinyproxy configuration file:

   sudo vi /etc/tinyproxy/tinyproxy.conf
   

4. In the file, do the following:

   1. Verify that the port is 8888.

   2. Search for the Allow section:

        /Allow 127
      

   3. Add the following line to the Allow section:

        Allow localhost
      

5. Save the file and restart Tinyproxy:

   sudo service tinyproxy restart
   

6. Exit the session:

   exit
   

Connect to your cluster from the remote client

After configuring Tinyproxy, you must set up the remote client with cluster credentials and specify the proxy. Do the following on the remote client:

1. Get credentials for the cluster:

   gcloud container clusters get-credentials CLUSTER_NAME \
       --location=CONTROL_PLANE_LOCATION \
       --project=PROJECT_ID
   

   Replace the following:

   • CLUSTER_NAME: the name of the private cluster.
   • CONTROL_PLANE_LOCATION: the Compute Engine location of the control plane of your cluster. Provide a region for regional clusters, or a zone for zonal clusters.
   • PROJECT_ID: the ID of the Google Cloud project of the cluster.

2. Tunnel to the bastion host using IAP:

   gcloud compute ssh INSTANCE_NAME \
       --tunnel-through-iap \
       --project=PROJECT_ID \
       --zone=COMPUTE_ZONE \
       --ssh-flag="-4 -L8888:localhost:8888 -N -q -f"
   

3. Specify the proxy:

   export HTTPS_PROXY=localhost:8888
   kubectl get ns
   

   The output is a list of namespaces in the private cluster.

Stop listening on the remote client

If you want to revert the change on the remote client at any time, you should end the listener process on TCP port 8888. The command to do this is different depending on the client operating system.

netstat -lnpt | grep 8888 | awk '{print $7}' | grep -o '[0-9]\+' | sort -u | xargs sudo kill

Troubleshooting

Firewall restrictions in enterprise networks

If you're on an enterprise network with a strict firewall, you might not be able to complete this tutorial without requesting an exception. If you request an exception, the source IP range for the bastion host is 35.235.240.0/20 by default.