Receive Pub/Sub events at a private HTTP endpoint in a private GKE cluster

Preview

This feature is subject to the "Pre-GA Offerings Terms" in the General Service Terms section of the Service Specific Terms. Pre-GA features are available "as is" and might have limited support. For more information, see the launch stage descriptions.

This tutorial shows you how to create a private HTTP endpoint in a private Google Kubernetes Engine (GKE) cluster that receives Pub/Sub message events using Eventarc. To learn more about this event destination, see Route events to an internal HTTP endpoint in a VPC network.

Private GKE clusters are a type of Virtual Private Cloud (VPC)-native cluster where nodes only have internal IP addresses, which means that nodes and Pods are isolated from the internet by default. You can choose to have no client access, limited access, or unrestricted access to the control plane. You can't convert an existing, non-private cluster to a private cluster. For more information, see About private clusters.

You can run the following commands using the Google Cloud CLI in either your terminal or Cloud Shell.

Create a proxy-only subnet

Unless you create an organizational policy that prohibits it, new projects start with a default network (an auto mode VPC network) that has one subnetwork (subnet) in each region. Each VPC network consists of one or more IP address ranges called subnets. Subnets are regional resources, and have IP address ranges associated with them.

1. Use the gcloud compute networks subnets create command to create a proxy-only subnet in the default network.

   gcloud compute networks subnets create proxy-only-subnet \
       --purpose=REGIONAL_MANAGED_PROXY \
       --role=ACTIVE \
       --region=us-central1 \
       --network=default \
       --range=10.10.10.0/24
   

   Note that a subnet with purpose=REGIONAL_MANAGED_PROXY is reserved for Envoy-based load balancers and that the range must provide 64 or more IP addresses.

2. Create a firewall rule that matches the proxy-only subnet's range and that allows traffic on TCP port 8080.

   gcloud compute firewall-rules create allow-proxy-connection \
       --allow tcp:8080 \
       --source-ranges 10.10.10.0/24 \
       --network=default
   

Create a private GKE cluster

Use the gcloud container clusters create-auto command to create a private GKE cluster in Autopilot mode that has private nodes, and that has no client access to the public endpoint.

The following example creates a private GKE cluster named private-cluster and also creates a subnet named my-subnet:

gcloud container clusters create-auto private-cluster \
    --create-subnetwork name=my-subnet \
    --enable-master-authorized-networks \
    --enable-private-nodes \
    --enable-private-endpoint \
    --region=us-central1

Note the following:

• --enable-master-authorized-networks specifies that access to the public endpoint is restricted to IP address ranges that you authorize.
• --enable-private-nodes indicates that the cluster's nodes don't have external IP addresses.
• --enable-private-endpoint indicates that the cluster is managed using the internal IP address of the control plane API endpoint.

It might take several minutes for the creation of the cluster to complete. Once the cluster is created, the output should indicate that the status of the cluster is RUNNING.

Create a VM instance in a specified subnet

A Compute Engine VM instance is a virtual machine that is hosted on Google's infrastructure. The terms Compute Engine instance, VM instance, and VM are synonymous and are used interchangeably. VM instances include GKE clusters, App Engine flexible environment instances, and other Google Cloud products built on Compute Engine VMs.

Use the gcloud compute instances create command to create a Compute Engine VM instance in the subnet you created previously. Attach a service account and set the VM's access scope to cloud-platform.

gcloud compute instances create my-vm \
    --service-account=[e-mail] \
    --scopes=https://www.googleapis.com/auth/cloud-platform \
    --zone=us-central1-a \
    --subnet=my-subnet

For more information, see Create and start a VM instance.

Deploy an event receiver on the VM

Using a prebuilt image, us-docker.pkg.dev/cloudrun/container/hello, deploy a service on your VM that listens on port 80, and that receives and logs events.

1. Establish an SSH connection to your VM instance by running the following command:

   gcloud compute ssh my-vm --project=PROJECT_ID --zone=us-central1-a
   

   After a connection to the SSH server is established, run the remaining commands on your VM instance.

2. If necessary, install kubectl and any required plugins.

3. From your VM instance, use the get-credentials command to enable kubectl to work with the cluster you created.

   gcloud container clusters get-credentials private-cluster \
       --region=us-central1 \
       --internal-ip
   

4. Use a Kubernetes command, kubectl create deployment, to deploy an application to the cluster.

   kubectl create deployment hello-app \
       --image=us-docker.pkg.dev/cloudrun/container/hello
   

   This creates a Deployment named hello-app. The Deployment's Pod runs the hello container image.

5. After deploying the application, you can expose your application to traffic by creating a Kubernetes Service. Run the following kubectl expose command:

   kubectl expose deployment hello-app \
       --type ClusterIP \
       --port 80 \
       --target-port 8080
   

   You should see service/hello-app exposed in the output.

   You can ignore any messages similar to the following:

   E0418 14:15:33.970933    1129 memcache.go:287] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
   

Configure Kubernetes traffic routing

A Gateway resource represents a data plane that routes traffic in Kubernetes. A Gateway can represent many different kinds of load balancing and routing depending on the GatewayClass it is derived from. For more information, see Deploying Gateways. An HTTPRoute manifest is deployed to create Routes and send traffic to application backends.

1. Deploy a Gateway in your cluster.

   kubectl apply -f - <<EOF
   kind: Gateway
   apiVersion: gateway.networking.k8s.io/v1beta1
   metadata:
     name: internal-http
   spec:
     gatewayClassName: gke-l7-rilb
     listeners:
     - name: http
       protocol: HTTP
       port: 80
   EOF
   

   Note the following:

   • gatewayClassName: gke-l7-rilb specifies the GatewayClass that this Gateway is derived from. gke-l7-rilb corresponds to the internal Application Load Balancer.
   • port: 80 specifies that the Gateway exposes only port 80 for listening for HTTP traffic.

2. Validate that the Gateway has deployed correctly. It might take a few minutes for it to deploy all of its resources.

   kubectl describe gateways.gateway.networking.k8s.io internal-http
   

   The output is similar to the following:

   Name:         internal-http
   Namespace:    default
   ...
   API Version:  gateway.networking.k8s.io/v1beta1
   Kind:         Gateway
   ...
   Spec:
     Gateway Class Name:  gke-l7-rilb
     Listeners:
       Allowed Routes:
         Namespaces:
           From:  Same
       Name:      http
       Port:      80
       Protocol:  HTTP
   Status:
     Addresses:
       Type:   IPAddress
       Value:  10.36.172.5
   ...
   Events:
     Type    Reason  Age                From                   Message
     ----    ------  ----               ----                   -------
     Normal  ADD     80s                sc-gateway-controller  default/internal-http
     Normal  UPDATE  20s (x3 over 80s)  sc-gateway-controller  default/internal-http
     Normal  SYNC    20s                sc-gateway-controller  SYNC on default/internal-http was a success
   

3. Deploy an HTTPRoute manifest to route HTTP traffic to the hello-app service at port 80.

   kubectl apply -f - <<EOF
   kind: HTTPRoute
   apiVersion: gateway.networking.k8s.io/v1beta1
   metadata:
     name: hello-app-route
   spec:
     parentRefs:
     - kind: Gateway
       name: internal-http
     rules:
     - backendRefs:
       - name: hello-app
         port: 80
   EOF
   

Create a network attachment

A network attachment is a resource that lets a producer VPC network initiate connections to a consumer VPC network through a Private Service Connect interface.

To publish events, Eventarc uses the network attachment to establish a connection to the internal HTTP endpoint hosted in a VPC network.

Note: Eventarc reserves the subnet CIDR IP range 240.0.0.0/20. The subnet used for the network attachment shouldn't overlap with this range.

You can create a network attachment that automatically accepts connections from any Private Service Connect interface that refers to the network attachment. Create the network attachment in the same network and region containing the HTTP destination service.

gcloud compute network-attachments create my-network-attachment \
    --region=us-central1 \
    --subnets=my-subnet\
    --connection-preference=ACCEPT_AUTOMATIC

For more information, see About network attachments.

Create an Eventarc trigger

Create an Eventarc trigger that creates a new Pub/Sub topic and routes events to the event receiver deployed on the VM when a message is published to the Pub/Sub topic.

1. Retrieve the Gateway address.

   GATEWAY_ADDRESS=$(kubectl get gateways.gateway.networking.k8s.io internal-http -o=jsonpath="{.status.addresses[0].value}")
   

2. Create a trigger.

   gcloud eventarc triggers create my-trigger \
       --location=us-central1 \
       --destination-http-endpoint-uri="http://$GATEWAY_ADDRESS:80/" \
       --network-attachment="projects/PROJECT_ID/regions/us-central1/networkAttachments/my-network-attachment" \
       --event-filters="type=google.cloud.pubsub.topic.v1.messagePublished" \
       --service-account=[e-mail]
   

   Replace PROJECT_NUMBER with your Google Cloud project number. You can find your project number on the Welcome page of the Google Cloud console or by running the following command:

   gcloud projects describe PROJECT_ID --format='value(projectNumber)'
   

For more information about configuring your trigger, see Route events to an internal HTTP endpoint in a VPC network.

Note: It might take several minutes to create your trigger, and then it can take up to two minutes for the trigger to be fully functional.

Generate and view a Pub/Sub topic event

You can generate an event by publishing a message to a Pub/Sub topic.

1. Find and set the Pub/Sub topic as an environment variable.

   export MY_TOPIC=$(gcloud eventarc triggers describe my-trigger \
       --location=us-central1 \
       --format='value(transport.pubsub.topic)')
   

2. Publish a message to the Pub/Sub topic to generate an event.

   gcloud pubsub topics publish $MY_TOPIC --message "Hello World"
   

   The Eventarc trigger routes the event to the internal HTTP endpoint in the private GKE cluster.

3. Check the application Pod logs and verify the event delivery.

   POD_NAME=$(kubectl get pod --selector app=hello-app --output=name)
   kubectl logs $POD_NAME
   

   The body of the event should be similar to the following:

   2024/04/18 20:31:43 Hello from Cloud Run! The container started successfully and is listening for HTTP requests on $PORT
   {"severity":"INFO","eventType":"google.cloud.pubsub.topic.v1.messagePublished","message":"Received event of type google.cloud.pubsub.topic.v1.messagePublished.
   Event data: Hello World","event":{"specversion":"1.0","id":"10935738681111260","source":"//pubsub.googleapis.com/projects/my-project/topics/eventarc-us-central1-my-trigger-224","type":"google.cloud.pubsub.topic.v1.messagePublished","datacontenttype":"application/json","time":"2024-04-18T20:40:03Z","data":
   {"message":{"data":"SGVsbG8gV29ybGQ=","messageId":"10935738681111260","publishTime":"2024-04-18T20:40:03Z"}}}}
   

You have successfully deployed an event receiver service to an internal HTTP endpoint in a private GKE cluster, created an Eventarc trigger, generated an event from Pub/Sub, and confirmed that the event was routed as expected by the trigger to the target endpoint.