Google Distributed Cloud (GDC) air-gapped provides a built-in managed Layer 4 (L4) load
balancer, but many enterprise applications require advanced Layer 7 (L7)
capabilities, such as host-based routing, centralized TLS management, and
complex traffic splitting. Historically, this is accomplished using the Ingress
API, which is now considered feature-frozen in the Kubernetes community.

This reference architecture provides a self-managed Layer 7 load balancing
solution. By deploying the popular NGINX Gateway Fabric open source controller
on a GDC Standard Cluster, customers can seamlessly
route L7 traffic to hybrid environments. This architecture uses TLS Termination
(HTTPRoute) to route traffic based on Server Name Indication (SNI) to both
built-in containerized pods and applications hosted on external
Virtual Machines.

## Architecture

![Architecture diagram for NGINX Layer 7 load balancing on GDC air-gapped.](https://docs.cloud.google.com/static/distributed-cloud/hosted/docs/latest/gdcag/solutions/images/l7-lb-nginx-arch.png)

The key components of the solution include:

- **Client:** An entity initiating HTTPS requests to interact with the applications.
- **GDC Standard Cluster:** GDC provides a built-in way to create Kubernetes Vanilla Clusters. In this solution, the cluster will host the L7 LB and its controllers, along with the workloads and healess service for external VMs
- **GDC L4 Load Balancer:** The built-in L4 load balancer serving as the entry point, distributing TCP/443 traffic directly to the Kubernetes Pods running the Gateway controllers.
- **Gateway Controllers:** NGINX operators running in the Standard Cluster. They monitor Gateway API resources (like `Gateway` and `HTTPRoute`) and dynamically update the underlying proxies. [Nginx Gateway
  Fabric](https://github.com/nginx/nginx-gateway-fabric) will be used for the solution.
- **Gateway (with HTTPRoute):** Standardized Kubernetes resources defining the physical listening port (443) and the SNI-based host routing rules with TLS Termination.
- **Containerized Workload (Pods):** A standard Kubernetes Deployment exposed internally with a regular Kubernetes `Service`.
- **VM-based Workload (External):** A workload hosted on an external VM on the project network, exposed to the proxy with a headless Kubernetes `Service` and a custom endpoint containing the VM's direct IP.
- **Harbor Registry:** A private container registry used to store and serve the proxy and application images in the air-gapped environment.

In the Standard Cluster, you create four namespaces:

- The `nginx-gateway` namespace that hosts the Nginx Gateway Fabric
  resources:

  ![The nginx-gateway namespace resources.](https://docs.cloud.google.com/static/distributed-cloud/hosted/docs/latest/gdcag/solutions/images/l7-lb-nginx-gateway-ns.png)
- The `load-balancer` namespace that hosts the `Gateway` resources:

  ![The load-balancer namespace resources.](https://docs.cloud.google.com/static/distributed-cloud/hosted/docs/latest/gdcag/solutions/images/l7-lb-nginx-lb-ns.png)
- The `vm-app` namespace hosts the external VM IP `vm-app`, a headless
  service, an `EndpointSlice` that points at the external IP, and an
  `HTTPRoute` for the `Gateway`:

  ![The vm-app namespace resources.](https://docs.cloud.google.com/static/distributed-cloud/hosted/docs/latest/gdcag/solutions/images/l7-lb-nginx-vm-app-ns.png)
- The `hello-app` namespace hosts the `Deployment`, a `Service`, a
  `HTTPRoute`, and a `Gateway` for the demo container workload:

  ![The hello-app namespace resources.](https://docs.cloud.google.com/static/distributed-cloud/hosted/docs/latest/gdcag/solutions/images/l7-lb-nginx-hello-app-ns.png)

## Before you begin

Before deploying this solution, ensure you have the following prerequisites in
place:

- **Software needed:** helm, docker, kubectl
- **CLI login and local setup:** Download gdcloud CLI from the GDC console and set
  up your environment locally.

  ```bash
  export USER_NAME="USER_NAME"
  export PROJECT_ID="PROJECT_ID"
  export ZONE="ZONE"
  export ORG_NAME="ORG_NAME"
  export GDC_URL="GDC_URL"

  gdcloud components install gdcloud-k8s-auth-plugin

  gdcloud config set core/organization_console_url \
    https://console.$ORG_NAME.$ZONE.$GDC_URL
  gdcloud config set core/zone $ZONE
  gdcloud config set core/project ${PROJECT_ID}

  gdcloud auth login # use --login-config-cert option in case of TLS error
  ```
- **Project Setup:** Create a
  [project](https://docs.cloud.google.com/distributed-cloud/hosted/docs/latest/gdcag/platform/pa-user/create-a-project.md.txt) in your
  GDC air-gapped environment to hold the resources.

  ```bash
  gdcloud projects create $PROJECT_ID
  ```
- **IAM Roles:** Grant your user the **Cluster Admin** and **Standard Cluster
  Admin** roles to manage Kubernetes resources, and the **Harbor Instance Admin**
  role to push images.

  ```bash
  # Grant standard cluster and cluster admin roles
  gdcloud projects add-iam-policy-binding ${PROJECT_ID} \
    --member="user:${USER_NAME}" \
    --role=cluster-admin

  gdcloud projects add-iam-policy-binding ${PROJECT_ID} \
    --member="user:${USER_NAME}" \
    --role=standard-cluster-admin

  # Grant Harbor instance admin role
  gdcloud projects add-iam-policy-binding ${PROJECT_ID} \
    --member="user:${USER_NAME}" \
    --role=harbor-instance-admin
  ```

## Create a standard cluster

This section guides you through the process of setting up a standard Kubernetes
cluster within your GDC air-gapped environment. A
standard cluster provides a flexible and robust foundation for deploying various
workloads, including the Nginx Gateway Controller, and your custom applications.
The following steps will ensure your cluster is properly configured and
accessible for subsequent deployments.

1. Identify the available virtual machine image types by running:

   ```bash
   gdcloud compute machine-types list
   ```
2. Select an appropriate machine type for your cluster worker nodes. For this
   tutorial, a machine type with *at least 4 vCPUs* is recommended.

   ```bash
   export MACHINE_TYPE="MACHINE_TYPE"
   ```
3. Get the management API server kubeconfig and set an alias:

   ```bash
   export CLUSTER_NAME="CLUSTER_NAME"

   KUBECONFIG=kubeconfig-admin.yaml gdcloud clusters \
     get-credentials ${ORG_NAME}-admin

   alias km="kubectl --kubeconfig kubeconfig-admin.yaml"
   ```
4. Create a standard cluster with 2 worker nodes:

   ```bash
   km create -f - <<EOF
   apiVersion: cluster.gdc.goog/v1
   kind: Cluster
   metadata:
     name: ${CLUSTER_NAME}
     namespace: ${PROJECT_ID}
   spec:
     nodePools:
     - machineTypeName: ${MACHINE_TYPE}
       nodeCount: 2
       name: ${CLUSTER_NAME}-node-pool
   EOF
   ```

   For more details on available options, refer to the
   [documentation](https://docs.cloud.google.com/distributed-cloud/hosted/docs/latest/gdcag/platform/pa-user/create-standard-cluster.md.txt).

   Standard cluster creation can take up to 60 minutes to complete. To
   check the status, use the following command:

   ```bash
   km get clusters/${CLUSTER_NAME} \
     -n ${PROJECT_ID} \
     --watch
   ```

   Once the cluster is ready, the output should show a state of
   Running, like this:

       NAME         STATE     K8S VERSION
       my-cluster   Running   1.30.12-gke.300

5. After the cluster is ready, retrieve its credentials:

   ```bash
   KUBECONFIG=kubeconfig-${CLUSTER_NAME}.yaml gdcloud clusters \
     get-credentials ${CLUSTER_NAME} \
     --standard \
     --project ${PROJECT_ID} \
     --zone ${ZONE}
   ```
6. Create an alias to keep `kubectl` commands more concise in the rest of this
   guide. This alias will be used to interact with the standard cluster:

   ```bash
   alias kk="kubectl --kubeconfig kubeconfig-${CLUSTER_NAME}.yaml"
   ```
7. Create namespaces for the controller, the "hello-app" demo containerized
   app, and the VM-based demo app:

   ```bash
   kk create namespace load-balancer
   kk create namespace hello-app
   kk create namespace vm-app
   ```

## Create and integrate Harbor Registry

Harbor is a container image registry with built-in support in
GDC air-gapped. This section guides you through the
steps to integrate a Harbor Registry with your standard cluster, including
configuring credentials and secrets to enable secure pulling and pushing of
images.

1. Create a [Harbor
   instance](https://docs.cloud.google.com/distributed-cloud/hosted/docs/latest/gdcag/platform-application/pa-ao-operations/create-harbor-instances) in your project.
2. Create a [Harbor
   project](https://docs.cloud.google.com/distributed-cloud/hosted/docs/latest/gdcag/platform-application/pa-ao-operations/create-harbor-projects) in your Harbor instance.
3. Set environment variables:

   ```bash
   export HARBOR_INSTANCE_NAME="HARBOR_INSTANCE_NAME"
   export HARBOR_INSTANCE_URL="HARBOR_INSTANCE_URL"
   export HARBOR_PROJECT="HARBOR_INSTANCE_PROJECT"
   export IMAGE_PULL_SECRET_NAME="harbor-secret"
   ```
4. Sign in to the Harbor instance using a [robot
   account](https://docs.cloud.google.com/distributed-cloud/hosted/docs/latest/gdcag/platform-application/deploy-container-workloads#create-image-pull-secret):

   ```bash
   docker --config=./docker login ${HARBOR_INSTANCE_URL}
   ```
5. Create the secrets in the standard cluster:

   ```bash
   kk create secret docker-registry ${IMAGE_PULL_SECRET_NAME} \
     --from-file=.dockerconfigjson=./docker/config.json \
     -n load-balancer

   kk create secret docker-registry ${IMAGE_PULL_SECRET_NAME} \
     --from-file=.dockerconfigjson=./docker/config.json \
     -n hello-app
   ```

## Deploy demo containerized app

This section details the deployment of a demo containerized application
(`hello-app`) within your GDC air-gapped Kubernetes
cluster. You will create the necessary Kubernetes Deployment and Service
resources to run the `hello-app` and expose it internally within the cluster,
preparing it for access using the L7 load balancer.

1. Upload a sample image for the demo containerized app into Harbor:

   ```bash
   docker pull gcr.io/google-samples/hello-app:1.0 \
     --platform linux/amd64
   docker tag gcr.io/google-samples/hello-app:1.0 \
     ${HARBOR_INSTANCE_URL}/${HARBOR_PROJECT}/hello-app:1.0
   docker --config=./docker push \
     ${HARBOR_INSTANCE_URL}/${HARBOR_PROJECT}/hello-app:1.0
   ```
2. Deploy the following manifest in the standard cluster:

   ```bash
   cat << EOF > hello-app.yaml
   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: hello-app
     namespace: hello-app
   spec:
     replicas: 2
     selector:
       matchLabels:
         app: hello-app
     template:
       metadata:
         labels:
           app: hello-app
       spec:
         containers:
         - name: hello-server
           image: ${HARBOR_INSTANCE_URL}/${HARBOR_PROJECT}/hello-app:1.0
           ports:
           - containerPort: 8080
         imagePullSecrets:
         - name: ${IMAGE_PULL_SECRET_NAME}
   ---
   apiVersion: v1
   kind: Service
   metadata:
     name: hello-app
     namespace: hello-app
   spec:
     type: ClusterIP
     selector:
       app: hello-app
     ports:
       - protocol: TCP
         port: 80
         targetPort: 8080
   EOF

   kk apply -f hello-app.yaml
   ```

Then verify that the deployment and the service are there:

```bash
kk get svc,deploy -n hello-app
```

## Deploy demo app in a VM

This section details the deployment of a demo application within a virtual
machine (VM) outside your Kubernetes cluster. By setting up an HTTP server
on a VM, you'll simulate an external application that the load balancer can
expose, demonstrating its capability to manage traffic to resources both inside
and outside the cluster.

First, create a VM for the demo app

1. Open the GDC console in your web browser.
2. Select the same project as where you created your standard Kubernetes cluster.
3. Open the menu then click **Virtual machines**.
4. Click **Create Instance**.
5. Give the VM the name `vm-workload`. A 2 vCPU image is enough for the example.
6. For the boot disk image, select an Ubuntu 22.04 distribution, which comes with Python pre-installed.
7. Click **Create**.
8. Wait a few minutes until the VM becomes ready.
9. Establish an SSH connection to the VM:
   1. In the GDC console, click the VM.
   2. Click **Connect with SSH**.

After you're connected to SSH console, run the following:

```bash
mkdir ~/simple-server
cd ~/simple-server
echo 'Welcome to my VM!' > index.html
python3 -m http.server --bind 0.0.0.0 8080 &
```

To route traffic to a VM, create a headless Service (without selectors). This
will be manually mapped to the VM's internal IP address using an `EndpointSlice`
resource.

```bash
kk apply -f - <<EOF
apiVersion: v1
kind: Service
metadata:
  name: vm-app-svc
  namespace: vm-app
spec:
  ports:
  - protocol: TCP
    port: 443
    targetPort: 443
EOF
```

Get the IP address of `vm-workload` VM by running

```bash
gdcloud compute instances list --project ${PROJECT_ID} \
  | grep workload-vm | awk '{print $3}'
```

The output will be the IP address of the VM that will be needed to set up the
EndpointSlice resource.

Create the resource `EndpointSlice` that will connect to the VM-app
selector-less Service and address the IP of the VM to which the traffic should
be routed.

```yaml
apiVersion: discovery.k8s.io/v1
kind: EndpointSlice
metadata:
  name: vm-app-endpoints
  namespace: vm-app
  labels:
    kubernetes.io/service-name: vm-app-svc
addressType: IPv4
ports:
  - port: 8080
endpoints:
  - addresses:
    - "VM_IP"
    conditions:
      ready: true
```

## Create self-signed certificates

This section guides you through the process of generating self-signed TLS
certificates for your applications. This guide uses self-signed certificates for
convenience, but in production environments, you must use production-grade
certificates, as described in
[Optional: Use production-ready certificates](https://docs.cloud.google.com/distributed-cloud/hosted/docs/latest/gdcag/solutions/l7-lb-nginx-impl.md.txt#optional-use-production-ready-certificates).
These certificates are crucial for enabling HTTPS termination at the Nginx
Gateway, ensuring encrypted communication between clients and the load balancer
for both containerized and VM-based workloads.

1. Create a certificate for the containerized app:

   ```bash
   openssl req -x509 -newkey rsa:2048 -nodes \
     -keyout tls-containerized.key \
     -out tls-containerized.crt \
     -subj "/CN=k8s-app.example.com" \
     -days 365

   kk create secret tls tls-containerized \
     --namespace load-balancer \
     --key tls-containerized.key \
     --cert tls-containerized.crt

   kk create secret tls tls-containerized \
     --namespace hello-app \
     --key tls-containerized.key \
     --cert tls-containerized.crt
   ```
2. Create a certificate for the VM app:

   ```bash
   openssl req -x509 -newkey rsa:2048 -nodes \
     -keyout tls-vm.key \
     -out tls-vm.crt \
     -subj "/CN=vm-app.example.com" \
     -days 365

   kk create secret tls tls-vm \
     --namespace load-balancer \
     --key tls-vm.key \
     --cert tls-vm.crt

   kk create secret tls tls-vm \
     --namespace vm-app \
     --key tls-vm.key \
     --cert tls-vm.crt
   ```

## Deploy NGINX

This section outlines the steps to deploy the Nginx Gateway Controller,
including installing the Gateway API Custom Resource Definitions (CRDs), setting
up Nginx Gateway Fabric using Helm, and verifying the installation within your
standard cluster. This prepares the infrastructure for advanced Layer 7 routing.

### Install the Gateway API CRDs

The Gateway API requires Custom Resource Definitions (CRDs) to be installed in
the cluster before deploying the controller. We will use the official
experimental custom resource definitions (CRD) from the Gateway API project (version 1.2.0 is used for this
guide).

1. Install the experimental CRDs:

   ```bash
   kk apply -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.2.0/experimental-install.yaml
   ```
2. Verify the CRDs are installed successfully by running `kk get crd
   gateways.gateway.networking.k8s.io`.

### Install NGINX Gateway Fabric

Deploy the NGINX Gateway Fabric controller using Helm. This controller will
watch Gateway API resources and configure NGINX to handle the traffic.

1. Set the NGINX image tag:

   ```bash
   helm template nfg oci://ghcr.io/nginx/charts/nginx-gateway-fabric | grep "image:" # get the tag of the nginx-gateway-fabric (in following case 2.4.2)

   export NGINX_TAG="2.4.2"
   ```
2. Pull the NGINX Gateway Fabric and NGINX images, then push them to your
   Harbor registry:

   ```bash
   docker pull ghcr.io/nginx/nginx-gateway-fabric:${NGINX_TAG}
   docker tag ghcr.io/nginx/nginx-gateway-fabric:${NGINX_TAG} \
     ${HARBOR_INSTANCE_URL}/${HARBOR_PROJECT}/nginx-gateway-fabric:${NGINX_TAG}
   docker --config=./docker push \
     ${HARBOR_INSTANCE_URL}/${HARBOR_PROJECT}/nginx-gateway-fabric:${NGINX_TAG}

   docker pull nginx:1.27.3
   docker tag nginx:1.27.3 \
     ${HARBOR_INSTANCE_URL}/${HARBOR_PROJECT}/nginx:1.27.3
   docker --config=./docker push \
     ${HARBOR_INSTANCE_URL}/${HARBOR_PROJECT}/nginx:1.27.3
   ```
3. Create the `nginx-gateway` namespace and add the Harbor image pull secret:

   ```bash
   kk create namespace nginx-gateway

   kk create secret docker-registry ${IMAGE_PULL_SECRET_NAME} \
     --from-file=.dockerconfigjson=./docker/config.json \
     -n nginx-gateway
   ```
4. Deploy NGINX Gateway Fabric with Helm, pointing to the images in your Harbor
   registry:

   ```bash
   KUBECONFIG=kubeconfig-${CLUSTER_NAME}.yaml helm upgrade --install ngf \
     oci://ghcr.io/nginx/charts/nginx-gateway-fabric \
     --create-namespace -n nginx-gateway \
     --set nginxGateway.image.tag="${NGINX_TAG}" \
     --set nginxGateway.image.repository="${HARBOR_INSTANCE_URL}/${HARBOR_PROJECT}/nginx-gateway-fabric" \
     --set nginx.image.repository="${HARBOR_INSTANCE_URL}/${HARBOR_PROJECT}/nginx" \
     --set nginxGateway.serviceAccount.imagePullSecret="${IMAGE_PULL_SECRET_NAME}" \
     --set nginx.imagePullSecret="${IMAGE_PULL_SECRET_NAME}"
   ```

   The Kubernetes deployment securely pulls images from Harbor using the
   provided secret (`${IMAGE_PULL_SECRET_NAME}`), which contains the
   Harbor robot account's credentials.
5. Verify that the `GatewayClass` resources are accepted:

   ```bash
   kk get gatewayclass
   ```

   You should see `nginx` listed with `ACCEPTED = True`.

### Create the Gateway instance

Define the logical load balancer instance listening on port 443. We will
configure it for `Terminate` mode, meaning the Gateway will perform TLS
termination and decrypt the traffic before passing it through.

Create and apply `gateway.yaml`:

```bash
cat << EOF > gateway.yaml 
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: my-gateway
  namespace: load-balancer
spec:
  gatewayClassName: nginx
  listeners:
  - name: https-k8s-workload
    hostname: "k8s-app.example.com"
    port: 443
    protocol: HTTPS
    allowedRoutes: 
      namespaces: 
        from: All
    tls:
      mode: Terminate
      certificateRefs:
      - kind: Secret
        name: tls-containerized 
  - name: https-vm-workload
    hostname: "vm-app.example.com"
    port: 443
    protocol: HTTPS
    allowedRoutes: 
      namespaces: 
        from: All
    tls:
      mode: Terminate
      certificateRefs:
      - kind: Secret
        name: tls-vm 
EOF

kk apply -f gateway.yaml
```

Verify the Gateway is deployed successfully by checking if `PROGRAMMED` is
`True`:

```bash
kk get gateway my-gateway --n load-balancer
```

Verify that a managed GDC L4 load balancer has been
deployed as a Service alongside the Gateway

```bash
kk get services -n load-balancer
```

Check that the Nginx service is created and configured, the output should look
like this:

    NAME               TYPE           CLUSTER-IP      EXTERNAL-IP    PORT(S)         AGE
    my-gateway-nginx   LoadBalancer   10.252.19.148   10.200.32.43   443:30649/TCP   45h

### Define the L7 routing logic (HTTPRoute)

Bind `HTTPRoute`s to your Gateway to define how traffic should be distributed
based on the requested hostname (SNI).

Create and apply `routing.yaml`:

```bash
cat << EOF > routing.yaml 
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: k8s-http-route
  namespace: hello-app
spec:
  parentRefs:
  - name: my-gateway 
    namespace: load-balancer
  hostnames:
  - "k8s-app.example.com"
  rules:
  - matches:
    - path:
        type: PathPrefix
        value: /
    backendRefs:
    - name: hello-app 
      namespace: hello-app
      port: 80          
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: vm-http-route
  namespace: vm-app
spec:
  parentRefs:
  - name: my-gateway
    namespace: load-balancer
  hostnames:
  - "vm-app.example.com"
  rules:
  - matches:
    - path:
        type: PathPrefix
        value: /
    backendRefs:
    - name: vm-app-svc  
      namespace: vm-app
      port: 443       
EOF

kk apply -f routing.yaml   
```

### Retrieve the load balancer IP

Run the command to get the IP of Nginx Load Balancer

```bash
kk get services/my-gateway-nginx \
  -n load-balancer \
  -o jsonpath='{.status.loadBalancer.ingress[0].ip}'
```

This will be needed when verifying the access to the apps. This will be referred
to as `LOAD_BALANCER_IP`.

## Create a client VM

Follow the steps to create a client VM

1. Open the GDC console in your web browser.
2. Open the menu then click **Virtual machines**.
3. Click **Create Instance**.
4. Create a VM named `client`, select a small machine type and select either Rocky Linux or Ubuntu, which come with `curl` preinstalled.
5. Click **Create**.
6. Wait a few minutes until the VM becomes ready.
7. After the VM is ready, establish an SSH connection to the VM:
   1. In the GDC console, click the VM.
   2. Click **Connect with SSH**.

## Verify access and routing

To test the routing, execute `curl` commands from your client VM. You can
connect to both applications using their defined hostnames with the load
balancer's IP address.

By passing the `--resolve` flag in curl, you can force the domain names to
resolve to your GDC air-gapped L4 Load Balancer's IP address.
Note that we pass the `-k` flag to trust the self-signed certificates.

Test the Kubernetes containerized app:

```bash
curl -k --resolve k8s-app.example.com:443:$LOAD_BALANCER_IP https://k8s-app.example.com -v
```

Test the External VM App:

```bash
curl -k --resolve vm-app.example.com:443:$LOAD_BALANCER_IP https://vm-app.example.com -v
```

If configured correctly, the Gateway controller will seamlessly act as TLS
terminator and pass-through the traffic to the destination.

## Optional: Use production-ready certificates

This section covers how to leverage the GDC air-gapped
CA Service to create a private Root Certificate Authority (CA), issue signed
certificates for your workloads, and securely update your
GDC air-gapped standard cluster and client VMs.

This section outlines how to use the GDC air-gapped [CA
Service](https://docs.cloud.google.com/distributed-cloud/hosted/docs/latest/gdcag/application/ao-user/ca-service/cas-overview) to create a
private Root Certificate Authority (CA) and issue valid certificates for your
applications. By installing this Root CA on your client VM, you can verify that
TLS termination works seamlessly with trusted certificates, without needing to
bypass SSL warnings (for example, using `curl -k`).

### Grant necessary permissions and get credentials

To manage the CA Service and issue certificates, your user needs the appropriate
IAM roles in the project.

1. Grant the `certificate-authority-service-admin` and `certificate-requester`
   roles:

   ```bash
   gdcloud projects add-iam-policy-binding ${PROJECT_ID} \
     --member=user:${USER_NAME} \
     --role=certificate-authority-service-admin

   gdcloud projects add-iam-policy-binding ${PROJECT_ID} \
     --member=user:${USER_NAME} \
     --role=certificate-requester
   ```
2. Get the credentials of the management API server:

   ```bash
   gdcloud clusters get-credentials ${ORG_NAME}-admin
   ```

### Create the root CA

You will create a Certificate Authority in the management API server within your
project namespace.

1. Apply the `CertificateAuthority` resource:

   ```bash
   km apply -f - <<EOF
   apiVersion: pki.security.gdc.goog/v1
   kind: CertificateAuthority
   metadata:
     name: my-root-ca
     namespace: ${PROJECT_ID}
   spec:
     caProfile:
       commonName: "My Root CA"
       duration: 87600h # 10 years
       keyAlgorithm: RSA_2048
       maxChainLength: 1
     caType: ROOT
     keyLocation: HSM
     rotationPolicy:
       cronTime: 0 0 1 1 *
   EOF
   ```

   > [!NOTE]
   > **Note:** Root CA creation can take up to 40 minutes to complete. You can monitor the status using the following command. Wait until it returns `True` before proceeding.

   ```bash
   km -n ${PROJECT_ID} get \
   certificateauthority.pki.security.gdc.goog/my-root-ca -ojson \
   | jq -r '
   .status.conditions[] | select( .type as $id | "Ready" | index($id)) .status'
   ```

### Issue and deploy certificates

After the CA is ready, you will request certificates for both the containerized
app and the VM-based app. These requests happen in the management API server, and
the resulting keys must be moved to your standard cluster.

Create requests for both domains:

```bash
km apply -f - <<EOF
apiVersion: pki.security.gdc.goog/v1
kind: CertificateRequest
metadata:
  name: tls-containerized-req
  namespace: ${PROJECT_ID}
spec:
  certificateAuthorityRef:
    name: my-root-ca
    namespace: ${PROJECT_ID}
  certificateConfig:
    subjectConfig:
      commonName: "k8s-app.example.com"
      dnsNames:
      - "k8s-app.example.com"
  signedCertificateSecret: tls-containerized-signed
---
apiVersion: pki.security.gdc.goog/v1
kind: CertificateRequest
metadata:
  name: tls-vm-req
  namespace: ${PROJECT_ID}
spec:
  certificateAuthorityRef:
    name: my-root-ca
    namespace: ${PROJECT_ID}
  certificateConfig:
    subjectConfig:
      commonName: "vm-app.example.com"
      dnsNames:
      - "vm-app.example.com"
  signedCertificateSecret: tls-vm-signed
EOF
```

Wait a few moments for the certificates to be issued. You can verify they are
ready when the Ready condition is True:

```bash
km get certificaterequests -n ${PROJECT_ID}
```

### Update the standard cluster

If you followed the previous sections of this guide, you have
self-signed secrets in your standard cluster. You must delete them before
creating the new, signed versions:

```bash
kk delete secret tls-containerized -n load-balancer
kk delete secret tls-vm -n load-balancer

kk delete secret tls-containerized -n hello-app
kk delete secret tls-vm -n vm-app
```

Now, extract the signed certificates from the management API server and create the
new secrets in the standard cluster.

```bash
km get secret -n ${PROJECT_ID} tls-containerized-signed \
  -o jsonpath='{.data.tls\.crt}' \
  | base64 -d > tls-containerized.crt

km get secret -n ${PROJECT_ID} tls-containerized-signed \
  -o jsonpath='{.data.tls\.key}' \
  | base64 -d > tls-containerized.key

kk create secret tls tls-containerized \
  --namespace load-balancer \
  --key tls-containerized.key \
  --cert tls-containerized.crt

kk create secret tls tls-containerized \
  --namespace hello-app \
  --key tls-containerized.key \
  --cert tls-containerized.crt

km get secret -n ${PROJECT_ID} tls-vm-signed \
  -o jsonpath='{.data.tls\.crt}' \
  | base64 -d > tls-vm.crt

km get secret -n ${PROJECT_ID} tls-vm-signed \
  -o jsonpath='{.data.tls\.key}' \
  | base64 -d > tls-vm.key

kk create secret tls tls-vm \
  --namespace load-balancer \
  --key tls-vm.key \
  --cert tls-vm.crt

kk create secret tls tls-vm \
  --namespace vm-app \
  --key tls-vm.key \
  --cert tls-vm.crt
```

The new secrets will be obtained automatically and refreshed by load balancers.

### Configure client trust

To verify the setup, you need to tell your client VM to trust your new root CA.

Extract the root CA certificate to a file:

```bash
km get secret -n ${PROJECT_ID} my-root-ca-secret \
  -o jsonpath='{.data.tls\.crt}' \
  | base64 -d > my-root-ca.crt
```

Transfer the certificate to your client VM. (You can copy the content of
my-root-ca.crt and paste it into a file on the client VM).

On the client VM, update the trust store.

If `client` VM is Ubuntu:

```bash
sudo cp my-root-ca.crt /usr/local/share/ca-certificates/
sudo chmod 644 /usr/local/share/ca-certificates/my-root-ca.crt
sudo update-ca-certificates
```

If `client` VM is Rocky Linux:

```bash
sudo cp my-root-ca.crt /etc/pki/ca-trust/source/anchors/
sudo update-ca-trust
```

### Verify access

You can now access your applications using curl without the `-k` flag. The
connection will be fully trusted.

Test the Kubernetes containerized app:

```bash
curl -v --resolve k8s-app.example.com:443:LOAD_BALANDER_IP https://k8s-app.example.com
```

Test the VM App:

```bash
curl -v --resolve vm.example.com:443:LOAD_BALANDER_IP https://vm-app.example.com
```

If successful, you will see the application output immediately without any SSL
certificate warnings.