This document describes how to configure Knative serving running on
Google Kubernetes Engine (GKE) to use GKE Sandbox for improved workload
isolation from the host kernel.

## Objectives

- Learn how to enable GKE Sandbox on your Knative serving cluster.
- Deploy a service specified to run on sandboxed nodes.

## Costs


In this document, you use the following billable components of Google Cloud:


- [Knative serving](https://docs.cloud.google.com/kubernetes-engine/enterprise/knative-serving/docs/architecture-overview.md.txt)


To generate a cost estimate based on your projected usage,
use the [pricing calculator](https://docs.cloud.google.com/products/calculator).
New Google Cloud users might be eligible for a [free trial](https://docs.cloud.google.com/free).

<br />

## Before you begin

- This tutorial assumes that you have Knative serving [installed and configured
  on your cluster](https://docs.cloud.google.com/kubernetes-engine/enterprise/knative-serving/docs/install).
- Ensure that your command-line environment is set up and the tools are up-to-date:
  - [GKE clusters on Google Cloud](https://docs.cloud.google.com/kubernetes-engine/enterprise/knative-serving/docs/install/on-gcp/command-line-tools.md.txt)
  - [GKE clusters outside Google Cloud](https://docs.cloud.google.com/kubernetes-engine/enterprise/knative-serving/docs/install/outside-gcp/command-line-tools.md.txt)

## Add GKE Sandbox node pool to an existing Knative serving Cluster

Follow the [steps to add a node pool enabled to use
GKE Sandbox](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/sandbox-pods#enabling-existing)
to your existing Knative serving cluster. This is where workloads that you opt in to use GKE Sandbox run.

## Deploy a workload on the GKE Sandbox node pool

Once you have a GKE cluster with a GKE Sandbox-enabled node
pool, update your Knative serving configuration. Also, update each service
specification to make use of the new nodes. These updates are required just one
time.

### Enable the runtime class feature flag

In this section, you enable the Kubernetes runtime class by using Knative
feature and extension flags in the `config-features` ConfigMap in the
`knative-serving` namespace. To enable the runtime class, you set the ConfigMap
flag `kubernetes.podspec-runtimeclassname` to `enabled`. For more information,
see [Kubernetes runtime
class](https://knative.dev/docs/serving/feature-flags/#kubernetes-runtime-class)
and [Configuring
flags](https://knative.dev/docs/serving/configuration/feature-flags/#configuring-flags)
in the Knative [Features and
extensions](https://knative.dev/docs/serving/configuration/feature-flags/)
documentation.

1. Open the `config-features` ConfigMap for editing:

   ```bash
   kubectl edit configmap config-features -n knative-serving
   ```
2. Add the `kubernetes.podspec-runtimeclassname: enabled` feature flag to the
   `data` field.

3. As an alternative to the preceding steps, you can run the following command
   to patch this flag into your Knative features:

   ```bash
   kubectl patch configmap config-features \
       --type='json' \
       -p="[{'op': 'replace', 'path': '/data/kubernetes.podspec-runtimeclassname', 'value': 'enabled'}]" \
       -n knative-serving
   ```
4. To verify your edits, run the following command to view the ConfigMap:

   ```bash
   kubectl get configmap config-features -n knative-serving -o yaml
   ```

   The ConfigMap should resemble the following:

   ```yaml
   apiVersion: v1
   kind: ConfigMap
   metadata:
       name: config-features
       namespace: knative-serving
       labels:
           serving.knative.dev/release: v0.22.0data:
   kubernetes.podspec-runtimeclassname: enabled
   ```

### Deploy a Service configured to run in GKE Sandbox

After you've enabled the runtime class feature flag, you can create Services
that specify a `runtimeClassName` in the spec. Set the `runtimeClassName` to
`gvisor` to indicate that a service should run on the GKE Sandbox node pool.

> [!IMPORTANT]
> **Important:** Not all workloads are compatible with GKE Sandbox, [learn more
> about its
> limitations](https://docs.cloud.google.com/kubernetes-engine/docs/concepts/sandbox-pods#limitations).

1. Deploy a "Hello World" Knative application by running the following
   command:

   ```bash
   cat <<EOF | kubectl apply -f -
   apiVersion: serving.knative.dev/v1
   kind: Service
   metadata:
       name: helloworld-go
   spec:
       template:
           spec:
               runtimeClassName: gvisor
               containers:
                   -   image: gcr.io/knative-samples/helloworld-go
                       env:
                       -   name: TARGET
                           value: "gVisor User"
   EOF
   ```

   If the Kubernetes runtime class name feature is enabled, the command should
   finish without errors.
2. Get the pods for your service running and their configured runtime class by
   running:

   ```bash
   kubectl get pods \
       -o=custom-columns='NAME:.metadata.name,RUNTIME CLASS:.spec.runtimeClassName,STATUS:.status.phase'
   ```

   The output should resemble the following:

       NAME                                              RUNTIME CLASS   STATUS
       helloworld-go-00001-deployment-55f58477f-fxm6m    gvisor          Running

## What's next

After verifying that you can deploy untrusted apps on your newly created GKE Sandbox node pool, you're ready to add `runtimeClassName: gvisor` to the
Kubernetes specs for other untrusted services you would like to execute on
GKE Sandbox node pools.

Additional reading:

- [GKE Sandbox documentation](https://docs.cloud.google.com/kubernetes-engine/docs/concepts/sandbox-pods.md.txt) docs to understand limitations that may break compatibility with some services.
- [gVisor docs on deploying Knative services](https://gvisor.dev/docs/tutorials/knative/) for a more detailed guide to deploying Knative apps on gVisor (also called GKE Sandbox).