Install Agent Substrate on GKE

Agent Substrate efficiently runs agentic workloads on GKE clusters by suspending idle agents and restoring those agents when they need to act again. Agent Substrate is open source software that you can install onto a GKE Standard cluster.

This document helps you prepare your Google Cloud project and GKE cluster for Agent Substrate, explains how to run the installer, and describes the resources that the installation creates.

Pricing

Agent Substrate is offered at no extra charge in GKE. GKE pricing applies to the resources that you create.

Limitations

Agent Substrate has the following installation limitations:

  • Cloud Shell: you can't install Agent Substrate in Cloud Shell because Cloud Shell provides 5 GB of persistent disk storage, which is insufficient for the installation. You must run the installer on your local machine.
  • Cluster mode: you can install Agent Substrate only on GKE Standard clusters. Autopilot clusters aren't supported.
  • Existing clusters and beta APIs: you can't enable the required beta APIs (podcertificaterequests and clustertrustbundles) on an existing cluster. If you install on an existing cluster running version 1.36, these APIs must have been enabled when the cluster was created.
  • Machine types: you can't use machine series that run on mixed CPU architectures (such as E2 machine types).

Before you begin

Before you start, make sure that you have performed the following tasks:

  • Enable the Google Kubernetes Engine API.
    • Enable Google Kubernetes Engine API
  • To use the Google Cloud CLI for this task, install and then initialize the gcloud CLI. If you previously installed the gcloud CLI, get the latest version by running the gcloud components update command. Earlier gcloud CLI versions might not support running the commands in this document.
  • Install the following components in the gcloud CLI:
    • kubectl
    • gke-gcloud-auth-plugin
  • Ensure that the following tools are installed on your local machine:
    • Git to clone the Agent Substrate repository.
    • Go to run the installer's programs, which are distributed as source code.
    • make to start the installer.

  • Because the installer acts on your behalf to enable APIs and provision resources, set up Application Default Credentials (ADC) by running the following command:

    gcloud auth application-default login

    For more information about authenticating with ADC, see Set up Application Default Credentials.

Required roles

To get the permissions that you need to install Agent Substrate, ask your administrator to grant you the following IAM roles on your project:

For more information about granting roles, see Manage access to projects, folders, and organizations.

You might also be able to get the required permissions through custom roles or other predefined roles.

Cluster requirements

If you let the installer create a cluster for you, it automatically configures a cluster that meets all requirements for Agent Substrate.

If you choose instead to install Agent Substrate on an existing cluster, that cluster must meet the following requirements:

  • Runs on GKE Standard (Agent Substrate is supported on Standard clusters only).
  • Runs GKE version 1.36 (with beta flags enabled) or version 1.37 or later.
  • Uses c3-standard-4 or larger machine types for nodes.
  • Has the PodCertificateRequest and ClusterTrustBundle Kubernetes APIs enabled (certificates.k8s.io/v1beta1/podcertificaterequests and certificates.k8s.io/v1beta1/clustertrustbundles). These APIs are GA in Kubernetes version 1.37 or later, which means that they are enabled on any cluster running version 1.37 or later. You only need to manually enable them on existing clusters running version 1.36 patch versions. For instructions, see Enable beta APIs on an existing cluster.

Install Agent Substrate

To install Agent Substrate, run the following command in your local terminal:

curl -sSL https://raw.githubusercontent.com/ai-on-gke/substrate-gke/main/install.sh | bash

The interactive installer prompts you for your project, cluster, and storage settings. It suggests recommended defaults where possible, which you can accept or customize:

  • Project ID (required): specify the Google Cloud project where resources will be provisioned. If you configured a default project for the gcloud CLI, then the installer uses that project ID.
  • Cluster (required): choose whether to let the installer create a new GKE cluster with recommended defaults or to install Agent Substrate on an existing cluster. If you choose to install on an existing cluster, verify that it meets the requirements in the Cluster requirements section.
  • Snapshot bucket (optional): specify a custom Cloud Storage bucket name, or leave it blank to let the installer create a default bucket named ate-snapshots-<project>-<zone>.

When prompted to select a container image source, you can choose from the following options:

  • Prebuilt images: pulls published images from a container registry. Because nothing is built or pushed, your Google Cloud project doesn't need a container registry of its own.
  • Build from source: builds the images from the upstream Agent Substrate repository (or a custom branch) and pushes them to your project's Artifact Registry registry.

What the installer does

The installer executes the following programs on your behalf:

  • setup-gcp: creates or verifies the required Google Cloud resources, enables APIs, creates the GKE cluster (if you chose not to use an existing cluster) and Cloud Storage bucket, and configures IAM policies.
  • ate-setup: deploys the prebuilt Agent Substrate container images and controllers onto your GKE cluster.

First, the installer makes the following changes to your Google Cloud project:

  1. Enables the Google Cloud APIs that Agent Substrate needs.
  2. Creates a GKE Standard cluster (if not using an existing cluster) that has the following settings:
    • Workload Identity Federation for GKE: allows Agent Substrate to access Cloud Storage without storing keys on the nodes.
    • GKE Dataplane V2: handles the networking required to route incoming requests to your agents.
    • Certificate APIs (PodCertificateRequest and ClusterTrustBundle): enables these Kubernetes beta APIs to issue certificates for TLS communication between Agent Substrate components.
    • Managed OpenTelemetry: collects metrics and traces from your cluster.
  3. Creates the Cloud Storage bucket that Agent Substrate writes your snapshots to.
  4. Grants Agent Substrate access to Cloud Storage for storing snapshots. If you choose to build from source, also grants access to Artifact Registry for container images.
  5. Creates Cloud Monitoring dashboards that chart Agent Substrate's routing latency, snapshot size, and gRPC traffic.

Next, the installer makes the following changes to your cluster:

  1. Adds the Agent Substrate API group and resources to your cluster. These are Kubernetes custom resources, such as ActorTemplate (which defines the container images and resources your agents use) and WorkerPool (which defines how many idle sandboxes are kept ready).
  2. Sets up the certificate authorities that issue the certificates identifying Agent Substrate's components and your Actors. Components present these certificates to authenticate to each other using mTLS.
  3. Deploys Agent Substrate's components to your cluster, along with a PostgreSQL database that tracks the state of your Actors and Workers. The installer waits for the database and all control plane Pods to become ready.

Agent Substrate deploys its components as ordinary workloads within the ate-system and podcertificate-controller-system namespaces. The installation doesn't modify the GKE control plane.

The components that get deployed include the gVisor-based runtime that Agent Substrate uses by default to isolate each of your agents from the node it runs on. The installer doesn't configure microVM runtimes. To use microVM runtimes, you must manually deploy them onto a node pool that has nested virtualization enabled. For details, see the microVM runtime instructions in the upstream Agent Substrate repository.

If the installation fails at any point (for example, due to a transient network error or token expiration), rerun the command. The installer preserves existing resources and resumes from where it left off.

To upgrade Agent Substrate on an existing cluster, run the installer and select Upgrade an installed cluster. For details, see Upgrading an installed cluster in the substrate-gke README.

Optional steps

After deploying the core components, the installer offers the following optional steps:

  • Filestore CSI driver: installs the Filestore CSI driver configured for Agent Substrate. Choose this option if your agents need shared file storage across Pods, such as shared workspaces or repositories. If your cluster has the managed GKE Filestore add-on enabled, the installer disables it before deploying the driver.
  • Node-pool autoscaling: enables GKE cluster autoscaling on your node pool. You specify the minimum and maximum number of nodes for the pool. Choose this option if you want GKE to automatically add nodes during demand surges and remove nodes when agents are suspended. Skip this option if you prefer a fixed node count with predictable compute costs.
  • Counter demo: deploys an example workload that demonstrates agent suspension and restoration. Choose this option if you want to test the cluster after installation. For details, see Try the Counter demo.

Verify the installation

To verify that the installation succeeded and the core components are running, run the following commands and ensure that all Pods have a status of Running:

kubectl get pods -n ate-system
kubectl get pods -n podcertificate-controller-system

Node labeling and versions

The installer applies the ate.dev/substrate-version label to every node in your cluster. This label specifies which version of Agent Substrate is installed on your cluster, and Agent Substrate runs Workers only on nodes that have this version label.

To check which version of Agent Substrate is installed on your cluster, run the following command:

kubectl get ds -n ate-system -l app=atelet -L ate.dev/substrate-version

Consider the following node labeling details:

  • Worker pools: when you create a WorkerPool, configure it to target this version tag. For examples, see Pin pools to the installed substrate version in the Substrate API guide.
  • New nodes: the installer labels individual nodes rather than the node pool. Because of this, any new nodes added after installation—even in a node pool created by the installer—start without the version label. You should label your GKE node pool so that new nodes automatically receive the label.

Try the Counter demo

If you chose to deploy the Counter demo when prompted by the installer, you can observe Agent Substrate's suspension and restoration capabilities. The demo runs a small web server that keeps a count in memory.

After deploying the demo's WorkerPool and ActorTemplate, the installer prints the commands you need to create an Actor from the ActorTemplate and to send a request to that Actor. The Counter demo README in the upstream Agent Substrate repository explains the next steps to take.

Uninstall Agent Substrate

When the installation completes and exits, the installer prints customized cleanup commands for your environment.

To remove resources created during installation, choose one of the following options:

  • Remove Agent Substrate from an existing cluster: if you installed on an existing cluster and want to retain the cluster and its other workloads, delete only the Agent Substrate components by running the ate-setup delete ate-system command that the installer prints. The installer pre-populates this command with the project, cluster, and location values configured during installation.

  • Delete all resources (including the cluster): to delete the GKE cluster, snapshot bucket, IAM bindings, and dashboards, run the cleanup-gcp command that the installer prints. The installer pre-populates this command with the project, cluster, location, and bucket values configured during installation. For details, see the "Tearing down" section of the substrate-gke README.

What's next