Performance testing on Google Kubernetes Engine

To test a Google Kubernetes Engine (GKE) workload's read and write performance from multiple GKE clients, use the IOR benchmark tool. The following instructions show you how to automate your client setup and use IOR with mpirun over passwordless SSH between Kubernetes Pods to test aggregate I/O.

Prerequisites

A Managed Lustre instance already provisioned.
A local Docker environment configured and authenticated to push to Google Artifact Registry or Container Registry (see Authentication methods).
Ensure that your network's mtu value is set to 8896.

Create a GKE cluster

To test performance, you need a GKE cluster with the Managed Lustre CSI driver enabled. For high-performance storage workloads, configure your GKE node pools with compute-optimized machine families (e.g., c2 or c3) and TIER_1 networking.

Run the following command to create a Standard GKE cluster optimized for performance testing:

gcloud container clusters create CLUSTER_NAME \
    --zone=ZONE \
    --machine-type=MACHINE_TYPE \
    --addons=LustreCsiDriver \
    --network-performance-configs=total-egress-bandwidth-tier=TIER_1 \
    --network=NETWORK \
    --num-nodes=NUM_NODES

Replace ZONE and NETWORK with your specific deployment values. The cluster must reside in the same VPC network as your Managed Lustre instance.
Choose a MACHINE_TYPE. See Performance considerations for information on choosing machine types to obtain the best throughput.
If your machine type doesn't support TIER_1 networking, delete the --network-performance-configs line from the command.
Specify the NUM_NODES. To saturate your file system, your cluster's aggregate network capacity should exceed your file system's provisioned throughput by ~20%.

For machines with Tier 1 networking enabled, a single node can push between 25 Gbps–200 Gbps (~3,000–25,000 MBps), depending on the VM family and the CPU count. For standard instances, egress is typically capped around 2 Gbps per vCPU.

For example, if your Managed Lustre instance capacity yields 100,000 MBps of theoretical throughput, you need an aggregate client egress of 120,000 MBps (100,000 * 1.2) to saturate it:
- With standard instances: If each node has a published egress of 2,000 MBps, you should provision at least 60 nodes (120,000 / 2,000).
- With Tier 1 networking: If each node has a published egress of 10,000 MBps (~80 Gbps), you should provision at least 12 nodes (120,000 / 10,000).

Create the IOR Docker Image

Build a container image with OpenMPI and IOR installed. Compile IOR with Asynchronous I/O (AIO) support for better performance.

Create a file named Dockerfile locally:

FROM ubuntu:22.04

# Prevent interactive prompts during installation
ENV DEBIAN_FRONTEND=noninteractive

# Install dependencies, SSH, and required Autotools packages
RUN apt-get update && apt-get install -y \
  openssh-server \
  openmpi-bin \
  libopenmpi-dev \
  wget \
  git \
  make \
  gcc \
  g++ \
  automake \
  autoconf \
  libtool \
  pkg-config \
  libaio-dev \
  sudo \
  && rm -rf /var/lib/apt/lists/*

# Build IOR from source (version 4.0.0) with Asynchronous I/O (AIO) support
RUN git clone -b 4.0.0 https://github.com/hpc/ior /tmp/ior \
  && cd /tmp/ior \
  && ./bootstrap \
  && ./configure --disable-dependency-tracking --with-aio \
  && make -j"$(nproc)" \
  && make install \
  && rm -rf /tmp/ior

# Configure SSH for OpenMPI passwordless communication
RUN mkdir /var/run/sshd
RUN echo 'root:root' | chpasswd
RUN sed -i 's/^#PermitRootLogin.*/PermitRootLogin yes/' /etc/ssh/sshd_config
RUN sed -i 's/^#PasswordAuthentication.*/PasswordAuthentication yes/' /etc/ssh/sshd_config

# SSH login fix so user isn't kicked out after container initialization
RUN sed 's@session\s*required\s*pam_loginuid.so@session optional pam_loginuid.so@g' -i /etc/pam.d/sshd

EXPOSE 22
CMD ["/usr/sbin/sshd", "-D"]

Build and push this image to your preferred container registry. The instructions in this document use Artifact Registry.

export IMAGE_TAG="gcr.io/PROJECT_ID/lustre-ior-benchmark:latest"
docker build -t $IMAGE_TAG .
docker push $IMAGE_TAG

Generate Passwordless SSH Keys for MPI

OpenMPI requires inter-node communication using passwordless SSH. Create an SSH key and store it in a Kubernetes Secret.

Generate the RSA keys:

ssh-keygen -t rsa -b 4096 -C "mpi-user" -N '' -f "./id_rsa"

Create the Kubernetes Secret:

kubectl create secret generic mpi-ssh-secret \
  --from-file=id_rsa=./id_rsa \
  --from-file=id_rsa.pub=./id_rsa.pub \
  --from-file=authorized_keys=./id_rsa.pub

Create Persistent Volume and Claim

Connect your GKE pods to the Managed Lustre instance using static provisioning.

Create a file named lustre-pv.yaml. Replace the following:

CAPACITY with your instance's storage capacity in GiB.
EXTENDED_LUSTRE_ID with your Managed Lustre identifier, in the format PROJECT_ID/ZONE/INSTANCE_NAME. For example, project-123/us-west1-a/my-lustre-instance.
LUSTRE_IP with the mount IP address of your instance.
FS_NAME with the instance's file system name.

These values can be retrieved with the gcloud lustre instances describe command.

apiVersion: v1
kind: PersistentVolume
metadata:
  name: my-lustre-pv
spec:
  storageClassName: ""
  claimRef:
    name: my-lustre-pvc
    namespace: default
  accessModes:
    - ReadWriteMany
  capacity:
    storage: CAPACITYGi   # retain `Gi` suffix
  persistentVolumeReclaimPolicy: Retain
  volumeMode: Filesystem
  csi:
    driver: lustre.csi.storage.gke.io
    volumeHandle: EXTENDED_LUSTRE_ID   # project-name/zone/instance-name
    volumeAttributes:
      ip: LUSTRE_IP
      filesystem: FS_NAME
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: my-lustre-pvc
spec:
  storageClassName: ""
  volumeName: my-lustre-pv
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: CAPACITYGi

Apply the manifest:
```
kubectl apply -f lustre-pv.yaml
```

Deploy the MPI Workers

To scale IOR tasks across multiple nodes, deploy a StatefulSet with your benchmark image.

Create a file named mpi-workers.yaml. Specify your PROJECT_ID, and set NUM_NODES to the number of nodes in your cluster.

apiVersion: v1
kind: Service
metadata:
  name: mpi-workers
  labels:
    app: mpi-worker
spec:
  clusterIP: None
  selector:
    app: mpi-worker
  ports:
  - port: 22
    name: ssh
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: mpi-worker
spec:
  serviceName: "mpi-workers"
  replicas: NUM_NODES
  selector:
    matchLabels:
      app: mpi-worker
  template:
    metadata:
      labels:
        app: mpi-worker
    spec:
      tolerations:
        - operator: "Exists"
      containers:
        - name: mpi-worker
          image: gcr.io/PROJECT_ID/lustre-ior-benchmark:latest
          command: ["/bin/sh", "-c"]
          args:
            - >-
              mkdir -p /var/run/sshd &&
              ssh-keygen -A &&
              mkdir -p /root/.ssh &&
              echo "Host *" > /root/.ssh/config &&
              echo "    StrictHostKeyChecking no" >> /root/.ssh/config &&
              echo "    UserKnownHostsFile=/dev/null" >> /root/.ssh/config &&
              cp /mnt/mpi-ssh-keys/id_rsa /root/.ssh/id_rsa &&
              cp /mnt/mpi-ssh-keys/id_rsa.pub /root/.ssh/id_rsa.pub &&
              cp /mnt/mpi-ssh-keys/authorized_keys /root/.ssh/authorized_keys &&
              chmod 700 /root/.ssh &&
              chmod 600 /root/.ssh/* &&
              exec /usr/sbin/sshd -D
          ports:
            - containerPort: 22
          volumeMounts:
            - name: lustre-mount
              mountPath: /lustre
            - name: ssh-key-secret
              mountPath: /mnt/mpi-ssh-keys
              readOnly: true
      volumes:
        - name: lustre-mount
          persistentVolumeClaim:
            claimName: my-lustre-pvc
        - name: ssh-key-secret
          secret:
            secretName: mpi-ssh-secret

Apply the manifest:
```
kubectl apply -f mpi-workers.yaml
```

Run the IOR Benchmark

Initiate the benchmark from the first pod (mpi-worker-0), treating it as the head node.

Generate a host file containing the workers' internal IP addresses and copy it to the head node:

kubectl get pods -l app=mpi-worker -o jsonpath='{range .items[*]}{.status.podIP}{"\n"}{end}' > hosts.txt
kubectl cp hosts.txt mpi-worker-0:/root/hostfile

Open a bash session inside your head pod:

kubectl exec -it mpi-worker-0 -- /bin/bash

Inside the pod, create a test directory:
```
mkdir -p /lustre/test
```
Define test variables:
```
export NUM_NODES="NUM_NODES"
export PROCESSES_PER_NODE="PROCESSES_PER_NODE"
export NUM_PROCESSES=$(( NUM_NODES * PROCESSES_PER_NODE ))
```
Where:
- NUM_NODES: The total number of worker Pods participating in the test.
- PROCESSES_PER_NODE: The number of MPI ranks to run on each container. We recommend that you start by setting this to match the number of physical cores (or half the number of vCPUs) on your client machines. For high-performance machine types, setting this between 8 and 16 typically yields the best network throughput.
Run the benchmark commands:

Write throughput

This command writes continuously for 60 seconds to test peak steady-state throughput. The per-task file size is set to an arbitrarily large 50 TiB ceiling to keep tasks writing until the 60-second timer expires.
```
mpirun \
  --allow-run-as-root \
  --mca plm_rsh_no_tree_spawn 1 \
  --mca opal_set_max_sys_limits 1 \
  --mca plm_rsh_num_concurrent ${NUM_NODES} \
  --mca plm_rsh_args "-o StrictHostKeyChecking=no" \
  --npernode ${PROCESSES_PER_NODE} \
  --np ${NUM_PROCESSES} \
  --hostfile ~/hostfile \
  /usr/local/bin/ior \
  -a AIO \
  --posix.odirect \
  -F -g -v -w -k \
  -t 4m -b 50t \
  -D 60 \
  -O stoneWallingWearOut=1 \
  -O stoneWallingStatusFile=/lustre/test/ior-easy.stonewall \
  -o /lustre/test/ior_file
```
Read throughput

This phase reads back exactly the amount of data that was successfully written during the 60-second write throughput test. Although the per-task file size (-b) is set to 50 TiB to match the write phase's geometry, the -O stoneWallingWearOut=1 flag instructs IOR to stop reading as soon as it hits the exact data boundary recorded in the stonewall status file.
```
mpirun \
  --allow-run-as-root \
  --mca plm_rsh_no_tree_spawn 1 \
  --mca opal_set_max_sys_limits 1 \
  --mca plm_rsh_num_concurrent ${NUM_NODES} \
  --mca plm_rsh_args "-o StrictHostKeyChecking=no" \
  --npernode ${PROCESSES_PER_NODE} \
  --np ${NUM_PROCESSES} \
  --hostfile ~/hostfile \
  /usr/local/bin/ior \
  -a AIO \
  --posix.odirect \
  -F -g -v -r -k \
  -t 4m -b 50t \
  -D 60 \
  -O stoneWallingWearOut=1 \
  -O stoneWallingStatusFile=/lustre/test/ior-easy.stonewall \
  -o /lustre/test/ior_file
```
Write IOPS

This test uses small 4 KiB transfer sizes and 8 GiB per-task file sizes to measure the maximum Input/Output Operations Per Second (IOPS) the file system can handle.
```
mpirun \
  --allow-run-as-root \
  --mca plm_rsh_no_tree_spawn 1 \
  --mca opal_set_max_sys_limits 1 \
  --mca plm_rsh_num_concurrent ${NUM_NODES} \
  --mca plm_rsh_args "-o StrictHostKeyChecking=no" \
  --np ${NUM_PROCESSES} \
  --oversubscribe \
  --map-by node \
  --bind-to socket \
  --hostfile ~/hostfile \
  /usr/local/bin/ior \
  -e \
  -t 4k \
  -b 8g \
  -s 1 \
  -a AIO \
  --posix.odirect \
  --aio.max-pending=256 \
  -w \
  -F \
  -z \
  -Q 1 \
  -G 1745405099 \
  -D 45 \
  -O stoneWallingWearOut=1 \
  -o /lustre/test/ior-random
```
Read IOPS

To prevent reading from a sparse file, this test uses two commands: a write to create a solid file using 4 MiB transfer sizes and 8 GiB per-task file sizes, followed by the actual 4 KiB random read IOPS test.
1. Create the file to read:
```
mpirun \
  --allow-run-as-root \
  --mca plm_rsh_no_tree_spawn 1 \
  --mca opal_set_max_sys_limits 1 \
  --mca plm_rsh_num_concurrent ${NUM_NODES} \
  --mca plm_rsh_args "-o StrictHostKeyChecking=no" \
  --npernode ${PROCESSES_PER_NODE} \
  --np ${NUM_PROCESSES} \
  --oversubscribe \
  --map-by node \
  --bind-to socket \
  --hostfile ~/hostfile \
  /usr/local/bin/ior \
  -a AIO \
  --posix.odirect \
  -w \
  -F \
  -k \
  -t 4m \
  -b 8g \
  -s 1 \
  -Q 1 \
  -G 1745405099 \
  -o /lustre/test/ior_rand_read
```
2. Run the IOR read test:
```
mpirun \
  --allow-run-as-root \
  --mca plm_rsh_no_tree_spawn 1 \
  --mca opal_set_max_sys_limits 1 \
  --mca plm_rsh_num_concurrent ${NUM_NODES} \
  --mca plm_rsh_args "-o StrictHostKeyChecking=no" \
  --oversubscribe \
  --map-by node \
  --bind-to socket \
  --npernode ${PROCESSES_PER_NODE} \
  --np ${NUM_PROCESSES} \
  --hostfile ~/hostfile \
  /usr/local/bin/ior \
  -a AIO \
  --posix.odirect \
  --aio.max-pending 256 \
  -r \
  -F \
  -z \
  -t 4k \
  -b 8g \
  -s 1 \
  -Q 1 \
  -G 1745405099 \
  -D 45 \
  -O stoneWallingWearOut=1 \
  -o /lustre/test/ior_rand_read
```
The mpirun flags are:
- --mca plm_rsh_no_tree_spawn 1: Disables tree-based daemon spawning to improve launch reliability across nodes.
- --mca opal_set_max_sys_limits 1: Automatically attempts to set system limits (like max open files) to their highest allowed values.
- --mca plm_rsh_num_concurrent: Sets the maximum number of concurrent SSH connections mpirun will use when launching worker daemons.
- --mca plm_rsh_args ...: Bypasses strict host key checking to prevent interactive SSH prompts from hanging the MPI process launch.
- --prefix ...: Explicitly defines the OpenMPI installation path for Rocky Linux and RHEL so worker nodes can find the required daemon (orted).
- --allow-run-as-root: Allows mpirun to execute as the root user.
- --oversubscribe: Allows MPI to schedule more processes on a node than there are available physical cores.
- --map-by node: Distributes MPI processes in a round-robin fashion evenly across the available nodes.
- --bind-to socket: Binds MPI processes to physical CPU sockets to optimize memory access and cache performance.
- --npernode: The number of processes per node.
- --np: The total number of MPI processes to launch.
- --hostfile: Specifies the file containing the list of hosts to run on.
The ior flags are:
- -a AIO --posix.odirect: Uses the Asynchronous I/O (AIO) engine combined with POSIX Direct I/O. This bypasses the client-side RAM page cache and forces concurrent non-blocking writes directly to the storage servers, ensuring the benchmark measures true network storage performance rather than memory buffers.
- --aio.max-pending=256: Determines the maximum number of concurrent Asynchronous I/O operations in flight per process.
- -C: Reorder tasks for optimal read performance.
- -F: File-per-process mode.
- -g: Use barriers to separate the write and read phases of the test.
- -v: Output verbose logging.
- -w / -r: Instructs IOR to run the write (-w) or read (-r) performance test.
- -k: Prevents IOR from deleting the test file after writing, making it available for the read test.
- -e: Performs an fsync after the write phase to guarantee that data is committed to the storage drives.
- -z: Instructs IOR to perform random access I/O instead of sequential access.
- -s 1: Sets the number of segments to 1.
- -Q 1: Sets the task-per-node offset, aligning tasks so the benchmark coordinates correctly across all nodes.
- -G 1745405099: Hardcodes the random seed timestamp so that the read phase generates the exact same random file offsets that the write phase used.
- -t: Sets the transfer size for each I/O operation (e.g., 4m for throughput, 4k for IOPS).
- -b: Sets the target block size per process (e.g., 50t for throughput, 8g for IOPS) to ensure the test does not run out of payload data during the timed run.
- -D: Restricts the test runtime duration to a specific number of seconds (e.g., 60 or 45). This "stonewalling" terminates the benchmark evenly to capture a true steady-state measurement.
- -O stoneWallingWearOut=1: Forces all concurrent threads to keep generating continuous write load for the entire duration, preventing faster threads from completing early and dropping the overall network pressure.
- -O stoneWallingStatusFile=<path>: Writes a state verification file at the end of the write phase. The subsequent read phase uses this file to read only the blocks that were successfully committed, preventing null pointer errors during reads.
- -o: The path to the test file on the Managed Lustre file system.

View the results

When the benchmark completes, it displays the aggregate performance metrics from all client VMs or pods directly in your terminal. Look for the Results table at the bottom of the output to find your maximum throughput or IOPS.

Key metrics

aggregate filesize: The total amount of data written or read during the test across all participating clients.
bw(MiB/s) / Max Write / Max Read: The most important metric for sequential tests. This shows the aggregate bandwidth achieved by the Managed Lustre file system.
IOPS: The most important metric for random I/O tests. This shows the maximum Input/Output Operations Per Second.

Export results to a file

If you want to programmatically parse your results, feed them into a database, or save them for later analysis, you can instruct IOR to export the summary data in JSON and CSV formats instead of just printing them to the screen.

To do this, append the -O flags to the end of your ior command string:

-O summaryFormat=JSON \
-O summaryFile=/lustre/test/perf-results/summary.json \
-O saveRankPerformanceDetailsCSV=/lustre/test/perf-results/details.csv

The output directory must exist on the file system before running the benchmark.

Expected versus real-world performance

You can calculate the mathematical maximum throughput of your file system based on its provisioned capacity and performance tier. Because storage capacity is provisioned in gibibytes (GiB) and tiers are rated in tebibytes (TiB), you must first convert your capacity:

(Capacity GiB / 1024) * Tier MBps = Theoretical Max MBps

For example, a 216,000 GiB instance in the 500 MBps per TiB tier mathematically provides 105,469 MBps of throughput ((216000 / 1024) * 500).

Maximum observed throughput will always be bounded by either your file system's provisioned throughput capacity or the combined network egress limits of your client machines, whichever is lower.

Common reasons your benchmark numbers may not reach theoretical speeds include:

TCP/IP overhead: Standard network encapsulation and packet headers consume roughly 5-10% of raw bandwidth. Your mathematical maximum includes this overhead, but the IOR benchmark only measures the raw payload written to disk.
Client network limits: Client machines have strict egress bandwidth caps. If you use a small number of clients or nodes, or machine types without Tier 1 networking enabled, the clients will throttle the benchmark before the Managed Lustre file system reaches its limit.
MPI context switching: If PROCESSES_PER_NODE is set higher than the number of physical cores on your client machines, CPU contention and context-switching overhead will artificially degrade the benchmark's I/O performance.
Missing Direct I/O: If the --posix.odirect flag is omitted, data passes through the client's RAM page cache. This introduces memory bottlenecks and CPU overhead that mask the true network storage performance.

Clean up

To avoid incurring charges to your Google Cloud account for the resources used on this page, follow these steps:

Delete the test files from the Managed Lustre volume:

kubectl exec mpi-worker-0 -- rm -rf /lustre/test

Delete the GKE cluster:
```
gcloud container clusters delete CLUSTER_NAME --zone=ZONE
```
Deleting the cluster also deletes the GKE pods, Kubernetes Secret, and Persistent Volume Claim.

If you pushed the benchmark Docker image and no longer need it, delete the image from your repository:

gcloud container images delete gcr.io/PROJECT_ID/lustre-ior-benchmark:latest --force-delete-tags

If you created the Managed Lustre instance specifically for this test and no longer need it, delete it:
```
gcloud lustre instances delete INSTANCE_ID --location=LOCATION
```

Troubleshooting common benchmarking bottlenecks

If your benchmark results are significantly lower than your expected storage performance tier, see Troubleshooting common bottlenecks.