Work with Onload

Private Preview — ULL Solution

This product or feature is subject to the "Pre-GA Offerings Terms" in the General Service Terms section of the Service Specific Terms. Pre-GA features are available "as is" and might have limited support. For more information, see the launch stage descriptions.

ULL Solution is available only to approved exchange operators and their exchange participants. If you are an approved exchange participant, contact [e-mail] to request access.

This page describes how to use Onload for Ultra Low Latency (ULL) Compute Engine instances.

Onload is a high-performance network stack for latency-sensitive applications that require ultra-low latency, minimal jitter, and consistent performance. Onload provides a TCP/IP implementation that bypasses the operating system kernel and runs directly in the user space while enabling applications to use standard BSD socket APIs.

Using Onload with ULL Solution includes support for the following:

• Flow steering: With ULL network interfaces, you can bypass default Receive Side Scaling (RSS) hashing by steering specific traffic flows directly to a designated Receive Queue (RX). For ULL unicast and multicast traffic, 3-tuple flow steering is supported (protocol, destination IP address, destination port).

Required instance type and OS image

To use Onload, you must create ULL Compute Engine instances (U4P or U4C) by using the image project and image provided by Google for testing as described in Create ULL Compute Engine instances.

When you use the required image, Onload comes pre-installed and loaded at boot. You can then run processes with Onload as described in the procedures on this page.

Start and stop onload

This section describes how to start and stop Onload. You must start Onload to complete the procedures in the following sections.

To start Onload, run the following command:

sudo /opt/gcp_tp-scripts/load-onload-and-tune

To stop Onload, run the following command:

sudo /opt/gcp_tp-scripts/unload-onload

Enable busy polling

To help achieve ultra-low latency and minimize jitter, enable busy polling.

To enable busy polling, do the following:

1. If you haven't already, connect to the instance by using SSH.

2. You can enable busy polling for all queues or more granularly. Choose the option that corresponds to your use case:

   • All queues:

     sudo /opt/gcp_tp-scripts/busypoll-all-queues
     

   • Specific RX queue that an Onload stack is using:

     1. If you haven't already, start an Onload stack by running the following command:

        gcp_tp-onload /opt/neper/tcp_rr &
        

     2. To get the stack-id of the stack that you started, run the following command:

        onload_stackdump
        

     3. To get the index numbers of your network interfaces, run the following command:

        cat /sys/class/net/NIC_NAME/ifindex
        

        Replace NIC_NAME with the name of the network interface in the guest OS, such as eth1, eth2.

     4. To enable busy polling run the following command:

        sudo /opt/gcp_tp-scripts/busypoll-onload-stack STACK_ID enable NIC_INDEX CPU_ID
        

        Replace the following:

        • STACK_ID: the stack ID that you got previously
        • NIC_INDEX: the index number of the network interface for which you want to enable busy polling, for example 1 or 2.

        • CPU_ID: the ID of the CPU for which to enable busy polling.

          • We recommend that you don't use CPU 0 or any CPU that is in use by an application.

          • Ensure that you use CPU cores local to the NUMA node of the network interface that you specified. For example:

            • eth1 uses NUMA node 0, which uses CPU cores 0-29.
            • eth2 uses NUMA node 2. which uses CPU cores 60-89.

You can disable busy polling by using the following command:

sudo /opt/gcp_tp-scripts/busypoll-onload-stack STACK_ID disable NIC_INDEX CPU_ID

What's next

• To synchronize your instance system clock to the physical NIC clock of its host server, see Configure accurate time.