This page explains the characteristics and best types of workloads for each GPU
sharing strategy available in Google Kubernetes Engine (GKE), such as
multi-instance GPUs, GPU time-sharing, and NVIDIA MPS. GPU sharing helps you to
minimize underutilized capacity in your cluster and to provide workloads with
just enough capacity to complete tasks.

This page is for Platform admins and operators and for
Data and AI specialists who want to run GPU-based workloads that consume GPU
capacity as efficiently as possible. To learn more about common roles that we
reference in Google Cloud content, see
[Common GKE user roles and tasks](https://docs.cloud.google.com/kubernetes-engine/enterprise/docs/concepts/roles-tasks.md.txt).

Before reading this page, ensure that you're familiar with the following
concepts:

- Kubernetes concepts, such as Pods, nodes, deployments, and namespaces.
- GKE concepts, such as [node pools](https://docs.cloud.google.com/kubernetes-engine/docs/concepts/node-pools.md.txt), [autoscaling](https://docs.cloud.google.com/kubernetes-engine/docs/concepts/cluster-autoscaler.md.txt), and [node auto-provisioning](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/node-auto-provisioning.md.txt).

## How GPU requests work in Kubernetes

Kubernetes enables workloads to request precisely the resource amounts they need
to function. Although you can request fractional *CPU* units for workloads, you
can't request fractional *GPU* units. Pod manifests must request GPU resources in
integers, which means that an entire physical GPU is allocated to one container
even if the container only needs a fraction of the resources to function
correctly. This is inefficient and can be costly, especially when you're running
multiple workloads with similar low GPU requirements.
**Best practice** :

Use GPU sharing strategies to improve GPU utilization when your workloads don't need all of the GPU resources.

## What are GPU sharing strategies?

GPU sharing strategies allow multiple containers to efficiently use your
attached GPUs and save running costs. GKE provides the following
GPU sharing strategies:

- **Multi-instance GPU**: GKE divides a single supported GPU in up to seven slices. Each slice can be allocated to one container on the node independently, for a maximum of seven containers per GPU. Multi-instance GPU provides hardware isolation between the workloads, plus consistent and predictable Quality of Service (QoS) for all containers running on the GPU.
- **GPU time-sharing** : GKE uses the built-in timesharing ability provided by the NVIDIA GPU and the software stack. Starting with the [Pascal architecture](https://www.nvidia.com/en-us/data-center/pascal-gpu-architecture/), NVIDIA GPUs support instruction level preemption. When doing context switching between processes running on a GPU, instruction-level preemption ensures every process gets a fair timeslice. GPU time-sharing provides software-level isolation between the workloads in terms of address space isolation, performance isolation, and error isolation.
- **NVIDIA MPS** : GKE uses [NVIDIA's Multi-Process Service (MPS)](https://docs.nvidia.com/deploy/pdf/CUDA_Multi_Process_Service_Overview.pdf). NVIDIA MPS is an alternative, binary-compatible implementation of the CUDA API designed to transparently enable co-operative multi-process CUDA workloads to run concurrently on a single GPU device. GPU with NVIDIA MPS provides software-level isolation in terms of resource limits ([active thread percentage](https://docs.nvidia.com/deploy/mps/index.html#topic_5_2_5) and [pinned device memory](https://docs.nvidia.com/deploy/mps/index.html#topic_5_2_7)).

## Which GPU sharing strategy to use

The following table summarizes and compares the characteristics of the available
GPU sharing strategies:

|---|---|---|---|
|   | Multi-instance GPU | GPU time-sharing | NVIDIA MPS |

**Best practice** :

To maximize your GPU utilization, combine GPU sharing
strategies. For each, multi-instance GPU partition,
use either time-sharing or NVIDIA MPS.
You can then run multiple containers on each partition, with those
containers sharing access to the resources on that partition. We recommend that
you use any of the following combinations:

- Multi-instance GPU and GPU time-sharing.
- Multi-instance GPU and NVIDIA MPS.

### How the GPU sharing strategies work

You can specify the maximum number of containers allowed to share a physical GPU:

- On Autopilot clusters, this is configured in your workload specification.
- On Standard clusters, this is configured when you create a new node pool with GPUs attached. Every GPU in the node pool is shared based on the setting you specify at the node pool level.

The following sections explain the scheduling behavior and operation of each GPU
sharing strategy.

#### Multi-instance GPU

You can request multi-instance GPU in workloads by specifying the
`cloud.google.com/gke-gpu-partition-size` label in the Pod spec
`nodeSelector` field, under `spec: nodeSelector`.

GKE schedules workloads to appropriate available nodes by matching these labels. If there are no appropriate available nodes, GKE uses autoscaling
and node auto-provisioning to create new nodes or node pools that match this
label.

#### GPU time-sharing or NVIDIA MPS

You can request GPU time-sharing or NVIDIA MPS in workloads by specifying the
following labels in the Pod spec `nodeSelector` field, under `spec:nodeSelector`.

- `cloud.google.com/gke-max-shared-clients-per-gpu`: Select nodes that allow a specific number of clients to share the underlying GPU.
- `cloud.google.com/gke-gpu-sharing-strategy`: Select nodes that use the time-sharing or NVIDIA MPS strategy for GPUs.

The following table describes how scheduling behavior changes based on the combination of node
labels that you specify in your manifests.

| Node labels ||
|---|---|
| `cloud.google.com/gke-max-shared-clients-per-gpu` *and* `cloud.google.com/gke-gpu-sharing-strategy` | GKE schedules workloads in available nodes that match both the labels. If there are no available nodes, GKE uses autoscaling and node auto-provisioning to create new nodes or node pools that match both the labels. |
| *Only* `cloud.google.com/gke-max-shared-clients-per-gpu` | **Autopilot:** GKE rejects the workload. **Standard:** GKE schedules workloads in available nodes that match the label. If there are no available nodes, GKE uses autoscaling and node auto-provisioning to create new nodes or node pools that match the label. By default, auto-provisioned nodes are given the following label and value for each strategy: - **GPU time-sharing:** `cloud.google.com/gke-gpu-sharing-strategy: time-sharing` - **NVIDIA MPS:** `cloud.google.com/gke-gpu-sharing-strategy: mps` |
| *Only* `cloud.google.com/gke-gpu-sharing-strategy` | **Autopilot:** GKE rejects the workload. **Standard:** GKE schedules workloads in available nodes that use specific sharing strategies. - If there are multiple shared node pools with different values for `cloud.google.com/gke-max-shared-clients-per-gpu`, the workload can be scheduled on any available node. - If there are no available nodes in any node pools, the cluster autoscaler scales up the node pool with the lowest value for `cloud.google.com/gke-max-shared-clients-per-gpu`. - If all node pools are at capacity, node auto-provisioning creates a new node pool with a default value of `cloud.google.com/gke-max-shared-clients-per-gpu=2` |

The GPU request process that you complete is the same for GPU time-sharing and
NVIDIA MPS strategy.

If you're developing GPU applications that run on GPU
time-sharing or NVIDIA MPS, you can only request one GPU for each container.
GKE rejects a request for more than one GPU in a
container to avoid unexpected behavior. In addition, the number of GPUs requested with
time-sharing and NVIDIA MPS is not a measure of the compute power
available to the container.

The following table shows you what to expect when you request specific quantities
of GPUs.

| GPU requests that apply to GPU time-sharing and NVIDIA MPS ||
|---|---|
| *One* GPU time-sharing or NVIDIA MPS per container | GKE allows the request, even if the node has one physical GPU or multiple physical GPUs. |
| *More than one* GPU time-sharing per container | GKE rejects the request. This behavior is the same when requesting more than one [multi-instance GPU](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/gpus-multi.md.txt) instance in a container, because each GPU instance is considered to be a discrete physical GPU. |
| *More than one* NVIDIA MPS per container | Based on the number of physical GPUs in the node, GKE does the following: - GKE allows the request when the node only has one physical GPU. - GKE rejects the request when the node has multiple physical GPUs. This behavior is the same when requesting more than one [multi-instance GPU](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/gpus-multi.md.txt) instance in a container, because each GPU instance is considered to be a discrete physical GPU. |

If GKE rejects the workload, you see an error message similar
to the following:

    status:
      message: 'Pod Allocate failed due to rpc error: code = Unknown desc = [invalid request
        for sharing GPU (time-sharing), at most 1 nvidia.com/gpu can be requested on GPU nodes], which is unexpected'
      phase: Failed
      reason: UnexpectedAdmissionError

## Monitor GPU time-sharing or NVIDIA MPS nodes

Use [Cloud Monitoring](https://docs.cloud.google.com/monitoring/docs.md.txt) to monitor the performance of your
GPU time-sharing or NVIDIA MPS nodes. GKE sends metrics for each
GPU node to Cloud Monitoring. These GPU time-sharing or NVIDIA MPS node metrics apply at the node level
(`node/accelerator/`).

You can check the following metrics for each GPU time-sharing or NVIDIA MPS node
in Cloud Monitoring:

- **Duty cycle (`node/accelerator/duty_cycle`)**: Percentage of time over the last sample period (10 seconds) during which the GPU node was actively processing. Ranges from 1% to 100%.
- **Memory usage (`node/accelerator/memory_used`)**: Amount of accelerator memory allocated in bytes for each GPU node.
- **Memory capacity (`node/accelerator/memory_total`)**: Total accelerator memory in bytes for each GPU node.

These metrics are *different* from the
[metrics for regular GPUs](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/gpus.md.txt#monitoring) that are not
time-shared or NVIDA MPS nodes.
The metrics for
[regular physical GPUs](https://docs.cloud.google.com/kubernetes-engine/docs/concepts/gpus.md.txt#monitoring)
apply at the container
level (`container/accelerator`) and
are *not* collected for containers scheduled on a GPU that uses GPU time-sharing
or NVIDIA MPS.

## What's next

- Learn how to [share GPUs with multiple workloads using GPU time-sharing](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/timesharing-gpus.md.txt).
- Learn how to [share GPUs with multiple workloads using NVIDIA MPS](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/nvidia-mps-gpus.md.txt).
- Learn how to [run multi-instance GPUs](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/gpus-multi.md.txt).
- Learn more about [GPUs](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/gpus.md.txt).
- For more information about compute preemption for the NVIDIA GPU, refer to the [NVIDIA Pascal Tuning Guide](https://docs.nvidia.com/cuda/pascal-tuning-guide/index.html#preemption).