Configure GPU for Cloud Run services

You can use GPU with your Cloud Run services for AI and other compute-intensive workloads. GPU works well for AI inference workloads like large language models (LLMs) or other compute-intensive, non-AI use cases such as video transcoding and 3D rendering. Google provides NVIDIA RTX PRO 6000 Blackwell GPU with 96 GB of GPU memory (VRAM) for a full GPU or 48 GB of GPU memory (VRAM) for a fractional (0.5) GPU, and NVIDIA L4 GPU with 24 GB of GPU memory (VRAM). GPU memory is separate from instance memory.

When using an LLM, be sure to also see Best practices: Cloud Run services with GPU.

GPU on Cloud Run is fully managed, with no extra drivers or libraries required. Cloud Run GPU offers on-demand availability with no reservations needed, similar to the way on-demand CPU and on-demand memory work in Cloud Run. To save on cost, Cloud Run services that use GPU can scale down to zero when not in use.

Cloud Run container instances with an NVIDIA RTX PRO 6000 Blackwell Blackwell or L4 GPU start in about 5 seconds. Your container can use the GPU as soon as the startup finishes.

You can configure up to one GPU per Cloud Run container instance. If you use sidecar containers, the GPU can only be attached to one container.

Supported GPU types

Cloud Run supports two GPU types:

  • NVIDIA RTX PRO 6000 Blackwell GPU:

    • Full GPU with the current NVIDIA driver version is 580.x.x (13.0). This GPU requires a minimum of 20 CPU and 80 GiB of memory.
    • Fractional GPU with the current NVIDIA driver version is 580.x.x (13.0), type: GRID. Fractional GPU (0.5 GPU, 48 GB VRAM) requires 10, 12, 14, or 16 CPU and at least 40 GiB of memory (up to 4 GiB per CPU).
  • L4 GPU with the current NVIDIA driver version: 580.x.x (13.0). For L4 GPU, you must use a minimum of 4 CPU and 16 GiB of memory. Fractional GPU is not supported on L4 GPU.

Fractional GPU

Fractional GPU lets you to reduce your GPU allocation for models that don't require a full GPU, while still benefiting from the GPU's advanced architecture and performance. With a fractional GPU, you can have a more cost effective solution for inference workloads that don't need the full capabilities of a full-sized NVIDIA RTX PRO 6000 Blackwell GPU. See Known issues for fractional GPU limitations.

Supported regions

The following regions are supported by the NVIDIA RTX PRO 6000 Blackwell GPU:

  • asia-southeast1 (Singapore).
  • asia-south2 (Delhi, India).
  • europe-west4 (Netherlands) leaf icon Low CO2
  • us-central1 (Iowa) leaf icon Low CO2

Fractional NVIDIA RTX PRO 6000 Blackwell GPU (Preview) are available in the following regions:

The following regions are supported by the L4 GPU:

  • asia-southeast1 (Singapore)
  • asia-south1 (Mumbai) . This region is available by invitation only. Contact your Google Account team if you are interested in this region.
  • europe-west1 (Belgium) leaf icon Low CO2
  • europe-west4 (Netherlands) leaf icon Low CO2
  • us-central1 (Iowa) leaf icon Low CO2 . Additional resource scaling in this region might require a quota increase request. Contact your Google Account team if you are interested in this region.
  • us-east4 (Northern Virginia) . Additional resource scaling in this region might require a quota increase request. Contact your Google Account team if you are interested in this region.

Pricing impact

See Cloud Run pricing for GPU pricing details. Note the following requirements and considerations:

  • There are no per-request fees. You must use instance-based billing to use the GPU feature, minimum instances are charged at the full rate even when idle.
  • There is a difference in cost between GPU zonal redundancy and non-zonal redundancy. See Cloud Run pricing for GPU pricing details.
  • When you deploy a GPU-enabled Cloud Run service or function from source code, Cloud Run builds your source code using the e2-highcpu-8 machine type instead of the default e2-standard-2 machine type. The larger machine type provides higher CPU support, and higher network bandwidth, which results in faster build times.
  • The CPU and memory configurations of your resource impact pricing.
  • GPU is billed for the entire duration of the instance lifecycle.

GPU zonal redundancy options

By default, Cloud Run deploys your service across multiple zones within a region. This architecture provides inherent resilience: if a zone experiences an outage, Cloud Run automatically routes traffic away from the affected zone to healthy zones within the same region.

When working with GPU resources, keep in mind GPU resources have specific capacity constraints. During a zonal outage, the standard failover mechanism for GPU workloads relies on sufficient unused GPU capacity being available in the remaining healthy zones. Due to the constrained nature of GPUs, this capacity might not always be available.

To increase the availability of your GPU-accelerated services during zonal outages, you can configure zonal redundancy specifically for GPU:

  • Zonal Redundancy Turned On (default): Cloud Run reserves GPU capacity for your service across multiple zones. This significantly increases the probability that your service can successfully handle traffic rerouted from an affected zone, offering higher reliability during zonal failures with additional cost per GPU second.

  • Zonal Redundancy Turned Off: Cloud Run attempts failover for GPU workloads on a best-effort basis. Traffic is routed to other zones only if sufficient GPU capacity is available at that moment. This option does not guarantee reserved capacity for failover scenarios but results in a lower cost per GPU second.

SLA

The SLA for Cloud Run GPU depends on whether the service uses the zonal redundancy or non-zonal redundancy option. Refer to the SLA page for details.

Request a quota increase

Quota for Cloud Run nvidia-rtx-pro-6000 GPU is granted in milliGPUs. A full (1) nvidia-rtx-pro-6000 GPU consumes 1,000 milliGPUs of quota, and a fractional (0.5) nvidia-rtx-pro-6000 GPU consumes 500 milliGPUs of quota. Projects using nvidia-rtx-pro-6000 GPU in a region for the first time will be automatically granted 3,000 milliGPU quota (zonal redundancy off) when the first deployment is created. This is equivalent to 3 full GPUs or 6 fractional (0.5) GPUs. Projects using Cloud Run nvidia-l4 GPU in a region for the first time are automatically granted 3 GPU quota (zonal redundancy off) when the first deployment is created.

If you need additional Cloud Run GPUs, you must request a quota increase for your Cloud Run service. Use the links provided in the following buttons to request the quota you need.

Quota needed Quota link
NVIDIA RTX PRO 6000 Blackwell GPU with zonal redundancy turned off (lower price) Request GPU quota without zonal redundancy
NVIDIA RTX PRO 6000 Blackwell GPU with zonal redundancy turned on (higher price) Request GPU quota with zonal redundancy
L4 GPU with zonal redundancy turned off (lower price) Request GPU quota without zonal redundancy
L4 GPU with zonal redundancy turned on (higher price) Request GPU quota with zonal redundancy

For more information on requesting quota increases, see How to increase quota.

Before you begin

The following list describes requirements and limitations when using GPU in Cloud Run:

  1. Sign in to your Google Cloud account. If you're new to Google Cloud, create an account to evaluate how our products perform in real-world scenarios. New customers also get $300 in free credits to run, test, and deploy workloads.
  2. In the Google Cloud console, on the project selector page, select or create a Google Cloud project.

    Roles required to select or create a project

    • Select a project: Selecting a project doesn't require a specific IAM role—you can select any project that you've been granted a role on.
    • Create a project: To create a project, you need the Project Creator role (roles/resourcemanager.projectCreator), which contains the resourcemanager.projects.create permission. Learn how to grant roles.

    Go to project selector

  3. Verify that billing is enabled for your Google Cloud project.

  4. In the Google Cloud console, on the project selector page, select or create a Google Cloud project.

    Roles required to select or create a project

    • Select a project: Selecting a project doesn't require a specific IAM role—you can select any project that you've been granted a role on.
    • Create a project: To create a project, you need the Project Creator role (roles/resourcemanager.projectCreator), which contains the resourcemanager.projects.create permission. Learn how to grant roles.

    Go to project selector

  5. Verify that billing is enabled for your Google Cloud project.

  6. Enable the Cloud Run API, if it is not already enabled.

    Roles required to enable APIs

    To enable APIs, you need the serviceusage.services.enable permission. If you created the project, then you likely already have this permission through the Owner role (roles/owner). Otherwise, you can get this permission through the Service Usage Admin role (roles/serviceusage.serviceUsageAdmin). Learn how to grant roles.

    Enable the API

  7. Request required quota.
  8. Consult Best practices: AI inference on Cloud Run with GPU for recommendations on building your container image and loading large models.
  9. Make sure your Cloud Run service has the following configurations:

Required roles

To get the permissions that you need to configure and deploy Cloud Run services, ask your administrator to grant you the following IAM roles on services:

  • Cloud Run Developer (roles/run.developer) - the Cloud Run service
  • Service Account User (roles/iam.serviceAccountUser) - the service identity

If you are deploying a service or function from source code, you must also have additional roles granted to you on your project and Cloud Build service account.

For a list of IAM roles and permissions that are associated with Cloud Run, see Cloud Run IAM roles and Cloud Run IAM permissions. If your Cloud Run service interfaces with Google Cloud APIs, such as Cloud Client Libraries, see the service identity configuration guide. For more information about granting roles, see deployment permissions and manage access.

Configure a Cloud Run service with GPU

Any configuration change leads to the creation of a new revision. Subsequent revisions will also automatically get this configuration setting unless you make explicit updates to change it.

You can use the Google Cloud console, Google Cloud CLI or YAML to configure GPU.

Console

  1. In the Google Cloud console, go to the Cloud Run page:

    Go to Cloud Run

  2. Select Services from the Cloud Run navigation menu:

    1. If you are configuring a new service, click Deploy container and fill out the initial service settings page. Then click Containers, Networking, Security to expand the service configuration page.

    2. If you are configuring an existing service, click the service.

  3. Click the Containers tab.

  4. Configure CPU, memory, concurrency, execution environment, and startup probe following the recommendations in Before you begin.

  5. Select GPU.

  6. Select the GPU type and Number of GPUs from the respective menus.

  7. By default, zonal redundancy is turned on for new services. Selecting GPU shows the GPU redundancy options.

    1. To turn off zonal redundancy, select No zonal redundancy.
    2. To turn on zonal redundancy, select Zonal redundancy.
  8. Click Create for a new service. Click View diff & redeploy, then Deploy changes for an existing service.

gcloud

To create a service with GPU enabled, use the gcloud run deploy command:

  • To deploy a container:

      gcloud run deploy SERVICE \
          --image IMAGE_URL \
          --gpu GPU_NUMBER \
          --gpu-type GPU_TYPE

    Replace the following:

    • SERVICE: the name of your Cloud Run service.
    • IMAGE_URL: a reference to the container image, for example, us-docker.pkg.dev/cloudrun/container/hello:latest. If you use Artifact Registry, the repository REPO_NAME must already be created. The URL follows the format of LOCATION-docker.pkg.dev/PROJECT_ID/REPO_NAME/PATH:TAG .
    • GPU_NUMBER: the number of GPU to attach (1 for a full GPU, or 0.5 for a fractional GPU).
    • GPU_TYPE: the GPU type (nvidia-rtx-pro-6000 or nvidia-l4).
  • To deploy from source code:

      gcloud run deploy SERVICE \
          --source . \
          --gpu GPU_NUMBER \
          --gpu-type GPU_TYPE

To update the GPU configuration for a service, use the gcloud run services update command. For example, to update an existing service that specifies a container image:

  gcloud run services update SERVICE \
      --image IMAGE_URL \
      --cpu CPU \
      --memory MEMORY \
      --no-cpu-throttling \
      --gpu GPU_NUMBER \
      --gpu-type GPU_TYPE \
      --max-instances MAX_INSTANCE \
      --GPU_ZONAL_REDUNDANCY
    

Replace the following:

  • SERVICE: the name of your Cloud Run service.
  • IMAGE_URL: a reference to the container image, for example, us-docker.pkg.dev/cloudrun/container/hello:latest. If you use Artifact Registry, the repository REPO_NAME must already be created. The URL follows the format of LOCATION-docker.pkg.dev/PROJECT_ID/REPO_NAME/PATH:TAG .
  • CPU: the number of CPUs. Ensure this meets the minimum requirements.
  • MEMORY: the amount of memory. Ensure this meets the minimum requirements.
  • GPU_NUMBER: the number of GPU to attach (1 for a full GPU, or 0.5 for a fractional GPU).
  • GPU_TYPE: the GPU type. For NVIDIA RTX PRO 6000 Blackwell GPU, enter nvidia-rtx-pro-6000. For L4 GPU, enter value nvidia-l4 (nvidia-L4 lowercase L, not numeric value fourteen).
  • MAX_INSTANCE: the maximum number of instances. This number can't exceed the GPU quota allocated for your project.
  • GPU_ZONAL_REDUNDANCY: no-gpu-zonal-redundancy to turn off zonal redundancy, or gpu-zonal-redundancy to turn on zonal redundancy.

YAML

  1. If you are creating a new service, skip this step. If you are updating an existing service, download its YAML configuration:

    gcloud run services describe SERVICE --format export > service.yaml
  2. Update the nvidia.com/gpu: attribute and nodeSelector:
    run.googleapis.com/accelerator:
    :

    apiVersion: serving.knative.dev/v1
    kind: Service
    metadata:
      name: SERVICE
    spec:
      template:
        metadata:
          annotations:
            autoscaling.knative.dev/maxScale: 'MAX_INSTANCE'
            run.googleapis.com/cpu-throttling: 'false'
            run.googleapis.com/gpu-zonal-redundancy-disabled: 'GPU_ZONAL_REDUNDANCY'
        spec:
          containers:
          - image: IMAGE_URL
            ports:
            - containerPort: CONTAINER_PORT
              name: http1
            resources:
              limits:
                cpu: 'CPU'
                memory: 'MEMORY'
                nvidia.com/gpu: 'GPU_NUMBER'
            # Optional: use a longer startup probe to allow long starting containers
            startupProbe:
              failureThreshold: 1800
              periodSeconds: 1
              tcpSocket:
                port: CONTAINER_PORT
              timeoutSeconds: 1
          nodeSelector:
            run.googleapis.com/accelerator: GPU_TYPE

    Replace the following:

    • SERVICE: the name of your Cloud Run service.
    • IMAGE_URL: a reference to the container image, for example, us-docker.pkg.dev/cloudrun/container/hello:latest. If you use Artifact Registry, the repository REPO_NAME must already be created. The URL follows the format of LOCATION-docker.pkg.dev/PROJECT_ID/REPO_NAME/PATH:TAG .
    • CONTAINER_PORT: the container port set for your service.
    • CPU: the number of CPUs. Ensure this meets the minimum requirements.
    • MEMORY: the amount of memory. Ensure this meets the minimum requirements.
    • GPU_NUMBER: the number of GPU to attach (1 for a full GPU, or 0.5 for a fractional GPU).
    • GPU_TYPE: the GPU type. For NVIDIA RTX PRO 6000 Blackwell GPU, enter nvidia-rtx-pro-6000. For L4 GPU, enter value nvidia-l4 (nvidia-L4 lowercase L, not numeric value fourteen).
    • MAX_INSTANCE: the maximum number of instances. This number can't exceed the GPU quota allocated for your project.
    • GPU_ZONAL_REDUNDANCY: false to turn on GPU zonal redundancy, or true to turn it off.
  3. Create or update the service using the following command:

    gcloud run services replace service.yaml

    The gcloud run services replace command defaults to using service.yaml file if present.

Terraform

To learn how to apply or remove a Terraform configuration, see Basic Terraform commands.

Add the following to a google_cloud_run_v2_service resource in your Terraform configuration:

resource "google_cloud_run_v2_service" "default" {
  provider = google-beta
  name     = "SERVICE"
  location = "europe-west1"

  template {
    gpu_zonal_redundancy_disabled = "GPU_ZONAL_REDUNDANCY"
    containers {
      image = "IMAGE_URL"
      resources {
        limits = {
          "cpu" = "CPU"
          "memory" = "MEMORY"
          "nvidia.com/gpu" = "GPU_NUMBER"
        }
      }
    }
    node_selector {
      accelerator = "GPU_TYPE"
    }
  }
}

Replace the following:

  • SERVICE: the name of your Cloud Run service.
  • GPU_ZONAL_REDUNDANCY: false to turn on GPU zonal redundancy, or true to turn it off.
  • IMAGE_URL: a reference to the container image, for example, us-docker.pkg.dev/cloudrun/container/hello:latest. If you use Artifact Registry, the repository REPO_NAME must already be created. The URL follows the format of LOCATION-docker.pkg.dev/PROJECT_ID/REPO_NAME/PATH:TAG .
  • CPU: the number of CPUs. Ensure this meets the minimum requirements.
  • MEMORY: the amount of memory. Ensure this meets the minimum requirements.
  • GPU_NUMBER: the number of GPU to attach (1 for a full GPU, or 0.5 for a fractional GPU).
  • GPU_TYPE: the GPU type. For NVIDIA RTX PRO 6000 Blackwell GPU, enter nvidia-rtx-pro-6000. For L4 GPU, enter value nvidia-l4 (nvidia-L4 lowercase L, not numeric value fourteen).

View GPU settings

To view the current GPU settings for your Cloud Run service:

Console

  1. In the Google Cloud console, go to the Cloud Run Services page:

    Go to Cloud Run

  2. Click the service you are interested in to open the Service details page.

  3. Click the Containers tab to view the settings.

gcloud

  1. Use the following command:

    gcloud run services describe SERVICE
  2. Locate the GPU setting in the returned configuration.

Remove GPU

You can remove GPU using the Google Cloud console, the Google Cloud CLI, or YAML.

Console

  1. In the Google Cloud console, go to the Cloud Run page:

    Go to Cloud Run

  2. Select Services from the Cloud Run navigation menu:

    1. If you are configuring a new service, click Deploy container and fill out the initial service settings page. Then click Containers, Networking, Security to expand the service configuration page.

    2. If you are configuring an existing service, click the service.

  3. Click the Containers tab.

  4. Clear GPU.

  5. Click Create for a new service. Click View diff & redeploy, then Deploy changes for an existing service.

gcloud

To remove GPU, set the number of GPU to 0 using the gcloud run services update command:

  gcloud run services update SERVICE --gpu 0
  

Replace SERVICE with the name of your Cloud Run service.

YAML

  1. If you are creating a new service, skip this step. If you are updating an existing service, download its YAML configuration:

    gcloud run services describe SERVICE --format export > service.yaml
  2. Delete the nvidia.com/gpu: and the nodeSelector: run.googleapis.com/accelerator: GPU_TYPE lines.

  3. Create or update the service using the following command:

    gcloud run services replace service.yaml

    The gcloud run services replace command defaults to using service.yaml file if present.

Driver libraries

By default, all of the NVIDIA RTX PRO 6000 Blackwell GPU and NVIDIA L4 GPU driver libraries are mounted under /usr/local/nvidia/lib64. Cloud Run automatically appends this path to the LD_LIBRARY_PATH environment variable (i.e. ${LD_LIBRARY_PATH}:/usr/local/nvidia/lib64) of the container with the GPU. This allows the dynamic linker to find the NVIDIA driver libraries. The linker searches and resolves paths in the order you list in the LD_LIBRARY_PATH environment variable. Any values you specify in this variable take precedence over the default Cloud Run driver libraries path /usr/local/nvidia/lib64.

If you want to use a CUDA version greater than 13.0, the easiest way is to depend on a newer NVIDIA base image with forward compatibility packages already installed. Another option is to manually install the NVIDIA forward compatibility packages and add them to LD_LIBRARY_PATH. Consult NVIDIA's compatibility matrix to determine which CUDA versions are forward compatible with the provided NVIDIA driver version.

About GPU and maximum instances

The number of instances with GPU is limited in two ways:

  • The Maximum instances setting limits the number of instances per service. This can't be set higher than the total GPU quota per project per region divided by the GPU allocation per instance. For example, if your project has 3,000 milliGPUs of nvidia-rtx-pro-6000 quota and your service uses --gpu 0.5 (500 milliGPUs per instance), you can set maximum instances up to 6.
  • The quota of GPUs allowed per project per region. This limits the number of instances across services in the same region.

What's next

See Run AI inference on Cloud Run with GPU for tutorials.