This document describes how to troubleshoot concurrent operation limits in Compute Engine that are caused by [`rateLimitExceeded`](https://cloud.google.com/compute/operations-quota) errors.

To help identify the cause of a `rateLimitExceeded` error, use the following troubleshooting steps:

1. In the Google Cloud console, go to the

   **Logs Explorer** page:

   [Go to **Logs Explorer**](https://console.cloud.google.com/logs/query)

   <br />

   If you use the search bar to find this page, then select the result whose subheading is
   **Logging**.
2. To filter the error logs generated by Compute Engine API, enter the following logging query in the query editor.

   ```
   log_id("cloudaudit.googleapis.com/activity")
   protoPayload.serviceName="compute.googleapis.com"
   protoPayload.response.error.code=403
   protoPayload.status.message="Rate Limit Exceeded"
   ```

   The filtered error logs should look similar to the following:

   ```
   "protoPayload":{
   "@type": "type.googleapis.com/google.cloud.audit.AuditLog",
   "status": {
     "message": "Rate Limit Exceeded"
   }
   "serviceName": "compute.googleapis.com",
   "methodName": "METHOD_NAME",
   ...
   ...
   "response": {
     "error": {
       "message": "Rate Limit Exceeded",
       "code": 403,
       "errors": [
         {
           "message": "Rate Limit Exceeded",
           "reason": "rateLimitExceeded",
           "domain": "usageLimits"
         }
       ]
     },
     "@type": "type.googleapis.com/error"
   },
   "resourceLocation": {
     "currentLocations": [
       "LOCATION"
     ]
   },
   ...
   ... 
   ```

   The error details contain the following values:
   - `METHOD_NAME`: The name of the service method or operation. For API calls, this name should be the name of the API method.

   - `LOCATION`: The location of the resource. This can be a region, zone or global. For example, the entries `currentLocations: "us-central1"`(Regional) or `currentLocations: "us-central1-a"` indicate that the region is `"us-central1"` or `currentLocations: "global"`.

3. Identify the associated `methodName` value from the error log. This name represents the API method for which the rate limit exceeded, for example: `v1.compute.instances.stop`.

4. Identify the relevant location in the audit logs.

5. Check the `Concurrent operations per project` quotas and limits.

   ### Console

   1. In the Google Cloud console, go to the **Quotas** page.

      [Go to Quotas](https://console.cloud.google.com/iam-admin/quotas)

      The **Quotas** page lists the quota usage and limits for your project.
      By default, the list is sorted to show your most used quotas first,
      which helps you see limits that are at risk of being exceeded.
   2. To filter the list and to view the usage and limit for any specific quota,
      select **Quota** or **Metric** in the **Filter** field.

      For example, either select the Quota by the name `Concurrent global operations per project operation type` or
      select `Service: Compute Engine API`, `type: System Limit` and a key word `concurrent` to list all the concurrent operation quotas. Filtering provides a list of concurrent quotas. From this list, you can search for the affected quota.
      To filter the quota for a specific operation, add **Dimensions** to the filter query
      and select the **operation_type** (shown as **methodName** in the audit logs).

      For example, select `instances_stop`
      to see the quota usage and limit for the operation `instances.stop`.

      To filter the quota for a specific region, add **Dimensions** to the filter query
      and select the **region** (shown as **currentLocations** in the audit logs).

      For example, select `us-central1` to see the quota usage and limit for `us-central1` region.

      Filters **region** and **operation_type** can be used together to view the quota usage and limit for a specific operation in a specific region.

   ### gcloud


   > [!WARNING]
   >
   > **Preview**
   >
   >
   > This product or feature is
   >
   > subject to the "Pre-GA Offerings Terms" in the General Service Terms section
   > of the [Service Specific
   > Terms](https://docs.cloud.google.com/terms/service-terms#1).
   >
   > Pre-GA products and features are available "as is" and might have limited support.
   >
   > For more information, see the
   > [launch stage descriptions](https://cloud.google.com/products/#product-launch-stages).

   <br />

   You can use the Google Cloud CLI to view the usage and limit of concurrent
   operation quotas.

   To use this approach, ensure that you have installed
   the [alpha commands](https://docs.cloud.google.com/sdk/docs/components#alpha_and_beta_components)
   component. If you're using Cloud Shell to interact with Google Cloud, the
   Google Cloud CLI is installed for you.

   To view an operation quota, use the following command:

   ```
   gcloud alpha services quota list \
     --service=compute.googleapis.com \
     --consumer=projects/PROJECT_ID
   ```

   Replace `PROJECT_ID` with the ID of the project for which you
   want to view the quota.

   These quotas are generally grouped into two categories:
   - Global or regional
   - Per project or per project per operation type

   For more information, see [Concurrent operation quota groups](https://cloud.google.com/compute/operations-quota#operation_groups).
6. Check the usage chart for the quota. If the usage is close to or above the quota limit during the period of error logs, it means that the quota or system limit is exceeded.

   To view charts that show current and peak usage, go to the quota and then
   click **Monitoring**.
   You might need to scroll right to the end of the table.

   The monitoring graph will confirm the cause of `rateLimitExceeded` error, with the specific quota/limit. If the current values are above the limit values, it means that the system limit is exceeded.
   Since system limits can't be modified, we recommend the following best practices to reduce the number of concurrent operations.

   Some of the best practices are as follows:
   - [Wait for operations to be done](https://docs.cloud.google.com/compute/docs/api/best-practices#wait-for-operations)
   - [Rely on error codes, not error messages](https://docs.cloud.google.com/compute/docs/api/best-practices#rely-on-errors-not-messages)
   - [Minimize client-side retries to preserve API rate limits](https://docs.cloud.google.com/compute/docs/api/best-practices#preserve-API-rate-limits)

     Any of the following methods can be used to minimize retries:
     - [Avoid short polling](https://docs.cloud.google.com/compute/docs/api/best-practices#avoid-short-polling)
     - [Use bursting sparingly and selectively](https://docs.cloud.google.com/compute/docs/api/best-practices#limit-bursting)
     - [Always make your calls in a retry loop with exponential backoff](https://docs.cloud.google.com/compute/docs/api/best-practices#retry-with-exponential-backoff)
     - [Use a client-side rate limiter](https://docs.cloud.google.com/compute/docs/api/best-practices#use-client-side-rate-limiter)
     - [Split up your applications across multiple projects](https://docs.cloud.google.com/compute/docs/api/best-practices#multiple-projects)