This page describes best practices for retrying failed requests to the
Identity and Access Management (IAM) API.

For requests that are safe to retry, we recommend using truncated exponential backoff with introduced jitter.

## Overview of truncated exponential backoff


Each request to the IAM API can succeed or fail. If your application retries
failed requests without waiting, it might send a large number of retries to IAM
in a short period of time. As a result, you might exceed [quotas and limits](https://docs.cloud.google.com/iam/quotas.md.txt) that apply to every
IAM resource in your Google Cloud project.


To avoid triggering this issue, we strongly recommend that you use
[truncated exponential backoff](https://en.wikipedia.org/wiki/Exponential_backoff) with
introduced [jitter](https://en.wikipedia.org/wiki/Jitter), which is a standard
error-handling strategy for network applications. In this approach, a client periodically retries
a failed request with exponentially increasing delays between retries. A small, random delay,
known as jitter, is also added between retries. This random delay helps prevent a synchronized
wave of retries from multiple clients, also known as the
[thundering herd problem](https://en.wikipedia.org/wiki/Thundering_herd_problem).

## Exponential backoff algorithm


The following algorithm implements truncated exponential backoff with jitter:

1. Send a request to IAM.
2. If the request fails, wait 1 + `random-fraction` seconds, then retry the request.
3. If the request fails, wait 2 + `random-fraction` seconds, then retry the request.
4. If the request fails, wait 4 + `random-fraction` seconds, then retry the request.
5. Continue this pattern, waiting 2^n^ + `random-fraction` seconds after each retry, up to a `maximum-backoff` time.
6. After `deadline` seconds, stop retrying the request.


Use the following values as you implement the algorithm:

- Before each retry, the wait time is `min((2n + random-fraction), maximum-backoff)`, with `n` starting at 0 and incremented by 1 for each retry.
- Replace `random-fraction` with a random fractional value less than or equal to 1. Use a different value for each retry. Adding this random value prevents clients from becoming synchronized and sending large numbers of retries at the same time.
- Replace `maximum-backoff` with the maximum amount of time, in seconds, to wait between retries. Typical values are 32 or 64 (2^5^ or 2^6^) seconds. Choose the value that works best for your use case.
- Replace `deadline` with the maximum number of seconds to keep sending retries. Choose a value that reflects your use case. For example, in a continuous integration/continuous deployment (CI/CD) pipeline that is not highly time-sensitive, you might set `deadline` to 300 seconds (5 minutes).

## Types of errors to retry

Use this retry strategy for all requests to the IAM API that
return the error codes `500`, `502`, `503`, or `504`.

Optionally, you can use this retry strategy for requests to the
IAM API that return the error code `404`.
[IAM reads are eventually consistent](https://docs.cloud.google.com/iam/docs/overview.md.txt#consistency); as a
result, resources might not be visible immediately after you create them, which
can lead to `404` errors.

In addition, use a modified version of this retry strategy for all requests to
the IAM API that return the error code `409` and the status
`ABORTED`. This type of error indicates a concurrency issue; for example, you
might be trying to update an [allow policy](https://docs.cloud.google.com/iam/docs/allow-policies.md.txt) that another client has
already overwritten. For this type of error, you should always retry the entire
[read-modify-write](https://docs.cloud.google.com/iam/docs/allow-policies.md.txt#etag) series of requests, using
truncated exponential backoff with introduced jitter. If you retry only the write operation, the request will
continue to fail.

## What's next

- Learn [how concurrency issues are managed](https://docs.cloud.google.com/iam/docs/allow-policies.md.txt#etag) in allow policies.
- Understand how to [implement the read-modify-write pattern](https://docs.cloud.google.com/iam/docs/granting-changing-revoking-access.md.txt#programmatic) for updating allow policies.