# Troubleshoot quota and limit errors

BigQuery has various [quotas and limits](https://docs.cloud.google.com/bigquery/quotas.md.txt)
that limit the rate and volume of different requests and operations. They exist
both to protect the infrastructure and to help guard against unexpected
customer usage. This document describes how to diagnose and mitigate
specific errors resulting from quotas and limits.

Some error messages specify quotas or limits that you can increase, while other
error messages specify quotas or limits that you can't increase. Reaching a hard
limit means that you need to implement temporary or permanent workarounds or
best practices for your workload. Doing so is a best practice, even for quotas
or limits that can be increased.

This document organizes error messages and their solutions according to these
categories, and the "Overview" section later in this document explains how to
read an error message and apply the correct solution for your issue.

If your error message is not listed in this document, then refer to
[the list of error messages](https://docs.cloud.google.com/bigquery/docs/error-messages.md.txt), which has more
generic error information.

## Overview

If a BigQuery operation fails because of exceeding a quota, the API
returns the HTTP `403 Forbidden` status code. The response body contains more
information about the quota that was reached. The response body looks similar to
the following:

```json
{
  "code" : 403,
  "errors" : [ {
    "domain" : "global",
    "message" : "Quota exceeded: ...",
    "reason" : "quotaExceeded"
  } ],
  "message" : "Quota exceeded: ..."
}
```

The `message` field in the payload describes which limit was exceeded. For
example, the `message` field might say `Exceeded rate limits: too many table
update operations for this table`.

In general, quota limits fall into two categories, indicated by the `reason`
field in the response payload.

- **`rateLimitExceeded`.** This value indicates a short-term
  limit. To resolve these limit issues, retry the operation after a few seconds.
  Use *exponential backoff* between retry attempts. That is, exponentially
  increase the delay between each retry.

- **`quotaExceeded`.** This value indicates a longer-term limit. If you reach a
  longer-term quota limit, you should wait 10 minutes or longer before trying
  the operation again. If you consistently reach one of these longer-term
  quota limits, you should analyze your workload for ways to mitigate the
  issue. Mitigations can include optimizing your workload or requesting a
  quota increase.

> [!NOTE]
> **Note:** Some quotas are expressed as daily quotas. For example, there is a daily quota on the number of [load jobs per table](https://docs.cloud.google.com/bigquery/quotas.md.txt#load_jobs). However, these quotas replenish incrementally over a 24-hour period, so you don't need to wait a full 24 hours after reaching the limit.

For `quotaExceeded` errors, examine the error message to understand which quota
limit was exceeded. Then, analyze your workload to see if you can avoid reaching
the quota.

In some cases, the quota can be raised by
[contacting BigQuery support](https://docs.cloud.google.com/bigquery/docs/getting-support.md.txt) or
[contacting Google Cloud sales](https://cloud.google.com/contact),
but we recommend trying the suggestions in this document first.

## Diagnosis

To diagnose issues, do the following:

- Use [`INFORMATION_SCHEMA` views](https://docs.cloud.google.com/bigquery/docs/information-schema-tables.md.txt) along with a
  [region qualifier](https://docs.cloud.google.com/bigquery/docs/information-schema-intro.md.txt#region_qualifier) to analyze the underlying issue. These views contain metadata about your BigQuery
  resources, including jobs, reservations, and streaming inserts.

  For example, the following query uses the
  [`INFORMATION_SCHEMA.JOBS`](https://docs.cloud.google.com/bigquery/docs/information-schema-jobs.md.txt) view to list all
  quota-related errors within the past day:

  ```googlesql
  SELECT
  job_id,
  creation_time,
  error_result
  FROM  `region-REGION_NAME`.INFORMATION_SCHEMA.JOBS
  WHERE creation_time > TIMESTAMP_SUB(CURRENT_TIMESTAMP, INTERVAL 1 DAY) AND
      error_result.reason IN ('rateLimitExceeded', 'quotaExceeded')
  ```

  Replace `REGION_NAME` with the region of the
  project. It must be preceded by `region-`. For example, for the `US`
  multi-region, use `region-us`.
- View errors in Cloud Audit Logs.

  For example, using
  [Logs Explorer](https://docs.cloud.google.com/logging/docs/view/logs-explorer-summary), the following query
  returns errors with either `Quota exceeded` or `limit` in the message string:

      resource.type = ("bigquery_project" OR "bigquery_dataset")
      protoPayload.status.code ="7"
      protoPayload.status.message: ("Quota exceeded" OR "limit")

  In this example, the status code `7` indicates
  [`PERMISSION_DENIED`](https://docs.cloud.google.com/tasks/docs/reference/rpc/google.rpc.md.txt#code), which
  corresponds to the HTTP `403` status code.

  For additional Cloud Audit Logs query samples, see [BigQuery
  queries](https://docs.cloud.google.com/logging/docs/view/query-library-preview#bigquery-filters).

## Troubleshoot quotas or limits that can be increased

You can increase the following quotas and limits; however, it's best to first
try any suggested workarounds or best practices.

### Your project exceeded quota for free query bytes scanned

BigQuery returns this error when you run a query in the free
usage tier and the account reaches the monthly query limit. For more information about query pricing, see [Free
usage tier](https://cloud.google.com/bigquery/pricing#free-usage-tier).

**Error message**

```
Your project exceeded quota for free query bytes scanned
```

<br />

#### Resolution

To continue using BigQuery, you need to [upgrade the account to a
paid Cloud Billing account](https://docs.cloud.google.com/free/docs/gcp-free-tier#how-to-upgrade).

### Streaming insert quota errors

This section provides tips for troubleshooting quota errors related to
streaming data into BigQuery.

In certain regions, streaming inserts have a higher quota if you don't populate
the `insertId` field for each row. For more information about quotas for
streaming inserts, see [Streaming inserts](https://docs.cloud.google.com/bigquery/quotas.md.txt#streaming_inserts).
The quota-related errors for BigQuery streaming depend on the
presence or absence of `insertId`.

**Error message**

If the `insertId` field is empty, the following quota error is possible:

| Quota limit | Error message |
|---|---|
| Bytes per second per project | Your entity with gaia_id: <var translate="no">GAIA_ID</var>, project: <var translate="no">PROJECT_ID</var> in region: <var translate="no">REGION</var> exceeded quota for insert bytes per second. |

If the `insertId` field is populated, the following quota errors are possible:

| Quota limit | Error message |
|---|---|
| Rows per second per project | Your project: <var translate="no">PROJECT_ID</var> in <var translate="no">REGION</var> exceeded quota for streaming insert rows per second. |
| Rows per second per table | Your table: <var translate="no">TABLE_ID</var> exceeded quota for streaming insert rows per second. |
| Bytes per second per table | Your table: <var translate="no">TABLE_ID</var> exceeded quota for streaming insert bytes per second. |

The purpose of the `insertId` field is to deduplicate inserted rows. If multiple
inserts with the same `insertId` arrive within a few minutes' window,
BigQuery writes a single version of the record. However, this
automatic deduplication is not guaranteed. For maximum streaming throughput, we
recommend that you don't include `insertId` and instead use
[manual deduplication](https://docs.cloud.google.com/bigquery/docs/streaming-data-into-bigquery#manually_removing_duplicates).
For more information, see
[Ensuring data consistency](https://docs.cloud.google.com/bigquery/docs/streaming-data-into-bigquery#dataconsistency).

When you encounter this error,
[diagnose the issue](https://docs.cloud.google.com/bigquery/docs/troubleshoot-quotas.md.txt#ts-streaming-insert-quota-diagnose), and then
[follow the recommended steps](https://docs.cloud.google.com/bigquery/docs/troubleshoot-quotas.md.txt#ts-streaming-insert-quota-resolution) to resolve
it.

#### Diagnosis

Use the [`STREAMING_TIMELINE_BY_*`](https://docs.cloud.google.com/bigquery/docs/information-schema-streaming.md.txt)
views to analyze the streaming traffic. These views aggregate streaming
statistics over one-minute intervals, grouped by `error_code`. Quota errors appear
in the results with `error_code` equal to `RATE_LIMIT_EXCEEDED` or
`QUOTA_EXCEEDED`.

Depending on the specific quota limit that was reached, look at `total_rows` or
`total_input_bytes`. If the error is a table-level quota, filter by `table_id`.

For example, the following query shows total bytes ingested per minute, and the
total number of quota errors:

```googlesql
SELECT
 start_timestamp,
 error_code,
 SUM(total_input_bytes) as sum_input_bytes,
 SUM(IF(error_code IN ('QUOTA_EXCEEDED', 'RATE_LIMIT_EXCEEDED'),
     total_requests, 0)) AS quota_error
FROM
 `region-REGION_NAME`.INFORMATION_SCHEMA.STREAMING_TIMELINE_BY_PROJECT
WHERE
  start_timestamp > TIMESTAMP_SUB(CURRENT_TIMESTAMP, INTERVAL 1 DAY)
GROUP BY
 start_timestamp,
 error_code
ORDER BY 1 DESC
```

#### Resolution

To resolve this quota error, do the following:

- If you are using the `insertId` field for deduplication, and your project is
  in a region that supports the higher streaming quota, we recommend removing the
  `insertId` field. This solution might require some additional steps to manually
  deduplicate the data. For more information, see
  [Manually removing duplicates](https://docs.cloud.google.com/bigquery/docs/streaming-data-into-bigquery#manually_removing_duplicates).

- If you are not using `insertId`, or if it's not feasible to remove it, monitor
  your streaming traffic over a 24-hour period and analyze the quota errors:

  - If you see mostly `RATE_LIMIT_EXCEEDED` errors rather than `QUOTA_EXCEEDED`
    errors, and your overall traffic is less than 80% of quota, the errors probably
    indicate temporary spikes. You can address these errors by retrying the
    operation using exponential backoff between retries.

  - If you are using a Dataflow job to insert data, consider using
    load jobs instead of streaming
    inserts. For more information, see [Setting the insertion
    method](https://beam.apache.org/documentation/io/built-in/google-bigquery/#setting-the-insertion-method).
    If you are using Dataflow with a custom I/O connector, consider
    using a built-in I/O connector instead. For more information, see [Custom
    I/O patterns](https://beam.apache.org/documentation/patterns/custom-io/).

  - If you see `QUOTA_EXCEEDED` errors or the overall traffic consistently
    exceeds 80% of the quota, submit a request for a quota increase. For more
    information, see
    [Request a quota adjustment](https://docs.cloud.google.com/docs/quotas/help/request_increase).

  - You might also want to consider replacing streaming inserts with the newer
    [Storage Write API](https://docs.cloud.google.com/bigquery/docs/write-api), which has higher throughput,
    lower price, and many useful features.

For more information about streaming inserts, see [streaming data into BigQuery](https://docs.cloud.google.com/bigquery/docs/write-api-rest.md.txt).

### Streaming table creation rate limit errors

When a streaming pipeline (`tabledata.insertAll`) automatically creates new
tables (for example, by using [template tables](https://docs.cloud.google.com/bigquery/docs/write-api-rest.md.txt#template-tables)
with `templateSuffix` or dynamic destination tables) faster than the per-project
[streaming table creation rate limit](https://docs.cloud.google.com/bigquery/docs/write-api-rest.md.txt#template_table_details),
BigQuery returns the following error:

**Error message**

```
Exceeded rate limits: Your Project: PROJECT_ID exceeded quota for
Streaming Table Creation Rate. For more information, see
https://docs.cloud.google.com/bigquery/troubleshooting-errors
```

#### Resolution

To resolve this error, do the following:

- **Pre-create destination tables.** If your pipeline generates many sharded or rapidly rotating tables dynamically, run a DDL script or use the [`tables.insert`](https://docs.cloud.google.com/bigquery/docs/reference/rest/v2/tables/insert.md.txt) API to pre-create the tables ahead of time instead of relying on the streaming API to create them during insertion. For more information, see [Template table details](https://docs.cloud.google.com/bigquery/docs/write-api-rest.md.txt#template_table_details).

### Maximum number of concurrent queries that contain remote functions

BigQuery returns this error when the number of concurrent
queries that contain remote functions exceeds the limit.

To learn more about remote functions limits, see
[Remote functions](https://docs.cloud.google.com/bigquery/quotas.md.txt#remote_function_limits).

**Error message**

```
Exceeded rate limits: too many concurrent queries with remote functions for
this project
```

This limit can be increased. Try the workarounds and best practices first.

#### Diagnosis

To see limits for concurrent queries that contain [remote
functions](https://docs.cloud.google.com/bigquery/docs/remote-functions), see
[Remote function limits](https://docs.cloud.google.com/bigquery/quotas.md.txt#remote_function_limits).

#### Resolution

- When using remote functions, adhere to [best practices for remote
  functions](https://docs.cloud.google.com/bigquery/docs/remote-functions#best_practices_for_remote_functions).
- You can request a quota increase by contacting [support](https://docs.cloud.google.com/bigquery/docs/getting-support.md.txt) or [sales](https://cloud.google.com/contact). It might take several days to review and process the request. We recommend stating the priority, use case, and the project ID in the request.

### Maximum number of `CREATE MODEL` statements

This error means that you have exceeded the quota for `CREATE MODEL` statements.

**Error message**

```
Quota exceeded: Your project exceeded quota for CREATE MODEL queries per
 project.
```

#### Resolution

If you exceed the [quota](https://docs.cloud.google.com/bigquery/quotas.md.txt#create_model_statements)
for `CREATE MODEL` statements, send an email to
[bqml-feedback@google.com](mailto:bqml-feedback@google.com)
and request a quota increase.

### Maximum number of copy jobs per day per project quota errors

BigQuery returns this error when the number of copy jobs running
in a project has exceeded the daily limit.
To learn more about the limit for copy jobs per day, see
[Copy jobs](https://docs.cloud.google.com/bigquery/quotas.md.txt#copy_jobs).

**Error message**

```
Your project exceeded quota for copies per project
```

#### Diagnosis

If you'd like to gather more data about where the copy jobs are coming from,
you can try the following:

- If your copy jobs are located in a single or only a few regions, you can try
  querying the [`INFORMATION_SCHEMA.JOBS`](https://docs.cloud.google.com/bigquery/docs/information-schema-jobs.md.txt)
  table for specific regions. For example:

  ```googlesql
  SELECT
  creation_time, job_id, user_email, destination_table.project_id, destination_table.dataset_id, destination_table.table_id
  FROM `PROJECT_ID`.`region-REGION_NAME`.INFORMATION_SCHEMA.JOBS
  WHERE
  creation_time BETWEEN TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 2 DAY) AND CURRENT_TIMESTAMP()
  AND job_type = "COPY"
  order by creation_time DESC
  ```

  You can also adjust the time interval depending on the time range you're interested in.
- To see all copy jobs in all regions, you can use the following filter in
  Cloud Logging:

  ```
  resource.type="bigquery_resource"
  protoPayload.methodName="jobservice.insert"
  protoPayload.serviceData.jobInsertRequest.resource.jobConfiguration.tableCopy:*
  ```

  <br />

#### Resolution

- If the goal of the frequent copy operations is to create a snapshot of data, consider using [table snapshots](https://docs.cloud.google.com/bigquery/docs/table-snapshots-intro.md.txt) instead. Table snapshots are a cheaper and faster alternative to copying full tables.
- You can request a quota increase by contacting [support](https://docs.cloud.google.com/bigquery/docs/getting-support.md.txt) or [sales](https://cloud.google.com/contact). It might take several days to review and process the request. We recommend stating the priority, use case, and the project ID in the request.

### Exceeded extract bytes per day quota error

BigQuery returns this error when the extraction exceeds the
default 50 TiB daily limit in a project.
For more information about extract job limits, see [Extract jobs](https://docs.cloud.google.com/bigquery/quotas.md.txt#export_jobs).

**Error message**

```
Your usage exceeded quota for ExtractBytesPerDay
```

#### Diagnosis

If you are exporting a table that is larger than 50 TiB, the export fails because
it [exceeds the extraction limit](https://docs.cloud.google.com/bigquery/quotas.md.txt#export_jobs).
If you want to
[export table data](https://docs.cloud.google.com/bigquery/docs/partitioned-tables.md.txt#export_table_data)
for specific table partitions, you can use a
[partition decorator](https://docs.cloud.google.com/bigquery/docs/partitioned-tables.md.txt#partition_decorators) to
identify the partitions to export.

If you would like to gather usages of exports data over recent days, you can
try the following:

- [View the quotas for your project](https://docs.cloud.google.com/docs/quotas/view-manage.md.txt#view_project_quotas)
  with filter criteria such as `Name: Extract bytes per day` or
  `Metric: bigquery.googleapis.com/quota/extract/bytes` along with the Show usage
  chart to see your usage trend over a few days.

- Alternatively you can query [`INFORMATION_SCHEMA.JOBS_BY_PROJECT`](https://docs.cloud.google.com/bigquery/docs/information-schema-jobs.md.txt)
  to see your total extract bytes over a few days. For example, the following
  query returns the daily total bytes processed by `EXTRACT` jobs in the past
  seven days.

  ```googlesql
  SELECT
  TIMESTAMP_TRUNC(creation_time, DAY) AS day,
  SUM ( total_bytes_processed ) / POW(1024, 3) AS total_gibibytes_processed
  FROM
  `region-REGION_NAME`.INFORMATION_SCHEMA.JOBS_BY_PROJECT
  WHERE
  creation_time BETWEEN TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 7 DAY) AND CURRENT_TIMESTAMP()
  AND job_type = "EXTRACT"
  GROUP BY 1
  ORDER BY 2 DESC
  ```
- You can then further refine the results by identifying the specific jobs that
  are consuming more bytes than expected. The following example returns the top
  100 `EXTRACT` jobs which are consuming more than 100 GB processed over the
  past seven days.

  ```googlesql
  SELECT
  creation_time,
  job_id,
  total_bytes_processed/POW(1024, 3) AS total_gigabytes_processed
  FROM
  `region-REGION_NAME`.INFORMATION_SCHEMA.JOBS_BY_PROJECT
  WHERE
  creation_time BETWEEN TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 7 DAY) AND CURRENT_TIMESTAMP()
  AND job_type="EXTRACT"
  AND total_bytes_processed > (POW(1024, 3) * 100)
  ORDER BY
  total_bytes_processed DESC
  LIMIT 100
  ```

You can alternatively use the [jobs explorer](https://docs.cloud.google.com/bigquery/docs/admin-jobs-explorer.md.txt)
with filters like `Bytes processed more than` to filter for high processing jobs
for a specified period of time.

#### Resolution

One method of resolving this quota error is to create a slot
[reservation](https://docs.cloud.google.com/bigquery/docs/reservations-intro.md.txt#reservations) and
[assign](https://docs.cloud.google.com/bigquery/docs/reservations-workload-management.md.txt#assignments) your
project into the reservation with the `PIPELINE` job type. This method can
bypass the limit check since it uses your dedicated reservations rather than
a free shared slot pool. If needed, the reservation can be deleted if you want
to use a shared slot pool later on.

For alternative approaches that allow exporting more *than* 50 TiB,
see the notes section in [Extract jobs](https://docs.cloud.google.com/bigquery/quotas.md.txt#export_jobs).

### Maximum `tabledata.list` bytes or requests per second per project quota errors

BigQuery returns this error when the project number mentioned
in the error message reaches the maximum size of data or maximum number of
requests (QPS) that can be read through the `tabledata.list` API call in a
project per second. For more information, see
[Maximum `tabledata.list` bytes per minute](https://docs.cloud.google.com/bigquery/quotas.md.txt#tabledata_list_bytes_per_minute).

**Error messages**

```
Your project:[project number] exceeded quota for tabledata.list bytes per second per project
```

```
Exceeded rate limits: Your project:[project number] exceeded quota for tabledata.list qps per project
```

#### Resolution

To resolve this error, do the following:

- In general, we recommend trying to stay under this limit. For example, by spacing out requests over a longer period with delays. If the error doesn't happen frequently, implementing retries with exponential backoff solves this issue.
- If you are reading data row by row or in small increments using `tabledata.list`, increase your batch size (for example, by using `maxResults`) to retrieve larger chunks over fewer HTTP requests.
- If the use case expects fast and frequent reading of a large amount of data from a table, we recommend using [BigQuery Storage Read API](https://docs.cloud.google.com/bigquery/docs/reference/storage.md.txt) instead of the `tabledata.list` API.
- If the preceding suggestions don't work, you can request a quota increase from
  Google Cloud console API dashboard by doing the following:

  1. Go to the [Google Cloud console API dashboard](https://console.cloud.google.com/apis/dashboard).
  2. In the dashboard, filter for Quota: `Tabledata list bytes per minute (default quota)`.
  3. Select the quota and follow the instructions in [Request a quota adjustment](https://docs.cloud.google.com/docs/quotas/help/request_increase).

  It might take several days to review and process the request.

### Maximum number of API requests limit errors

BigQuery returns this error when you reach the rate limit for the
number of API requests to a BigQuery API per user per method---for
example, the [`tables.get` method](https://docs.cloud.google.com/bigquery/docs/reference/rest/v2/tables/get.md.txt)
calls from a service account, or the
[`jobs.insert` method](https://docs.cloud.google.com/bigquery/docs/reference/rest/v2/jobs/insert.md.txt) calls from
a different user email---or when a user exceeds the rate limit for
`INFORMATION_SCHEMA` queries.

Most BigQuery API core methods have a maximum of 100 API requests per second
per user per method; however, some of these core methods can have different rate
limits;
for example:

- The [`jobs.get` method](https://docs.cloud.google.com/bigquery/docs/reference/rest/v2/jobs/get.md.txt) has a limit of 1000 API requests per user per second.
- The [`projects.list` method](https://docs.cloud.google.com/bigquery/docs/reference/rest/v2/projects/list.md.txt) and the [`tables.insert` method](https://docs.cloud.google.com/bigquery/docs/reference/rest/v2/tables/insert.md.txt) both have a limit of 10 requests per user per second.

In addition, BigQuery API rate limits [don't apply](https://docs.cloud.google.com/bigquery/quotas.md.txt#streaming_inserts)
to streaming inserts.

For more information on individual rate limits, see the
[**Maximum number of API requests per second per user per method** rate limit](https://docs.cloud.google.com/bigquery/quotas.md.txt#api_request_quotas).

**Error messages**

```
Exceeded rate limits: too many api requests per user per method for this
user_method. For more information, see
https://cloud.google.com/bigquery/docs/troubleshoot-quotas
```

```
Exceeded rate limits: too many rate INFORMATION_SCHEMA queries per user for
this user. For more information, see
https://cloud.google.com/bigquery/docs/troubleshoot-quotas
```

```
Exceeded rate limits: Your user exceeded quota for rate INFORMATION_SCHEMA
queries per user
```

```
Quota exceeded: Your user_method exceeded quota for concurrent api requests
per user per method.
```

When you encounter this error, [diagnose](https://docs.cloud.google.com/bigquery/docs/troubleshoot-quotas.md.txt#ts-maximum-api-request-limit-diagnose) the
issue and then [follow the recommended steps](https://docs.cloud.google.com/bigquery/docs/troubleshoot-quotas.md.txt#ts-maximum-api-request-limit-resolution)
to resolve it.

If the error has reason `apiLimitExceeded`, see
[API rate and metadata evaluation limit errors](https://docs.cloud.google.com/bigquery/docs/troubleshoot-quotas.md.txt#ts-api-limit-rate-and-metadata)
or, for dataset replica errors,
[Cross-region dataset replica creation limit errors](https://docs.cloud.google.com/bigquery/docs/troubleshoot-quotas.md.txt#ts-dataset-replica-limit).

#### Diagnosis

If you have not identified the method that has reached this rate limit, do the
following:

**For service account**

1. [Go to the project](https://console.cloud.google.com/projectselector2/home/dashboard)
   that hosts the service account.

2. In the Google Cloud console, go to the [API Dashboard](https://console.cloud.google.com/apis/dashboard).

   For instructions on how to view the detailed usage information of an API,
   see [Using the API Dashboard](https://docs.cloud.google.com/apis/docs/monitoring.md.txt#using_the_api_dashboard).
3. In the API Dashboard, select **BigQuery API**.

4. To view more detailed usage information, select **Metrics**, and then do
   the following:

   1. For **Select Graphs** , select **Traffic by API method**.

   2. Filter the chart by the service account's credentials. You might see
      spikes for a method in the time range where you noticed the error.

**For API calls**

Some API calls log errors in BigQuery audit
logs in Cloud Logging. To identify the method that reached the limit, do the
following:

1. In the Google Cloud console, go to the Google Cloud navigation
   menu and then
   select **Logging** \> **Logs Explorer** for your project:

   [Go to the Logs Explorer](https://console.cloud.google.com/logs/query)
2. Filter logs by running the following query:

   <br />

   ```
    resource.type="bigquery_resource"
    protoPayload.authenticationInfo.principalEmail="<user email or service account>"
    ("too many api requests per user per method" OR "too many rate INFORMATION_SCHEMA queries per user")
    In the log entry, you can find the method name under the property protoPayload.methodName.
    
   ```

   <br />

   For more information, see [BigQuery audit logs
   overview](https://docs.cloud.google.com/bigquery/docs/reference/auditlogs).

#### Resolution

To resolve this quota error, do the following:

- Reduce the number of API requests or add a delay between multiple API requests
  so that the number of requests stays under this limit.

- If the limit is only exceeded occasionally, you can implement retries on this
  specific error with exponential backoff.

- Avoid polling `INFORMATION_SCHEMA` views directly from live BI dashboards or
  high-frequency workflows. If your workflow requires frequent access to job
  history or metadata, schedule a periodic query that materializes the
  `INFORMATION_SCHEMA` output into a standard BigQuery table, and
  point your applications or dashboards to that materialized table. If you need
  to run many `INFORMATION_SCHEMA` queries, space them out over time instead of
  running them concurrently.

- If you frequently insert data, consider using the
  [BigQuery Storage Write API (gRPC)](https://docs.cloud.google.com/bigquery/docs/write-api-streaming.md.txt). Streaming
  data with this API isn't affected by the BigQuery API quota.

- While loading data to BigQuery using Dataflow
  with the [BigQuery I/O connector](https://beam.apache.org/documentation/io/built-in/google-bigquery/), you
  might encounter this error for the [`tables.get` method](https://docs.cloud.google.com/bigquery/docs/reference/rest/v2/tables/get.md.txt). To resolve this issue, do the following:

  - Set the create disposition of the destination table to `CREATE_NEVER`. For
    more information, see [Create disposition](https://beam.apache.org/documentation/io/built-in/google-bigquery/#create-disposition).

  - Use the Apache Beam SDK version 2.24.0 or higher. In the
    previous versions of the SDK, the `CREATE_IF_NEEDED` disposition
    calls the `tables.get` method to check if the table exists.

  - For additional quota, see [Request a quota increase](https://docs.cloud.google.com/bigquery/quotas.md.txt#requesting_a_quota_increase).
    Requesting a quota increase might take several days to process. To provide more
    information for your request, we recommend that
    your request includes the priority of the job, the user running the query, and
    the affected method.

### API rate and metadata evaluation limit errors

BigQuery returns an HTTP `403 Forbidden` error with reason
`apiLimitExceeded` when a request exceeds API rate limits or concurrent metadata
and access evaluation limits.

**Error messages**

- `Quota exceeded for the project...`
- `API limit exceeded: Quota exceeded for quota metric 'Read requests'...`

#### Diagnosis

This error can occur in the following scenarios:

- **High volume of concurrent API requests.** An application submits a burst of concurrent API calls (such as `jobs.insert`, `tables.get`, or rapid job status polling) that exceeds BigQuery API request limits.
- **BigLake Metastore concurrency.** Queries against external tables backed by BigLake Metastore generate a high volume of concurrent metadata read requests.
- **Complex IAM or row-level security evaluation.** Concurrent queries reference a large number of datasets, tables, or row-level security (RLS) policies, exhausting backend access-evaluation limits.

If the error message is `Quota exceeded: Your user_method exceeded quota for
concurrent api requests per user per method`, see
[Maximum number of API requests limit errors](https://docs.cloud.google.com/bigquery/docs/troubleshoot-quotas.md.txt#ts-maximum-api-request-limit).

#### Resolution

To resolve this error, do the following:

- **Retry with truncated exponential backoff.** Configure your client application or script to retry failed requests by using truncated exponential backoff.
- **Batch and consolidate API operations.** Load or query data in fewer, larger batches rather than issuing high-frequency single-row or per-table API calls, and reduce the frequency of `jobs.get` polling calls.
- **Simplify IAM and security configurations.** Prefer Google Groups and dataset-level IAM role bindings over large numbers of granular per-table IAM bindings or overlapping row-level security policies to reduce access-check overhead.
- **Check quota usage in the Google Cloud console.** Review the [**Quotas \& System Limits** page](https://docs.cloud.google.com/docs/quotas/view-manage.md.txt#viewing_your_quota_console) to identify whether an adjustable BigQuery or BigLake API quota is being reached. If you have a sustained need for more capacity, request a quota adjustment.

### Maximum Python UDF image storage bytes per project per region

The total size of all stored container images used by active Python UDFs can't
exceed `10 GiB` per project per region. This limit is enforced when you create
or update a Python UDF. If the total size exceeds the regional limit, the
request fails immediately, and the following error is returned.

**Error message**

`Resources exceeded during query execution: Quota exceeded: Your project:
PROJECT_ID exceeded quota for Python UDF image storage usage per
region per project.`

#### Resolution

To resolve this error, do the following:

1. Request an increase: go to the [**Quotas \& System Limits** page](https://docs.cloud.google.com/docs/quotas/view-manage.md.txt#viewing_your_quota_console),
   search for the `Python UDF image storage bytes per project per region`
   limit, and then request an increase.

2. Optimize storage: delete unused Python UDFs using the `DROP FUNCTION`
   statement to free space. After a UDF is deleted, its image size no longer
   counts towards the quota. You can find the image size by using the API's
   [`RoutineBuildStatus`](https://docs.cloud.google.com/bigquery/docs/reference/rest/v2/routines.md.txt#routinebuildstatus).
   For more information on viewing build status, see
   [Container build status](https://docs.cloud.google.com/bigquery/docs/user-defined-functions-python.md.txt#container_build_status).

### Maximum concurrent Python UDFs

You are limited to 10 concurrent Python UDFs per project. If you run more than
10, you receive the following error.

**Error message**

`Exceeded rate limits: too many concurrent queries with Python UDFs in this
project. Your project:
PROJECT_ID exceeded quota for maximum concurrent Python UDFs per
project.`

To view all Python UDF quotas, see [Routines](https://docs.cloud.google.com/bigquery/quotas.md.txt#udf_limits) in
the Quotas and limits reference.

#### Resolution

To resolve this error, you can request a quota increase by contacting
[support](https://docs.cloud.google.com/bigquery/docs/getting-support.md.txt) or
[sales](https://cloud.google.com/contact). It might take several days to review
and process the request. We recommend stating the priority, use case, and the
project ID in the request.

### Python UDF execution rate limits or capacity errors

This error occurs when a query that invokes a Python UDF exceeds the rate
limits or maximum container scaling capacity for the UDF. This issue typically
happens in the following situations:

- A query processes a large volume of data with high parallelism.
- Each container instance is configured to handle a small number of concurrent requests.
- Individual UDF invocations take a long time to complete.

**Error message**

`Resources exceeded during query execution: Python user-defined function
execution exceeded rate limits or capacity. Check
https://docs.cloud.google.com/bigquery/docs/troubleshoot-quotas for more
details.`

#### Resolution

To resolve this error, try the following actions:

- **Tune container limits** : check [Python UDF metrics](https://docs.cloud.google.com/bigquery/docs/user-defined-functions-python.md.txt#view_python_udf_metrics) in Cloud Monitoring to inspect CPU utilization, memory utilization, and concurrent requests. If `container_request_concurrency` is set to a low value, increase it so that each container instance can process more requests in parallel. If `container_cpu` is set to less than `1.0` vCPU without an explicit `container_request_concurrency` value, concurrency defaults to `1` at run time. For more information, see [Configure container limits for Python UDFs](https://docs.cloud.google.com/bigquery/docs/user-defined-functions-python.md.txt#configure-container-limits).
- **Optimize UDF execution and query structure** : follow the [best practices for Python UDFs](https://docs.cloud.google.com/bigquery/docs/user-defined-functions-python.md.txt#best_practices) to reduce the execution time and request volume per query. For example, filter your data before invoking the UDF, keep input row payloads small, and structure your query to support batching.
- **Request a capacity increase** : if the error persists after you optimize your UDF, contact [support](https://docs.cloud.google.com/bigquery/docs/getting-support.md.txt) or [sales](https://cloud.google.com/contact) to request a higher container scaling limit. Include the priority, use case, and project ID in your request.

### Custom daily query quota errors

BigQuery returns this error (`reason: quotaExceeded`) when a
project or user reaches a custom daily query usage quota (`QueryUsagePerDay` or
`QueryUsagePerUserPerDay`) configured by an administrator. For more information
about how custom query quotas work, see
[Custom quotas](https://docs.cloud.google.com/bigquery/docs/custom-quotas.md.txt).

**Error messages**

- `Custom quota exceeded: Your usage exceeded the custom quota for
  QueryUsagePerDay, which is set by your administrator.`
- `Custom quota exceeded: Your usage exceeded the custom quota for
  QueryUsagePerUserPerDay, which is set by your administrator.`

#### Diagnosis

If your query usage reached the custom daily quota earlier than expected, check
the following common causes:

- **Daily reset window and timezone alignment.** Custom daily query quotas reset at midnight Pacific Time (US and Canada). If your local timezone differs from Pacific Time, queries that ran on the previous calendar day in your local timezone might still fall within the current 24-hour Pacific Time quota window.
- **Normalized usage for BigQuery ML statements.** The **Quotas \& System Limits** page in the Google Cloud console reports normalized query usage, which can be higher than the billed bytes in `INFORMATION_SCHEMA.JOBS`. Certain BigQuery ML operations, such as creating a logistic regression model, are billed at a higher rate, as described in [BigQuery ML pricing](https://cloud.google.com/bigquery/pricing#bqml). For example, if an operation is priced at 50 times the standard on-demand rate, a `CREATE MODEL` job that reports 1 TiB of `total_bytes_billed` in `INFORMATION_SCHEMA.JOBS` counts as 50 TiB of query usage toward your custom daily quota. For more information, see [Set or modify custom quotas](https://docs.cloud.google.com/bigquery/docs/custom-quotas.md.txt#set-custom-quotas).

To identify which jobs and users consumed the most billed bytes during the
current Pacific Time day, query the
[`INFORMATION_SCHEMA.JOBS_BY_PROJECT`](https://docs.cloud.google.com/bigquery/docs/information-schema-jobs.md.txt)
view:

```sql
SELECT
  user_email,
  job_id,
  statement_type,
  creation_time,
  total_bytes_billed / POWER(1024, 4) AS tib_billed
FROM
  `region-REGION_NAME`.INFORMATION_SCHEMA.JOBS_BY_PROJECT
WHERE
  creation_time >= TIMESTAMP(
    CURRENT_DATE('America/Los_Angeles'), 'America/Los_Angeles')
  AND job_type = 'QUERY'
ORDER BY
  tib_billed DESC
LIMIT 50;
```

Replace `REGION_NAME` with your dataset region (for
example, `us` or `eu`).

Jobs with a `statement_type` of `CREATE_MODEL` might count toward the quota at
a higher rate than `tib_billed` shows.

#### Resolution

To resolve or mitigate custom daily query quota errors, do the following:

- **Optimize high-usage queries to scan fewer bytes.** For example, filter on partitioned and clustered columns, select only the columns you need instead of using `SELECT *`, and use a [dry run](https://docs.cloud.google.com/bigquery/docs/running-queries.md.txt#dry-run) to preview bytes processed before you run a query. For more information, see [Reduce data processed](https://docs.cloud.google.com/bigquery/docs/best-practices-performance-compute.md.txt#reduce-data-processed).
- **Run queries from a different billing project.** If your datasets are shared with another Google Cloud project that has access to the underlying tables and either doesn't have a custom daily quota or has remaining quota available, you can submit your query jobs from that alternative project.
- **Increase the custom daily quota.** If your workload's daily query volume has grown, ask your Google Cloud administrator or Quota Administrator (`roles/servicemanagement.quotaAdmin`) to increase the **Query usage per
  day** or **Query usage per day per user** limit on the **Quotas \& System
  Limits** page. For instructions, see [Set or modify custom quotas](https://docs.cloud.google.com/bigquery/docs/custom-quotas.md.txt#set-custom-quotas).

## Troubleshoot quotas or limits that can't be increased

You can't increase the following quotas or limits, but you can apply the
suggested workarounds or best practices to mitigate them.

### Table snapshot limit exceeded

This error occurs when a BigQuery table exceeds the maximum number of
table snapshots allowed within a 24-hour period. This is a table-level limit,
which means that it applies to each individual table in your project.

For more information about the limits on table operations per day, see [Standard
tables](https://docs.cloud.google.com/bigquery/quotas#standard_tables).

This limit [cannot be increased](https://docs.cloud.google.com/bigquery/quotas.md.txt#table_snapshots).

**Error message**

```
Your table exceeded quota for SNAPSHOT operations per table
```

#### Resolution

To resolve this error, consider the following actions:

- Reduce the number of snapshot operations performed on the affected table within a 24-hour window.
- If reducing the snapshot frequency isn't an option, contact [Cloud Customer Care](https://docs.cloud.google.com/bigquery/docs/getting-support.md.txt) or your technical account manager.

### Maximum number of pending jobs errors

This limit [can't be increased](https://docs.cloud.google.com/bigquery/quotas.md.txt#pending_jobs).

**Error message**

```
Quota exceeded: concurrent number of jobs that can be in the pending state
```

#### Resolution

This quota cannot be increased.

To resolve this error, do the following:

- **Reduce submission rate.** Reduce the rate at which you submit jobs to BigQuery, and retry failed job submissions using [truncated exponential backoff](https://cloud.google.com/monitoring/api/troubleshooting#exponential-retry).
- **Limit concurrency and batch smaller operations.** Avoid submitting a large number of jobs in parallel that might accumulate in the `PENDING` state. Instead, combine smaller load, copy, or DML operations into fewer, larger jobs.
- **Inspect active and pending jobs.** Query the [`INFORMATION_SCHEMA.JOBS` view](https://docs.cloud.google.com/bigquery/docs/information-schema-jobs.md.txt) for jobs with `state IN ('PENDING', 'RUNNING')` to identify the users, service accounts, or pipelines submitting the backlog of jobs.

### Query queue limit errors

You might encounter this error if a project queues more interactive or batch
queries than its [queue limit](https://docs.cloud.google.com/bigquery/quotas.md.txt#queued_interactive_queries)
permits.

**Error message**

```
Quota exceeded: Your project and region exceeded quota for max number of jobs
that can be queued per project.
```

#### Resolution

This limit [can't be increased](https://docs.cloud.google.com/bigquery/quotas.md.txt#jobs). To resolve this error,
see the following general guidance. For high-volume interactive queries, see
[Avoid limits for high-volume interactive queries](https://docs.cloud.google.com/bigquery/docs/troubleshoot-quotas.md.txt#interactive-query-queue-resolution).

- **Pause the job.** If you identify a process or pipeline responsible for an increase in queries, then pause that process or pipeline.
- **Distribute run times.** Distribute the load across a larger timeframe. If your reporting solution needs to run many queries, try to introduce some randomness for when queries start. For example, don't start all reports at the same time.
- **Optimize queries and the data model.** Often, a query can be rewritten
  so that it runs more efficiently.

  For example, if your query contains a *Common table expression (CTE)* --a `WITH`
  clause--which is referenced in more than one place in the query, then this
  computation is done multiple times. It is better to persist calculations done
  by the CTE in a temporary table,
  and then reference it in the query.

  Multiple joins can also be the source of lack of efficiency. In this case, you
  might want to consider using [nested and repeated columns](https://docs.cloud.google.com/bigquery/docs/nested-repeated.md.txt). Using this often improves locality of the
  data, eliminates the need for some joins, and overall reduces resource
  consumption and the query runtime.

  Optimizing queries makes them cheaper, so when you use capacity-based pricing,
  you can run more queries with your slots. For more information, see
  [Introduction to optimizing query
  performance](https://docs.cloud.google.com/bigquery/docs/best-practices-performance-overview).
- **Optimize the query model.** BigQuery is not a relational
  database. It is not optimized for an infinite number of small queries. Running a
  large number of small queries quickly depletes your quotas. Such queries don't
  run as efficiently as they do with the smaller database products.
  BigQuery is a large data warehouse and this is its primary use
  case. It performs best with analytical queries over large amounts of data.

  Use the recommendations in the following list to optimize the query model to
  avoid reaching the query queue limit.
  - **Persist data (saved tables).** Pre-process the data in BigQuery and store that data in additional tables. For example, if you execute many similar, computationally intensive queries using different `WHERE` conditions, then their results are not cached. Such queries also consume resources each time they run. You can improve the performance of such queries and decrease their processing time by pre-computing the data and storing it in a table. This pre-computed data in the table can be queried by `SELECT` queries. It can often be done during ingestion within the ETL process, or by using [scheduled queries](https://docs.cloud.google.com/bigquery/docs/scheduling-queries.md.txt) or [materialized views](https://docs.cloud.google.com/bigquery/docs/materialized-views-intro.md.txt).
  - **Use dry run mode.** Run queries in [dry run mode](https://docs.cloud.google.com/bigquery/docs/running-queries.md.txt#dry-run), which estimates the number of bytes read but does not actually process the query.
  - **Preview table data.** To experiment with or explore data rather than running queries, preview table data with the [table preview capability](https://docs.cloud.google.com/bigquery/docs/best-practices-costs.md.txt#preview-data) in BigQuery.
  - **Use [cached query results](https://docs.cloud.google.com/bigquery/docs/cached-results.md.txt).** All query results, including both interactive and batch queries, are cached in temporary tables for approximately 24 hours with some [exceptions](https://docs.cloud.google.com/bigquery/docs/cached-results.md.txt#cache-exceptions). While running a cached query does still count against your concurrent query limit, queries that use cached results are significantly faster than queries that don't use cached results because BigQuery does not need to compute the result set.
  - **Use BigQuery BI Engine.** If you have encountered this error while using a business intelligence (BI) tool to create dashboards that query data in BigQuery, then we recommend that you use [BigQuery BI Engine](https://docs.cloud.google.com/bigquery/docs/bi-engine-intro.md.txt). Using BigQuery BI Engine is optimal for this use case.
- **Use jobs with batch priority.** You can queue more
  [batch queries](https://docs.cloud.google.com/bigquery/docs/running-queries.md.txt#batch) than interactive
  queries.

- **Distribute queries.** Organize and distribute the load across different
  projects as informed by the nature of your queries and your business needs.

- **Increase slots in your reservation.** [Increase slots](https://docs.cloud.google.com/bigquery/docs/reservations-workload-management.md.txt#estimate-slots) or, if you have a
  high-demand workload, [switch](https://docs.cloud.google.com/bigquery/docs/reservations-intro.md.txt#choosing_a_model)
  to reservations (capacity-based model) from on-demand (pay-per query model).

- **Adjust the reservation concurrency target.** If you use reservations, you
  can
  [set the maximum concurrency target](https://docs.cloud.google.com/bigquery/docs/query-queues.md.txt#set_the_maximum_concurrency_target)
  on your reservation to control how many queries run concurrently before
  additional queries are queued.

##### Avoid limits for high-volume interactive queries

Running high-volume interactive queries can cause those queries to reach
the [limit](https://docs.cloud.google.com/bigquery/quotas.md.txt#query_jobs) for the maximum number of queued
interactive queries. This limit can't be increased.

If you run high-volume interactive queries, especially in scenarios
that involve automated triggers like Cloud Run functions, first
[monitor](https://docs.cloud.google.com/bigquery/docs/admin-jobs-explorer.md.txt)
the behavior of, and stop, the Cloud Run function causing the
error, and then use one of the following recommended strategies for avoiding
this limit:

- [Increase slots
  in your reservation](https://docs.cloud.google.com/bigquery/docs/reservations-workload-management#estimate-slots). If you have a high-demand workload, [switch](https://docs.cloud.google.com/bigquery/docs/reservations-intro.md.txt#choosing_a_model) to reservations (capacity-based model) from on-demand (pay per query model).
- Spread the workload across interactive queries.
- Because you can queue more batch queries than interactive queries, use [batch
  priority jobs](https://docs.cloud.google.com/bigquery/docs/running-queries#batch) instead of interactive queries.

#### Query admission denied or continuous query slot limits

If query queuing is disabled (`default_interactive_query_queue_timeout_ms` or
`default_batch_query_queue_timeout_ms` is set to `-1`) and the
[query queue concurrency limit](https://docs.cloud.google.com/bigquery/docs/query-queues.md.txt) is reached,
BigQuery rejects new queries with
`Query was not admitted as the maximum concurrency has been reached`
(`rateLimitExceeded` with HTTP `403`, or `jobRateLimitExceeded` with HTTP `400`
on synchronous `jobs.query` and `jobs.getQueryResults` calls).

To resolve query admission errors:

- Reduce concurrent query submissions, run non-urgent workloads with [batch priority](https://docs.cloud.google.com/bigquery/docs/running-queries.md.txt#batch), or adjust the [reservation concurrency target](https://docs.cloud.google.com/bigquery/docs/query-queues.md.txt#set_the_maximum_concurrency_target).
- Retry failed operations using [truncated exponential backoff](https://docs.cloud.google.com/monitoring/api/troubleshooting.md.txt#retry). If `jobs.query` or `jobs.getQueryResults` failed with `jobRateLimitExceeded`, submit the retry with a new `jobId` because calling `jobs.getQueryResults` again on the failed `jobId` returns the same error.
- **Continuous queries:** BigQuery admits a continuous query only when the `CONTINUOUS` reservation assignment provides at least 10 slots for each concurrent continuous query. Increase the reservation's maximum size (for example, by using [slot autoscaling](https://docs.cloud.google.com/bigquery/docs/continuous-queries-introduction.md.txt#slots_autoscaling)) or reduce the number of concurrent continuous queries. For more information, see [Reservation limitations](https://docs.cloud.google.com/bigquery/docs/continuous-queries-introduction.md.txt#reservation_limitations).

### Shuffle size limit errors

BigQuery returns this error when your project exceeds the maximum
disk and memory size limit available for shuffle operations.

This quota is computed per-reservation and sliced across projects for the
reservations. The quota cannot be modified by Cloud Customer Care. You can learn
more about your usage by querying the [`INFORMATION_SCHEMA.JOBS_TIMELINE` view](https://docs.cloud.google.com/bigquery/docs/information-schema-jobs-timeline.md.txt#schema).

If you are using the [on-demand](https://cloud.google.com/bigquery/pricing#on_demand_pricing)
pricing model, you are limited to a [maximum of 2,000 slots](https://docs.cloud.google.com/bigquery/quotas.md.txt#query_jobs). To increase the
number of slots, you must change to a [capacity-based](https://cloud.google.com/bigquery/pricing#capacity_compute_analysis_pricing)
pricing model.

**Error message**

You receive one of the following error messages:

```
Quota exceeded: Your project exceeded quota for total shuffle size limit.
```

```
Resources exceeded: Your project or organization exceeded the maximum
  disk and memory limit available for shuffle operations. Consider provisioning
  more slots, reducing query concurrency, or using more efficient logic in this
  job.
```

#### Resolution

To resolve this error, do the following:

- [Increase your reservation](https://docs.cloud.google.com/bigquery/docs/reservations-tasks.md.txt#update_reservations).
- [Optimize queries](https://docs.cloud.google.com/bigquery/docs/best-practices-performance-overview.md.txt).
- Reduce concurrency of queries or materializing intermediate results to reduce dependence on resources. For more information, see [Query queues](https://docs.cloud.google.com/bigquery/docs/query-queues.md.txt) and [Create materialized views](https://docs.cloud.google.com/bigquery/docs/materialized-views-create.md.txt).
- [Review detailed steps](https://docs.cloud.google.com/bigquery/docs/query-insights.md.txt#insufficient_shuffle_quota) for mitigating insufficient shuffle quota.

### Number of partition modifications for column-partitioned tables quota errors

BigQuery returns this error when your column-partitioned table
reaches the
quota of the number of partition modifications permitted per day.
Partition modifications include the total of all [load jobs](https://docs.cloud.google.com/bigquery/quotas.md.txt#load_jobs),
[copy jobs](https://docs.cloud.google.com/bigquery/quotas.md.txt#copy_jobs), and [query jobs](https://docs.cloud.google.com/bigquery/quotas.md.txt#query_jobs)
that append or overwrite a destination partition.

To see the value of the **Number of partition
modifications per column-partitioned table per day** limit, see [Partitioned
tables](https://docs.cloud.google.com/bigquery/quotas#partitioned_tables).

**Error message**

```
Quota exceeded: Your table exceeded quota for
Number of partition modifications to a column partitioned table
```

#### Resolution

This quota cannot be increased. To resolve this quota error, do the following:

- Change the partitioning on the table to have more data in each partition, in order to decrease the total number of partitions. For example, change from [partitioning by day to partitioning by month](https://docs.cloud.google.com/bigquery/docs/partitioned-tables.md.txt#select_daily_hourly_monthly_or_yearly_partitioning) or change [how you partition the table](https://docs.cloud.google.com/bigquery/docs/partitioned-tables.md.txt).
- Use [clustering](https://docs.cloud.google.com/bigquery/docs/clustered-tables.md.txt#when_to_use_clustering) instead of partitioning.
- If you frequently load data from multiple small files stored in Cloud Storage that uses a job per file, then combine multiple load jobs into a single job. You can load from multiple Cloud Storage URIs with a comma-separated list (for example, `gs://my_path/file_1,gs://my_path/file_2`), or by using wildcards (for example, `gs://my_path/*`).


  For more information, see
  [Batch loading data](https://docs.cloud.google.com/bigquery/docs/batch-loading-data.md.txt#permissions-load-data-from-cloud-storage).
- If you use load, select or copy jobs to append single rows of data to a table, for example, then you should consider batching multiple jobs into one job. BigQuery doesn't perform well when used as a relational database. As a best practice, avoid running frequent, single-row append actions.
- To append data at a high rate, consider using [BigQuery Storage Write API (gRPC)](https://docs.cloud.google.com/bigquery/docs/write-api). It is a recommended solution for high-performance data ingestion. The BigQuery Storage Write API (gRPC) has robust features, including exactly-once delivery semantics. To learn about limits and quotas, see [Storage Write API](https://cloud.google.com/bigquery/quotas#write-api-limits) and to see costs of using this API, see [BigQuery data ingestion pricing](https://cloud.google.com/bigquery/pricing#data_ingestion_pricing).
- To monitor the number of modified partitions on a table, use the [`INFORMATION_SCHEMA` view](https://docs.cloud.google.com/bigquery/docs/information-schema-jobs.md.txt#partitions-modified-by).
- For information about optimizing table load jobs to avoid reaching quota limits, see [Optimize load jobs](https://docs.cloud.google.com/bigquery/docs/optimize-load-jobs.md.txt).

### Maximum rate of table metadata update operations limit errors

BigQuery returns this error when your table reaches the limit for
maximum rate of table metadata update operations per table for standard or
partitioned tables.
Table operations include the combined total of all [load jobs](https://docs.cloud.google.com/bigquery/quotas.md.txt#load_jobs),
[copy jobs](https://docs.cloud.google.com/bigquery/quotas.md.txt#copy_jobs), and [query jobs](https://docs.cloud.google.com/bigquery/quotas.md.txt#query_jobs)
that append to or overwrite a destination table or that use
a [DML](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/dml-syntax.md.txt) `DELETE`, `INSERT`,
`MERGE`, `TRUNCATE TABLE`, or `UPDATE` to write data to a table.

To see the value of the **Maximum rate of table metadata update
operations per table** limit, see [Standard tables](https://docs.cloud.google.com/bigquery/quotas.md.txt#standard_tables).

**Error messages**

```
Exceeded rate limits: too many table update operations for this table. For
more information, see
https://cloud.google.com/bigquery/docs/troubleshoot-quotas
```

```
Exceeded rate limits: too many partitioned table update operations for this
table. For more information, see
https://cloud.google.com/bigquery/docs/troubleshoot-quotas
```

When you encounter this error, [diagnose](https://docs.cloud.google.com/bigquery/docs/troubleshoot-quotas.md.txt#ts-maximum-update-table-metadata-limit-diagnose) the
issue, and then [follow the recommended steps](https://docs.cloud.google.com/bigquery/docs/troubleshoot-quotas.md.txt#ts-maximum-update-table-metadata-limit-resolution)
to resolve it.

#### Diagnosis

Metadata table updates can originate from *API calls* that modify a table's
metadata or from *jobs* that modify a table's content. If you have not
identified the source from where most update operations to a table's metadata
are originating, do the following:

- [Identify API calls](https://docs.cloud.google.com/bigquery/docs/troubleshoot-quotas.md.txt#identify-api-calls).
- [Identify jobs](https://docs.cloud.google.com/bigquery/docs/troubleshoot-quotas.md.txt#identify-jobs).

##### Identify API calls

1. Go to the Google Cloud navigation
   menu, and then
   select **Logging \> Logs Explorer**:

   [Go to the Logs Explorer](https://console.cloud.google.com/logs/query)
2. Filter logs to view table operations by running the following query:

   ```
   resource.type="bigquery_dataset"
   protoPayload.resourceName="projects/my-project-id/datasets/my_dataset/tables/my_table"
   (protoPayload.methodName="google.cloud.bigquery.v2.TableService.PatchTable" OR
   protoPayload.methodName="google.cloud.bigquery.v2.TableService.UpdateTable" OR
   protoPayload.methodName="google.cloud.bigquery.v2.TableService.InsertTable")
   ```

##### Identify jobs

The following query returns a list of jobs that modify the affected table in
the project within the past day. If you expect multiple projects in an organization
to write to the table, replace `JOBS_BY_PROJECT` with `JOBS_BY_ORGANIZATION`.

```googlesql
SELECT
 job_id,
 user_email,
 query
FROM  `region-REGION_NAME`.INFORMATION_SCHEMA.JOBS_BY_PROJECT
WHERE creation_time > TIMESTAMP_SUB(CURRENT_TIMESTAMP, INTERVAL 1 DAY)
AND destination_table.project_id = "my-project-id"
AND destination_table.dataset_id = "my_dataset"
AND destination_table.table_id = "my_table"
```

For more information, see [BigQuery audit logs
overview](https://docs.cloud.google.com/bigquery/docs/reference/auditlogs).

#### Resolution

This quota cannot be increased. To resolve this quota error, do the following:

- Reduce the update rate for the table metadata.
- Add a delay between jobs or table operations to ensure that the update rate is within the limit, and retry failed operations with truncated exponential backoff.
- Consolidate consecutive DDL statements (such as multiple `ALTER TABLE` statements) into a single [`CREATE OR REPLACE TABLE` statement](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/data-definition-language.md.txt#create_table_statement) or batch schema changes into a single table metadata update rather than running consecutive metadata update commands.
- For data inserts or modification, consider using DML operations. DML
  operations are not affected by the **Maximum rate of table metadata update
  operations per table** rate limit.

  DML operations have other [limits](https://docs.cloud.google.com/bigquery/quotas.md.txt#data-manipulation-language-statements)
  and quotas. For more information, see [Using
  data manipulation language (DML)](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/data-manipulation-language).
- If you frequently load data from multiple small files stored in Cloud Storage that uses a job per file, then combine multiple load jobs into a single job. You can load from multiple Cloud Storage URIs with a comma-separated list (for example, `gs://my_path/file_1,gs://my_path/file_2`), or by using wildcards (for example, `gs://my_path/*`).


  For more information, see
  [Batch loading data](https://docs.cloud.google.com/bigquery/docs/batch-loading-data.md.txt#permissions-load-data-from-cloud-storage).
- If you use load, select or copy jobs to append single rows of data to a table, for example, then you should consider batching multiple jobs into one job. BigQuery doesn't perform well when used as a relational database. As a best practice, avoid running frequent, single-row append actions.
- To append data at a high rate, consider using [BigQuery Storage Write API (gRPC)](https://docs.cloud.google.com/bigquery/docs/write-api). It is a recommended solution for high-performance data ingestion. The BigQuery Storage Write API (gRPC) has robust features, including exactly-once delivery semantics. To learn about limits and quotas, see [Storage Write API](https://cloud.google.com/bigquery/quotas#write-api-limits) and to see costs of using this API, see [BigQuery data ingestion pricing](https://cloud.google.com/bigquery/pricing#data_ingestion_pricing).
- To monitor the number of modified partitions on a table, use the [`INFORMATION_SCHEMA` view](https://docs.cloud.google.com/bigquery/docs/information-schema-jobs.md.txt#partitions-modified-by).
- For information about optimizing table load jobs to avoid reaching quota limits, see [Optimize load jobs](https://docs.cloud.google.com/bigquery/docs/optimize-load-jobs.md.txt).

### Maximum concurrent table and dataset metadata reads limit errors

BigQuery returns this error when too many queries concurrently
read the same table or concurrently query deprecated legacy dataset metadata
tables (`__TABLES__` or `__TABLES_SUMMARY__`).

**Error messages**

- `Exceeded rate limits: too many concurrent reads on a table for this table.
  For more information, see
  https://cloud.google.com/bigquery/docs/troubleshoot-quotas`
- `Exceeded rate limits: too many concurrent dataset meta table reads per
  dataset for this dataset. For more information, see
  https://cloud.google.com/bigquery/docs/troubleshoot-quotas`
- `Exceeded rate limits: too many concurrent dataset meta table reads per
  project for this project. For more information, see
  https://cloud.google.com/bigquery/docs/troubleshoot-quotas`

#### Resolution

To resolve this error, do the following:

- **Throttle concurrent reads and add jitter.** If an orchestrator (such as Apache Airflow or dbt) launches many parallel queries against the same table simultaneously, configure concurrency limits or add randomized start delays (jitter).
- **Enable [cached query results](https://docs.cloud.google.com/bigquery/docs/cached-results.md.txt).** Use cached results for repeated identical read queries against the same table.
- **Retry with exponential backoff.** Retry failed queries using truncated exponential backoff.
- **Distribute high-concurrency read workloads.** If a table must serve a high number of concurrent readers, distribute the load by creating [materialized views](https://docs.cloud.google.com/bigquery/docs/materialized-views-intro.md.txt) to serve different queries, maintaining separate table copies to distribute the read load, or [partitioning](https://docs.cloud.google.com/bigquery/docs/partitioned-tables.md.txt) and [clustering](https://docs.cloud.google.com/bigquery/docs/clustered-tables.md.txt) the table so that different queries read different partitions or clusters.
- **Replace legacy `__TABLES__` and `__TABLES_SUMMARY__` views with
  `INFORMATION_SCHEMA` views.** If you encounter the `concurrent dataset meta
  table reads` error, migrate queries from the deprecated `__TABLES__` and `__TABLES_SUMMARY__` views to [`INFORMATION_SCHEMA.TABLES`](https://docs.cloud.google.com/bigquery/docs/information-schema-tables) or [`INFORMATION_SCHEMA.PARTITIONS`](https://docs.cloud.google.com/bigquery/docs/information-schema-partitions).

### Maximum job creation rate limit errors

BigQuery returns this error when your workload creates new jobs
in a project and region faster than the per-project regional job creation rate
limit allows, including when queries using
[`jobCreationMode: JOB_CREATION_OPTIONAL`](https://docs.cloud.google.com/bigquery/docs/running-queries.md.txt#optional-job-creation)
fall back to creating standard BigQuery jobs.

**Error messages**

- `Exceeded rate limits: too many creating jobs for this project_and_region.
  For more information, see
  https://cloud.google.com/bigquery/docs/troubleshoot-quotas`
- `Exceeded rate limits: Your project_and_region exceeded quota for creating
  jobs. For more information, see
  https://cloud.google.com/bigquery/troubleshooting-errors`

#### Resolution

To resolve this error, do the following:

- **Retry with exponential backoff.** Catch HTTP `403` `rateLimitExceeded` errors in your client application and retry job submissions using truncated exponential backoff.
- **Combine individual DML statements.** Instead of submitting many separate single-statement DML jobs concurrently, group them into a [multi-statement query](https://docs.cloud.google.com/bigquery/docs/multi-statement-queries.md.txt).
- **Inspect and optimize `JOB_CREATION_OPTIONAL` fallback queries.** If your workload uses `JOB_CREATION_OPTIONAL`, query [`job_creation_reason.code` in `INFORMATION_SCHEMA.JOBS`](https://docs.cloud.google.com/bigquery/docs/information-schema-jobs.md.txt#optional-job-creation) to identify why queries fall back to creating a job (for example, `LARGE_RESULTS` or `LONG_RUNNING`), and optimize those queries to return smaller result sets or execute faster. Alternatively, if your workload consistently requires full job creation, set [`jobCreationMode`](https://docs.cloud.google.com/bigquery/docs/reference/rest/v2/jobs/query.md.txt#JobCreationMode) to `JOB_CREATION_REQUIRED` (the default) and pace your job submission rate.
- **Reduce slot contention or increase reservation capacity.** If queries running in a reservation fall back to `LONG_RUNNING` because of slot contention, smooth out query concurrency or [increase your reservation size](https://docs.cloud.google.com/bigquery/docs/reservations-tasks.md.txt#update_reservations).

### Table imports or query appends quota errors

BigQuery returns this error message when your table reaches the
limit for
table operations per day for standard tables. Table operations include the
combined total of all [load jobs](https://docs.cloud.google.com/bigquery/quotas.md.txt#load_jobs),
[copy jobs](https://docs.cloud.google.com/bigquery/quotas.md.txt#copy_jobs), and [query jobs](https://docs.cloud.google.com/bigquery/quotas.md.txt#query_jobs)
that append or overwrite a destination table.

To see the value of the **Table operations per day** limit, see [Standard
tables](https://docs.cloud.google.com/bigquery/quotas#standard_tables).

**Error message**

```
Your table exceeded quota for imports or query appends per table
```

When you encounter this error, [diagnose](https://docs.cloud.google.com/bigquery/docs/troubleshoot-quotas.md.txt#ts-table-import-quota-diagnose) the
issue, and then [follow the recommended steps](https://docs.cloud.google.com/bigquery/docs/troubleshoot-quotas.md.txt#ts-table-import-quota-resolution)
to resolve it.

#### Diagnosis

If you have not identified the source from where most table operations are
originating, do the following:

1. Make a note of the project, dataset, and table that the failed query, load,
   or the copy job is writing to.

2. Use `INFORMATION_SCHEMA.JOBS_BY_*` tables to learn more about jobs
   that modify the table.

   The following example finds the hourly count of jobs grouped by job type for
   the last 24-hour period using `JOBS_BY_PROJECT`. If you expect multiple
   projects to write to the table, replace `JOBS_BY_PROJECT` with
   `JOBS_BY_ORGANIZATION`.

   ```googlesql
   SELECT
   TIMESTAMP_TRUNC(creation_time, HOUR),
   job_type,
   count(1)
   FROM `region-REGION_NAME`.INFORMATION_SCHEMA.JOBS_BY_PROJECT
   WHERE creation_time > TIMESTAMP_SUB(CURRENT_TIMESTAMP, INTERVAL 1 DAY)
   AND destination_table.project_id = "my-project-id"
   AND destination_table.dataset_id = "my_dataset"
   AND destination_table.table_id = "my_table"
   GROUP BY 1, 2
   ORDER BY 1 DESC
   ```

#### Resolution

This quota cannot be increased. To resolve this quota error, do the following:

- If you frequently load data from multiple small files stored in Cloud Storage that uses a job per file, then combine multiple load jobs into a single job. You can load from multiple Cloud Storage URIs with a comma-separated list (for example, `gs://my_path/file_1,gs://my_path/file_2`), or by using wildcards (for example, `gs://my_path/*`).


  For more information, see
  [Batch loading data](https://docs.cloud.google.com/bigquery/docs/batch-loading-data.md.txt#permissions-load-data-from-cloud-storage).
- If you use load, select or copy jobs to append single rows of data to a table, for example, then you should consider batching multiple jobs into one job. BigQuery doesn't perform well when used as a relational database. As a best practice, avoid running frequent, single-row append actions.
- To append data at a high rate, consider using [BigQuery Storage Write API (gRPC)](https://docs.cloud.google.com/bigquery/docs/write-api). It is a recommended solution for high-performance data ingestion. The BigQuery Storage Write API (gRPC) has robust features, including exactly-once delivery semantics. To learn about limits and quotas, see [Storage Write API](https://cloud.google.com/bigquery/quotas#write-api-limits) and to see costs of using this API, see [BigQuery data ingestion pricing](https://cloud.google.com/bigquery/pricing#data_ingestion_pricing).
- To monitor the number of modified partitions on a table, use the [`INFORMATION_SCHEMA` view](https://docs.cloud.google.com/bigquery/docs/information-schema-jobs.md.txt#partitions-modified-by).
- For information about optimizing table load jobs to avoid reaching quota limits, see [Optimize load jobs](https://docs.cloud.google.com/bigquery/docs/optimize-load-jobs.md.txt).

### Quota exceeded for quota metric 'Authorized Views per dataset' and limit '2500'.

If a dataset's access control list exceeds the combined limit for authorized
resources, BigQuery returns this error message.

**Error message**

```
Quota exceeded for quota metric 'Authorized Views per dataset' and limit '2500'.
```

#### Resolution


A dataset's access control list can have up to 2,500 total authorized
resources, including
[authorized views](https://docs.cloud.google.com/bigquery/docs/authorized-views.md.txt),
[authorized datasets](https://docs.cloud.google.com/bigquery/docs/authorized-datasets.md.txt),
and
[authorized functions](https://docs.cloud.google.com/bigquery/docs/authorized-functions).

If you exceed this limit due to a large number of authorized views, consider grouping the
views into authorized datasets.

As a best practice, group related views into authorized datasets when you design new
BigQuery architectures, especially multi-tenant architectures.

### Too many DML statements outstanding against table

Error string: `Resources exceeded during query execution: Too many DML
statements outstanding against table
PROJECT_ID:DATASET.TABLE, limit is 20.`

This error means that the number of
[concurrent mutating DML statements](https://docs.cloud.google.com/bigquery/docs/data-manipulation-language.md.txt#update_delete_merge_dml_concurrency)
(`UPDATE`, `DELETE`, and `MERGE`), plus `TRUNCATE TABLE` statements, queued in
`PENDING` status against a single table has exceeded the
[data manipulation language (DML) quota limit](https://docs.cloud.google.com/bigquery/quotas.md.txt#data-manipulation-language-statements)
of 20. `INSERT` statements use a separate queue with a limit of 100.

#### Resolution

- Query the [`INFORMATION_SCHEMA.JOBS` view](https://docs.cloud.google.com/bigquery/docs/troubleshoot-queries.md.txt#too_many_DML) for `RUNNING` and `PENDING` jobs with `statement_type IN ('UPDATE', 'DELETE',
  'MERGE', 'TRUNCATE_TABLE')` to identify the jobs filling the table's DML queue.
- Group multiple smaller mutating DML operations into fewer, larger jobs---for example, by combining separate `UPDATE` statements into a single `UPDATE ...
  SET column = CASE WHEN ... END` statement, or by staging frequent updates in a separate table and periodically applying them to the target table with a single `MERGE` statement. See [Avoid DML statements that update or insert single rows](https://docs.cloud.google.com/bigquery/docs/best-practices-performance-compute.md.txt#avoid-dml-update-single-rows) and [Too many DML statements](https://docs.cloud.google.com/bigquery/docs/troubleshoot-queries.md.txt#too_many_DML).
- Partition or cluster the target table to improve DML execution speed so queued jobs drain faster. For more information, see [Best practices for DML statements](https://docs.cloud.google.com/bigquery/docs/data-manipulation-language.md.txt#best_practices).

### Loading CSV files quota errors

If you load a large CSV file using the `bq load` command with the
[`--allow_quoted_newlines` flag](https://docs.cloud.google.com/bigquery/docs/reference/bq-cli-reference.md.txt#flags_and_arguments_9),
you might encounter this error.

**Error message**

    Input CSV files are not splittable and at least one of the files is larger than
    the maximum allowed size. Size is: ...

#### Resolution

To resolve this quota error, do the following:

- Set the `--allow_quoted_newlines` flag to `false`.
- Split the CSV file into smaller chunks that are each less than 4 GB.

For more information about limits that apply when you load data into
BigQuery, see [Load jobs](https://docs.cloud.google.com/bigquery/quotas.md.txt#load_jobs).

### Cross-region dataset replica creation limit errors

BigQuery returns this error (`reason: apiLimitExceeded`) when you
configure [cross-region dataset replication](https://docs.cloud.google.com/bigquery/docs/data-replication.md.txt)
(`ALTER SCHEMA ... ADD REPLICA`) and exceed the maximum number of dataset
replicas that can be added for a region within a rolling time window.

**Error message**

```
API limit exceeded: Cannot add more than LIMIT replicas for a region within WINDOW day(s)
```

#### Diagnosis

This error frequently happens when an automated script or
Infrastructure-as-Code (IaC) tool repeatedly creates and drops dataset replicas
(`ADD REPLICA` and `DROP REPLICA`) in a loop.

#### Resolution

To resolve this error, do the following:

- **Audit automation and Infrastructure-as-Code scripts.** Inspect any Terraform, deployment scripts, or scheduled jobs managing dataset replicas to ensure they aren't repeatedly dropping and recreating replicas on every run.
- **Wait for the rolling window to elapse.** Wait until the rolling day window specified in the error message passes before running `ALTER SCHEMA ... ADD
  REPLICA` again for that region.

### Your user exceeded quota for concurrent `project.lists` requests

This error occurs when Microsoft Power BI jobs that communicate with
BigQuery through a Simba ODBC driver or DataHub fail because they
exceed the `project.list` API limit. To resolve this issue, use the short-term
or long-term workarounds described in this section.

**Error message**

    Your user exceeded quota for concurrent project.lists requests

#### Diagnosis

This error occurs during the connection and discovery phase for Power BI when
a Power BI report refreshes and the Simba driver establishes a connection to a
specific BigQuery project.

#### Short-term resolution

To address this issue in the short term, use the following workarounds, which
are ordered from most to least effective. Implement fixes three or four,
depending on whether you connect to BigQuery using the Simba
driver or using DataHub.

For long-term fixes, see [Long-term resolution](https://docs.cloud.google.com/bigquery/docs/troubleshoot-quotas.md.txt#long-term-resolution).

1. **Stagger report refreshes**. If you can't modify the DSN, mitigate the quota
   issue by reducing the number of simultaneous requests. Instead of having all
   reports refresh simultaneously (for example, at
   9:00 AM), stagger their schedules by a few minutes (for example, at 9:01 AM,
   9:03 AM, and 9:05 AM). This practice spreads out API calls over time, making
   it less likely that you reach the concurrent limit.

2. **Implement retries in Power BI**. This reactive strategy helps a report recover
   from a temporary failure. Power BI has built-in retry logic for data refresh
   failures. While this practice doesn't prevent the quota error, it makes your
   pipeline more resilient by allowing a report to succeed on a subsequent
   attempt after the initial spike in API calls subsides. To implement this
   fix, do the following:

   1. In the Power BI service, go to **Settings** for your dataset.
   2. Expand the **Scheduled refresh** section. Under **Retry**, configure Power BI to automatically rerun a failed refresh.
3. **For earlier versions of the Simba driver, specify the project ID in the ODBC
   connection** . This action prevents the driver from performing the
   `projects.list` discovery call. Instead, the driver connects directly to the
   specified project, which prevents unnecessary API calls and resolves the
   quota issue.

   Newer drivers fail immediately if the project isn't specified with a message
   similar to `Unable to establish connection with data source. Missing settings: {[Catalog]}`.

   To make this fix, do the following:
   1. On the machine running the Power BI Gateway or Power BI Desktop, open the **ODBC Data Sources (64-bit)** application.
   2. On the main setup screen for the Simba ODBC driver for BigQuery, fill the **Catalog (Project)** field with your specific Google Cloud project ID---for example, `my-gcp-project-id`.
4. **For earlier versions of DataHub, specify the project ID in the
   DataHub ingestion configuration**. Make this fix if you are using DataHub
   instead of the Simba driver. Similar to Simba, later versions of DataHub
   require you to specify the project ID or they won't connect to
   BigQuery.

   To avoid exceeding DataHub limits, modify your DataHub ingestion configuration
   to provide an explicit list of project IDs to scan. This prevents the DataHub
   configuration from finding all projects that the service account can see.

   In your BigQuery source recipe file (typically a YAML
   file), use the `project_ids` configuration to enumerate the projects that you
   want to ingest. Then, redeploy the DataHub ingestion recipe with the new
   configuration. See the following example and [this longer example](https://github.com/datahub-project/datahub/blob/master/metadata-ingestion/examples/recipes/bigquery_to_datahub.dhub.yaml)
   provided by DataHub.

   The following is an example of a DataHub configuration snippet:

   ```yaml
     source:
     type: "bigquery"
     config:
       # Instead of relying on discovery, explicitly list the projects.
       # This avoids the problematic projects.list() API call.
       project_ids:
       -  "YOUR_PRODUCTION_PROJECT_ID"
       -  "YOUR_ANALYTICS_PROJECT_ID"
       -  "ANOTHER_BQ_PROJECT"
   ```

#### Long-term resolution

The best long-term fix for this error message is to create separate, dedicated
Google Cloud service accounts for each function. For example, create a
service account for all Power BI reports and a service account for DataHub
ingestion.

This best practice isolates API usage into separate quota buckets and
prevents a high-load job in DataHub from causing critical business reports in
Power BI to fail.

Use the action plan in the following sections to resolve long-term quota errors
across Power BI and DataHub.

##### Phase 1: Preparation

1. Inform owners of the Power BI gateways and DataHub configuration that you will make coordinated changes to resolve ongoing job failures.
2. In the Google Cloud console, [create](https://docs.cloud.google.com/iam/docs/service-accounts-create.md.txt) two new service accounts---for example, `sa-powerbi-gateway@...` and `sa-datahub-ingestion@...`.
3. [Create](https://docs.cloud.google.com/iam/docs/keys-create-delete.md.txt) service account keys for the Power BI and DataHub service accounts.
4. [Grant](https://docs.cloud.google.com/iam/docs/manage-access-service-accounts.md.txt) each new service account least-privilege permissions by assigning the following Identity and Access Management roles that enable it to perform its tasks in relevant Identity and Access Management (IAM). Avoid assigning overly broad roles---for example, **ProjectEditor**.

###### Required roles

The service account for Power BI runs queries and reads data from tables. Grant
the following roles to service accounts in each Google Cloud project that
contains data that Power BI must access. To learn more about these roles, see
[BigQuery roles](https://docs.cloud.google.com/bigquery/docs/access-control.md.txt#bigquery-roles).

- [**BigQuery Data Viewer**](https://docs.cloud.google.com/bigquery/docs/access-control.md.txt#bigquery.dataViewer): provides read-only access to datasets, tables, and views.
- [**BigQuery Job User**](https://docs.cloud.google.com/bigquery/docs/access-control.md.txt#bigquery.jobUser): provides permissions to run jobs, including queries, which is essential for Power BI to execute its requests.

The service account for DataHub ingestion only needs to read metadata, such as
table names, schemas, and descriptions, not the data within the tables. Grant
the following role at the project level for each project that DataHub scans.
To learn more about these roles, see
[IAM roles for BigQuery](https://docs.cloud.google.com/iam/docs/roles-permissions/bigquery.md.txt).

[**BigQuery Metadata Viewer**](https://docs.cloud.google.com/bigquery/docs/access-control.md.txt#bigquery.metadataViewer): this role is designed specifically to read metadata. It grants permissions to
list datasets and tables and view their metadata without granting access to the
underlying data.

##### Phase 2: Coordinated rollout

During a low-usage period, the Power BI administrator updates the ODBC DSN
configurations on the gateway machines by performing the following steps:

1. Changes the authentication method to use the new `sa-powerbi-gateway@...` service account key created in a previous step.
2. If not already performed as a short-term fix, enters the Google Cloud project ID in the **Catalog (Project)** field of the ODBC driver.
3. Has the DataHub owner update the BigQuery source configuration YAML file.
4. Points to the new `sa-datahub-ingestion@...` service account key created in a previous step.
5. If not already performed as a short-term fix, uses the `project_ids` parameter to explicitly list the projects to be scanned.
6. Redeploys the DataHub ingestion recipe with the new configuration.

##### Phase 3: Verification and monitoring

To verify and monitor the effects of the fixes, the Power BI and DataHub
administrators perform the following steps:

1. Manually trigger a refresh for a few key Power BI reports and a new ingestion run in DataHub. Confirm that these jobs complete successfully without incurring quota errors.
2. To ensure the `Quota exceeded: Your user exceeded quota for concurrent
   project.lists requests` error no longer occurs, monitor your application logs or business intelligence tool run histories.
3. Optional: verify that the overall volume of `projects.list` API calls or
   related `403` errors has decreased for the associated service accounts:

   1. In the Google Cloud console, go to the BigQuery API **API/Service
      Details** page

      [Go to APIs \& services](https://console.cloud.google.com/apis/api/bigquery.googleapis.com/metrics)
   2. View the dashboards on the **Metrics** tab to verify the decrease in call
      volume and related 403 errors.

Administrators should see a dramatic and permanent drop in the usage of this
specific API call, confirming that the fix was successful.

### Job exceeded rate limits

Error string:
`Job exceeded rate limits: Your SCOPE exceeded quota for QUOTA_METRIC. For more information, see https://cloud.google.com/bigquery/docs/troubleshoot-quotas`

This HTTP `400 Bad Request` error (`reason: jobRateLimitExceeded`) occurs when a
job is created, and then fails during execution with a `rateLimitExceeded`
error---for example, when the job exceeds the
[table metadata update rate limit](https://docs.cloud.google.com/bigquery/docs/troubleshoot-quotas.md.txt#ts-maximum-update-table-metadata-limit) or
the [concurrent table and dataset metadata read limit](https://docs.cloud.google.com/bigquery/docs/troubleshoot-quotas.md.txt#ts-concurrent-table-and-meta-reads).
BigQuery returns `jobRateLimitExceeded` when you call
`jobs.query` or `jobs.getQueryResults` for that job, or when a synchronous query
isn't admitted because queuing is disabled (see
[Query admission denied or continuous query slot limits](https://docs.cloud.google.com/bigquery/docs/troubleshoot-quotas.md.txt#ts-query-admission-denied)).

To resolve this error:

- Use [truncated exponential backoff](https://docs.cloud.google.com/monitoring/api/troubleshooting.md.txt#retry) to reduce the request rate, and then retry the job with a **new `jobId`** . Calling `jobs.getQueryResults` again on the failed `jobId` returns the same error.
- Ensure your client retry logic handles HTTP `400` responses with `reason: jobRateLimitExceeded`, because retry policies that check only for `429` or `5xx` status codes skip this error.
- Inspect `<var>QUOTA_METRIC</var>` and `<var>SCOPE</var>` in the error message and apply the mitigation for that specific quota on this page.