This document lists the quotas and system limits that apply to Lakehouse.
- Quotas have default values, but you can typically request adjustments.
- System limits are fixed values that can't be changed.
Google Cloud uses quotas to help ensure fairness and reduce spikes in resource use and availability. A quota restricts how much of a Google Cloud resource your Google Cloud project can use. Quotas apply to a range of resource types, including hardware, software, and network components. For example, quotas can restrict the number of API calls to a service, the number of load balancers used concurrently by your project, or the number of projects that you can create. Quotas protect the community of Google Cloud users by preventing the overloading of services. Quotas also help you to manage your own Google Cloud resources.
The Cloud Quotas system does the following:
- Monitors your consumption of Google Cloud products and services
- Restricts your consumption of those resources
- Provides a way to request changes to the quota value and automate quota adjustments
In most cases, when you attempt to consume more of a resource than its quota allows, the system blocks access to the resource, and the task that you're trying to perform fails.
Quotas generally apply at the Google Cloud project level. Your use of a resource in one project doesn't affect your available quota in another project. Within a Google Cloud project, quotas are shared across all applications and IP addresses.
For more information, see the Cloud Quotas overview.
There are also system limits on Lakehouse resources. System limits can't be changed.
Get quota and system limit metric names
Quotas and system limits have two types of names: display names and metric names. Display names have spaces and capitalization that make them easier for humans to read. Metric names are more likely to be lowercase and delimited by underscores instead of spaces; the exact format depends on the service.
The following instructions show how to get metric names for quotas and system limits by using either the Google Cloud console or the Google Cloud CLI.
Console
In the Google Cloud console, go to the Identity and Access Management (IAM) & Admin > Quotas & System Limits page:
The table on this page displays quotas and system limits that have usage or have adjusted values, and a reference entry for other quotas. The reference entry has the word "default" in parentheses at the end of the listing in the Name column. For example,
SetIAM Policy requests per minute per region (default)is the reference entry for the quotaSetIamPolicyRequestsPerMinutePerProject.If you don't see the Metric column, take the following steps.
- Click Column display options.
- Select Metric.
- Click OK. The Metric column appears in the table.
The Metric column shows the metric names. To filter the results, enter a property name or value in the field next to Filter.
gcloud
To get the metric names for a service by
using the gcloud CLI, run the quotas info list
command. To skip lines that don't list metric names, pass the output to a
command such as grep with metric: as the search term, or use the
gcloud CLI
--format flag:
gcloud quotas info list --project=PROJECT_ID_OR_NUMBER \
--service=SERVICE_NAME --format="value(metric)"
Replace the following:
PROJECT_ID_OR_NUMBER: the project ID or project number.SERVICE_NAME: the name of the service whose quota metrics you want to see—for example, the service name for Compute Engine iscompute.googleapis.com. Include thegoogleapis.comportion of the service name.
Request a quota increase
To adjust most quotas, use the Google Cloud console. For more information, see Request a quota adjustment.
Quotas
The following quotas apply to Lakehouse:
Lakehouse runtime catalog
The following limits apply to Lakehouse runtime catalog:
| Limit | Default | Notes |
|---|---|---|
| Table modifications per day | 1,500 modifications | Your project can make up to 1,500 table modifications per table per day. A load job, copy job, or query job that appends or overwrites table data counts as one modification to the table. This limit cannot be changed. DML statements are excluded and don't count toward the number of table modifications per day. Streaming data is excluded and doesn't count toward the number of table modifications per day. |
| Maximum rate of table metadata update operations per table | 5 operations per 10 seconds |
Your project can make up to five table metadata update operations per 10 seconds
per table. This limit applies to all table metadata update operations,
performed by the following:
DELETE, INSERT, MERGE,
TRUNCATE TABLE, or UPDATE statements to write
data to a table. Note that while DML statements count toward this limit, they are not subject to it if it is reached. DML operations have dedicated rate limits.
If you exceed this limit, you get an error message like
To identify the operations that count toward this limit, you can Inspect your logs. Refer to Troubleshoot quota errors for guidance on diagnosing and resolving this error. |
| Maximum number of columns per table | 10,000 columns | Each table, query result, or view definition can have up to 10,000 columns. This includes nested and repeated columns. |
Cross-cloud data access
| Quota | Value |
|---|---|
| Maximum total number of namespaces in a federated catalog | 100 namespaces |
| Maximum number of tables per namespace in a federated catalog | 300 tables per namespace |
| Maximum total number of tables in a federated catalog | 10,000 tables |
| Maximum number of namespace filters in a federated catalog | 16 namespace filters |
System limits
The following table lists the system limits that apply to Lakehouse and the value for each system limit:
| System limit | Value |
|---|---|
| Maximum number of Apache Iceberg tables managed by Lakehouse per Cloud Storage bucket that a discovery scan supports | 1,000 tables per bucket |