AI indexes overview
AI indexes let you perform natural language searches across table columns
without manually generating embeddings or managing models. You can use the
AI.SEARCH
function to query your data while BigQuery automatically
handles column selection, index optimization, and ongoing maintenance.
About AI indexes
An AI index is a managed, self-optimizing indexing abstraction that lets you use
natural language search across one or more columns in your
BigQuery tables with the AI.SEARCH function.
Use an AI index when you want natural-language search across one or more text
columns without managing embedding models, embedding columns, or vector
indexes. If you need a specific embedding model, or custom vector index options,
generate embeddings and
create a vector index.
BigQuery AI indexes help you perform the following tasks more efficiently:
- Perform natural language searches without managing models: you can search across your data using natural language queries without generating embeddings or configuring vector indexes.
- Automatic column selection: when you create an AI index with the
AUTO COLUMNSoption, BigQuery automatically identifies and selects candidate columns in the table to index, or lets you specify an explicit list of columns to include. - Adaptive indexing: BigQuery automatically determines and applies the optimal indexing and retrieval strategies for your data and queries.
- Fully managed lifecycle: as your table data changes, BigQuery automatically manages index creation, maintenance, updates, and refreshes, without requiring you to manage underlying AI artifacts.
Workload management
By default, BigQuery uses on-demand slots and the shared slot pool to handle the background processing required to build, refresh, and optimize an AI index. To ensure predictable and consistent indexing performance on larger production workloads, or to create an AI index on a table with more than 1,000,000 rows, do the following:
- Create a reservation in the region where your table is located
- Assign your project to the reservation with the
job_typeset toBACKGROUND.
When a BACKGROUND reservation is present,
BigQuery uses it to run the background jobs that maintain the
AI index.
Pricing
AI index costs fall into the following categories:
- DDL statements: there are no additional charges for running the
CREATE AI INDEXandDROP AI INDEXstatements. - Background index data processing (
gc_jobs): all data processing runs directly within BigQuery using BigQuery slots. By default, BigQuery uses on-demand slots to handle these jobs, and the table's project is billed following the DML on-demand billing model. If aBACKGROUNDreservation is assigned to the project, BigQuery uses theBACKGROUNDreservation to run these background DML jobs, and the reservation is billed for the slot time usage. To learn more, see Track background job costs. - Embedding generation: embedding generation for AI indexes, including the
embeddings generated for
AI.SEARCHqueries on tables with an AI index, is billed at the published rate of the underlying model that the AI index uses. Google might update the default model used by AI indexes over time for model lifecycle or performance improvements, and bill at the published BigQuery pricing for the model. - Index training and optimization: there are no additional charges for the processing required to train and optimize the index structures when you use the default shared slot pool, as long as the total size of the indexed table data is below your organizational limit.
- Storage: AI indexes incur storage costs while they exist on the table. You
can view the storage size of an AI index in the
total_logical_bytescolumn of theINFORMATION_SCHEMA.AI_INDEXESview. - Queries: queries that use the
AI.SEARCHfunction are billed at standard BigQuery compute pricing. Embedding generation for the query is billed as described previously in this list.
Limitations on AI indexes
When you use AI indexes, the following limitations apply:
- If the combined text of a row's indexed columns is very long, such as full
documents or many long
STRINGcolumns, only part of the text is used for search. To ensure the most informative fields are used, specify an explicit column list inCREATE AI INDEXwith the highest-signal columns listed first. - Queries that require exact matching on identifier values with no natural
language meaning, such as IDs, UUIDs, hashes, or
system codes, aren't supported. By default,
AUTO COLUMNSincludes all top-level STRING columns without filtering out opaque ID columns. To exclude opaque ID columns from the AI index, specify an explicit column list inCREATE AI INDEX. - Queries that contain strict numeric constraints or require structured
filtering on non-
STRINGcolumns aren't supported by natural-language search text alone. Apply structured or numeric filters using aWHEREclause in a pre-filter subquery instead. - Search quality might degrade if the AI index includes columns with text that
doesn't describe the row, such as raw logs, stack traces, URLs, or
machine-generated strings. To exclude these columns, list the columns to
index instead of using
AUTO COLUMNS. CREATE OR REPLACE AI INDEXandALTER AI INDEXstatements aren't supported. You also can't drop, rename, or change the data type of a source column included in an AI index. To change indexed columns, drop the AI index and create it again.The following table operations aren't supported on tables with AI indexes. To perform these operations, drop the AI index.
- Create a table from the table's schema (
CREATE TABLE ... LIKE). - Copy the table (
bq cporCREATE TABLE ... COPY), including cross-region copies, or copying data into an existing table that has an AI index. - Clone the table (
CREATE TABLE ... CLONE). - Create a table snapshot (
CREATE SNAPSHOT TABLE). - Rename the table (
ALTER TABLE ... RENAME TO). - Restore the table (
RESTORE TABLE).
- Create a table from the table's schema (
What's next
- Learn how to use AI indexes.
- Learn how to search tables using the
AI.SEARCHfunction.