This page lists the metrics related to the vector indexes that you generate in AlloyDB for PostgreSQL. You can view these metrics using the pg_stat_ann_indexes and pg_stat_ann_indexes_creation views that are available when you install the alloydb_scann extension.
For more information about viewing the metrics, see View vector index metrics.
Usability metrics
The usability metrics include metrics that help you understand the state of index utilization with metrics such as, index configuration and number of index scans.
| Metric name | Data type | Description |
|---|---|---|
relid |
OID |
Unique identifier of the table that contains the vector index. |
indexrelid |
OID |
Unique identifier of the vector index. |
schemaname |
NAME |
Name of the schema to which index belongs. |
relname |
NAME |
Name of the table that contains the index. |
indexrelname |
NAME |
Name of the index. |
indextype |
NAME |
Type of the index. This value is always set to scann. |
indexconfig |
TEXT[] |
Configuration, such as leaves count and quantizer, defined for the index when it was created. |
indexsize |
TEXT |
Size of the index. |
indexscan |
BIGINT |
Number of index scans initiated on the index. |
partitioncount |
BIGINT |
Number of partitions (leaf nodes) in the tree. |
Tuning metrics
Tuning metrics provide insights into your current index optimization, allowing you to apply recommendations for faster query performance.
| Metric name | Data type | Description |
|---|---|---|
insertcount |
BIGINT |
Number of insert operations on the index. This metric also includes any number of rows that existed before the index was created. |
updatecount |
BIGINT |
Number of update operations on the index. This metric doesn't take into account any HOT updates. |
deletecount |
BIGINT |
Number of delete operations on the index. |
distribution |
JSONB |
Vector distributions across all partitions for the index. The following fields show the distribution:
Note: Due to the inherent characteristics of the K-means clustering algorithm, there will always be some degree of variance in the distribution of vectors across partitions, even when the index is initially created. |
distributionpercentile |
JSONB |
Vector index distribution helps you understand the distribution of vectors between partitions of your ScaNN index. The partitions are created based on num_leaves value defined during index creation.Vector index distribution contains buckets for 10, 25, 50, 75, 90, 95, 99, and 100th percentiles. Each bucket contains the following values:
Note: Due to the inherent characteristics of the K-means clustering algorithm, there is always some degree of variance in the distribution of vectors across partitions, even when the index is initially created. |
Index creation metrics
This section lists the metrics available at the time of index creation for the vector indexes that you generate in AlloyDB. You can view these metrics using the pg_stat_ann_index_creation view that is available when you install the alloydb_scann extension.
Index creation metrics help you understand the state of the index when you created it using metrics like index_rows_at_creation_time.
This information can be used to analyze performance and recall if there is huge difference between index_rows_at_creation_time and current number of rows in index. This information can also be used to decide whether to change the predicate, if any used to create index and to rebuild an index if the index contains too few rows.
| Metric name | Data type | Description |
|---|---|---|
relid |
OID |
Unique identifier of the table that contains the vector index. |
indexrelid |
OID |
Unique identifier of the vector index. |
schemaname |
NAME |
Name of the schema to which the index belongs. |
relname |
NAME |
Name of the table that contains the index. |
indexrelname |
NAME |
Name of the index. |
index_rows_at_creation_time |
BIGINT |
Number of rows in the index at the time of index creation. |
Tuning recommendation based on the metrics
Your index is made of many leaves. When you create an index, it is created with a specific number of leaves and with vectors present in the table. The selection of leaves and the distribution of vectors into different leaves depends on the vectors present in the table at index build time. A clustering algorithm is used to distribute the vectors across leaves.
Due to the nature of clustering algorithms, it is normal for this process to create an uneven distribution of vectors, with some leaves having 5-10x more vectors than others.
Over time, as you insert, update, and delete vectors, the imbalance can become extreme. Some leaves can become nearly empty, while others become over-populated with vectors.
When queries search these imbalanced leaves, your search speed (QPS) and accuracy (recall) can drop. When you experience slowness and accuracy drop, consider checking the metrics and potentially reindexing.
The insertcount, updatecount, and deletecount metrics show you that your data is changing which in turn changes your index structure. The distribution and distributionpercentile metrics show you how these data changes affect the index's structure.
Use your distribution metrics to spot these common problems.
High data mutation
High data mutation occurs when a high volume of INSERT, UPDATE, or DELETE operations cause the index's original structure to become outdated and inefficient for the current data.
How to Check?
Compare the mutation metrics from pg_stat_ann_indexes against the current row count of your index and the row count at creation time. The following example code shows how to verify if high data mutation has occurred:
Retrieve the current row count of your index:
Replace INDEX_NAME with the name of your ScaNN index.SELECT reltuples FROM pg_class WHERE relname='INDEX_NAME';
Retrieves the cumulative counts of write operations performed on your vector index:
SELECT insertcount, updatecount, deletecount from pg_stat_ann_indexes;
Retrieve the number of vectors that were in the table at the exact time the index was built.
SELECT index_rows_at_creation_time from pg_stat_ann_index_creation;
Based on the result of the queries, observe the following. The index is considered outdated if any of these conditions are met, especially if you also observe a drop in QPS or recall:
- If auto-maintenance is not enabled - The current row count is more than double (2x) of the row count derived using
index_rows_at_creation_time. - If auto-maintenance is enabled - The current row count is more than triple (3x) the row count derived using
index_rows_at_creation_time. - The current row count is less than half (x/2) of the row count derived using
index_rows_at_creation_time. - The total mutations (
insertcount+updatecount+deletecount) is more than double (2x) the row count derived usingindex_rows_at_creation_time.
How to resolve high data mutation?
If you observe that vectors have changed significantly in the index, you may need to increase or decrease the number of leaves and levels to ensure that the leaves are not overpopulated or empty.
If the mutations did not change the vector count, only updated the vectors, you don't need to re-tune the index and only REINDEX should suffice. For more information about how to decide num_leaves and max_num_levels values, see Tune a ScaNN index. You can also use AUTO mode index which decides the best possible index configuration and does not need manual tuning.
- AUTO Mode Index
- Manual Mode Index
REINDEX INDEX CONCURRENTLY INDEX_NAME;
ALTER INDEX SET (num_leaves=NUM_LEAVES, max_num_levels=MAX_NUM_LEVELS);
REINDEX INDEX CONCURRENTLY INDEX_NAME;
num_leaves_to_search and pct_leaves_to_search may need to be re-tuned if the num_leaves have changed, see Tune a ScaNN index.
Skewed index due to few leaves being over-populated
Data mutations are skewed, which means that a few leaves have become hotspots and are significantly larger than the others.
How to Check?
To check for over-populated leaves, examine the distributionpercentile using the following command:
SELECT distributionpercentile FROM pg_stat_ann_indexes;
You can observe that some variance is normal and expected when the index is initially created, for example, a 5-10x ratio between max and median value is expected. The index is considered skewed when this ratio becomes extreme, for example if the ratio between max and median value becomes 50-100x. These few over-populated leaves might be slowing down the query response.
How to resolve a skewed index?
To resolve a skewed index, you must REINDEX your index using the following command.
REINDEX INDEX CONCURRENTLY INDEX_NAME;
This reindexing re-balances the index, taking the vectors from the over-populated leaves and distributing them more evenly.