Migrate from Vertex AI Feature Store (Legacy) to Bigtable

Migrating a machine learning feature management workload from Vertex AI Feature Store (Legacy) to Bigtable can result in improved performance and flexibility. This guide provides an overview of related concepts and the migration process.

Vertex AI Feature Store (Legacy) is a managed environment that uses Bigtable for its online serving layer. Running your AI platform or feature store directly on Bigtable without using Vertex AI Feature Store (Legacy) can result in faster speeds and reduced costs.

We recommend a minimal viable migration path focused on migrating data from the underlying Bigtable tables in Vertex AI Feature Store (Legacy) to a Bigtable instance that you create in your Google Cloud project.

Benefits of migration

Migration to Bigtable offers several strategic and operational advantages:

  • Cost efficiency: You eliminate the Vertex AI Feature Store (Legacy) specific node management premium, often reducing infrastructure costs.
  • Direct control: You gain full access to Bigtable capabilities. Bigtable monitoring exposes many more metrics than Vertex AI Feature Store (Legacy). You also have greater control over custom architecture layouts and scaling.
  • High performance: Bigtable supports high-performance workloads and high-performance features such as aggregation on writes and vector search.

  • Cross-product integration: You get access to Bigtable integrations such as BigQuery external tables, connectors for Apache Spark, Apache Flink, and Kafka Connect, and reverse ETL from BigQuery.

  • Change data capture: You can enable change streams to capture changes to your Bigtable feature store table as the changes happen.

Key concepts

This section describes how core Vertex AI Feature Store (Legacy) concepts are implemented by Bigtable and BigQuery.

Data retention

In Bigtable, you manage data retention with garbage collection. Garbage collection is the automatic, ongoing process of removing expired and obsolete data from Bigtable tables. A garbage collection policy is a set of rules you create that state when data in a specific feature, defined in Bigtable as a column family, is no longer needed. Garbage collection policies are set based on the timestamp associated with the data or the number of versions you want to keep.

Garbage collection is a built-in, asynchronous background process that takes place during compaction. Garbage collection occurs on a fixed schedule. Until data is deleted, it appears in read results, but you can filter your reads to exclude this data. For more information, see Garbage collection overview.

Additionally, Bigtable tiered storage can be a cost-effective solution for online feature stores that need to retain historical data for model training or regulatory compliance. Tiered storage manages the movement of infrequently accessed data from online serving on SSD storage to a lower-cost storage tier.

Feature development

In Bigtable, you can implement online feature development with Bigtable SQL, and you can implement offline feature development using BigQuery DataFrames.

When you use Vertex AI Feature Store (Legacy), you work with developer APIs and data models that map to underlying data sources prepared in BigQuery. You then register these data sources and specific feature columns in the feature registry. With a Bigtable feature store, you work directly with data in the underlying BigQuery and Bigtable instances, without needing to map to the Vertex AI Feature Store (Legacy) data model.

Online feature development

For online feature development, Bigtable offers several tools:

Offline feature development

For offline feature development, BigQuery DataFrames provides a Python interface with more than 750 pandas and scikit-learn APIs. These APIs are implemented through SQL conversion to BigQuery and BigQuery ML APIs. BigQuery DataFrames feature generation allows for both built-in and user-defined Python functions. It also provides automated data synchronization to Bigtable for serving features created in batch and offline processes, described in the next section.

Online and offline feature synchronization

When using Bigtable directly for ML workloads, you can ensure that an offline feature value is imported from BigQuery and that same value is reused for both training and serving, keeping code paths in sync for generating features between training and serving. The following technologies enable feature synchronization:

  • Batch synchronization: Reverse ETL from BigQuery to Bigtable provides the ability to export the results of a BigQuery query into Bigtable. These queries are run in batch and can be scheduled directly from BigQuery.
  • Streaming synchronization: BigQuery continuous queries are SQL statements that continuously run and output rows into a Bigtable table.
  • Synchronization from BigQuery DataFrames: To capture offline features developed in Python, you can use a BigFrames StreamingDataFrame to generate a BigQuery continuous query that captures your Python logic for feature generation and synchronizes the data results with Bigtable.
  • Offline feature development directly on Bigtable data: You can build offline features in BigQuery against data stored in Bigtable by using a BigQuery external table. An external table mirrors the look of a BigQuery table and offers most of the same functionality such as joins, scheduled queries, and advanced BigQuery SQL functions without the need to move data back into BigQuery storage. To avoid impacting your application-serving traffic, you can use Data Boost serverless compute when you read Bigtable data with BigQuery external tables. Using Data Boost is especially cost-effective for ad hoc queries. To use Data Boost, specify a Data Boost app profile when you create the external table definition. For more information about Data Boost, see Bigtable Data Boost overview.

After migration, you can continue using Vertex AI Model Monitoring to track the quality of your models.

The practice of using Bigtable and BigQuery together is a common pattern for building real-time analytics databases.

Migration phases

To ensure service continuity, migration is typically executed in the following distinct phases.

Phase 1: Prepare the infrastructure

Before beginning your migration, set up the destination environment:

Phase 2: Define your schema mapping between Vertex AI Feature Store (Legacy) and Bigtable

  1. Review and understand Bigtable schema design best practices. A general mapping of the Vertex AI Feature Store (Legacy) API to the Bigtable API is the following:

    Vertex AI Feature Store (Legacy) resource

    Bigtable component

    FeatureOnlineStore

    Bigtable instance

    FeatureView

    Column family

    featureValues (batch)

    Column (single cell per key)

    featureValues (continuous)

    Column (multiple cells per key [versioning])

  2. After you've defined your schema mapping, create a Bigtable table that has a column family for each feature in the source feature store.

Phase 3: Data extraction and synchronization

In this phase, you migrate data using a tiered approach based on the update frequency of the data.

Real-time feature synchronization

For features that you are writing with write_feature_values or equivalent API calls, begin writing the same data to the new Bigtable table.

  1. Install the Python client library for Bigtable.
  2. Configure your application to simultaneously write feature data to Vertex AI Feature Store (Legacy) and Bigtable. For more information about writing data to Bigtable, see Writes.

Batch feature migration

Next, migrate the data that was stored before you began dual writes. This involves moving the data from Vertex AI Feature Store (Legacy) to BigQuery and then to Bigtable.

  1. Export feature store data to BigQuery using Vertex AI Feature Store (Legacy) exporting capabilities, which let you export all values or snapshots. This lets BigQuery serve as the Vertex AI Feature Store (Legacy) offline store.
  2. Migrate the historical data from BigQuery to Bigtable using one of the following:
    1. Reverse ETL
    2. Bigtable Spark connector
    3. BigQuery to Bigtable Dataflow template

Phase 4: Application and SDK transition

The final step is cutting over the application layer.

  1. After the migration has been completed and tested, stop writing to Vertex AI Feature Store (Legacy).

  2. Modify your application to use only the Python client library for Bigtable.

    The following example demonstrates using Python to pull a single feature from Bigtable.

    from google.cloud import bigtable
from google.cloud.bigtable import row_filters
# Replace 'project_id' and 'instance_id' with your actual IDs.
client = bigtable.Client(project=project_id)
instance = client.instance(instance_id)

#return only the latest feature
row_filter = bigtable.row_filters.CellsColumnLimitFilter(1)

# Replace 'user1' and 'feature0` with your actual row key and column qualifier.

print("Getting a single feature by row key.")
key = "user1".encode()

row = table.read_row(key, row_filter)
cell = row.cells[column_family_id.decode("utf-8")][feature0][0]
print(cell.value.decode("utf-8"))

    For another example of how to read and write data using the Bigtable data and admin APIs, see the Python hello world.

    The Python client library for Bigtable also lets you use GoogleSQL to return features that meet your filter criteria or to perform transformations of the features. The following example shows how to call a SQL query asynchronously from the Bigtable Python client library. For additional information on GoogleSQL for Bigtable, see Additional SQL examples.

    import asyncio
from google.cloud.bigtable.data_async import BigtableDataClient
from google.cloud.bigtable_v2.types import ExecuteQueryRequest

async def run_bigtable_sql_query(project_id, instance_id, table_id):
    """
    Runs a GoogleSQL query on a Bigtable table using the async client.
    """
    client = BigtableDataClient(project_id=project_id)
    instance = client.instance(instance_id)
    table = instance.table(table_id)

    # Example query: Select a specific row and all columns from a column family
    # Replace 'my_table' and 'my_cf' with your actual table and column family IDs.
    # The table name in the SQL must be in the format `dataset.table`,
    # where dataset is the instance ID and table is the table ID (in backticks).
    sql_query = f"SELECT _key, my_cf FROM `{instance_id}`.`{table_id}` WHERE _key = 'user_123'"

    print(f"Executing query: {sql_query}")

    # The client library automatically handles the SQL execution
    try:
        # The query method returns an AsyncPartialRowsIterator
        results_iterator = await table.query(query=sql_query)

        async for row in results_iterator:
            print(f"Row key: {row.row_key.decode('utf-8')}")
            # Iterate through the cells in the row
            for col_family, cells in row.cells.items():
                for cell in cells:
                    print(f"  Column Family: {col_family}, Qualifier: {cell.qualifier.decode('utf-8')}, Value: {cell.value.decode('utf-8')}, Timestamp: {cell.timestamp_micros}")

    except Exception as e:
        print(f"An error occurred: {e}")
    finally:
        await client.close()

if __name__ == "__main__":
    # TODO(developer): Replace with your project, instance, and table IDs
    your_project_id = "your-gcp-project-id"
    your_instance_id = "your-bigtable-instance-id"
    your_table_id = "your-bigtable-table-id"

    # Run the asynchronous function
    asyncio.run(run_bigtable_sql_query(your_project_id, your_instance_id, your_table_id))

  3. Begin using Bigtable metrics to monitor latency and throughput. For more information, see Monitoring.

Best practices

After you move from Vertex AI Feature Store (Legacy) to a Bigtable feature store implementation, you need to replicate the internal preprocessing and optimization logic that was previously handled by the service to maintain stability and performance.

Client-side adaptive throttling

The Vertex AI Feature Store (Legacy) backend utilizes a client-side adaptive throttler to protect its underlying Bigtable instances from being overloaded during spikes in traffic or when the storage backend is experiencing high latency or errors. We recommend that you implement a similar throttler in your application code to register backend responses and proactively throttle requests when needed.

Request partitioning and batch size optimization

Bigtable row filters have a hard limit of 20 KB. Requesting too many features or entity IDs in a single filtered read can cause requests to fail. To mirror the behavior of Vertex AI Feature Store (Legacy), do the following:

  • Chunk feature IDs: Limit the number of feature IDs per Bigtable read to approximately 100.
  • Balance entity batches: To prevent saturating client or server resources when performing multi-entity reads, take the following precautions:
    • Partition entities into small concurrent batches, such as 10 entities per batch.
    • Limit the maximum number of concurrent batch requests, such as 10-20.

Intelligent filter selection

Calculating and applying column filters on the server side adds overhead. If your application typically requests nearly all features in a column family, such as >99.9%, it's more efficient to skip the column filter and read the full row, filtering the results client-side.

Concurrency and asynchronous execution

To minimize time-to-first-result in streaming scenarios, use asynchronous patterns or thread bundles to fetch entity batches in parallel. This ensures that the application can begin processing results as soon as the first batch returns, rather than waiting for a large serial read to complete.

What's next