Build and optimize context sets with the context engineering agent

To achieve high QueryData query accuracy and low online latency in your data agent applications, you can use the context engineering agent to build, evaluate, and optimize context sets. By automating context creation, upload, and optimization, the agent helps your applications return fast, deterministic, and explainable answers to natural language questions.

To learn about context sets and QueryData, see Context sets overview and QueryData overview.

By providing QueryData with context tailored to your business application, you supply the precise business rules that the system needs to resolve nuanced user intent and deliver near-100% query quality. As you work in your CLI or IDE, the context engineering agent automatically uploads your context sets to Google Cloud while saving local files in your workspace for version control.

Context engineering agent

The context engineering agent acts as your design-time assistant directly in your CLI or IDE. Manual context authoring requires writing detailed query patterns by hand, anticipating every phrasing variation, and manually diagnosing query failures across iterations. The context engineering agent eliminates this overhead by bootstrapping baseline context, curating ground-truth evaluation datasets, iteratively hill-climbing to fix accuracy gaps, and proving that your context generalizes across new questions.

You converse with the agent by using natural language to drive the optimization workflow. Start with a prompt like "Optimize the context set against my application requirements."

Before you begin

Before you use the context engineering agent, complete the prerequisites in the following sections. You can create and store context sets in the us-central1, us-east1, europe-west4, and asia-southeast1 regions, even if your database instance is in a different region.

Enable required services

Enable the following services for your project:

Prepare your database

Prepare your Cloud SQL for MySQL instance and configure access for the agent:

Required roles and permissions

To get the permissions that you need to use the context engineering agent and manage context sets, configure the following Identity and Access Management (IAM) roles and database privileges:

  • Add an IAM user or service account to the instance. For more information, see Add an individual IAM user or service account to a Cloud SQL instance.
  • Grant the roles/cloudsql.instanceUser (or roles/cloudsql.studioUser if you also test queries in Cloud SQL Studio), roles/serviceusage.serviceUsageConsumer, roles/geminidataanalytics.queryDataUser, and roles/mcp.toolUser roles to the IAM user or service account at the project level. For more information, see Add IAM policy binding for a project.
  • To create, upload, and manage context sets, grant the Dataplex Catalog Editor (roles/dataplex.catalogEditor) or Dataplex Catalog Admin (roles/dataplex.catalogAdmin) role to the IAM user or service account at the project level. For more information, see Context sets permissions.

  • Have a privileged user grant read-only database privileges (and execution privileges on mysql.ml_embedding if you use semantic value searches) to the IAM user or service account:

    GRANT SELECT ON *.* TO "IAM_USERNAME";
GRANT EXECUTE ON FUNCTION mysql.ml_embedding TO "IAM_USERNAME";

    Replace IAM_USERNAME with the database username of the IAM user or service account (the email address without the @ and domain name). For more information, see Grant database privileges to an individual IAM user or service account.

Prepare the database for value searches

To use semantic and trigram value searches, you must configure your Cloud SQL for MySQL instance to support vector embeddings and n-gram indexing.

  1. Enable the Vertex AI API and Compute Engine API, and grant the roles/aiplatform.user role to your Cloud SQL for MySQL instance's service account. For more information, see Enable database integration with Vertex AI.

  2. Enable the cloudsql_vector and ngram_token_size database flags and Vertex AI integration on your Cloud SQL for MySQL instance:

    gcloud sql instances patch INSTANCE_NAME \
    --enable-google-ml-integration \
    --database-flags=cloudsql_iam_authentication=on,cloudsql_vector=on,ngram_token_size=3

    Replace INSTANCE_NAME with the name of your Cloud SQL for MySQL instance.

  3. To let the Cloud SQL for MySQL instance perform semantic value searches:

    1. Create a vector column to store embeddings (for example, city embeddings on the properties table):

      ALTER TABLE `properties`
ADD COLUMN `city_embedding` VECTOR(768) USING VARBINARY;

    2. Generate and store vector embeddings for the column values:

      UPDATE `properties`
SET `city_embedding` = mysql.ml_embedding('text-embedding-005', `city`)
WHERE `city` IS NOT NULL;

  4. To let the Cloud SQL for MySQL instance perform trigram value searches, create a FULLTEXT index with the ngram parser on the target text column (for example, address on the properties table):

    CREATE FULLTEXT INDEX `idx_ngram_properties_address`
ON `properties`(`address`)
WITH PARSER ngram;

Prepare your environment

You can build and optimize context sets from your local development environment or IDE.

Install uv

The context engineering agent requires uv to manage underlying Python packages:

  1. Install uv by following the instructions in Install uv.

  2. Verify that uv is installed and accessible from your command line:

    uv --version

Install the context engineering agent

Install the context engineering agent plugin in your preferred development environment:

Antigravity CLI

  1. Install the Antigravity CLI. See Get started with Antigravity CLI.
  2. Install the context engineering agent plugin: 
    agy plugin install https://github.com/GoogleCloudPlatform/db-context-enrichment/tree/main/plugin
  3. Start the Antigravity CLI: 
    agy

To update the plugin later, run agy plugin uninstall db-context-engineering followed by agy plugin install https://github.com/GoogleCloudPlatform/db-context-enrichment/tree/main/plugin.

Claude Code

  1. Add the plugin marketplace: 
    /plugin marketplace add https://github.com/GoogleCloudPlatform/db-context-enrichment.git
  2. Install the plugin: 
    /plugin install db-context-engineering@db-context-enrichment-marketplace
  3. Reload plugins to activate the changes: 
    /reload-plugins

To update the plugin later, run /plugin update db-context-engineering@db-context-enrichment-marketplace.

Gemini CLI (Deprecated)

  1. Install the Gemini CLI. See Get Started with Gemini CLI.
  2. Install the extension: 
    gemini extensions install https://github.com/GoogleCloudPlatform/db-context-enrichment

To update the extension later, run gemini extensions update google-cloud-db-context-engineering.

Authenticate with Google Cloud

  1. Configure Application Default Credentials (ADC) to authenticate your development environment:

    gcloud auth application-default login

  2. Set the quota project for your credentials so that the Model Context Protocol (MCP) tools can use your Google Cloud project:

    gcloud auth application-default set-quota-project PROJECT_ID

    Replace PROJECT_ID with your Google Cloud project ID.

Connect to your database

  1. Start your agent environment and ask the agent to establish the database connection (for example, "Help me set up the database connection"). The agent guides you through creating the .context-engineering/tools.yaml configuration file so that it can read schemas and validate SQL. (Before .context-engineering/tools.yaml is created, an initial toolbox MCP server connection warning on startup is expected.)

  2. Reload the MCP server connection to apply the .context-engineering/tools.yaml configuration:

    • Antigravity CLI: Run /mcp and select toolbox (or db-context-engineering_toolbox) to restart.
    • Claude Code: Run /mcp, select toolbox, and select Reconnect.
    • Gemini CLI: Run /mcp refresh (or /mcp reload).

Build and optimize context

The context engineering agent guides your context set through an end-to-end lifecycle, from initial bootstrapping to verified generalizability.

Bootstrap a baseline context set

To avoid creating context from a blank page, the agent inspects your database schema along with any available application artifacts, such as business requirements, documentation, or sample SQL queries. The agent derives an initial baseline context set containing foundational query templates and query facets grounded in your database model.

After generating and validating the baseline context, the agent uploads it directly to Google Cloud and returns the context set resource name.

Create evaluation and held-out datasets from application artifacts

To systematically improve accuracy, you need ground-truth questions paired with verified SQL queries. If you don't already have an evaluation benchmark, the agent can generate one by extracting business scenarios from your application code, historical queries, and database schema.

During dataset generation, the agent creates two distinct sets:

  • Training benchmark: A collection of representative question-SQL pairs used during iterative optimization.
  • Held-out dataset: A separate set of questions similar in domain to your training benchmark, reserved for testing for generalizability.

Optimize accuracy with hill-climbing

In hill-climbing optimization, the agent iteratively refines your context set to drive QueryData accuracy toward near-100%:

  • Evaluate: The agent tests your current context set against the training benchmark by using the QueryData API and scores accuracy.
  • Diagnose gaps: The agent analyzes query failures, identifies their root causes (such as missing table joins, unlinked private entities, or complex filter conditions), and formulates targeted fixes.
  • Refine and upload: The agent adds or improves query templates, query facets, and value searches to resolve the gaps, and automatically uploads the updated context set to Google Cloud from your CLI.
  • Iterate: The agent repeats this cycle across rounds until accuracy converges on the optimal score.

The following diagram shows the systematic hill-climbing workflow:

Workflow for iteratively building context.

Prove generalizability with the held-out dataset

Achieving high accuracy on a training benchmark is necessary, but production traffic includes phrasing variations and complex compound questions that were never seen during development.

After hill-climbing optimization converges, the agent tests the resulting context set against the held-out dataset. This step verifies that the context set generalizes cleanly to new user inputs without overfitting to the initial benchmark. The agent produces a generalizability report showing accuracy across unseen question variations and composite filters (such as "Find me homes near good schools in Del Mar for a starter family with an ocean view").

On-demand authoring and reasoning

In addition to full optimization runs, you can converse with the agent for targeted updates as business requirements evolve:

  • Targeted authoring: Describe a specific business requirement in natural language, and the agent formats the context item and uploads it directly from your CLI without manual editing:

    • "Create a template for: 'Find me homes near good schools in Del Mar' with SQL: SELECT p.property_id, p.address FROM properties AS p JOIN schools AS s ON p.school_id = s.school_id WHERE p.city = 'Del Mar' AND s.rating >= 7."
    • "Create a facet for the filter properties.bedrooms BETWEEN 3 AND 4 AND properties.square_feet BETWEEN 1500 AND 2500 labeled 'for a starter family'."
    • "Create a value search for properties.city."

  • Bulk updates across a context set: If database columns are renamed or table structures change, the agent can update all affected context items simultaneously without modifying unrelated entries:

    • "In my context set, replace properties.property_type = 'SFR' with properties.property_type = 'Single Family Residential' for any item referencing single-family homes. Leave unrelated items alone."

What's next