About data context in Knowledge Catalog

In a distributed data estate, raw data without context is rarely enough to solve a business problem. When structured databases, unstructured logs, and analytical metrics lack business meaning, users and autonomous systems face a context ceiling. Human analysts must perform manual validation to verify data sources, while AI agents often generate inaccurate answers (hallucinations) or fail to run actions.

Knowledge Catalog resolves this problem by transitioning metadata management from a passive catalog into a dynamic control plane of active context. By aggregating technical structures, business rules, and real-world usage patterns, Knowledge Catalog provides a unified context layer that helps ensure both you and your AI applications can discover, evaluate, and interpret data assets accurately and securely.

What is context?

Context is the rich fabric of structural, operational, and semantic details that describes a data asset. Instead of representing a table as a standalone schema of names and data types, context provides the complete story of your data.

Knowledge Catalog organizes context into the following layers:

  • Technical metadata. The structural details ingested directly from source systems, including database schemas, field types, partitioning keys, and security tags (such as Identity and Access Management (IAM) policies).

  • Semantic metadata. The business meaning of the data, which includes business glossaries, metric relationships, classifications, domain terminology, and golden queries (verified natural-language questions mapped to correct SQL code statements).

  • Operational metadata. The data's behavior and health over time, including column-level data lineage, execution profiles, profiling statistics, and data quality scores.

  • Relationships. The inferred or defined links between physical storage tables and conceptual business indicators.

  • Trust signals. Indicators of verification, such as stamps of ownership and certification, that verify if an asset is an authoritative source.

  • Agent guidance. Unstructured descriptions (such as README files, usage notes, and execution logs) that guide autonomous tools on how and when to query a resource.

By unifying these layers, Knowledge Catalog helps ensure that your data is fully understood, helping you avoid the operational bottlenecks and uncertainty associated with siloed and undocumented data.

How data context differs from metadata

While metadata provides the foundational building blocks, metadata and context are not interchangeable:

  • Metadata. Descriptive attributes that define the physical structure, location, and technical properties of an isolated data asset (for example, column names, data types, partition keys, and table sizes).
  • Context. The synthesized, multi-dimensional knowledge graph that combines technical schemas, business semantics, data relationships, operational behavior, and trust signals to give data meaning and guide reasoning.

The following table highlights the key differences between passive metadata and active data context:

Dimension Technical metadata Active data context
Core focus Physical specifications and storage format. Business meaning, operational behavior, and relationships.
State Passive: Stored in catalog registries for manual lookup by engineers. Active: Continuously enriched by AI, mined from query histories, and assembled dynamically for LLM prompts.
Primary consumers Database engines, data engineers, and compliance auditors. AI agents, Large Language Models (LLMs), business analysts, and conversational interfaces.
Primary purpose Storage management, indexing, and compliance verification. Grounding agent reasoning, resolving joins, and preventing hallucinations.

Example: Metadata compared to context

To understand how context expands upon technical metadata, consider how Knowledge Catalog represents a transaction column:

1. Technical metadata alone

Technical metadata provides structural information, but lacks business logic and operational guidelines:

  • Table: fact_orders
  • Column: disc_val
  • Data type: NUMERIC
  • Storage properties: Partitioned by order_date

An AI agent using only this technical metadata might guess that disc_val represents total customer discounts, but it cannot determine whether that value includes tax, manufacturer rebates, or promotional coupons.

2. Enriched data context

Knowledge Catalog enriches the technical metadata with semantic, relational, and operational layers into a unified context payload:

  • Technical schema: fact_orders.disc_val (NUMERIC).
  • Semantic definition: Mapped to the business glossary term "Promotional Discount", defined as customer discounts applied at checkout, excluding manufacturer rebates and shipping allowances.
  • Discovered relationships: Automatically links fact_orders.customer_id to dim_customers.id and fact_orders.promo_id to dim_promotions.id.
  • Operational usage: Query history reveals that 90% of business queries filter this table by status = 'COMPLETED' and join it with dim_promotions.
  • Trust and quality: Scored with a 99.8% validity rating by automated data quality scans, with verified data freshness within the last 2 hours.
  • Golden query: Includes a verified SQL template demonstrating how to calculate net recognized revenue using disc_val.

With this complete context, an AI agent or data analyst can accurately generate queries and interpret data without manual human validation or hallucinations.

The challenges of using data without context

Most enterprise data lack necessary metadata, making it difficult to discover, trust, or operationalize for AI. Without structured context, organizations face several operational bottlenecks:

  • Collaboration toil. A lack of clear business definitions forces analytics and AI teams into continuous, manual coordination cycles with subject matter experts (SMEs) to build basic understanding of tables.

  • Context drift. Standard documentation remains static and decays over time. Without continuous monitoring, outdated metadata leads to incorrect assumptions and broken AI agent workflows.

  • Reasoning failures. AI agents and large language models might generate syntactically correct SQL queries but fail to respect the underlying enterprise logic, yielding calculations that are incorrect from a business perspective.

  • Siloed information. Internal knowledge regarding how to correctly filter or join datasets remains trapped in undocumented scripts or dashboard setups rather than being registered in a centralized inventory.

To overcome these operational bottlenecks and make data actionable, organizations require a reliable context engine across their data estate.

Core pillars of context management

To maintain this reliable context engine across your data estate, Knowledge Catalog operates on three pillars:

Aggregation

Knowledge Catalog automatically gathers metadata from across your built-in databases, Iceberg REST catalogs, and third-party platforms. It consolidates these fragmented inputs into a single, unified zone of governance, helping ensure that security policies and classification definitions inherit consistently.

Enrichment

Knowledge Catalog continually updates and curates your context graph using AI-driven ingestion. By mining transaction logs, database schemas, and business intelligence models, Knowledge Catalog automatically discovers column descriptions and table relations. This continuous learning captures the institutional knowledge of data owners and applies it to your assets dynamically.

Search and retrieval

Knowledge Catalog provides high-precision semantic search that lets you query your data estate in natural language. Because Knowledge Catalog understands permission groups (ACLs) and business terminology, both users and autonomous agents can locate relevant data context with low latency.

How metadata and context are fed into Knowledge Catalog

To build a reliable context engine, Knowledge Catalog ingests and curates information through automated platform connectors, continuous AI enrichment, and extensible custom interfaces.

First-party data sources and automated harvesting

Knowledge Catalog automatically harvests structural and operational metadata from core Google Cloud services without requiring manual configuration:

  • Databases and analytical warehouses. Services like BigQuery, Spanner, Cloud SQL, and AlloyDB for PostgreSQL continuously sync technical metadata, including table schemas, field data types, partitioning keys, clustering specifications, and dataset boundaries.
  • Unstructured data insights. For unstructured files stored in Cloud Storage (such as PDF documentation or policy manuals), Knowledge Catalog runs discovery and profiling scans that extract key concepts and entity relationships, generating a graph profile and registering structured object tables in BigQuery. For more information, see About unstructured data insights.
  • Out-of-the-box BI semantic and lineage sync. For Looker (Google Cloud core), Knowledge Catalog automatically synchronizes business intelligence assets:
    • LookML semantic definitions. Models, Explores, views, dimensions, and measures.
    • BI dashboard assets. Dashboards, dashboard elements, and Looks.
    • End-to-end data lineage. Relationships tracing data flow from underlying BigQuery tables through to downstream Looker dashboards. For more information, see Manage Looker (Google Cloud core) resources with Knowledge Catalog.
  • Data integration and transformations. Services like Datastream capture change data capture (CDC) stream metadata and lineage, Dataform pushes table definitions and SQL transformations, and Cortex Framework publishes registered data products.

Continuous AI-driven context curation

Beyond static technical schemas, Knowledge Catalog dynamically updates its context graph using built-in intelligence engines:

  • Data insights. Powered by Gemini in BigQuery, data insights mines historical query logs, user transactions, and schema patterns to generate natural-language column descriptions, discover table relationships (such as join keys), and propose verified example queries ("golden queries") that capture enterprise logic. For more information, see About data insights for structured data.
  • Automated data profiling and data quality. Scans compute statistical distributions (such as distinct counts, null ratios, and sample values) and evaluate data against rule-based data quality checks, attaching live health scores to catalog entries. For more information, see About auto data quality.
  • Automated data lineage. Aggregates column-level transformation flows and job execution histories across your data pipelines. For more information, see About data lineage.

Custom metadata ingestion and third-party systems

For organizations with bespoke knowledge bases, third-party governance tools, or on-premises systems, Knowledge Catalog provides flexible ingestion mechanisms:

  • Knowledge Catalog APIs (CRUD operations). Use the CreateEntry, UpdateEntry, CreateAspectType, and SetAspect API methods to programmatically register external assets and attach custom metadata aspects (such as compliance ratings or system documentation).
  • Custom enrichment agents. Use agent frameworks such as the Agent Development Kit (ADK) or LangChain to build custom enrichment agents. For example, an ADK agent can ingest technical documentation from internal wikis or Git repositories, use an LLM to parse them into standardized business glossary terms, and attach them to catalog entries. For more information, see Build an AI agent to enrich metadata.
  • Export and import operations. Use file-based (JSON or CSV) bulk export and import workflows to review AI-generated business definitions, collaborate with data stewards, and batch-update metadata. For more information, see Export metadata.

How context is consumed across 1P and 3P services

Services and applications interact with Knowledge Catalog in two distinct roles:

  • Metadata providers. Services that send technical metadata, schemas, LookML definitions, and lineage into Knowledge Catalog (for example, Looker, Cloud SQL, and Spanner). Metadata providers push metadata into the catalog but don't necessarily consume business glossary definitions into their own user interfaces.
  • Context consumers. Services, developer tools, and AI agents that retrieve business definitions, operational profiles, and governance rules from Knowledge Catalog to ground their reasoning and execution.

Before applications and agents can consume context, you should establish foundational business terms and quality rules. For more information, see Establish foundational data context and Build a policy-as-code data quality workflow.

AI applications and services consume context from Knowledge Catalog through the following mechanisms:

Built-in first-party (1P) integrations

Several Google Cloud services natively query Knowledge Catalog to provide context-aware AI experiences:

  • Gemini in BigQuery (Conversational Agent). When you enter natural-language prompts in BigQuery Studio, Gemini automatically queries Knowledge Catalog business glossary definitions, column descriptions, and historical query relationships in the background. This grounds Gemini in enterprise business terminology, ensuring that generated SQL queries use the correct table joins and metric definitions.
  • Data Engineering Agent in BigQuery. Discovers and queries Apache Iceberg and BigQuery tables, automatically invokes semantic catalog searches to resolve asset dependencies, and generates catalog metadata during pipeline execution.
  • Vertex AI and Agent Builder. Custom GenAI agents and search applications built on Vertex AI use catalog metadata and business rules to ground model reasoning in verified enterprise facts.
  • Google Cloud console and Business UI. End users and data stewards explore assets using natural-language semantic search, view interactive data relationship graphs, and manage business glossaries.

Third-party applications and AI agent ecosystems

Services and autonomous systems without built-in integrations consume context through open protocols and dedicated retrieval APIs:

Model Context Protocol (MCP)

Model Context Protocol (MCP) is an open standard that lets AI agents, developer tools, and IDEs connect directly to Knowledge Catalog tools. MCP provides a standardized interface to inspect table schemas, look up glossary definitions, verify data quality scores, and trace data lineage.

Knowledge Catalog supports two MCP deployment options:

Retrieve pre-formatted context with the LookupContext API

For custom AI pipelines and LLM applications, Knowledge Catalog provides the LookupContext API method (projects/<var>PROJECT_ID</var>/locations/<var>LOCATION</var>:lookupContext).

Instead of requiring an agent to make multiple sequential calls to inspect schemas, look up glossary terms, fetch data quality scores, and inspect query logs, the LookupContext method retrieves a unified, pre-formatted bundle of rich metadata in a single request.

Key capabilities of the LookupContext method include the following:

  • LLM-ready formats. Returns context formatted as yaml (default), json, or xml, designed for direct insertion into LLM system prompts.
  • Token budgeting (context_budget). Lets you specify a target character budget. The API intelligently truncates and prioritizes metadata to fit within your model's context window.
  • Configurable context views.
    • BASIC. Returns fundamental entry metadata, schemas, column descriptions, overview text, and agent guidelines.
    • STANDARD. In addition to basic metadata, traverses the context graph to include multi-table join relationships, attached business glossary terms and synonyms, runtime usage patterns, and verified sample queries.
  • Rich operational and usage signals. Returns detailed usage statistics mined from query history, such as partitionBy, clusterBy, topReadFields, topFilterFields, topGroupByFields, topSortFields, and topAggregations.
  • Discovered joins and golden queries. Includes inferred table join conditions (for example, orders.customer_id = customers.id) and verified example SQL queries.

For more information, see Retrieve context for data assets and the lookupContext REST reference.

How context helps end users and AI agents

Knowledge Catalog provides universal context within its agentic data cloud by unifying technical schemas, business semantics, data lineage, and quality metrics into an active context layer. By exposing this metadata through open interfaces like the Model Context Protocol (MCP), context retrieval APIs, and semantic search, Knowledge Catalog assists both users and autonomous systems.

Eliminate manual trust work for users

Without a centralized semantic layer, you must investigate database lineage and query histories to answer basic business questions. Standard catalogs act as passive inventories of schemas, which requires you to manually look up metric definitions or verify table cleanliness.

Knowledge Catalog converts this passive governance into active intelligence. By presenting quality scores, semantic descriptions, and usage warnings directly alongside data assets, Knowledge Catalog lets you trust and use data without waiting for engineering reviews.

Ground autonomous agents and enable dynamic governance

AI agents and Large Language Models (LLMs) require strict, deterministic rules to interact with databases. Without explicit semantic context, agents make incorrect assumptions about column names or generate faulty SQL queries.

Additionally, active context transitions metadata management from a passive registry into an active governance layer. By defining verified business rules, metrics, and agent instructions directly within Knowledge Catalog, organizations can govern how agents retrieve, evaluate, and act upon corporate data.

Knowledge Catalog solves this gap. It exports unified active context profiles to your AI applications through standard interfaces like the Model Context Protocol (MCP) or the LookupContext API. Grounding your models in this metadata minimizes hallucinations, enforces governance rules, and helps ensure that autonomous actions are based on verified organizational truth.

Coordinate multi-agent workflows with shared context

In a multi-agent framework (or swarm), separate specialized agents collaborate to run complex workflows. When these agents operate across siloed environments without a shared context plane, they face a context ceiling—they might use conflicting business rules, fail to locate relevant dependencies, or duplicate retrieval effort.

Using Knowledge Catalog as a centralized context registry helps you coordinate and scale multi-agent swarms. Grounding your swarm in Knowledge Catalog provides the following benefits:

  • Shared business semantics. All agents reference the same business meanings and definitions. For example, if a data analysis agent and a reporting agent both retrieve a table containing sales figures, they use the same glossary definition of a metric like Gross Merchandise Value (GMV) to help ensure consistency.

  • Dynamic dependency resolution. Agents can use data lineage to discover upstream and downstream resources programmatically. This lets a pipeline troubleshooting agent trace a data failure to its source table and hand off the task to a repair agent without hardcoding dependency rules.

  • Unified connection interface. By using Model Context Protocol (MCP), you expose a single semantic endpoint to the entire swarm. This eliminates the need to configure custom API integrations for each individual agent.

  • Perimeter-level security governance. Knowledge Catalog checks agent queries against preconfigured IAM roles. This helps ensure that agents retrieve only the resource context that they are explicitly authorized to access, preventing accidental data exposure across the swarm.

Use cases for using data context

To understand how data context resolves complex inquiries, consider how Knowledge Catalog applies technical, semantic, and operational layers in practice:

Ecommerce: Personalize search recommendations

To align seasonal marketing campaigns with product inventories, search agents must understand how unstructured promotional emails relate to structured product catalogs and user transaction tables. Knowledge Catalog maps these entities, letting recommendation models suggest relevant products in real-time.

Manufacturing: Optimize machinery maintenance

Predictive maintenance models require integrating structured parts logs with unstructured technician repair guides and live sensor feeds. By mapping relations between part numbers and manual chapters, Knowledge Catalog lets analytical systems predict equipment fatigue and schedule repairs before failures occur.

Healthcare: Validate clinical trial data

Clinical research centers must verify that patient statistics conform to strict study rules, and that patient consent records map directly to trial result datasets. Knowledge Catalog tracks column-level data lineage to audit consent pipelines and alerts researchers to data profile anomalies, helping preserve trial integrity.

Financial services: Evaluate credit risk

Loan underwriting models use customer credit scores, transaction histories, and active debt balances across separate source systems. Knowledge Catalog provides a unified graph of loan metrics and maps data lineage, letting underwriting tools confirm that risk assessments use certified, compliant assets.

Telecommunications: Diagnose network operations

Autonomous network management systems and agents analyze real-time signal behaviors, detect performance degradations, and apply self-healing corrective actions. By grounding these agents in Knowledge Catalog context, the systems can safely trace network components through lineage graphs and map relationships between technical logs and business service areas.

What's next

Tutorial

Build a business glossary, define custom aspect types, and enrich assets in BigQuery.

Ai integration

Learn how Model Context Protocol connects agents to Knowledge Catalog tools and resources.

Api

Extract pre-formatted, token-budgeted YAML or JSON context for direct LLM prompt injection.

Ai agent

Automate metadata curation and glossary mapping using the Agent Development Kit.

Concept

Learn about the flexible metamodel, entry types, aspect types, and entry groups.

Governance

Visualize column-level data transformations to understand data origin and downstream impact.