Knowledge Catalog (formerly Dataplex Universal Catalog) lets you automate metadata enrichment for data
assets across your organization at scale. Adding business context from external
sources helps users and downstream AI agents discover, understand, and query
the data required to answer questions. In this tutorial, you use sample code
from the
[`knowledge-catalog`](https://github.com/GoogleCloudPlatform/knowledge-catalog/tree/main/samples/enrichment)
repository to build an agent that generates and publishes BigQuery
table documentation to Knowledge Catalog.

## How metadata enrichment works

While Knowledge Catalog automatically manages resources, tracks technical
schemas, and generates descriptions and
[data profiles](https://docs.cloud.google.com/knowledge-catalog/docs/data-profiling-overview.md.txt), valuable
business context often resides in other locations, such as the following:

- Internal documents and wikis
- Code repositories
- Communication channels, such as Google Chat and Slack

You can build AI agents to extract context from these sources, combine it with
technical metadata, and continuously enrich catalog entries in
Knowledge Catalog.

## Objectives

In this tutorial, you build an agent that performs the following tasks:

- Extract business context from knowledge bases, documents, code, or chat to enrich technical metadata.
- Generate documentation for BigQuery tables based on extracted context and other information sources.
- Publish generated documentation to Knowledge Catalog to improve search and discovery, making entries easier to find and understand.

## Before you begin

To run the Knowledge Catalog enrichment agent, you must meet the following requirements:

### Required roles


To get the permissions that
you need to use the enrichment agent,

ask your administrator to grant you the
following IAM roles on your Google Cloud project iam.gserviceaccount.com:

- To manage sample BigQuery resources: [**BigQuery Data Editor**](https://docs.cloud.google.com/iam/docs/roles-permissions/bigquery.md.txt#bigquery.dataEditor) (`roles/bigquery.dataEditor`)
- To search for catalog metadata: [**Dataplex Viewer**](https://docs.cloud.google.com/iam/docs/roles-permissions/dataplex.md.txt#dataplex.viewer) (`roles/dataplex.viewer`)
- To manage catalog metadata: [**Dataplex Catalog Editor**](https://docs.cloud.google.com/iam/docs/roles-permissions/dataplex.md.txt#dataplex.catalogEditor) (`roles/dataplex.catalogEditor`)
- To access Vertex AI features (Gemini LLM APIs): [**Vertex AI User**](https://docs.cloud.google.com/iam/docs/roles-permissions/aiplatform.md.txt#aiplatform.user) (`roles/aiplatform.user`)
- To consume service APIs: [**Service Usage Consumer**](https://docs.cloud.google.com/iam/docs/roles-permissions/serviceusage.md.txt#serviceusage.serviceUsageConsumer) (`roles/serviceusage.serviceUsageConsumer`)


For more information about granting roles, see [Manage access to projects, folders, and organizations](https://docs.cloud.google.com/iam/docs/granting-changing-revoking-access.md.txt).


These predefined roles contain

the permissions required to use the enrichment agent. To see the exact permissions that are
required, expand the **Required permissions** section:


#### Required permissions

The following permissions are required to use the enrichment agent:

- `bigquery.projects.get/createDatasets`
- `dataplex.projects.search`
- `dataplex.entryGroups.get/updateEntries`
- `aiplatform.endpoints.predict`
- `serviceusage.services.use`


You might also be able to get
these permissions
with [custom roles](https://docs.cloud.google.com/iam/docs/creating-custom-roles.md.txt) or
other [predefined roles](https://docs.cloud.google.com/iam/docs/roles-overview.md.txt#predefined).

### Enable APIs

To use the Knowledge Catalog enrichment agent, enable the following APIs in your project:

- BigQuery API
- Dataplex API
- Vertex AI API
- Service Usage API

<br />

**Roles required to enable APIs**


To enable APIs, you need the `serviceusage.services.enable` permission. If you
created the project, then you likely already have this permission through the
Owner role (`roles/owner`). Otherwise, you can get this permission through the
Service Usage Admin role (`roles/serviceusage.serviceUsageAdmin`).
[Learn how to grant roles](https://docs.cloud.google.com/iam/docs/granting-changing-revoking-access.md.txt).

[Enable the APIs](https://console.cloud.google.com/apis/enableflow?apiid=bigquery.googleapis.com,dataplex.googleapis.com,aiplatform.googleapis.com,serviceusage.googleapis.com)

### Install dependencies

You need the following Python packages and tools to run the sample:

- `google-adk` (Agent Development Kit (ADK))
- `google-cloud-dataplex` Knowledge Catalog Python Client
- `google-auth` manages [Application Default Credentials](https://docs.cloud.google.com/docs/authentication/application-default-credentials.md.txt)
- `mcp[cli]` for building a sample MCP server
- `gcloud` for authentication and configuration. To install Google Cloud CLI, see the [Google Cloud SDK](https://docs.cloud.google.com/sdk/docs/install) documentation.

## Set up the environment

1. Configure `gcloud` and sign in:

   ```bash
   gcloud auth application-default login
   gcloud config set core/project PROJECT_ID
   ```

   Replace the following:
   - `PROJECT_ID` with the ID of your project
2. Clone the [`knowledge-catalog`](https://github.com/GoogleCloudPlatform/knowledge-catalog) repository and navigate to the sample source directory:

   ```bash
   git clone https://github.com/GoogleCloudPlatform/knowledge-catalog.git
   cd knowledge-catalog/samples/enrichment/src
   ```
3. To install dependencies, use the provided script that sets up a Python virtual environment and the necessary environment variables:

   ```bash
   source env.sh --install
   ```
4. To create a sample BigQuery dataset named `kc_sample_analytics` in the `us` region of your cloud project, run the `create_data.py` script:

   ```bash
   python3 ../sample/data/create_data.py
   ```

   The sample also includes a number of documents in the `sample/docs` directory. These documents form a local knowledge base. The enrichment agent uses this knowledge base to extract information and produce documentation.

## Download metadata

Start by running the download tool to extract a metadata snapshot from Knowledge Catalog for the BigQuery dataset and its tables. This creates local metadata artifacts.

The `--dir` argument specifies the directory where the metadata files are written.

```bash
python3 -m enrichment.download \
  --dir ../sample/metadata.initial \
  --dataset ${KC_ENRICH_SAMPLE_PROJECT}.kc_sample_analytics
```

The script creates one Markdown file per table in the `sample/metadata` directory using the following naming convention: `<project_id>.<dataset_id>.<table_id>.md`.

## Enrich the metadata

After you create the local Markdown files, run the enrichment agent. The agent iterates over each file, finds information relevant to the tables, and summarizes findings along with citations to generate updated Markdown files.

- `--dir`: Specifies the directory containing the local metadata files.
- `--output-dir`: Specifies the target directory for the updated metadata files.
- `--config-dir`: Specifies the directory that contains agent instructions, MCP tools, and skills.

```bash
python3 -m enrichment.enrich \
  --dir ../sample/metadata.initial \
  --output-dir ../sample/metadata.new \
  --config-dir ../sample/config
```

## Review the metadata

The enriched metadata files contain the agent-produced documentation. Review and modify the files as needed before publishing the changes to Knowledge Catalog.

```bash
git diff --no-index ../sample/metadata.initial ../sample/metadata.new
```

## Publish the metadata

Run the publish tool to deploy the enriched metadata to Knowledge Catalog.

```bash
python3 -m src.enrichment.publish --dir ../sample/metadata.new
```

## Customize for your data

In the previous step, you used the `--config-dir` argument to point the agent to the `../sample/config` directory for its configuration. This is how the agent knows where to find information and how to interact with different sources.

The sample comes with a default configuration that instructs the agent to use a local MCP server to access files in the local knowledge base (`sample/docs`). To apply this workflow in your environment, you can customize these configuration files to connect the agent to your internal wikis, code repositories, Google Drive, or other systems.

The `sample/config/` directory contains the following files:

```bash
sample/config/
├─ instructions.md
├─ mcp.json
└─ skills/
    └─ kb-search/
        └─ SKILL.md
```

- `instructions.md`: Augments the agent's baseline instructions with details relevant to your organization, such as telling it to search a specific knowledge base.
- `mcp.json`: Configures MCP servers that the agent can use to access tools for your information sources, such as a tool to read files from a local directory.
- `SKILL.md`: Describes how the agent should use specific tools to interact with an information source, such as using `list_contents`, `read_file`, and `search_content` to find information in local documents.

## Explore the sample Knowledge Catalog code

The `download` and `publish` tools in the enrichment flow section use Knowledge Catalog APIs to read and write metadata.

This section covers how these APIs work so you can adapt the sample for your own integrations.

### Search for and retrieve metadata

The sample uses the following APIs to search for and retrieve metadata:

- `SearchEntries` to retrieve the entry and location metadata for the dataset.
- `ListEntries` to enumerate BigQuery tables within a Catalog EntryGroup.
- `GetEntry` to fetch the specific metadata for each BigQuery table.

The following code shows how to search for a dataset to locate its entry group, list all contained tables, and retrieve their specific metadata:

```python
import google.cloud.dataplex_v1 as dataplex

BIGQUERY_TABLE_TYPE = "projects/dataplex-types/locations/global/entryTypes/bigquery-table"
OVERVIEW_ASPECT_TYPE = "projects/dataplex-types/locations/global/aspectTypes/overview"

catalog = dataplex.CatalogServiceClient()

dataset_reference = '...'   # project_id.dataset_id
project_id, dataset_id = dataset_reference.split('.')

# 1. Search for dataset to determine its location
search_response = catalog.search_entries(
    request=dataplex.SearchEntriesRequest(
        name=f"projects/{project_id}/locations/global",
        query=f"type=dataset name={dataset_id}",
        page_size=1
    )
)
dataset_entry = search_response.results[0].dataplex_entry
location_id = dataset_entry.entry_source.location

# 2. List resources in the underlying group
entry_group_name = f"projects/{project_id}/locations/{location_id}/entryGroups/@bigquery"
entry_filter = f'parent_entry="{dataset_entry.name}"'
list_response = catalog.list_entries(
    request=dataplex.ListEntriesRequest(
        parent=entry_group_name,
        entry_filter=entry_filter,
    )
)

# 3. Retrieve metadata for each table in the list
for table_entry in list_response.entries:
    entry = catalog.get_entry(
        request=dataplex.GetEntryRequest(
            name=table_entry.name,
            view="CUSTOM",
            aspect_types=[OVERVIEW_ASPECT_TYPE]
        )
    )
```

### Update table metadata

The following code shows how to publish the generated documentation to the Overview aspect for a table and update its metadata:

```python
import google.cloud.dataplex_v1 as dataplex
import google.protobuf.field_mask_pb2 as field_mask_pb2
import google.protobuf.json_format as jsonpb

OVERVIEW_ASPECT_TYPE = "projects/dataplex-types/locations/global/aspectTypes/overview"
OVERVIEW_ASPECT_KEY = "dataplex-types.global.overview"

catalog = dataplex.CatalogServiceClient()

table_reference = "..."    # project_id.dataset_id.table_id
project_id, dataset_id, table_id = table_reference.split('.')

entry_data = {
    "name": f"bigquery.googleapis.com/projects/{project_id}/datasets/{dataset_id}/tables/{table_id}",
    "aspects": {
        OVERVIEW_ASPECT_KEY: {
            "aspectType": OVERVIEW_ASPECT_TYPE,
            "data": {
                "content": "...", # content parsed from local markdown file
                "contentType": "MARKDOWN"
            }
        }
    }
}

entry = dataplex.Entry()
jsonpb.ParseDict(entry_data, entry._pb)

catalog.update_entry(
    request=dataplex.UpdateEntryRequest(
        entry=entry,
        update_mask=field_mask_pb2.FieldMask(paths=["aspects"]),
        aspect_keys=[OVERVIEW_ASPECT_KEY],
    )
)
```

## What's next

- Learn more about working with [metadata](https://docs.cloud.google.com/knowledge-catalog/docs/metadata).
- Use the Antigravity CLI to [test your data context](https://docs.cloud.google.com/knowledge-catalog/docs/use-antigravity-cli-agent-to-test-data-context.md.txt).
- Learn about [managing aspects and enriching metadata](https://docs.cloud.google.com/knowledge-catalog/docs/enrich-entries-metadata.md.txt).
- Explore other classes and methods available in the [Knowledge Catalog client library for Python](https://docs.cloud.google.com/python/docs/reference/dataplex/latest.md.txt).
- Try other [Knowledge Catalog use cases](https://docs.cloud.google.com/knowledge-catalog/docs/use-cases).