Evaluate and approve enrichment proposals

This guide is intended for data stewards and catalog administrators. It describes how to evaluate and approve the metadata proposals generated by enrichment runs.

Every time you execute an enrichment task, it generates an enrichment run. The run processes your context set against your knowledge signals, generates evaluation metrics, and proposes new metadata properties.

You can view the details of historical runs, evaluate their performance, and decide whether to approve or deny the proposed metadata changes.

Before you begin

To get the permissions that you need to evaluate and approve enrichment runs, ask your administrator to grant you one of the following IAM roles on the project:

  • Dataplex Catalog Editor (roles/dataplex.catalogEditor)
  • Dataplex Catalog Admin (roles/dataplex.catalogAdmin)

For more information about granting roles, see Manage access to projects, folders, and organizations.

You must also have an active enrichment task executed in your project. For more information, see Create enrichment tasks.

Evaluation workflow

You can review an enrichment run in three phases:

  1. Assess overall viability: Check the aggregate scores on the Overview and Evaluation tabs. This tells you if the run is broadly successful or a complete hallucination before you dig into the details.

  2. Scrutinize individual proposals: Filter the Overview tab for Low confidence proposals. If a proposal's decision is unclear, use the Artifacts tab to audit the raw conversation traces to understand why the agent made its suggestion.

  3. Approve or deny the run: Conduct a final check. If the majority of proposals are sound and properly grounded, approve the run to merge the metadata into your catalog. If it is heavily flawed, deny the run, improve your context sets, and try again.

Open an enrichment run

To view the details of an enrichment run:

  1. In the Google Cloud console, go to the Knowledge Studio page.

    Go to Knowledge Studio

  2. In the left navigation pane, select Enrichments.

    The Enrichment page opens and displays a list of active, failed, and succeeded tasks. The task list includes the task name, status, run summary, trigger, and the date it was last modified.

  3. From the list of tasks, click the name of the enrichment task you want to investigate.

  4. Click the Jobs tab.

  5. In the list of jobs, click the linked Name (ID) of the run you want to view. Alternatively, you can click Actions next to a run, and select View result.

The Enrichment run details page opens.

The details page organizes your results into the following tabs:

Tab Use this tab to...
Overview Assess the high-level health and confidence, and review and approve the generated metadata proposals. Answers the questions: Is this run mostly good? and What exactly will be published to the catalog?
Evaluation Check quantitative scoring and pass/fail metrics. Answers the question: Did the enrichment logic actually improve the data's usability?
Artifacts Inspect diagnostics, tacit learning, and grounding scores. Answers the question: Why did the agent make this decision and is the logic trustworthy?
Logs Troubleshoot system execution traces. Answers the question: Did the job fail technically?

Assess overall run viability

The Overview tab is your starting point. Visit this tab to gauge the overall success of the run, browse the high-level metrics, and review or approve the specific metadata changes proposed by the AI.

To review the run summary:

  1. On the Enrichment run details page, select the Overview tab.

  2. Review the Enrichment report. This section displays the overall health of the run, including the aggregate score, status (for example, "Needs review"), duration, total tokens used, and a summary describing the overall confidence of the proposals.

    What to look for: Review the aggregate score and summary to quickly gauge if the run succeeded at a macroscopic level before digging into individual proposals. For example, a score of 80% or higher generally indicates a high-confidence run, while lower scores might require you to review the proposals more carefully.

  3. Review the End-to-end evaluation section. This displays the baseline and projected pass rates for your evaluation cases. An acceptable run typically results in a positive Delta (for example, +5%), indicating that the proposed metadata improves overall data usability compared to the baseline.

  4. Review the Proposals section. This section displays a high-level summary of proposed changes (such as the total number of creations, updates, or deletions), followed by a list of proposals grouped by resource type (for example, Knowledge base documents). You can filter the proposals by confidence level (High, Medium, Low, N/A) and expand the resource types to view specific entries.

    What to look for: Look for Low confidence proposals in particular, as these typically indicate missing context or hallucinations that must be rejected.

Scrutinize individual proposals

Unless you enabled auto-publish on the task, enrichment runs don't modify your metadata automatically. The run generates proposed changes which you must approve.

When reviewing proposals, we recommend filtering by confidence score and reviewing the Low confidence proposals first, as they require the most scrutiny and are most likely to need manual adjustments or rejection. Verify that the generated Markdown, tags, or descriptions accurately represent your business logic in Preview mode, and look for any unintended attribute overrides in the Changes view.

To review the specific proposals generated by the run:

  1. On the Enrichment run details tab, in the Proposals section, locate the proposal you want to review.

  2. Click View proposal for the entry. A Proposed changes side panel opens.

  3. In the side panel, review and optionally edit the AI's generated metadata.

    • Under the specific resource being modified (for example, the Overview aspect), use the view toggle to switch between views.

      The Changes view lets you inspect the exact diff being pushed to the catalog.

      The Preview view displays the human-friendly formatting of the generated documents and tables.

    • To manually adjust the AI's proposal, click Edit next to the resource name, make your changes in the text editor, and then click Save.

  4. If the proposal is inaccurate, click Remove from request in the side panel, and then click Remove to confirm. This permanently removes the proposed change, preventing it from being added to the catalog when you approve the run.

  5. After reviewing all proposals, click Approve in the Proposals section header to merge the remaining proposals into your catalog. Alternatively, if the entire run is inaccurate, you can click Deny.

View end-to-end evaluation metrics

The Evaluation tab measures the impact of the enrichment run. It compares how well your data answers test questions with the proposed metadata versus without it. This helps you determine whether to approve the run or adjust your context sets.

When evaluating results, use the Why column to understand the reasoning. The agent provides detailed reasoning comparing its logic to the reference. You can use this feedback to understand why the agent failed (for example, missing a critical table join). You can then use that information to improve your source data or clarify your context set.

To interpret the evaluation results:

  1. On the Enrichment run details page, click the Evaluation tab.
  2. Review the scores at the top of the tab. Focus on the Delta, which shows the overall percentage change in the pass rate, indicating the high-level value added by the enrichment proposals.
  3. Review the Summary metrics to see statistical changes (such as Average or Maximum) across specific criteria, like Data result accuracy.
  4. Use the Per-case results table to debug specific cognitive gaps. This table lists every test question evaluated during the run.
    • Filter the list to locate Fail results or Still failing entries in the Change column.
    • Read the explanation in the Why column to understand AI failures.

Audit run artifacts

The Artifacts tab contains detailed diagnostics and evidence supporting the AI's decisions. Visit this tab when you need to understand why a proposal received a low confidence score. You can also use it to audit raw conversation traces and learned rules extracted by the agent.

When viewing the Learnings tab, check the Applied versus Needs attention counts. The Applied count shows how many new facts were successfully added to the system, while Needs attention flags contradictory or ambiguous findings that require manual cleanup. A count of 0 means no new knowledge was extracted this round, which is a normal outcome. If items are flagged as ambiguous or conflicts, it means the agent found contradictory information in your sources that you need to resolve in your context set.

When viewing the Confidence scores tab, read the summary sentence (for example, What held most proposals back). Go to the Changes proposed section to isolate the lowest-rated changes. If a change is listed here, it likely failed grounding checks or lacked sufficient evidence. Use this list to prioritize which proposals to reject.

To view run artifacts:

  1. On the Enrichment run details page, click the Artifacts tab.

  2. Review the following sub-tabs to inspect diagnostic data:

    • Learnings: Displays whether the agent successfully mined new tacit knowledge (for example, from conversation logs or chat histories). You can also view the conversation traces that led to specific proposals.

    • Confidence scores: Breaks down the AI's confidence rating across three dimensions: Grounding, Corroboration, and Source quality. You can also access the raw dynamic_eval.json dataset from this section if you need to audit the evaluations.

Troubleshoot failed runs

The Logs tab contains detailed system execution traces. Visit this tab to troubleshoot backend errors or investigate why an enrichment job failed to run to completion.

To troubleshoot a failed run or inspect execution details:

  1. On the Enrichment run details page, click the Logs tab.
  2. Use the Severity drop-down or the Filter bar to narrow the log results.
  3. Review the execution traces and system logging.
    • What to look for: Look for red ERROR or FATAL severity entries that indicate backend failures, such as quota exhaustion or API timeouts.

Alternatively, you can query diagnostic logs directly using Cloud Logging. Detailed execution logs from the enrichment agent are automatically routed to Cloud Logging. Sensitive system instructions and raw data are redacted.

Use the following queries in the Logs Explorer to filter logs:

  • View all logs for enrichment tasks:

    resource.type="dataplex.googleapis.com/EnrichmentWorkflow"

  • Filter logs for a specific task: text resource.labels.workflow_id="YOUR_TASK_NAME"

  • Filter logs for a specific run execution: text labels.run_id="YOUR_RUN_ID"

  • Filter for warnings and errors: text severity>=WARNING

You can also use these queries to configure Cloud Monitoring alert policies based on your Cloud Logging metrics. For example, you can proactively route an alert to Slack, PagerDuty, or an email distribution list whenever an enrichment run fails (severity>=ERROR).

Iterate on an enrichment task

If an enrichment run generates inaccurate proposals or fails grounding checks (reporting Low confidence), don't approve the run. Instead, you must improve the information given to the agent and run the task again.

  1. On the Overview tab, in the Proposals section, click Deny.

  2. If the agent misinterpreted your data schemas, go to the source definitions in Knowledge Catalog and improve the schema descriptions.

  3. If the agent lacked sufficient business logic or used the wrong tables to answer questions, go to the task's attached Context set.

    • Add resources: Add missing internal documentation, valid query examples, or missing related tables to the context set to give the agent more facts.
    • Remove resources: Remove outdated documents or conflicting glossary terms if they confused the agent.

  4. Go to the Enrichment page and select your task.

  5. Click Edit to verify it points to the updated context set, and then click Run to generate a new set of proposals.

What's next