Manage connector jobs

This product or feature is subject to the "Pre-GA Offerings Terms" in the General Service Terms section of the Service Specific Terms. Pre-GA products and features are available "as is" and might have limited support. For more information, see the launch stage descriptions.

This document describes how to view, update, and delete Knowledge Catalog (formerly Dataplex Universal Catalog) connector jobs used for metadata import.

Before you begin

To manage connector jobs, you need the appropriate IAM roles. See IAM roles for details.

• To view jobs and run history:
  • BigQuery User (roles/bigquery.user)
  • Dataplex Viewer (roles/dataplex.viewer)
• To edit, delete, or manually trigger jobs:
  • BigQuery Admin (roles/bigquery.admin)
  • Dataplex Administrator (roles/dataplex.admin) or Dataplex Editor (roles/dataplex.editor)

You need to enable the BigQuery Data Transfer Service.

Additionally, you must grant the BigQuery Data Transfer Service service agent ([e-mail]) the dataplex.entryGroups.import permission or the Dataplex Entry Group Importer (roles/dataplex.entryGroupImporter) role. This can be granted either at the project level or at the entry group level.

View connector jobs

1. In the Google Cloud console, go to the Knowledge Catalog page.

   Go to Knowledge Catalog

2. In the navigation menu, in the Manage section, click Connectors. The Connectors page lists all metadata import jobs in your project.

3. To view the details, run history, or configuration of a specific job, click its display name in the list.

View job run history and logs

For any connector job, you can view the history of its runs, including successful and failed runs. Job runs older than 90 days are automatically deleted.

1. On the Connectors page, click the display name of the job you want to view.
2. Click the Run history tab. This tab displays recent job runs.
3. To view logs for a specific run, find the run in the list and click View details.

Logging and monitoring

Knowledge Catalog metadata connectors export logs and metrics to Cloud Monitoring and Cloud Logging that provide observability into your import configurations. You can use Monitoring to set up dashboards to monitor metadata import configurations, evaluate run performance, and view error messages to troubleshoot run failures. You can use Logging to view logs related to a run or a configuration.

Update a connector job

You can modify the configuration of an existing job, including its schedule, destination entry group, notification settings, and the source objects to import.

1. On the Connectors page, click the display name of the job you want to update.
2. Click Edit.
3. Modify the job configuration as needed.
4. Click Save.

Disable or enable a connector job

When you disable a job, its name is updated with a disabled status, and no new job runs are scheduled. Any runs already in progress are completed. Disabling a job does not remove metadata already imported into Knowledge Catalog.

1. On the Connectors page, find the job you want to disable or enable.
2. Click Actions, and then select Disable or Enable.

Delete a connector job

When you delete a job, no new runs are scheduled, and any runs in progress are stopped. Deleting a job does not remove metadata already imported into Knowledge Catalog.

1. On the Connectors page, find the job you want to delete.
2. Click Actions, and then select Delete.
3. To confirm, type delete and click Delete.

Manually trigger a job

You can start a job run outside of its regular schedule. This is sometimes called a backfill.

1. On the Connectors page, click the display name of the job you want to run.

2. Click Run transfer now, and select Run one time transfer.

   Note: Knowledge Catalog connector jobs only support full metadata overwrites for each run.