About database connectors

This document describes how to use Knowledge Catalog (formerly Dataplex Universal Catalog) connectors to import metadata from external, third-party data sources into the Knowledge Catalog.

Knowledge Catalog connectors leverage the BigQuery Data Transfer Service to provide prebuilt integrations that automatically extract metadata from source systems and map it to entries within a Knowledge Catalog entry group.

The connectors extract the following types of metadata:

  • Technical metadata: Database, schema, table, and view definitions.
  • Operational metadata: Created and last updated timestamps for assets, such as tables, views, and routines.
  • Business metadata: Asset owners and annotations.

After you configure a connector, Knowledge Catalog schedules metadata import runs on the schedule you specify, ensuring that the catalog stays synchronized with the source system.

Supported data sources

You can import metadata from the following sources using Knowledge Catalog connectors:

  • Oracle
  • MySQL

Integrations with data governance partners

You can integrate Knowledge Catalog with other enterprise data cataloging and governance platforms. These integrations help ensure that metadata is discoverable and consistent across your organization's data landscape, allowing you to enrich metadata in Knowledge Catalog or view Knowledge Catalog metadata within third-party tools.

The following partners provide integrations for Knowledge Catalog:

How it works

Knowledge Catalog connectors use transfer configurations managed by BigQuery Data Transfer Service to connect to your data source, extract metadata, and ingest it into a Knowledge Catalog entry group that you specify.

Each time a metadata import run occurs, it performs a full overwrite of the entries in the destination entry group managed by that configuration. Metadata objects that are no longer present in the source are removed from the entry group, and new objects are added. Objects in the entry group that were not imported by the connector configuration are unaffected.

You configure and manage these configurations from the Connectors page in the Knowledge Catalog section of the Google Cloud console.

Terminology

The following terms are used in this documentation:

  • Metadata import configuration: The settings for a scheduled or on-demand task that uses BigQuery Data Transfer Service to extract metadata from a source and load it into Knowledge Catalog.
  • Run: A single execution instance of a metadata import configuration.
  • Entry group: A resource in Knowledge Catalog that contains metadata entries. When you configure a connector, you designate an entry group as the destination for the imported metadata.
  • Entry: A resource in Knowledge Catalog that represents a data asset, such as a table or view.

Pricing

There are no additional charges from Knowledge Catalog or BigQuery Data Transfer Service for importing metadata using connectors.

Standard charges apply for other Knowledge Catalog usage (such as metadata storage). For more information, see Knowledge Catalog pricing.

Limitations

  • Only metadata is imported, not the underlying data from the source tables or views. To transfer the underlying data, you can use the BigQuery Data Transfer Service.
  • Each run performs a full overwrite of connector-managed entries in the destination entry group. Incremental metadata updates are not supported.
  • Run history is retained for 90 days.

What's next