Enable Knowledge Catalog

Knowledge Catalog (formerly Dataplex Universal Catalog) lets you automatically ingest and maintain up-to-date metadata from supported Google Cloud data sources. To use Knowledge Catalog in your Google Cloud project, you must enable the Dataplex API, Dataproc API, and Dataproc Metastore API.

Before you begin

• Create a Google Cloud project and enable billing. Because enabling APIs can incur billing, your organization might restrict who can enable services.

• To enable and disable the Dataplex API, you need the Owner (roles/owner), Editor (roles/editor), or Service Usage Admin (roles/serviceusage.serviceUsageAdmin) role. For more information, see Manage access with IAM and Enabling and disabling services.

Enable Knowledge Catalog

You can use an existing project with Knowledge Catalog or create a new one. We recommend that you create a new project. If you are using a new Google Cloud project or a project in which the BigQuery API is enabled, the Dataplex API is enabled by default. However, you still need to enable the Dataproc API and Dataproc Metastore API.

To keep resources created by the lake within your project, separate from one another, create a project per lake instance.

To create a project and enable Knowledge Catalog, follow these steps:

1. In the Google Cloud console, select or create a project.
2. Make sure that billing is enabled for your project.

3. Enable the Dataplex, Managed Service for Apache Spark, and Dataproc Metastore APIs, if any are not already enabled.

   Roles required to enable APIs

   To enable APIs, you need the serviceusage.services.enable permission. If you created the project, then you likely already have this permission through the Owner role (roles/owner). Otherwise, you can get this permission through the Service Usage Admin role (roles/serviceusage.serviceUsageAdmin). Learn how to grant roles.

   Enable the APIs

   Note: The Data Lineage API is billed under the Knowledge Catalog Premium Processing SKU. If you also enable the Data Lineage API, it incurs charges separately from the Dataplex API. For more information, see Knowledge Catalog pricing.

Disable Knowledge Catalog

If you use Knowledge Catalog in a project that also uses BigQuery, be aware of the effects on BigQuery behavior when you disable Knowledge Catalog.

To avoid additional charges, complete the following steps before you disable Knowledge Catalog:

1. In the Google Cloud console, delete each lake instance in your project. To delete a lake, you must first delete the individual assets and zones. For more information, see projects.locations.lakes.delete.

   To avoid charges, delete all Cloud Storage buckets for zones within your lake. You can delete the Cloud Storage buckets before or after disabling the API.

   Wait until in-progress service operations for deleting lakes complete. Look for an empty list of lake instances on the Knowledge Catalog page.

2. In the Google Cloud console, delete all custom entry groups in your project. Deleting an entry group removes all entries that belong to the entry group. For more information, see Delete an entry group.

3. Select the project to disable the Dataplex API in.

4. Go to the Enabled APIs & services page.

   Go to Enabled APIs & services

5. Click Cloud Dataplex API.

6. Click Disable API. Confirm when prompted.

7. If you also enabled the Data Lineage API, repeat the steps to disable it. Disabling the Dataplex API does not disable the Data Lineage API or stop its Premium Processing charges. For more information, see Turn off data lineage.

What's next

• Learn how to search for resources.
• Learn how to measure data quality.
• Learn about metadata in Knowledge Catalog.
• Follow a quickstart: Track data lineage for a BigQuery table.
• Try Knowledge Catalog use cases.