Manage data products

This document is intended for data product owners who want to update, delete, and grant access to data products in Knowledge Catalog (formerly Dataplex Universal Catalog).

For more information about the architecture and key concepts of data products, see About data products.

Before you begin

1. Enable the Dataplex, BigQuery APIs.

   Roles required to enable APIs

   To enable APIs, you need the Service Usage Admin IAM role (roles/serviceusage.serviceUsageAdmin), which contains the serviceusage.services.enable permission. Learn how to grant roles.

   Enable the APIs

Note: This guide assumes you have existing data products to manage. For more information about creating and configuring data products, see Create data products.

Required roles

To get the permissions that you need to manage data products, ask your administrator to grant you the following IAM roles on the project:

• Full permissions to create, update, delete, and manage permissions for data products: Dataplex Data Products Admin (roles/dataplex.dataProductsAdmin)
• Update and manage existing data products: Dataplex Data Products Editor (roles/dataplex.dataProductsEditor)
• Add aspects such as schema, overview, and contacts: Dataplex Entry and EntryLink Owner (roles/dataplex.entryOwner)
• Search for and add assets: Dataplex Catalog Viewer (roles/dataplex.catalogViewer)
• Edit system aspect types such as documentation and contract aspect of refresh cadence: Dataplex Catalog Editor (roles/dataplex.catalogEditor)

For more information about granting roles, see Manage access to projects, folders, and organizations.

These predefined roles contain the permissions required to manage data products. To see the exact permissions that are required, expand the Required permissions section:

Required permissions

The following permissions are required to manage data products:

• Edit the overview system aspect type: dataplex.entryGroups.useOverviewAspect
• Edit the refresh cadence system aspect type: dataplex.entryGroups.useRefreshCadenceAspect
• Approve access request for a data product: dataplex.dataProducts.approve

You might also be able to get these permissions with custom roles or other predefined roles.

Access control for BigQuery models

Access to BigQuery models within a data product is managed through IAM conditions applied to the parent dataset's IAM policy.

If you delete and then recreate a BigQuery model using the same name, Knowledge Catalog restores the permissions previously granted to Google Groups or service accounts through the data product.

Set permissions to make data product discoverable

To enable potential consumers to search for, view, and request access to a data product, data product owners must ensure the product is discoverable by granting the appropriate IAM roles on the data product resource:

• Search for, view, and request access to data products: Dataplex Data Product Consumer (dataplex.dataProductsConsumer)

• View data product definitions and metadata (read-only access): Dataplex Data Product Viewer (dataplex.dataProductsViewer)

Manage data product access requests

When a data product consumer requests access, the data product owner receives an email notification. Owners can review, approve, or reject these requests using either the Google Cloud console or the API.

Console

Review access requests for a specific data product

1. In the Google Cloud console, go to the Knowledge Catalog Data products page.

   Go to Data products

2. Click the data product for which you want to review access requests.

3. Click the Access request management tab.

4. Select the requester whose request you want to review.

5. Click Actions and then click Approve or Reject.

6. Click Save.

Review access requests across multiple data products

Preview

This product or feature is subject to the "Pre-GA Offerings Terms" in the General Service Terms section of the Service Specific Terms. Pre-GA products and features are available "as is" and might have limited support. For more information, see the launch stage descriptions.

1. In the Google Cloud console, go to the Knowledge Catalog Governance workflows page.

   Go to Governance workflows

2. Click the Pending Approvals tab.

3. Optional: Filter the list by the specific data product name to isolate its requests.

4. Select the target request and click Approve or Reject.

5. Click Save.

Post-approval behavior

• User requests: Consumers who requested access for their user identity are automatically added as members to the Google Group mapped to the access group.

• Service account requests: Consumers who requested access for a service account are granted permissions to impersonate the data producer service account mapped to the access group.

Once processed, the request status updates automatically and the entry moves to the Approval log tab.

REST

List all pending access requests

To list all access requests that are pending your review within a specific project and location, send a GET request using the custom collection method :listReviewable:

curl -X GET \
-H "Authorization: Bearer $(gcloud auth print-access-token)" \
-H "Content-Type: application/json" \
"https://dataplex.googleapis.com/v1/projects/PROJECT_ID/locations/LOCATION/changeRequests:listReviewable"

Replace the following:

• PROJECT_ID: the ID of your Google Cloud project

• LOCATION: the region where the data product exists (for example, us-central1)

Approve an access request

Data product owners with the dataplex.dataProducts.approve permission can approve a pending request by sending a POST request to the custom method :approve:

curl -X POST \
-H "Authorization: Bearer $(gcloud auth print-access-token)" \
-H "Content-Type: application/json" \
"https://dataplex.googleapis.com/v1/projects/PROJECT_ID/locations/LOCATION/changeRequests/CHANGE_REQUEST_ID:approve"

Replace the following:

• PROJECT_ID: the ID of your Google Cloud project

• LOCATION: the region where the data product exists (for example, us-central1)

• CHANGE_REQUEST_ID: the unique ID of your access request you want to approve

Unlike console-driven approvals, approving a change request using the REST API doesn't automatically modify Google Group memberships or configure service account token impersonation. You must complete these downstream access management steps manually:

• For user identity access requests: Add the data product consumer to the mapped Google Group. For more information, see Add a membership to a Google Group.

• For service account access requests: Grant the Service Account Token Creator (roles/iam.serviceAccountTokenCreator) IAM role to the data product consumer's service account on the resource by running the following command:

  gcloud iam service-accounts add-iam-policy-binding PRODUCER_SERVICE_ACCOUNT \
    --member="serviceAccount:CONSUMER_SERVICE_ACCOUNT" \
    --role="roles/iam.serviceAccountTokenCreator"
  

  Replace the following:

  • PRODUCER_SERVICE_ACCOUNT: the fully qualified email address of the service account mapped to the data product's access group

  • CONSUMER_SERVICE_ACCOUNT: the fully qualified email address of the service account belonging to the consumer requesting access

Reject an access request

Data product owners with the dataplex.dataProducts.approve permission can reject a pending request by sending a POST request to the custom method :reject along with a justification payload:

curl -X POST \
-H "Authorization: Bearer $(gcloud auth print-access-token)" \
-H "Content-Type: application/json" \
-d '{
  "comment": "REJECTION_REASON"
}' \
"https://dataplex.googleapis.com/v1/projects/PROJECT_ID/locations/LOCATION/changeRequests/CHANGE_REQUEST_ID:reject"

Replace the following:

• PROJECT_ID: the ID of your Google Cloud project

• LOCATION: the region where the data product exists (for example, us-central1)

• CHANGE_REQUEST_ID: the unique ID of your access request you want to reject

Update a data product

You can update the basic details (for example, data product name, description, owner details), assets, access groups, permissions, contract, aspects, and additional documentation for an existing data product.

Note: The REST API methods for updating data products and assets are long-running operations. When you call these methods, the API returns an Operation object that you can use to check the status of the task.

Update the basic details of a data product

Console

1. In the Google Cloud console, go to the Knowledge Catalog Data products page.

   Go to Data products

2. Click the data product that you want to update.

3. Click Edit.

4. Update the Data product name, Data product icon, Description, Data product owner(s) email address, Data product approver(s) email address, and Labels as required.

5. Click Save.

REST

To update the basic details of a data product, use the dataProducts.patch method.

For example, to update the description of a data product, send the following PATCH request:

curl -X PATCH \
-H "Authorization: Bearer $(gcloud auth print-access-token)" \
-H "Content-Type: application/json" \
-d '{"description": "NEW_DESCRIPTION"}' \
https://dataplex.googleapis.com/v1/projects/PROJECT_ID/locations/ LOCATION/dataProducts/DATA_PRODUCT_ID?update_mask="description"

Replace the following:

• NEW_DESCRIPTION: a new description for the data product
• PROJECT_ID: the ID of your Google Cloud project
• LOCATION: the region where the data product exists
• DATA_PRODUCT_ID: the ID of your data product

Update assets of a data product

Console

1. In the Google Cloud console, go to the Knowledge Catalog Data products page.

   Go to Data products

2. Click the data product for which you want to update assets.

3. Click the Assets tab.

4. To add a new asset, follow these steps:

   1. Click +Add.

   2. Search for and select the assets that you want to add to your data product. The assets you select must reside in the same region as the data product.

      If you have necessary permissions, you can view the metadata of assets by clicking the asset.

   3. To refine the search results, use Filters.

   4. After you select the assets, click Add.

5. To remove an asset from the data product, follow these steps:

   1. For the asset that you want to remove, click > Remove.

   2. To confirm the action, click Remove.

   Note: You can remove up to 10 assets at a time.

REST

To add a new data asset to the data product, use the dataAssets.create method. For a code example, see Optional: Add assets.

To delete a data asset from the data product, use the dataAssets.delete method. For example, send the following DELETE request:

curl -X DELETE \
-H "Authorization: Bearer $(gcloud auth print-access-token)" \
"https://dataplex.googleapis.com/v1/projects/PROJECT_ID/locations/LOCATION/dataProducts/DATA_PRODUCT_ID/dataAssets/DATA_ASSET_ID"

Replace the following:

• PROJECT_ID: the ID of your Google Cloud project
• LOCATION: the region in which you want to create the data product
• DATA_PRODUCT_ID: a unique ID for your data product
• DATA_ASSET_ID: the ID of the data asset you want to remove from the data product

Update access groups and asset permissions

Console

1. In the Google Cloud console, go to the Knowledge Catalog Data products page.

   Go to Data products

2. Click the data product for which you want to update the access groups and permissions.

3. Click the Access groups & permissions tab.

4. To update an access group, follow these steps:

   1. Click Edit.

   2. Update the access group name, description, and identifier as required.

   3. To add a new access group, click Add access group and follow these steps:

      1. In the Access group name field, enter a name for the new access group. For example, Analyst.

      2. In the Access group description field, enter a description for the access group.

      3. In the Access group identifier field, enter the email address of a Google Group or service account or both that you want to assign to this access group:

         • Google Group: Data product consumers who request access for themselves are added as members to the mapped Google Group.

         • Service account: Data product consumers who request access for their service accounts are granted the Service Account Token Creator (roles/iam.serviceAccountTokenCreator) IAM role to impersonate the data producer service account mapped to the access group.

         Note: You can add a maximum of three access groups per data product.

   4. Click Add.

5. To remove an access group, follow these steps:

   1. For the access group that you want to remove, click > Remove.

   2. To confirm the action, click Remove.

6. If no access groups are configured for the selected data product, you can add a new access group by clicking Add access group.

7. To update asset permissions, follow these steps:

   1. For the asset you want to update permissions, click > Permissions.

   2. Update the required access groups and IAM roles.

   3. Click Configure.

REST

To update the access groups associated with a data product, use the dataProducts.patch method. For a code example, see Configure access groups.

To update the asset permissions, use the dataAssets.patch method. For a code example, see Configure asset permissions.

Update the data product contract

Console

1. In the Google Cloud console, go to the Knowledge Catalog Data products page.

   Go to Data products

2. Click the data product for which you want to update the contract.

3. Click the Contract tab.

4. To update the existing contract, follow these steps:

   1. Click Edit.

   2. Update the necessary fields.

   3. Click Save.

5. If no contract is attached to the selected data product, you can add one by clicking Add contract.

REST

To update a contract, use the entries.patch method. For a code example, see Add a contract.

Update aspects of a data product

Console

1. In the Google Cloud console, go to the Knowledge Catalog Data products page.

   Go to Data products

2. Click the data product for which you want to update the aspects.

3. Click the Aspects tab.

4. To update an aspect, follow these steps:

   1. Click the aspect you want to update.

   2. Click Edit.

   3. Update the Aspect type, Country, and Region as required.

   4. Click Save.

5. To add a new aspect for the data product, follow these steps:

   1. Click + Add aspect.

   2. In the Select aspect type field, search for and select an aspect type from the list. For example, Geo context.

   3. In the Country field, select the country to which the asset belongs.

   4. In the Region field, select the business region to which the asset belongs.

   5. Click Save.

6. To delete an aspect, follow these steps:

   1. Click the aspect you want to delete.

   2. Click Delete.

   3. To confirm the action, click Confirm.

REST

To update any aspect on the data product entry, use the entries.patch method.

Update the data product documentation

Console

1. In the Google Cloud console, go to the Knowledge Catalog Data products page.

   Go to Data products

2. Click the data product for which you want to update the documentation.

3. Click the Overview tab.

4. Click Edit against Documentation.

5. Update user guide, sample queries, and other documentation as required.

6. Click Save.

REST

Documentation is part of the overview aspect. To update the data product documentation, use the entries.patch method.

Delete a data product

To delete a data product, you must first remove all the assets from the data product.

Console

1. In the Google Cloud console, go to the Knowledge Catalog Data products page.

   Go to Data products

2. Click the data product that you want to delete.

3. Click the Assets tab.

4. For every asset listed, click > Remove. Confirm the action by clicking Remove.

5. After you remove all the assets, delete the data product by clicking Delete.

6. In the confirmation dialog, enter delete and click Delete.

REST

To delete a data product, use the dataProducts.delete method.

For example, send the following DELETE request:

curl -X DELETE \
-H "Authorization: Bearer $(gcloud auth print-access-token)" \
-H "Content-Type: application/json" \
https://dataplex.googleapis.com/v1/projects/PROJECT_ID/locations/ LOCATION/dataProducts/DATA_PRODUCT_ID

Replace the following:

• PROJECT_ID: the ID of your Google Cloud project
• LOCATION: the region where the data product exists
• DATA_PRODUCT_ID: the ID of your data product that you want to delete

Data retention and project deletion

When you delete a Google Cloud project that contains data products, the associated resources follow the standard Google Cloud deletion lifecycle, also referred to as a wipeout. For a detailed overview of the technical processes and timelines governing this behavior, see Data deletion on Google Cloud.

Resource lifecycle during project deletion

The lifecycle of your data products and data assets is managed in two primary stages following a project deletion request:

• Soft deletion (Hidden state): Immediately upon initiating project deletion, all data products and data assets within the project are marked as Hidden. During this 30-day grace period, the metadata is preserved but is no longer accessible through Dataplex APIs or the Google Cloud console. You can restore the project and its resources at any point during this window.

• Permanent purge (Purged state): After the 30-day grace period expires, the project and all its Knowledge Catalog metadata are permanently deleted. Knowledge Catalog utilizes an internal row deletion policy to verifiably purge these records from storage.

Behavior of IAM policies on cross-project resources

A critical operational distinction exists between Knowledge Catalog metadata and the IAM policies enforced on your underlying physical data (such as BigQuery datasets or Cloud Storage buckets).

While Knowledge Catalog metadata within the project is automatically purged, any IAM bindings applied to data resources located in a different project (Project B) aren't automatically revoked when the data product owner project (Project A) is deleted.

• Orphaned bindings: Because the physical resources in the target project remain active, the IAM system doesn't automatically remove the principal identifiers associated with the deleted project.

• Prevent zombie permissions: To ensure complete decommissioning of data access, manually delete data assets from your data products, or explicitly strip the associated IAM bindings from your cross-project resources, before you initiate a project deletion.

Project restoration (Undelete)

If you restore a deleted project within the 30-day grace period, Knowledge Catalog attempts to restore your resources and their visibility in the catalog. For cross-project assets, because the underlying IAM bindings weren't automatically revoked during the deletion process, data access typically resumes immediately for the restored project.

What's next

• Learn more about data products.
• Learn how to search for data products.
• Learn how to create a data product.