Troubleshoot metadata change feeds

This document helps you troubleshoot and resolve common issues with Dataplex Universal Catalog metadata change feeds.

Metadata change feeds creation errors

This section describes how to resolve errors that you might encounter when you create metadata change feeds.

Permission denied errors

You receive a PERMISSION_DENIED error when trying to create a metadata change feed.

The issue occurs because you or the Dataplex Universal Catalog service account might be missing required permissions.

To resolve this issue, review these requirements:

  • Ensure that you have the dataplex.metadataFeeds.create permission.

  • Ensure that you have the dataplex.entryGroups.export permission on the scope of the feed (Organization, Project, or Entry Group).

  • Ensure that the Dataplex Universal Catalog service account (service-{PROJECT_NUMBER}@gcp-sa-dataplex.iam.gserviceaccount.com) has the roles/pubsub.publisher role on the destination Pub/Sub topic.

VPC Service Controls errors

You receive a PERMISSION_DENIED or VPC Service Controls error when creating a feed.

To resolve the issue, check that the metadata change feed and the resources it monitors belong to the same VPC Service Controls perimeter:

  • If the feed is organization-scoped, only projects within the same perimeter are monitored.

  • If the feed is project-scoped or entry-group-scoped, ensure that all specified projects or entry groups are in the same perimeter as the project where you are creating the feed.

Notification delivery issues

This section describes how to resolve issues with notification delivery.

Notifications not received

You are not receiving notifications in your Pub/Sub topic. This can be caused by several factors:

  • Missing permissions: the Dataplex Universal Catalog service account lost the roles/pubsub.publisher role on the topic.

  • Activation delay: a newly created or updated feed can take up to 10 minutes to become active.

  • Topic deleted: if the destination Pub/Sub topic is deleted, Dataplex Universal Catalog stops publishing.

  • Scope or filter mismatch: the changes might not match the configured scope or filters.

To resolve the issue, check the following:

  • Verify that the Dataplex Universal Catalog service account has the roles/pubsub.publisher role on the topic.

  • Wait at least 10 minutes after creating or updating a feed.

  • Check if the Pub/Sub topic exists. If it was deleted, you must recreate it.

  • Review your feed configuration to ensure the scope and filters match the metadata changes you expect to receive.

Duplicate notifications

You receive duplicate notification messages. Dataplex Universal Catalog ensures "at least once" delivery, which means duplicate messages might occur, especially during retries.

Configure your subscriber application to handle potential duplicates. You can use the messageId or the change timestamp to identify and deduplicate messages.

Content issues

This section describes how to resolve issues with the content of metadata change feed notifications.

Missing full metadata in payload

The notification message contains only the resource name and change type, but not the full metadata or the specific changes.

This is because by design, metadata change feed notifications provide "change signatures" (what changed) rather than the full payload. This keeps messages lightweight and avoids payload size limits.

Use the resource name provided in the notification to call the Dataplex Universal Catalog API (for example, GetEntry) and retrieve the current state of the resource.