Receive notifications with metadata change feeds

This document describes how to configure Dataplex Universal Catalog metadata change feeds to receive notifications through Pub/Sub when metadata within Dataplex Universal Catalog is created, updated, or deleted.

For more information about metadata change feeds, see Metadata change feeds overview.

Before you begin

Familiarize yourself with Pub/Sub and the Dataplex Universal Catalog API.

Enable the Dataplex Universal Catalog and Pub/Sub APIs.
Roles required to enable APIs
To enable APIs, you need the Service Usage Admin IAM role (roles/serviceusage.serviceUsageAdmin), which contains the serviceusage.services.enable permission. Learn how to grant roles.
Enable the APIs
Create a Pub/Sub topic to receive notifications. For more information, see Create a topic.
Install gcloud. Make sure the short alias of gcloud is set.
Set an alias of gcurl. This creates a shortcut that includes your authentication token and sets the JSON content-type for API requests:
```
alias gcurl='curl -H "Authorization: Bearer $(gcloud auth print-access-token)" -H "Content-Type: application/json"'
```
Set the DATAPLEX_API variable:
```
DATAPLEX_API="dataplex.googleapis.com/v1/projects/PROJECT_ID/locations/LOCATION"
```
Replace the following:
- PROJECT_ID: your project ID where the Dataplex API is enabled
- LOCATION: the location where the job runs—for example, us-central1, europe-west3, or asia-south1

Required roles and permissions

To configure metadata change feeds, ensure that you and the Dataplex Universal Catalog service account have the required IAM roles and permissions.

User roles and permissions

To get the permissions that you need to export metadata and access metadata change feeds messages, ask your administrator to grant you the following IAM roles on the project or organization:

Export metadata: Dataplex Entry Group Exporter (roles/dataplex.entryGroupExporter)
Access metadata change feeds messages: Pub/Sub Subscriber (roles/pubsub.subscriber)

For more information about granting roles, see Manage access to projects, folders, and organizations.

These predefined roles contain the permissions required to export metadata and access metadata change feeds messages. To see the exact permissions that are required, expand the Required permissions section:

Required permissions

The following permissions are required to export metadata and access metadata change feeds messages:

Export metadata:
- dataplex.metadataFeeds.create on the project
- dataplex.entryGroups.export on the resource defining the scope of the feed (Organization, Project, or Entry Group)
- resourcemanager.projects.get on the project
- resourcemanager.projects.list on the project
Access metadata change feeds messages:
- pubsub.subscriptions.consume on the subscription
- pubsub.topics.attachSubscription on the topic

You might also be able to get these permissions with custom roles or other predefined roles.

Dataplex Universal Catalog service account roles and permissions

To ensure that the Dataplex Universal Catalog service account has the necessary permission to publish metadata change feeds messages, ask your administrator to grant the Dataplex Universal Catalog service account the Pub/Sub Publisher (roles/pubsub.publisher) IAM role on the Pub/Sub topic.

For more information about granting roles, see Manage access to projects, folders, and organizations.

This predefined role contains the pubsub.topics.publish permission, which is required to publish metadata change feeds messages.

Your administrator might also be able to give the Dataplex Universal Catalog service account this permission with custom roles or other predefined roles.

Grant permissions to the Dataplex Universal Catalog service account

The Dataplex Universal Catalog service agent is created when you enable the Dataplex API. You can identify the service agent by its email:

service-PROJECT_NUMBER@gcp-sa-dataplex.iam.gserviceaccount.com

Here, PROJECT_NUMBER is the project number of the project where you enabled the Dataplex API.

The Dataplex Universal Catalog service account requires the ability to publish messages to your Pub/Sub topic. To grant this permission, give the service account the Pub/Sub Publisher role (roles/pubsub.publisher) on the Pub/Sub topic:

gcloud

Run the gcloud pubsub topics add-iam-policy-binding command:

gcloud pubsub topics add-iam-policy-binding TOPIC_ID \
    --member="serviceAccount:service-PROJECT_NUMBER@gcp-sa-dataplex.iam.gserviceaccount.com" \
    --role="roles/pubsub.publisher"

Replace the following:

TOPIC_ID: the ID of your Pub/Sub topic to which the metadata change feed messages are published.
PROJECT_NUMBER: the project number of your Dataplex Universal Catalog project where the service account is located.

Console

In the Google Cloud console, go to the Pub/Sub Topics page.

Go to topics
Select the topic you're using for metadata feed notifications and if required click Show info panel.
In the Permissions tab, click Add principal.
In the New principals field, enter the Dataplex Universal Catalog service account: service-PROJECT_NUMBER@gcp-sa-dataplex.iam.gserviceaccount.com.
In the Assign roles field, select Pub/Sub Publisher.
Click Save.

Create a metadata change feed

To control which changes generate notifications, you can configure a metadata change feed to monitor specific resources. You do this by specifying a scope, such as your entire organization, specific projects, or specific entry groups. While scope lets you define which resources to monitor, you can use filters to further refine when Dataplex Universal Catalog sends notifications.

For more information, see Metadata change feed.

REST

To create a metadata change feed, use the projects.locations.metadataFeeds.create method.

Organization scope

Run the following command to create a metadata change feed that monitors the entire organization:

gcurl -X POST -d "$(cat <<EOF
{
  "scope": {
    "organizationLevel": true
  },
  "pubsubTopic": "projects/PROJECT_ID_PUBSUB/topics/TOPIC_ID"
}
EOF
)" "https://${DATAPLEX_API}/metadataFeeds?metadataFeedId=FEED_ID"

Replace the following:

PROJECT_ID_PUBSUB: the project ID where the Pub/Sub topic is located—for example, example-project
TOPIC_ID: the Pub/Sub topic ID to which the metadata change feed messages are published—for example, example-topic
FEED_ID: the metadata change feed ID that you want to create—for example, example-feed

Project scope

Run the following command to create a metadata change feed that monitors specific projects:

gcurl -X POST -d "$(cat <<EOF
{
  "scope": {
    "projects": [
      "projects/PROJECT_ID_1",
      "projects/PROJECT_ID_2"
    ]
  },
  "pubsubTopic": "projects/PROJECT_ID_PUBSUB/topics/TOPIC_ID"
}
EOF
)" "https://${DATAPLEX_API}/metadataFeeds?metadataFeedId=FEED_ID"

Replace the following:

PROJECT_ID_PUBSUB: the project ID where the Pub/Sub topic is located—for example, example-project
TOPIC_ID: the Pub/Sub topic ID to which the metadata change feed messages are published—for example, example-topic
FEED_ID: the metadata change feed ID that you want to create—for example, example-feed

Entry group scope

Run the following command to create a metadata change feed that monitors specific entry groups:

gcurl -X POST -d "$(cat <<EOF
{
  "scope": {
    "entryGroups": [
      "projects/PROJECT_ID/locations/LOCATION/entryGroups/ENTRY_GROUP_ID_1",
      "projects/PROJECT_ID/locations/LOCATION/entryGroups/ENTRY_GROUP_ID_2"
    ]
  },
  "pubsubTopic": "projects/PROJECT_ID_PUBSUB/topics/TOPIC_ID"
}
EOF
)" "https://${DATAPLEX_API}/metadataFeeds?metadataFeedId=FEED_ID"

Replace the following:

PROJECT_ID_PUBSUB: the project ID where the Pub/Sub topic is located—for example, example-project
LOCATION: the location where you want to create the feed—for example, us-central1
ENTRY_GROUP_ID: the entry group ID to monitor—for example, example-entry-group
TOPIC_ID: the Pub/Sub topic ID to which the metadata change feed messages are published—for example, example-topic
FEED_ID: the metadata change feed ID that you want to create—for example, example-feed

With filters

Run the following command to create a metadata change feed with filters (filtering by entry type):

gcurl -X POST -d "$(cat <<EOF
{
  "scope": {
    "projects": [
      "projects/PROJECT_ID_1",
      "projects/PROJECT_ID_2"
    ]
  },
  "filter": {
    "entryTypes": [
      "projects/PROJECT_ID/locations/global/entryTypes/bigquery-table"
    ]
  },
  "pubsubTopic": "projects/PROJECT_ID_PUBSUB/topics/TOPIC_ID"
}
EOF
)" "https://${DATAPLEX_API}/metadataFeeds?metadataFeedId=FEED_ID"

Replace the following:

PROJECT_ID_PUBSUB: the project ID where the Pub/Sub topic is located—for example, example-project
TOPIC_ID: the Pub/Sub topic ID to which the metadata change feed messages are published—for example, example-topic
FEED_ID: the metadata change feed ID that you want to create—for example, example-feed

Response:

{
  "name": "projects/PROJECT_ID/locations/LOCATION/operations/OPERATION_ID",
  "metadata": {
    "@type": "type.googleapis.com/google.cloud.dataplex.v1.OperationMetadata",
    "createTime": "2023-10-02T15:01:23Z",
    "target": "projects/PROJECT_ID/locations/LOCATION/metadataFeeds/FEED_ID",
    "verb": "create",
    "apiVersion": "v1"
  },
  "done": false
}

View metadata change feeds

You can view the details of a metadata change feed.

REST

To view a metadata change feed, use the projects.locations.metadataFeeds.get method.

Run the following command:

gcurl "https://${DATAPLEX_API}/metadataFeeds/FEED_ID"

Replace the following:

FEED_ID: the metadata change feed ID that you want to view—for example, example-feed

List metadata change feeds

You can list the metadata change feeds in a project and location.

REST

To list metadata change feeds, use the projects.locations.metadataFeeds.list method.

Run the following command:

gcurl "https://${DATAPLEX_API}/metadataFeeds"

Update a metadata change feed

You can update the scope or filters of an existing metadata change feed.

REST

To update a metadata change feed, use the projects.locations.metadataFeeds.patch method.

Run the following command to update the metadata change feed to remove the entry type filters:

gcurl -X PATCH -d "$(cat <<EOF
{
  "filter": {
    "entryTypes": []
  }
}
EOF
)" "https://${DATAPLEX_API}/metadataFeeds/FEED_ID?updateMask=filter"

Replace the following:

FEED_ID: the metadata change feed ID that you want to update—for example, example-feed

Delete a metadata change feed

You can delete a metadata change feed if you no longer need to receive notifications.

Deleting a metadata change feed stops the feed from publishing new metadata changes to your Pub/Sub topic. However, it doesn't delete the topic or the subscription associated with the feed. You must delete them manually if they're no longer needed.

Before you delete the feed or the subscription, ensure that your subscriber applications have processed all outstanding messages in the Pub/Sub subscription.

REST

To delete a metadata change feed, use the projects.locations.metadataFeeds.delete method.

Run the following command:

gcurl -X DELETE \
"https://${DATAPLEX_API}/metadataFeeds/FEED_ID"

Replace the following:

FEED_ID: the metadata change feed ID that you want to delete—for example, example-feed

Consume notification messages

After you configure a metadata change feed, Dataplex Universal Catalog publishes messages to the specified Pub/Sub topic when metadata changes occur. To consume these messages, create a Pub/Sub subscription to the topic.

For example, you can create a pull subscription and use the Google Cloud CLI to view messages:

Create a subscription:
```
gcloud pubsub subscriptions create SUBSCRIPTION_ID --topic=TOPIC_ID
```
Replace the following:
- SUBSCRIPTION_ID: the subscription ID that you want to create
- TOPIC_ID: the Pub/Sub topic ID to which the metadata change feed messages are published.
Pull messages from the subscription:
```
gcloud pubsub subscriptions pull SUBSCRIPTION_ID --auto-ack --limit=10
```
Replace the following:
- SUBSCRIPTION_ID: the subscription ID from which you want to pull messages

For more information about processing Pub/Sub messages, see Receive messages from a pull subscription. For information about the message format, see Data payload.

What's next

Read the Metadata change feeds overview.
Learn about Dataplex Universal Catalog IAM and access control.
Troubleshoot metadata change feeds.

Receive notifications with metadata change feeds Stay organized with collections Save and categorize content based on your preferences.

Before you begin

Required roles and permissions

User roles and permissions

Required permissions

Dataplex Universal Catalog service account roles and permissions

Grant permissions to the Dataplex Universal Catalog service account

gcloud

Console

Create a metadata change feed

REST

View metadata change feeds

REST

List metadata change feeds

REST

Update a metadata change feed

REST

Delete a metadata change feed

REST

Consume notification messages

What's next

Receive notifications with metadata change feeds