Manage data assets using Knowledge Catalog

Knowledge Catalog automatically catalogs metadata about Bigtable resources. This feature helps you search for and manage your Bigtable data assets.

Metadata is automatically synced from Bigtable to Knowledge Catalog for the following Bigtable resources:

  • Instances
  • Clusters
  • Tables, including column family details

Knowledge Catalog is a fully managed metadata management service. It provides unified view and tagging mechanisms for technical and business metadata. To learn more about Knowledge Catalog and why you should use it, see Knowledge Catalog overview.

Benefits of using Knowledge Catalog with Bigtable

Information from Knowledge Catalog about your data can help facilitate analysis, data reuse, application development, and data management.

You can search entries in Knowledge Catalog by details, such as resource name. For example, you can search for an instance name to find the entry for an instance or search for a column family to find the entry for a table. You can also use a faceted search, applying filters to narrow down your search. For more information, see Search for resources in Knowledge Catalog.

Tagging data assets can help you understand and catalog the data that is available in your Bigtable tables to make the data more discoverable and searchable by Knowledge Catalog users, and as part of your larger data estate. You can use aspects to add your own metadata to your cataloged metadata, and then you can use those aspects to conduct searches.

Get started with Knowledge Catalog

To start using Knowledge Catalog to manage your Bigtable data assets, complete the following tasks:

  1. Enable the Dataplex API.

  2. Grant IAM roles to principals who require access to Bigtable entries in Knowledge Catalog.

  3. Learn how to search for data assets.

  4. Review Knowledge Catalog search syntax to understand how to find your metadata.

  5. Read the concepts on tagging Knowledge Catalog entries. For more information, see About metadata management in Knowledge Catalog.

Ways to search for Bigtable metadata

You can search synced Bigtable metadata using Knowledge Catalog.

In addition to qualified predicates, you can search for Bigtable resources with the following metadata fields:

  • Instance

    • Bigtable instance name
    • Bigtable instance ID
    • Cluster location
    • Cluster ID

  • Table

    • Table ID
    • Column family name
    • Bigtable instance name
    • Bigtable instance ID

Required roles

Viewing, tagging, and searching Bigtable resources in Knowledge Catalog have different permission requirements. In general, grant principals in your organization the minimum role required to accomplish a task.

Permission to view public and private aspects

To get the permissions that you need to view public and private aspects on Bigtable resources, ask your administrator to grant you the following IAM roles:

These predefined roles contain the permissions required to view public and private aspects. The exact permissions that are required are the following:

  • bigtable.instances.get
  • bigtable.clusters.get
  • bigtable.tables.get
  • dataplex.aspectTypes.get

Permission to search Bigtable resources

To get the permissions that you need to search Bigtable resources in Knowledge Catalog, ask your administrator to grant you the following IAM roles:

These predefined roles contain the permissions required to search Bigtable resources. The exact permissions that are required are the following:

  • bigtable.instances.get
  • bigtable.clusters.get
  • bigtable.tables.get

Permission to attach aspects to Bigtable resources

To get the permissions that you need to attach aspects to Bigtable resources in Knowledge Catalog, ask your administrator to grant you the following IAM roles:

These predefined roles contain the permissions required to add public and private aspects. The exact permissions that are required are the following:

  • bigtable.instances.update
  • bigtable.clusters.update
  • bigtable.tables.update
  • dataplex.aspectTypes.use
  • dataplex.entries.update

Optional: Create a custom role

You can create a custom role to grant to users who require complete access to all Bigtable metadata in Knowledge Catalog across your project or organization. Then you can make any user or service account that needs access to a principal of the custom role. To learn more about custom roles, see Roles and permissions.

Choose a meaningful name for the role, such as Bigtable Dataplex Editor, and then assign only the principals who need access to the role. For setup instructions, see Create and manage custom roles.

Grant the following permissions to the custom role:

  • bigtable.instances.get
  • bigtable.instances.update
  • bigtable.clusters.get
  • bigtable.clusters.update
  • bigtable.tables.get
  • bigtable.tables.update
  • dataplex.aspectTypes.get
  • dataplex.aspectTypes.use

For more information, see Knowledge Catalog IAM permissions. For descriptions of Bigtable permissions, see Permissions.

Limitations

The following section lists constraints that you must consider before you use Knowledge Catalog to manage your Bigtable metadata.

Location constraints

If your organization has set up location constraints, metadata entries for your Bigtable resources might not be synced to Knowledge Catalog. Any entries created before enforcing the location constraints are retained by Knowledge Catalog.

To allow Knowledge Catalog to load resources, we recommend that you add the global multi-region to your location policy.

Column family name exception

The Dataplex API rejects requests where the column family name in your request includes a period (.). For example, if the column family name is personal.info.

What's next