Regional resources

When you attach a Cloud Storage bucket as a Knowledge Catalog (formerly Dataplex Universal Catalog) asset, Knowledge Catalog creates a publishing dataset in the corresponding region to publish tables discovered in the bucket.

This page describes how Knowledge Catalog maps single, dual, and multiple regions in Cloud Storage to BigQuery publishing datasets.

Mapping of Cloud Storage regions to BigQuery datasets

Knowledge Catalog lakes, zones, and assets are regional resources, and can reside in one or more regions. BigQuery datasets and Cloud Storage buckets are also regional resources that can reside in one or more regions.

The following are the differences between the regional resources available in Cloud Storage and in BigQuery:

  • Both Cloud Storage and BigQuery support single-region resources.

  • Cloud Storage has dual-regions, whereas BigQuery doesn't.

  • Both Cloud Storage and BigQuery have multi-regions, but they are different.

You can attach Cloud Storage buckets and BigQuery datasets to Knowledge Catalog zones or lakes as Knowledge Catalog assets. Knowledge Catalog automates the creation of publishing datasets for Cloud Storage buckets attached as assets.

Knowledge Catalog ensures that the BigQuery and Cloud Storage regions match. If there is no overlap between your Knowledge Catalog lake's region and one of the Cloud Storage bucket's regions, you can't add the bucket to your lake's zone.

  • In the case of a single-region Cloud Storage bucket, Knowledge Catalog creates a single-region publishing dataset in the same region as the bucket.

  • In the case of the Cloud Storage bucket located in either the Cloud Storage us multi-region or the Cloud Storage eu multi-region, Knowledge Catalog creates a publishing dataset in the corresponding BigQuery us or eu multi-region.

  • In the case of dual-region Cloud Storage buckets, Knowledge Catalog creates a publishing dataset in the region corresponding to the region of the lake. When you attach the Cloud Storage bucket as an asset to the lake, Knowledge Catalog validates that one of the data locations of the Cloud Storage bucket matches the region of the Knowledge Catalog lake.

What's next