Create data products

Preview

This product or feature is subject to the "Pre-GA Offerings Terms" in the General Service Terms section of the Service Specific Terms. Pre-GA products and features are available "as is" and might have limited support. For more information, see the launch stage descriptions.

This document is intended for data product owners who want to create and configure data products in Dataplex Universal Catalog.

For more information about the architecture and key concepts of data products, see About data products.

Before you begin

1. Enable the Dataplex, BigQuery APIs.

   Roles required to enable APIs

   To enable APIs, you need the Service Usage Admin IAM role (roles/serviceusage.serviceUsageAdmin), which contains the serviceusage.services.enable permission. Learn how to grant roles.

   Enable the APIs

2. Ensure that your data assets (for example, BigQuery datasets, tables, and views) are created and populated.

   For more information about creating data assets, see the following documents:

   • Create BigQuery datasets
   • Create and use BigQuery tables
   • Create logical views in BigQuery
   • Create materialized views in BigQuery

3. Identify or create the Google groups that you want to configure in your data product. Each data product must have a unique Google group.

Required roles

This section outlines the minimum IAM roles required for two primary user groups: data product owners (those who create and manage data products) and data product consumers (those who search for and use data products).

Roles for a data product owner

To get the permissions that you need to create and manage data products, ask your administrator to grant you the following IAM roles on the project:

• Full permissions to create, update, delete, and manage permissions for data products: Dataplex Data Products Admin (roles/dataplex.dataProductsAdmin)
• Update and manage existing data products: Dataplex Data Products Editor (roles/dataplex.dataProductsEditor)
• Add aspects such as schema, overview, and contacts: Dataplex Entry and EntryLink Owner (roles/dataplex.entryOwner)
• Search for and add assets: Dataplex Catalog Viewer (roles/dataplex.catalogViewer)
• Edit system aspect types such as documentation and contract aspect of refresh cadence: Dataplex Catalog Editor (roles/dataplex.catalogEditor)

For more information about granting roles, see Manage access to projects, folders, and organizations.

These predefined roles contain the permissions required to create and manage data products. To see the exact permissions that are required, expand the Required permissions section:

Required permissions

The following permissions are required to create and manage data products:

• Edit the overview system aspect type: dataplex.entryGroups.useOverviewAspect
• Edit the refresh cadence system aspect type: dataplex.entryGroups.useRefreshCadenceAspect

You might also be able to get these permissions with custom roles or other predefined roles.

Roles for a data product consumer

For the data product consumers to search for, view, and request access to data products, as a data product owner, you must ensure the data product is discoverable. To do this, grant the data product consumers the following IAM roles on the data product:

• Search for data products and access them in Dataplex Universal Catalog Search: Dataplex Data Product Consumer (dataplex.dataProductsConsumer)
• Read-only access to view data product definitions and metadata: Dataplex Data Product Viewer (dataplex.dataProductViewer)
• Request access to data products: Dataplex Data Product Consumer (dataplex.dataProductsConsumer)

Create and configure a data product

Creating a data product involves the following high-level tasks:

1. Create a data product

   This mandatory initial step requires defining core details such as a unique data product name, description, region where the data product is created, and owner details.

2. Optional: Add assets

   In this phase, you select assets to include in the data product. A key constraint is that assets must reside in the same region as the data product itself. You can add a maximum of 10 assets to a data product.

   For the list of supported assets, see Assets supported.

3. Optional: Configure access groups and asset permissions

   In this optional phase, you simplify access control by creating access groups. These access groups act as user-friendly aliases (for example, Analyst or Reader) for underlying Google groups. You then assign permissions by selecting a specific IAM role and mapping it to an access group for a specific asset.

4. Optional: Add additional details such as contracts, aspects, and documentation

   This optional phase enhances governance and metadata. You can add a contract, which is a first-party aspect type, to formally communicate the agreed-upon data refresh cadence, specifying parameters like refresh frequency, refresh time, and threshold. You also include Aspects to provide additional metadata for your data product. Additionally, you add rich text documentation, such as user guides and sample queries.

Note: You can perform the optional steps during data product creation or post creation.

To create and configure a data product, complete the steps in the following sections:

Create a data product

1. In the Google Cloud console, go to the Dataplex Universal Catalog Data products page.

   Go to Data products

2. Click Create.

3. In the Create data products pane, enter the following details:

   • Data product name: Enter a unique name for your data product.
   • Data product ID: This is an auto-generated unique identifier. You can edit this field.
   • Project ID: This is a unique identifier of the project where the data product is created. Browse and select the project.
   • Region: Select the region or multi-region where the data product is created.
   • Icon: Browse and select an icon to visually identify the data product. This is optional.
   • Description: Enter a brief description of the data product.
   • Contacts: Enter the email ID of the data product owner.
   • Labels: Add key-value labels to organize your resources. This is optional.

4. Click Create data product.

Optional: Add assets

1. In the Add assets pane, click +Add.

2. Search for and select the assets that you want to add to your data product. The assets you select must reside in the same region as the data product.

   If you have necessary permissions, you can view the metadata of assets by clicking the asset.

3. To refine the search results, use Filters.

4. After you select the assets, click Add.

   Note: In preview, you can add a maximum of 10 assets to a data product.

5. Click Continue.

Optional: Configure access groups and asset permissions

In the Configure access groups and asset permissions pane, you can create access groups and assign permissions to assets.

Configure access groups

To configure access groups, follow these steps:

1. Click Add access group.

2. In the Access group name field, enter a name for the access group. For example, Analyst.

3. In the Access group description field, enter a description for the access group.

4. In the Access group identifier field, enter the email address of a Google group that you want to assign to this access group. Data product consumers who request access to this access group can be added as members to the mapped Google group.

   If you don't have a Google group, you can create one. For more information, see Create and manage Google groups in the Google Cloud console.

   Note: You can add a maximum of three access groups per data product.

5. Click Add.

Configure asset permissions

After you configure access groups, you can configure permissions for the assets in the data product:

1. In the Asset permissions section, select the asset for which you want to configure permissions.

2. Click Configure permissions.

3. In the Select access group field, select an access group.

4. In the Assign IAM role field, select an IAM role that you want to assign to the access group.

   For example, if your asset is a BigQuery table named Sales, and if you selected Analyst access group, and assigned BigQuery Metadata Viewer role to this access group, the data product consumers who are part of the Analyst access group have BigQuery Metadata Viewer permission on the Sales table.

   You can add multiple roles to an asset.

5. Click Configure. The asset now shows its assigned permissions.

6. To configure permissions for other assets, repeat the steps.

7. Click Continue.

Optional: Add additional details

In the Add additional details pane, you can add contracts, aspects, and additional documentation for the data product.

Add a contract

To add a contract, follow these steps:

1. Click Add contract.

2. In the Select contract field, select Refresh cadence.

3. In the Frequency field, select an agreed-upon schedule for how often data is updated or delivered, ensuring a predictable flow from data producer to data consumer. For example, Weekly.

4. In the Refresh time field, enter a maximum acceptable time when data is updated at its source and when it becomes available to the consumer. For example, 23:00 PST.

5. In the Threshold (in minutes) field, enter a measurable limit in minutes for the acceptable delay in data delivery. For example, enter 30 to set threshold as 30 minutes.

6. Optional: In the Cron schedule field, enter a cron expression that defines the schedule for data generation and delivery in the format: MINUTE HOUR DAY_OF_MONTH MONTH DAY_OF_WEEK

   The following are the accepted values:

   • MINUTE: 0-59
   • HOUR: 0-23
   • DAY_OF_MONTH: 1-31
   • MONTH: 1-31 or JAN-DEC
   • DAY_OF_WEEK: 0-6 or SUN-SAT

   For example, 0 8 * * 1-5 runs at 8:00 AM on weekdays (Monday-Friday).

7. Click Save.

Add additional metadata

To add additional metadata for the data product as aspects, follow these steps:

1. Click + Add aspect.

2. In the Select aspect type field, search for and select an aspect type from the list. For example, Geo context.

3. In the Country field, select the country to which the asset belongs.

4. In the Region field, select the business region to which the asset belongs.

5. Click Save.

   To add additional documentation such as user guide or sample queries, click Edit next to Documentation. This opens a rich text editor. Add content and click Save.

6. Click Save.

   The newly created data product appears on the Dataplex Universal Catalog Data products page.

What's next

• Learn more about managing data products.
• Learn how to search for data products.
• As a data consumer, learn how to request access to data products.