Manage Looker (Google Cloud core) resources with Dataplex Universal Catalog

Preview

This product or feature is subject to the "Pre-GA Offerings Terms" in the General Service Terms section of the Service Specific Terms. Pre-GA products and features are available "as is" and might have limited support. For more information, see the launch stage descriptions.

Note: Use the issue report and feedback form to provide feedback or report issues with this preview feature.

This page explains how you can search for and manage your Looker (Google Cloud core) metadata resources by using Dataplex Universal Catalog.

Dataplex Universal Catalog is a platform that stores, manages, and provides access to your metadata. It acts as a cataloging tool that indexes production LookML, LookML project, and dashboard metadata from Looker (Google Cloud core). This integration provides a unified and comprehensive view of data assets and their relationships, including data lineage.

Dataplex Universal Catalog captures multiple entity levels from Looker, which include the following:

• Looker instance
• Dashboard
• Dashboard element
• Look
• LookML project
• Model
• Explore
• View (which contains dimensions and measures)

How the integration works

The Looker and Dataplex Universal Catalog integration uses a first-party connector for metadata ingestion. Looker (Google Cloud core) publishes LookML metadata to regional Cloud Storage repositories for Dataplex Universal Catalog to consume through a batch sync process.

Data transmission from Looker (Google Cloud core) to Dataplex Universal Catalog is automatic by default and requires no configuration changes. Looker (Google Cloud core) uses internal services to perform the synchronization. The internal batch sync process runs approximately every four hours, and data from Looker refreshes approximately hourly into Dataplex Universal Catalog.

When you update your LookML models and push them to production, the system automatically synchronizes key metadata with Dataplex Universal Catalog. This metadata includes view and Explore details, field definitions, and model information. Similarly, dashboard metadata, including titles, ownership, and usage statistics, appears in the catalog.

When to use Dataplex Universal Catalog with Looker

Use Dataplex Universal Catalog to discover and understand your Looker metadata. The catalog aids with the following activities:

• Impact Analysis: Use data lineage to visualize how data flows from BigQuery to Looker dashboards.
• Discovery: Search for dashboards, Explores, and views across your organization.
• Change management: Understand dependencies before making changes.
• Data movement pipelines: Track data provenance.

With Dataplex Universal Catalog, you can also curate metadata by attaching aspects to Looker metadata entries. Attaching aspects to Looker assets lets you add business metadata to the assets and search for assets by that metadata.

Required permissions

To get the permissions that you need to use the Dataplex Universal Catalog and Looker (Google Cloud core) integration, ask your administrator to grant you the following IAM roles on project where your Looker (Google Cloud core) instance resides:

• To view Dataplex Universal Catalog resources: Dataplex Viewer (roles/dataplex.viewer)
• To access the Looker (Google Cloud core) metadata on Dataplex Universal Catalog: Looker Schema Viewer (roles/looker.schemaViewer)
  • The Looker Schema Viewer predefined role contains the looker.schemas.view permission, which is required to access Looker (Google Cloud core) metadata on Dataplex Universal Catalog.
• To disable or enable the integration between Dataplex Universal Catalog and Looker (Google Cloud core): Looker Admin (roles/looker.admin)
• To view data lineage graphs: Data Lineage Viewer (roles/datalineage.viewer)

For more information about granting roles, see Manage access to projects, folders, and organizations.

You might also be able to get the required permissions through custom roles or other predefined roles.

Caution: You must grant the Looker Schema Viewer role or the looker.schema.view permission at the project level. Users with this permission can view the metadata for all Looker (Google Cloud core) instances in the project that have the integration enabled.

Enable the integration of Dataplex Universal Catalog and your Looker (Google Cloud core) instance

Dataplex Universal Catalog is enabled on Looker (Google Cloud core) instances by default. To disable the integration, see the Disable the integration of Dataplex Universal Catalog and your Looker (Google Cloud core) instance section of this documentation page.

If the integration is disabled on your instance and you would like to enable it, complete the following steps:

console

1. Enable the Dataplex API. In order to view the metadata being ingested, you must enable the Cloud Dataplex API (dataplex.googleapis.com) on the Google Cloud project that hosts your Looker (Google Cloud core) instance.

   Enable the Dataplex API

2. On the Details tab of the instance's Instance page, click Edit.
3. Expand the Universal catalog integration section.
4. Deselect the Opt out of Dataplex Catalog integration checkbox.
   • Note: Enabling this setting enables both metadata ingestion and data lineage (if the Data Lineage API is enabled).
5. Click Save.

gcloud

Be sure the Dataplex API is enabled on the Google Cloud project that hosts your Looker (Google Cloud core) instance.

Use the gcloud looker instances update command with the --catalog-integration-enabled flag to enable the integration:

gcloud looker instances update INSTANCE_NAME --catalog-integration-enabled

Replace INSTANCE_NAME with the name of the Looker (Google Cloud core) instance.

Once the integration is enabled, it will take up to a day for Looker data to be available in Dataplex Universal Catalog.

Search for Looker assets

To view Looker (Google Cloud core) data in Dataplex Universal Catalog, you must first enable the Dataplex API. To get the permission that you need to enable APIs, you must have the Service Usage Admin (roles/serviceusage.serviceUsageAdmin) IAM role.

Enable the Dataplex API

Use the Dataplex Universal Catalog Search page in the Google Cloud console to search for Looker (Google Cloud core) assets.

Go to Search

In the Filters panel, you can filter for Looker (Google Cloud core) assets by using the Project, System and Type aliases sections. In the System section, you can filter for Looker. In Type aliases, you can filter the search results to a specific type of Looker asset by selecting one or more of the following:

• Dashboard
• Dashboard element
• Explore
• LookML project
• Model
• Instance
• View
• Other

You can also use the search field to perform search queries. You can perform a keyword or natural language search. For example, to view all Looker assets through keyword search, enter system=Looker. To view all Looker dashboards, enter the query system=Looker AND type=DASHBOARD. The following assets are available for search with the type keyword:

• dashboard
• dashboard_element
• explore
• instance
• look
• lookml_project
• model
• view

To learn more about searching for resources, see the Search for resources in Dataplex Universal Catalog documentation page. To learn more about the expressions that you can use in the search field, see the Search syntax for Dataplex Universal Catalog documentation page.

Disable the integration of Dataplex Universal Catalog and your Looker (Google Cloud core) instance

To disable the integration of Dataplex Universal Catalog and your Looker (Google Cloud core) instance, complete the following steps.

console

1. On the Details tab of the instance's Instance page, click Edit.
2. Expand the Universal catalog integration section.
3. Select the Opt out of Dataplex Catalog integration checkbox.
   • Note: Disabling this setting disables both metadata ingestion and data lineage (if the Data Lineage API is enabled).
4. Click Save.

gcloud

Use the gcloud looker instances update command with the --no-catalog-integration-enabled flag to enable the integration:

gcloud looker instances update INSTANCE_NAME --no-catalog-integration-enabled

Replace INSTANCE_NAME with the name of the Looker (Google Cloud core) instance.

When the integration is disabled after having been enabled, the Looker metadata that is already in Dataplex Universal Catalog will remain in Dataplex Universal Catalog, but it won't be updated with new metadata.

VPC Service Controls

The integration between Looker (Google Cloud core) and Dataplex Universal Catalog supports VPC Service Controls.

If your Looker (Google Cloud core) instance or Dataplex Universal Catalog resources are within a VPC Service Controls service perimeter, you must ensure that the services required for metadata export are accessible.

• Ensure that Dataplex API (dataplex.googleapis.com) is configured as a restricted service within your perimeter.
• The integration uses internal Cloud Storage buckets to transfer metadata. Ensure that your service perimeter allows access to the necessary Cloud Storage resources. Contact your Google Cloud account team or support if you encounter PERMISSION_DENIED errors that arerelated to metadata export.

Pricing

There is no charge for storing Looker (Google Cloud core) technical metadata in Dataplex Universal Catalog. Standard Dataplex Universal Catalog pricing applies for API calls and additional business metadata enrichment. For more information, see the Dataplex Universal Catalog pricing page.

Limitations

This section lists limitations with using Looker (Google Cloud core) and Dataplex Universal Catalog:

• The metadata from Looker is not instantly available in the catalog. The batch sync process runs on a set interval, and data refresh occurs approximately hourly.
• There is no public API exposed for this integration.
• For data lineage limitations, see the Track data lineage with Dataplex Universal Catalog page.

What's next

• Track data lineage with Dataplex Universal Catalog.
• Learn about metadata management in Dataplex Universal Catalog.
• Learn how to search for resources in Dataplex Universal Catalog.
• Learn how to manage aspects and enrich metadata.