Import metadata from dbt Core

Preview

This feature is subject to the "Pre-GA Offerings Terms" in the General Service Terms section of the Service Specific Terms. Pre-GA features are available "as is" and might have limited support. For more information, see the launch stage descriptions.

This document describes how to import metadata from dbt Core and MetricFlow into Knowledge Catalog (formerly Dataplex Universal Catalog) using the gcloud command.

The following metadata is captured by the dbt integration:

• Technical metadata: this includes key resources (sources, seeds, models) and their technical properties (column names, data types, row counts).
• Business and semantic metadata: powered by dbt MetricFlow, this includes business definitions and logic like semantic models, metrics, and saved queries.
• Operational and data quality metadata: this includes execution metadata like timing, success or failure status, data freshness, test and test results.
• Lineage and relationship metadata: this includes transformation graphs (DAGs) and dependencies between dbt resources, physical lineage that tracks and links physical transformation blocks, join keys and dynamic joins, and parent-child relationships.
• Consumption metadata: this includes metadata captured in exposures that map how data is being used outside of dbt.

Before you can import metadata from dbt Core and MetricFlow, complete the following tasks:

1. Grant the required roles and permissions.
2. Enable the Knowledge Catalog API.
3. Meet the dbt prerequisites.
4. Create the destination entry group if it does not already exist.
5. Understand the Cloud Storage roles.

IAM roles and permissions

To create and manage a Knowledge Catalog connector job, you need Identity and Access Management (IAM) roles that grant permissions for Knowledge Catalog and Cloud Storage.

To get the permissions that you need to configure a dbt connector, ask your administrator to grant you the following IAM roles:

• To create and manage entry groups: Dataplex Catalog Admin (roles/dataplex.catalogAdmin), Dataplex Catalog Editor (roles/dataplex.catalogEditor), or Dataplex Entry Group Owner (roles/dataplex.entryGroupOwner) on the project.

• To execute the dbt gcloud command and create metadata import jobs: To follow the principle of least privilege, grant the following roles:

  • Dataplex Metadata Job Owner (roles/dataplex.metadataJobOwner) on the project.
  • Dataplex Entry Group Importer (roles/dataplex.entryGroupImporter) on the target entry group or the project.

  Alternatively, you can grant the Dataplex Catalog Admin (roles/dataplex.catalogAdmin) role and the Dataplex Metadata Job Owner (roles/dataplex.metadataJobOwner) role on the project.

• To upload transformed metadata to the output staging bucket (--storage-uri): Storage Object Creator (roles/storage.objectCreator) or Storage Object Admin (roles/storage.objectAdmin) on the staging bucket.

• To read dbt artifacts from an input Cloud Storage bucket (--artifacts-path, if using Cloud Storage): Storage Object Viewer (roles/storage.objectViewer) or Storage Object Admin (roles/storage.objectAdmin) on the input artifacts bucket. If you have the Storage Object Admin role, the Storage Object Viewer role is not required.

• To view dbt metadata: Dataplex Catalog Viewer (roles/dataplex.catalogViewer) on the project.

• To view logs in Cloud Logging: Logs Viewer (roles/logging.viewer) on the project.

Additionally, you must grant the Knowledge Catalog service agent ([e-mail]) the Storage Object Viewer (roles/storage.objectViewer) role on the output staging Cloud Storage bucket (--storage-uri) so that the import job can read the staged metadata file.

Note: The import job also requires permission to use the first-party dbt aspect types (dataplex.aspectTypes.use). This permission is granted by default on the public system projects hosting the types.

For more information about granting roles, see Manage access.

Enable APIs

Enable the Knowledge Catalog API.

Enable the API

dbt prerequisites

To import the full set of dbt metadata, we recommend producing all four dbt JSON artifact files. Only manifest.json is required; the others enrich the import and the transform degrades gracefully without them:

• manifest.json (required): Core project structure and execution graph. Also carries the MetricFlow semantic models, metrics, and saved queries.
• catalog.json: Column names and data types. Without catalog.json, the schema aspect is imported with untyped columns.
• run_results.json: Test outcomes and execution metadata.
• sources.json: Source freshness.

To generate the full set of dbt metadata artifact JSON files, you can execute the following dbt commands in this order:

1. dbt source freshness
2. dbt build

3. dbt docs generate --no-compile

   Warning: Without --no-compile, dbt docs generate overwrites run_results.json with a success status for every node. Test results from the preceding dbt build are lost and the catalog reports every data quality test as passing.

Understand Cloud Storage roles

Importing dbt metadata involves two distinct Cloud Storage locations that serve different purposes and shouldn't be conflated:

• Input (dbt source artifacts): Where your generated dbt JSON files reside. This can be a local directory path on your machine or CI runner (such as ./target/ or .) or an input Cloud Storage bucket URI prefix (such as gs://my-dbt-artifacts-bucket/target/). You provide this path using the --artifacts-path flag. The gcloud command reads these input files during job preparation. The caller executing the gcloud command needs read access (roles/storage.objectViewer or roles/storage.objectAdmin) if using Cloud Storage. The Knowledge Catalog service agent does not need access to the input artifacts bucket.
• Output (Knowledge Catalog import staging bucket): A Cloud Storage bucket URI prefix (such as gs://my-staging-bucket/dbt-imports/) where the gcloud command uploads the transformed metadata import file (dbt_metadata.jsonl), and from which the Knowledge Catalog import job reads during ingestion. You provide this URI using the --storage-uri flag. The caller executing the gcloud command needs write access (roles/storage.objectCreator or roles/storage.objectAdmin) to upload the file, and the Knowledge Catalog service agent needs read access (roles/storage.objectViewer) to import it.

Configure dbt connectivity

To establish dbt connectivity, you must first run the appropriate dbt commands to generate the metadata artifacts. Once the JSON files are stored and accessible, you can use the gcloud alpha dataplex dbt metadata-jobs create command to:

1. Read input artifacts: Read the JSON artifacts generated by dbt Core and MetricFlow from the input location (local directory or Cloud Storage URI specified in --artifacts-path).
2. Transform metadata: Transform the content into the Knowledge Catalog metadata import format (dbt_metadata.jsonl).
3. Upload to staging: Upload the transformed metadata import file to the output staging Cloud Storage location specified in --storage-uri.
4. Trigger import job: Trigger a Knowledge Catalog metadata import job that instructs the Knowledge Catalog service agent to read and ingest the staged metadata from --storage-uri into Knowledge Catalog resources.

To create a dbt metadata job, complete the following steps:

1. Ensure the dbt metadata artifact files are stored locally or in an input Cloud Storage bucket.
2. Ensure you have an output staging Cloud Storage bucket configured with the appropriate permissions for both the caller and the Knowledge Catalog service agent.

3. From Cloud Shell, a local terminal, or an automated workflow tool, execute the gcloud command:

   gcloud alpha dataplex dbt metadata-jobs create my-dbt-import \
       --project=my-project \
       --location=us-central1 \
       --artifacts-path=. \
       --entry-group=dbt-metadata-ingestion \
       --storage-uri=gs://my-bucket/dbt-imports/
   

   Required flags

   • --storage-uri=STORAGE_URI: (Output/Staging) Cloud Storage URI prefix (gs://bucket/path/) where the transformed JSONL is uploaded to and where the import job reads from during ingestion. The caller must have write access (roles/storage.objectCreator or roles/storage.objectAdmin), and the Knowledge Catalog service agent must have read access (roles/storage.objectViewer).

   Optional flags

   • --artifacts-path=ARTIFACTS_PATH: (Input) Path to the source dbt artifacts. This can be a local directory path (such as . or ./target) or a Cloud Storage URI prefix (such as gs://my-bucket/dbt-artifacts/). May point at the dbt project root (the target/ subdirectory is detected automatically) or directly at the directory containing manifest.json. Defaults to .. If a Cloud Storage URI is provided, the caller must have read access (roles/storage.objectViewer or roles/storage.objectAdmin) to the input bucket.
   • --async: Return immediately, without waiting for the operation in progress to complete.
   • --entry-group=ENTRY_GROUP: Short ID of the entry group that receives the dbt entries. Must already exist in the project and location (default is dbt-metadata-ingestion).
   • --aspects-only: Update only the metadata this dbt run observed and leave the rest of the entry group untouched. No entry is created, deleted, or re-parented, and an aspect whose dbt artifact was absent from this run keeps the value a previous run gave it. Use this for routine, repeated ingestion. See Re-run ingestion.
   • --validate-only: Build and upload the JSON and validate the metadata job, but don't actually ingest.

4. Confirm you received a Created status.

5. After you create the job, Knowledge Catalog schedules the first run according to your configuration, or you can start it manually.

Re-run ingestion

After the first import, most runs only need to refresh metadata for resources that already exist. Use --aspects-only for those runs. It updates only what the dbt run observed and leaves everything else in the entry group alone, so it is safe to run repeatedly, on any schedule, and from more than one job.

Run a full ingestion (omit --aspects-only) when the set of entries changes:

• The first ingestion into an entry group.
• A dbt resource is added, renamed, or deleted.
• An entry's display name, description, or labels change.
• The entry hierarchy changes.

A full run rewrites every entry's required aspects from the artifacts on disk, so run it from as complete an artifact set as your pipeline can produce.

Run --aspects-only for routine refreshes:

• After whichever dbt command your pipeline runs: dbt build, dbt test, dbt source freshness, or a --select-narrowed rebuild.
• A column is added, removed, retyped, or re-described.
• Model SQL changed and the run also wrote catalog.json.
• New test results or source freshness.

--aspects-only can add and refresh metadata, but cannot remove it.

Search and view dbt metadata

1. In the Google Cloud console, go to the Knowledge Catalog Search page.

   Go to Search

2. In the Filters panel, you can filter for dbt assets by using the Project, System, and Type aliases sections. In the System section, select Imported Context. Selecting this filter opens a Managed Connectors sub-section. Select dbt to filter for all dbt metadata.

3. You can use the search field to perform search queries. You can perform a keyword or natural language search. For example, to view all dbt assets through keyword search, enter system=DBT.

   To learn more about searching for resources, see Search for resources in Knowledge Catalog. To learn more about the expressions that you can use in the search field, see Search syntax for Knowledge Catalog.

4. You can also use the LookupContext API to retrieve LLM context for specific dbt resources.

Limitations

• Supports recent dbt Core v1 versions (validated against versions 1.11 and 1.12). dbt Core v2 and dbt Fusion are not supported.
• dbt models that use model versioning are not supported.
• dbt Cloud is not supported.
• Very large or deeply nested schemas are truncated: a single aspect cannot exceed the per-aspect size cap, so deeply nested schemas might lose trailing fields.
• --aspects-only can add and refresh metadata, but cannot remove it. Deleting a dbt resource requires a full run.
• Entry links are not supported.
• This integration supports only dbt lineage events on BigQuery resources in Data Lineage API and graph. dbt entries (source, seeds, models) for external 3P sources are not captured in data lineage.
  • To ingest all dbt lineage events in Data Lineage API, use the OpenLineage dbt integration. Then, integrate OpenLineage with Knowledge Catalog to import and visualize data lineage from dbt.

What's next

• Learn how to Manage connector jobs.