As of April 10, 2026, Dataplex Universal Catalog is now called Knowledge Catalog. The API, client library, CLI, and IAM names remain unchanged. For more information, see Introducing the Google Cloud Knowledge Catalog.

Import metadata from dbt Core

This document describes how to import metadata from dbt Core and MetricFlow into Knowledge Catalog (formerly Dataplex Universal Catalog) using the gcloud command.

The following metadata is captured by the dbt integration:

Technical metadata: this includes key resources (sources, seeds, models) and their technical properties (column names, data types, row counts).
Business and semantic metadata: powered by dbt MetricFlow, this includes business definitions and logic like semantic models, metrics, and saved queries.
Operational and data quality metadata: this includes execution metadata like timing, success or failure status, data freshness, test and test results.
Lineage and relationship metadata: this includes transformation graphs (DAGs) and dependencies between dbt resources, physical lineage that tracks and links physical transformation blocks, join keys and dynamic joins, and parent-child relationships.
Consumption metadata: this includes metadata captured in exposures that map how data is being used outside of dbt.

Before you can import metadata from dbt Core and MetricFlow, complete the following tasks:

Grant the required roles and permissions.
Enable the Knowledge Catalog API.
Meet the dbt prerequisites.
Create the destination entry group if it does not already exist.
Understand the Cloud Storage roles.

IAM roles and permissions

To create and manage a Knowledge Catalog connector job, you need Identity and Access Management (IAM) roles that grant permissions for Knowledge Catalog and Cloud Storage.

To get the permissions that you need to configure a dbt connector, ask your administrator to grant you the following IAM roles:

To create and manage entry groups: Dataplex Catalog Admin (roles/dataplex.catalogAdmin), Dataplex Catalog Editor (roles/dataplex.catalogEditor), or Dataplex Entry Group Owner (roles/dataplex.entryGroupOwner) on the project.
To execute the dbt gcloud command and create metadata import jobs: To follow the principle of least privilege, grant the following roles:
- Dataplex Metadata Job Owner (roles/dataplex.metadataJobOwner) on the project.
- Dataplex Entry Group Importer (roles/dataplex.entryGroupImporter) on the target entry group or the project.
Alternatively, you can grant the Dataplex Catalog Admin (roles/dataplex.catalogAdmin) role and the Dataplex Metadata Job Owner (roles/dataplex.metadataJobOwner) role on the project.
To upload transformed metadata to the output staging bucket (--storage-uri): Storage Object Creator (roles/storage.objectCreator) or Storage Object Admin (roles/storage.objectAdmin) on the staging bucket.
To read dbt artifacts from an input Cloud Storage bucket (--artifacts-path, if using Cloud Storage): Storage Object Viewer (roles/storage.objectViewer) or Storage Object Admin (roles/storage.objectAdmin) on the input artifacts bucket. If you have the Storage Object Admin role, the Storage Object Viewer role is not required.
To view dbt metadata: Dataplex Catalog Viewer (roles/dataplex.catalogViewer) on the project.
To view logs in Cloud Logging: Logs Viewer (roles/logging.viewer) on the project.

Additionally, you must grant the Knowledge Catalog service agent (service-PROJECT_NUMBER@gcp-sa-dataplex.iam.gserviceaccount.com) the Storage Object Viewer (roles/storage.objectViewer) role on the output staging Cloud Storage bucket (--storage-uri) so that the import job can read the staged metadata file.

For more information about granting roles, see Manage access.

Enable APIs

Enable the Knowledge Catalog API.

Enable the API

dbt prerequisites

To import the full set of dbt metadata, we recommend producing all four dbt JSON artifact files. Only manifest.json is required; the others enrich the import and the transform degrades gracefully without them:

manifest.json (required): Core project structure and execution graph. Also carries the MetricFlow semantic models, metrics, and saved queries.
catalog.json: Column names and data types. Without catalog.json, the schema aspect is imported with untyped columns.
run_results.json: Test outcomes and execution metadata.
sources.json: Source freshness.

To generate the full set of dbt metadata artifact JSON files, you can execute the following dbt commands in this order:

dbt source freshness
dbt build
dbt docs generate --no-compile

Warning: Without --no-compile, dbt docs generate overwrites run_results.json with a success status for every node. Test results from the preceding dbt build are lost and the catalog reports every data quality test as passing.

Understand Cloud Storage roles

Importing dbt metadata involves two distinct Cloud Storage locations that serve different purposes and shouldn't be conflated:

Input (dbt source artifacts): Where your generated dbt JSON files reside. This can be a local directory path on your machine or CI runner (such as ./target/ or .) or an input Cloud Storage bucket URI prefix (such as gs://my-dbt-artifacts-bucket/target/). You provide this path using the --artifacts-path flag. The gcloud command reads these input files during job preparation. The caller executing the gcloud command needs read access (roles/storage.objectViewer or roles/storage.objectAdmin) if using Cloud Storage. The Knowledge Catalog service agent does not need access to the input artifacts bucket.
Output (Knowledge Catalog import staging bucket): A Cloud Storage bucket URI prefix (such as gs://my-staging-bucket/dbt-imports/) where the gcloud command uploads the transformed metadata import file (dbt_metadata.jsonl), and from which the Knowledge Catalog import job reads during ingestion. You provide this URI using the --storage-uri flag. The caller executing the gcloud command needs write access (roles/storage.objectCreator or roles/storage.objectAdmin) to upload the file, and the Knowledge Catalog service agent needs read access (roles/storage.objectViewer) to import it.

Configure dbt connectivity

To establish dbt connectivity, you must first run the appropriate dbt commands to generate the metadata artifacts. Once the JSON files are stored and accessible, the import process performs the following actions:

Read input artifacts: Read the JSON artifacts generated by dbt Core and MetricFlow from the input location (local directory or Cloud Storage URI specified in --artifacts-path).
Transform metadata: Transform the content into the Knowledge Catalog metadata import format (dbt_metadata.jsonl).
Upload to staging: Upload the transformed metadata import file to the output staging Cloud Storage location specified in --storage-uri.
Trigger import job: Trigger a Knowledge Catalog metadata import job that instructs the Knowledge Catalog service agent to read and ingest the staged metadata from --storage-uri into Knowledge Catalog resources.

Console

In the Google Cloud console, go to the Knowledge Catalog Connectors page.

Go to Connectors
Click Add connection.
In the Connectors list, select the dbt Core and MetricFlow card.

Note: Because dbt Core metadata ingestion requires transforming client-side dbt artifact JSON files (manifest.json, catalog.json, run_results.json, sources.json) into the Knowledge Catalog import format before ingestion, selecting this card directs you to the CLI or API instructions.
To view your imported dbt assets, go to the Search page or view the destination Entry groups page.

gcloud

To create a dbt metadata job, complete the following steps:

Ensure the dbt metadata artifact files are stored locally or in an input Cloud Storage bucket.
Ensure you have an output staging Cloud Storage bucket configured with the appropriate permissions for both the caller and the Knowledge Catalog service agent.
From Cloud Shell, a local terminal, or an automated workflow tool, execute the gcloud command:
```
gcloud alpha dataplex dbt metadata-jobs create my-dbt-import \
    --project=my-project \
    --location=us-central1 \
    --artifacts-path=. \
    --entry-group=dbt-metadata-ingestion \
    --storage-uri=gs://my-bucket/dbt-imports/
```
Required flags
- --storage-uri=STORAGE_URI: (Output/Staging) Cloud Storage URI prefix (gs://bucket/path/) where the transformed JSONL is uploaded to and where the import job reads from during ingestion. The caller must have write access (roles/storage.objectCreator or roles/storage.objectAdmin), and the Knowledge Catalog service agent must have read access (roles/storage.objectViewer).
Optional flags
- --artifacts-path=ARTIFACTS_PATH: (Input) Path to the source dbt artifacts. This can be a local directory path (such as . or ./target) or a Cloud Storage URI prefix (such as gs://my-bucket/dbt-artifacts/). May point at the dbt project root (the target/ subdirectory is detected automatically) or directly at the directory containing manifest.json. Defaults to .. If a Cloud Storage URI is provided, the caller must have read access (roles/storage.objectViewer or roles/storage.objectAdmin) to the input bucket.
- --async: Return immediately, without waiting for the operation in progress to complete.
- --entry-group=ENTRY_GROUP: Short ID of the entry group that receives the dbt entries. Must already exist in the project and location (default is dbt-metadata-ingestion).
- --aspects-only: Update only the metadata this dbt run observed and leave the rest of the entry group untouched. No entry is created, deleted, or re-parented, and an aspect whose dbt artifact was absent from this run keeps the value a previous run gave it. Use this for routine, repeated ingestion. See Re-run ingestion.
- --validate-only: Build and upload the JSON and validate the metadata job, but don't actually ingest.
Confirm you received a Created status.

REST

To import dbt metadata using the REST API:

Generate the dbt artifacts and transform them into the Knowledge Catalog JSON import file (dbt_metadata.jsonl).
Upload the transformed file to your Cloud Storage staging bucket (gs://BUCKET_NAME/PATH/).

Call the projects.locations.metadataJobs.create method:

curl -X POST \
    -H "Authorization: Bearer $(gcloud auth print-access-token)" \
    -H "Content-Type: application/json" \
    https://dataplex.googleapis.com/v1/projects/PROJECT_ID/locations/LOCATION/metadataJobs?metadataJobId=JOB_ID \
    -d '{
      "type": "IMPORT",
      "importSpec": {
        "sourceStorageUri": "gs://BUCKET_NAME/PATH/",
        "entrySyncMode": "FULL",
        "aspectSyncMode": "INCREMENTAL",
        "scope": {
          "entryGroups": [
            "projects/PROJECT_ID/locations/LOCATION/entryGroups/ENTRY_GROUP"
          ],
          "entryTypes": [
            "projects/dataplex-connector-types/locations/global/entryTypes/dbt-project",
            "projects/dataplex-connector-types/locations/global/entryTypes/dbt-model",
            "projects/dataplex-connector-types/locations/global/entryTypes/dbt-source",
            "projects/dataplex-connector-types/locations/global/entryTypes/dbt-seed",
            "projects/dataplex-connector-types/locations/global/entryTypes/dbt-snapshot",
            "projects/dataplex-connector-types/locations/global/entryTypes/dbt-group",
            "projects/dataplex-connector-types/locations/global/entryTypes/dbt-exposure",
            "projects/dataplex-connector-types/locations/global/entryTypes/dbt-metric",
            "projects/dataplex-connector-types/locations/global/entryTypes/dbt-macro",
            "projects/dataplex-connector-types/locations/global/entryTypes/dbt-semantic-model",
            "projects/dataplex-connector-types/locations/global/entryTypes/dbt-saved-query",
            "projects/dataplex-connector-types/locations/global/entryTypes/dbt-test"
          ],
          "aspectTypes": [
            "projects/dataplex-connector-types/locations/global/aspectTypes/dbt-node",
            "projects/dataplex-connector-types/locations/global/aspectTypes/dbt-project",
            "projects/dataplex-connector-types/locations/global/aspectTypes/dbt-model",
            "projects/dataplex-connector-types/locations/global/aspectTypes/dbt-source",
            "projects/dataplex-connector-types/locations/global/aspectTypes/dbt-seed",
            "projects/dataplex-connector-types/locations/global/aspectTypes/dbt-snapshot",
            "projects/dataplex-connector-types/locations/global/aspectTypes/dbt-group",
            "projects/dataplex-connector-types/locations/global/aspectTypes/dbt-exposure",
            "projects/dataplex-connector-types/locations/global/aspectTypes/dbt-metric",
            "projects/dataplex-connector-types/locations/global/aspectTypes/dbt-macro",
            "projects/dataplex-connector-types/locations/global/aspectTypes/dbt-semantic-model",
            "projects/dataplex-connector-types/locations/global/aspectTypes/dbt-saved-query",
            "projects/dataplex-connector-types/locations/global/aspectTypes/dbt-data-quality",
            "projects/dataplex-connector-types/locations/global/aspectTypes/dbt-model-contracts"
          ]
        }
      }
    }'

Replace the following:

PROJECT_ID: the Google Cloud project ID where your entry group is located.
LOCATION: the region of your entry group (for example, us-central1).
JOB_ID: a unique identifier for the metadata job.
BUCKET_NAME/PATH: the Cloud Storage URI prefix where dbt_metadata.jsonl was uploaded.
ENTRY_GROUP: the short ID of the destination entry group.

To track the status of your import job, use the projects.locations.metadataJobs.get method:

curl -X GET \
    -H "Authorization: Bearer $(gcloud auth print-access-token)" \
    https://dataplex.googleapis.com/v1/projects/PROJECT_ID/locations/LOCATION/metadataJobs/JOB_ID

After you create the job, Knowledge Catalog schedules the first run according to your configuration, or you can start it manually.

Re-run ingestion

After the first import, most runs only need to refresh metadata for resources that already exist. Use --aspects-only for those runs. It updates only what the dbt run observed and leaves everything else in the entry group alone, so it is safe to run repeatedly, on any schedule, and from more than one job.

Run a full ingestion (omit --aspects-only) when the set of entries changes:

The first ingestion into an entry group.
A dbt resource is added, renamed, or deleted.
An entry's display name, description, or labels change.
The entry hierarchy changes.

A full run rewrites every entry's required aspects from the artifacts on disk, so run it from as complete an artifact set as your pipeline can produce.

Run --aspects-only for routine refreshes:

After whichever dbt command your pipeline runs: dbt build, dbt test, dbt source freshness, or a --select-narrowed rebuild.
A column is added, removed, retyped, or re-described.
Model SQL changed and the run also wrote catalog.json.
New test results or source freshness.

--aspects-only can add and refresh metadata, but cannot remove it.

Search and view dbt metadata

Console

In the Google Cloud console, go to the Knowledge Catalog Search page.

Go to Search
In the Filters panel, filter for dbt assets:
- In the System section, select Imported Context.
- In the Managed Connectors subsection that appears, select dbt.
In the search field, enter your query using keyword or natural language search. For example, to view all dbt assets using keyword search, enter system=DBT or system=DBT AND type=dbt-model.
In the search results, click any dbt asset to open its entry details page to view its schema, lineage, and technical aspects.

gcloud

To search for dbt entries across your project, use the gcloud dataplex entries search command:

gcloud dataplex entries search 'system=DBT' \
    --project=PROJECT_ID

To filter by a specific dbt entry type (such as models or sources):

gcloud dataplex entries search 'system=DBT AND type=dbt-model' \
    --project=PROJECT_ID

To view the full details and aspects of a specific dbt entry, use the gcloud dataplex entries lookup command:
```
gcloud dataplex entries lookup ENTRY_ID \
    --project=PROJECT_ID \
    --location=LOCATION \
    --entry-group=ENTRY_GROUP \
    --view=FULL
```
Replace the following:
- PROJECT_ID: your Google Cloud project ID.
- LOCATION: the location of the entry group (for example, us-central1).
- ENTRY_GROUP: the short ID of your destination entry group (for example, dbt-metadata-ingestion).
- ENTRY_ID: the short ID or relative resource name of the dbt entry.

REST

To search for dbt entries, call the projects.locations:searchEntries method:

curl -X POST \
    -H "Authorization: Bearer $(gcloud auth print-access-token)" \
    -H "Content-Type: application/json" \
    https://dataplex.googleapis.com/v1/projects/PROJECT_ID/locations/global:searchEntries \
    -d '{
      "query": "system=DBT"
    }'

To filter by a specific dbt resource type:

curl -X POST \
    -H "Authorization: Bearer $(gcloud auth print-access-token)" \
    -H "Content-Type: application/json" \
    https://dataplex.googleapis.com/v1/projects/PROJECT_ID/locations/global:searchEntries \
    -d '{
      "query": "system=DBT AND type=dbt-model"
    }'

To retrieve full metadata details and aspects for a specific entry, call the projects.locations.entryGroups.entries.get method:

curl -X GET \
    -H "Authorization: Bearer $(gcloud auth print-access-token)" \
    https://dataplex.googleapis.com/v1/projects/PROJECT_ID/locations/LOCATION/entryGroups/ENTRY_GROUP/entries/ENTRY_ID?view=FULL

To retrieve LLM context for specific dbt resources, use the projects.locations:lookupContext API:

curl -X POST \
    -H "Authorization: Bearer $(gcloud auth print-access-token)" \
    -H "Content-Type: application/json" \
    https://dataplex.googleapis.com/v1/projects/PROJECT_ID/locations/LOCATION:lookupContext \
    -d '{
      "resources": [
        "projects/PROJECT_ID/locations/LOCATION/entryGroups/ENTRY_GROUP/entries/ENTRY_ID"
      ]
    }'

Replace the following:

PROJECT_ID: your Google Cloud project ID.
LOCATION: the location of the entry group (for example, us-central1).
ENTRY_GROUP: the short ID of your destination entry group (for example, dbt-metadata-ingestion).
ENTRY_ID: the short ID or relative resource name of the dbt entry.

To learn more about searching for resources, see Search for resources in Knowledge Catalog. To learn more about query expressions and filters, see Search syntax for Knowledge Catalog.

Limitations

Supports recent dbt Core v1 versions (validated against versions 1.11 and 1.12). dbt Core v2 and dbt Fusion are not supported.
dbt models that use model versioning are not supported.
dbt Cloud is not supported.
Very large or deeply nested schemas are truncated: a single aspect cannot exceed the per-aspect size cap, so deeply nested schemas might lose trailing fields.
--aspects-only can add and refresh metadata, but cannot remove it. Deleting a dbt resource requires a full run.
Entry links are not supported.
This integration supports only dbt lineage events on BigQuery resources in Data Lineage API and graph. dbt entries (source, seeds, models) for external 3P sources are not captured in data lineage.
- To ingest all dbt lineage events in Data Lineage API, use the OpenLineage dbt integration. Then, integrate OpenLineage with Knowledge Catalog to import and visualize data lineage from dbt.

What's next

Learn how to Manage connector jobs.

Import metadata from dbt Core Stay organized with collections Save and categorize content based on your preferences.

IAM roles and permissions

Enable APIs

dbt prerequisites

Understand Cloud Storage roles

Configure dbt connectivity

Console

gcloud

Required flags

Optional flags

REST

Re-run ingestion

Search and view dbt metadata

Console

gcloud

REST

Limitations

What's next

Import metadata from dbt Core