Importar glossários empresariais e links de entrada usando arquivos JSON

Nesta página, descrevemos como importar glossários, categorias, termos e links de entrada em massa para o Knowledge Catalog (antigo Dataplex Universal Catalog) usando a API Dataplex. Você pode usar esse processo para migrar metadados de outras ferramentas de catalogação ou para fazer atualizações em massa nos glossários atuais enviando arquivos JSON para o Cloud Storage.

Antes de começar

Antes de iniciar o processo de importação, atenda aos seguintes pré-requisitos:

Criar um bucket do Cloud Storage

Crie um bucket do Cloud Storage para servir como uma área de preparação para arquivos de importação.

Funções exigidas

Para receber as permissões necessárias para importar glossários e links de entrada usando arquivos JSON, peça ao administrador para conceder a você os papéis do IAM a seguir:

Para mais informações sobre a concessão de papéis, consulte Gerenciar o acesso a projetos, pastas e organizações.

Também é possível conseguir as permissões necessárias com papéis personalizados ou outros papéis predefinidos.

Ativar APIs

Para importar glossários e links de entrada, ative a API Dataplex no seu projeto.

Funções necessárias para ativar APIs

Para ativar APIs, você precisa da permissão serviceusage.services.enable. Se você criou o projeto, provavelmente já tem essa permissão pelo papel Proprietário (roles/owner). Caso contrário, você pode receber essa permissão pelo papel Administrador de uso do serviço (roles/serviceusage.serviceUsageAdmin). Saiba como conceder papéis.

Ativar as APIs

Importar glossários usando arquivos JSON

Para importar glossários, categorias e termos, conclua as seguintes tarefas.

Criar um glossário de destino

Crie um glossário de destino para importar metadados.

alias gcurl='curl -H "Authorization: Bearer $(gcloud auth print-access-token)" -H "Content-Type: application/json"'

gcurl -X POST https://dataplex.googleapis.com/v1/projects/PROJECT_ID/locations/LOCATION_ID/glossaries?glossary_id=GLOSSARY_ID -d "$(cat<<EOF

{
   "displayName": "DISPLAY_NAME",
   "description": "DESCRIPTION"
}
EOF
)"

Substitua:

  • PROJECT_ID: o ID do projeto em que você está criando o glossário
  • LOCATION_ID: o local em que você quer criar o glossário
  • GLOSSARY_ID: o ID do glossário
  • DISPLAY_NAME: o nome de exibição do glossário
  • DESCRIPTION: a descrição do glossário

Preparar arquivos JSON

Crie um arquivo JSON delimitado por novas linhas com o glossário, as categorias e os termos que você quer enviar para o bucket do Cloud Storage.

Use o esquema JSON a seguir para estruturar seus arquivos de importação:

   {"entry":{"name":"projects/PROJECT_NUMBER/locations/LOCATION_ID/entryGroups/@dataplex/entries/projects/PROJECT_NUMBER/locations/LOCATION_ID/glossaries/GLOSSARY_ID/categories/CATEGORY_ID","entryType":"projects/dataplex-types/locations/global/entryTypes/glossary-category","aspects":{"dataplex-types.global.glossary-category-aspect":{"data":{}},"dataplex-types.global.overview":{"data":{"content":"CONTENT"}},"dataplex-types.global.contacts":{"data":{"identities":[{role: "steward", name: "CONTACT_DISPLAY_NAME", id: "CONTACT_EMAIL"}]}}},"parentEntry":"projects/PROJECT_NUMBER/locations/LOCATION_ID/entryGroups/@dataplex/entries/projects/PROJECT_NUMBER/locations/LOCATION_ID/glossaries/GLOSSARY_ID","entrySource":{"resource":"projects/PROJECT_NUMBER/locations/LOCATION_ID/glossaries/GLOSSARY_ID/categories/CATEGORY_ID","displayName":"CATEGORY_NAME","description":"CATEGORY_DESCRIPTION","ancestors":[{"name":"projects/PROJECT_NUMBER/locations/LOCATION_ID/entryGroups/@dataplex/entries/projects/PROJECT_NUMBER/locations/LOCATION_ID/glossaries/GLOSSARY_ID","type":"projects/dataplex-types/locations/global/entryTypes/glossary"}]}}}
   {"entry":{"name":"projects/PROJECT_NUMBER/locations/LOCATION_ID/entryGroups/@dataplex/entries/projects/PROJECT_NUMBER/locations/LOCATION_ID/glossaries/GLOSSARY_ID/terms/TERM1_ID","entryType":"projects/dataplex-types/locations/global/entryTypes/glossary-term","aspects":{"dataplex-types.global.glossary-term-aspect":{"data":{}},"dataplex-types.global.overview":{"data":{"content":"TERM1_CONTENT"}},"dataplex-types.global.contacts":{"data":{"identities":[{role: "steward", name: "CONTACT_DISPLAY_NAME", id: "CONTACT_EMAIL"}]}}},"parentEntry":"projects/PROJECT_NUMBER/locations/LOCATION_ID/entryGroups/@dataplex/entries/projects/PROJECT_NUMBER/locations/LOCATION_ID/glossaries/GLOSSARY_ID","entrySource":{"resource":"projects/PROJECT_NUMBER/locations/LOCATION_ID/glossaries/GLOSSARY_ID/terms/TERM1_ID","displayName":"TERM1_DISPLAY_NAME","description":"TERM1_DESCRIPTION","ancestors":[{"name":"projects/PROJECT_NUMBER/locations/LOCATION_ID/entryGroups/@dataplex/entries/projects/PROJECT_NUMBER/locations/LOCATION_ID/glossaries/GLOSSARY_ID","type":"projects/dataplex-types/locations/global/entryTypes/glossary"},{"name":"projects/PROJECT_NUMBER/locations/LOCATION_ID/entryGroups/@dataplex/entries/projects/PROJECT_NUMBER/locations/LOCATION_ID/glossaries/GLOSSARY_ID/categories/CATEGORY_ID","type":"projects/dataplex-types/locations/global/entryTypes/glossary-category"}]}}}
   {"entry":{"name":"projects/PROJECT_NUMBER/locations/LOCATION_ID/entryGroups/@dataplex/entries/projects/PROJECT_NUMBER/locations/LOCATION_ID/glossaries/GLOSSARY_ID/terms/TERM2_ID","entryType":"projects/dataplex-types/locations/global/entryTypes/glossary-term","aspects":{"dataplex-types.global.glossary-term-aspect":{"data":{}},"dataplex-types.global.overview":{"data":{"content":"TERM1_CONTENT"}},"dataplex-types.global.contacts":{"data":{"identities":[{role: "steward", name: "CONTACT_DISPLAY_NAME", id: "CONTACT_EMAIL"}]}}},"parentEntry":"projects/PROJECT_NUMBER/locations/LOCATION_ID/entryGroups/@dataplex/entries/projects/PROJECT_NUMBER/locations/LOCATION_ID/glossaries/GLOSSARY_ID","entrySource":{"resource":"projects/PROJECT_NUMBER/locations/LOCATION_ID/glossaries/GLOSSARY_ID/terms/TERM2_ID","displayName":"TERM2_DISPLAY_NAME","description":"TERM2_DESCRIPTION","ancestors":[{"name":"projects/PROJECT_NUMBER/locations/LOCATION_ID/entryGroups/@dataplex/entries/projects/PROJECT_NUMBER/locations/LOCATION_ID/glossaries/GLOSSARY_ID","type":"projects/dataplex-types/locations/global/entryTypes/glossary"},{"name":"projects/PROJECT_NUMBER/locations/LOCATION_ID/entryGroups/@dataplex/entries/projects/PROJECT_NUMBER/locations/LOCATION_ID/glossaries/GLOSSARY_ID/categories/CATEGORY_ID","type":"projects/dataplex-types/locations/global/entryTypes/glossary-category"}]}}}

Importar glossário, categorias e termos

Importe o glossário, as categorias e os termos para o bucket do Cloud Storage:

# Set GCURL alias
alias gcurl='curl -H "Authorization: Bearer $(gcloud auth print-access-token)" -H "Content-Type: application/json"'

gcurl https://DATAPLEX_API/metadataJobs?metadata_job_id=JOB_ID -d "$(cat<<EOF
{
   "type":"IMPORT",
   "import_spec":{
      "log_level":"DEBUG",
      "source_storage_uri":"gs://STORAGE_BUCKET/",
      "entry_sync_mode":"FULL",
      "aspect_sync_mode":"INCREMENTAL",
      "scope":{
         "glossaries": "GLOSSARY_NAME"
      }
   }
}
EOF
)"

Substitua DATAPLEX_API pelo endpoint da API Dataplex no formato dataplex.googleapis.com/v1/projects/PROJECT_ID/locations/LOCATION_ID.

Para importar links de entrada em massa, conclua as seguintes tarefas.

Crie um arquivo JSON delimitado por novas linhas com os links de entrada que você quer enviar para o bucket do Cloud Storage.

Use o esquema JSON a seguir para estruturar seus arquivos de importação:

Formato de exemplo para links entre termos

   {"entryLink":{"name":"projects/PROJECT_NUMBER/locations/global/entryGroups/@dataplex/entryLinks/el-import-0606e3f2-8206-4f3a-aba9-32c6196f6048","entryLinkType":"projects/dataplex-types/locations/global/entryLinkTypes/synonym","entryReferences":[{"name":"projects/PROJECT_NUMBER/locations/global/entryGroups/@dataplex/entries/projects/PROJECT_NUMBER/locations/global/glossaries/import-glossary/terms/term-1"},{"name":"projects/PROJECT_NUMBER/locations/global/entryGroups/@dataplex/entries/projects/PROJECT_NUMBER/locations/global/glossaries/import-glossary/terms/term-2"}]}}
   {"entryLink":{"name":"projects/PROJECT_NUMBER/locations/global/entryGroups/@dataplex/entryLinks/el-import-2f7408e3-af3d-405d-81bb-861cf9ec5146","entryLinkType":"projects/dataplex-types/locations/global/entryLinkTypes/related","entryReferences":[{"name":"projects/PROJECT_NUMBER/locations/global/entryGroups/@dataplex/entries/projects/PROJECT_NUMBER/locations/global/glossaries/import-glossary/terms/term-1"},{"name":"projects/PROJECT_NUMBER/locations/global/entryGroups/@dataplex/entries/projects/PROJECT_NUMBER/locations/global/glossaries/import-glossary/terms/term-2"}]}}

Formato de exemplo para links entre termos e recursos de dados

   {"entryLink":{"name":"projects/PROJECT_NUMBER/locations/global/entryGroups/@dataplex/entryLinks/el-import-0606e3f2-8206-4f3a-aba9-32c6196f6048","entryLinkType":"projects/dataplex-types/locations/global/entryLinkTypes/definition","entryReferences":[{"name":"projects/PROJECT_NUMBER/locations/global/entryGroups/@dataplex/entries/projects/PROJECT_NUMBER/locations/global/glossaries/import-glossary/terms/term-1"},{"name":"projects/PROJECT_NUMBER/locations/us-central1/entryGroups/entry-group-1/entries/entry-1"}]}}

Para iniciar um job de importação de metadados que estabeleça conexões synonym ou related entre os termos do catálogo, use o seguinte comando:

gcurl https://DATAPLEX_API/metadataJobs?metadata_job_id=JOB_ID -d "$(cat<<EOF
   {
      "type":"IMPORT",
      "import_spec":{
         "log_level":"DEBUG",
         "source_storage_uri":"gs://STORAGE_BUCKET/",
         "entry_sync_mode":"FULL",
         "aspect_sync_mode":"INCREMENTAL",
         "scope":{
            "entry_groups":[  "projects/PROJECT_ID/locations/LOCATION_ID/entryGroups/@dataplex"
            ],
            "entry_link_types":[
               "projects/dataplex-types/locations/global/entryLinkTypes/synonym",
               "projects/dataplex-types/locations/global/entryLinkTypes/related"
            ],
            "referenced_entry_scopes":[
               "PROJECT_IDS"
            ]
         }
      }
   }
EOF
)"

Substitua DATAPLEX_API por dataplex.googleapis.com/v1/projects/PROJECT_ID/locations/LOCATION_ID.

Para criar um link entre termos do glossário e recursos de dados, execute o job de importação para cada grupo de entrada a que a entrada do recurso de dados pertence. Todos os links de entrada de definição são criados nesse grupo.

   gcurl https://DATAPLEX_API/metadataJobs?metadata_job_id=JOB_ID -d "$(cat<<EOF
   {
   "type":"IMPORT",
   "import_spec":{
      "log_level":"DEBUG",
      "source_storage_uri":"gs://STORAGE_BUCKET/",
      "entry_sync_mode":"FULL",
      "aspect_sync_mode":"INCREMENTAL",
      "scope":{
         "entry_groups":[  "projects/PROJECT_ID/locations/ENTRY_GROUP_LOCATION_ID/entryGroups/@dataplex"
         ],
         "entry_link_types":[
            "projects/dataplex-types/locations/global/entryLinkTypes/definition"
         ],
         "referenced_entry_scopes":[
            "PROJECT_IDS"
         ]
      }
   }
   }
EOF
)"

Substitua DATAPLEX_API por dataplex.googleapis.com/v1/projects/PROJECT_ID/locations/LOCATION_ID.

Monitorar o status do job de importação

  1. Para receber o status da operação import, execute o seguinte comando:

    gcurl https://dataplex.googleapis.com/v1/projects/PROJECT_ID/locations/LOCATION_ID/operations/operation-OPERATION_ID

    Substitua OPERATION_ID pelo ID da operação.

  2. Para receber o status do job de metadados, execute o seguinte comando:

    gcurl -X GET https://dataplex.googleapis.com/v1/projects/PROJECT_ID/locations/LOCATION_ID/metadataJobs/JOB_ID

A seguir