使用 API 创建存储在 Cloud Storage 中的数据的去标识化副本

本页面介绍如何检查 Cloud Storage 资源，以及如何使用 Cloud Data Loss Prevention API 创建去标识化的数据副本。

此操作有助于确保您在业务流程中使用的文件不包含敏感数据，例如个人身份信息 (PII)。Sensitive Data Protection 可以检查 Cloud Storage 存储桶中的文件是否存在敏感数据，并在单独的存储桶中创建这些文件的去标识化副本。然后，您可以在业务流程中使用去标识化的副本。

如需详细了解此功能，请参阅对 Cloud Storage 中的敏感数据进行去标识化处理。

准备工作

本页面假定您已完成以下操作：

• 您已启用结算功能。

  了解如何启用结算功能

• 您已启用 Sensitive Data Protection。

  启用 Sensitive Data Protection

• 您有一个 Cloud Storage 存储桶，其中包含要进行去标识化处理的数据。

• 您知道如何向 DLP API 发送 HTTP 请求。如需了解详情，请参阅使用 DLP API 检查敏感文本。

了解此操作的 限制和注意事项 。

存储检查需要以下 OAuth 范围：https://www.googleapis.com/auth/cloud-platform。如需了解详情，请参阅 对 DLP API 进行身份验证。

所需 IAM 角色

如果此操作的所有资源都位于同一项目中，则 服务代理的 DLP API 服务代理角色 (roles/dlp.serviceAgent) 就足够了。借助该角色，您可以执行以下操作：

• 创建检查作业
• 读取输入目录中的文件
• 将去标识化的文件写入输出目录
• 将转换详情写入 BigQuery 表

相关资源包括检查作业、去标识化模板、输入存储桶、输出存储桶和转换详情表。

如果您必须将资源放在不同的项目中，请确保项目的服务代理还具有以下角色：

• 输入存储桶或包含该存储桶的项目的 Storage Object Viewer 角色 (roles/storage.objectViewer)。
• 输出存储桶或包含该存储桶的项目的 Storage Object Creator 角色 (roles/storage.objectCreator)。
• 转换详情表或包含该表的项目的 BigQuery Data Editor 角色 (roles/bigquery.dataEditor)。

如需向服务代理授予角色，请参阅授予单个角色。您还可以在以下级别控制访问权限：

• Cloud Storage 存储桶
• BigQuery 数据集
• BigQuery 表

API 概览

如需创建存储在 Cloud Storage 中的内容的去标识化副本， 您可以配置 一个检查作业 ，该作业会根据您指定的条件查找敏感数据 。然后，在检查作业中，您 以Deidentify操作的形式提供去标识化说明。

注意： 在 Sensitive Data Protection 中， “操作”是指在 Sensitive Data Protection 作业 成功完成后发生的事情。如需详细了解操作，请参阅以下内容：

1. 操作概念 主题
2. 操作参考文档
3. 检索检查结果

如果您只想扫描存储桶中的部分文件，可以 限制作业扫描的文件。对于包含 去标识化的作业，支持的选项包括按类型 (FileType) 和正则表达式 (FileSet) 进行文件过滤。

启用 Deidentify 操作后，默认情况下，Sensitive Data Protection 会创建扫描中包含的所有 受支持文件类型 的去标识化（转换后）副本。不过，您可以将作业配置为仅转换部分受支持的文件类型。

可选：创建去标识化模板

如果您想控制发现结果的转换方式， 请创建以下模板。这些模板提供了有关转换结构化文件、非结构化文件和图片中的发现结果的说明。

**注意**： 如果您选择加密方法，则 必须先使用 Cloud Key Management Service 创建封装的密钥，并在去标识化模板中提供该密钥 。不支持临时（原始）密钥。

• 去标识化模板：用于非结构化文件（例如自由格式文本文件）的默认DeidentifyTemplate，可 用于非结构化文件，例如自由格式文本文件。此类 DeidentifyTemplate不能包含RecordTransformations对象，该对象 仅受结构化内容支持。如果此模板不存在， Sensitive Data Protection 会使用 ReplaceWithInfoTypeConfig 方法转换 非结构化文件。

• 结构化去标识化模板 ：用于结构化文件（例如 CSV 文件）的 DeidentifyTemplate。此 DeidentifyTemplate 可以包含 RecordTransformations。如果此模板不存在，Sensitive Data Protection 会使用您创建的默认去标识化模板。 如果该模板也不存在，Sensitive Data Protection 会使用 ReplaceWithInfoTypeConfig 方法转换结构化文件。

• 图片遮盖模板 ：用于图片的 DeidentifyTemplate。此 模板必须包含 ImageTransformations 对象。 如果此模板不存在，Sensitive Data Protection 会使用黑框遮盖图片中的所有发现结果。

详细了解如何创建去标识化模板。

创建包含去标识化操作的检查作业

DlpJob 对象提供了有关要检查的内容、要标记为敏感的数据类型以及如何处理发现结果的说明。 如需对 Cloud Storage 目录中的敏感数据进行去标识化处理，您的 DlpJob 必须至少定义以下内容：

• 一个 StorageConfig 对象，用于指定要检查的 Cloud Storage 目录 。
• 一个 InspectConfig 对象，其中包含要查找的数据类型 以及有关如何查找敏感数据的其他检查说明。

• 包含以下内容的 Deidentify 操作：

  • 一个 TransformationConfig 对象，用于指定您为结构化和非结构化文件中的数据去标识化而创建的任何 模板。您还可以包含用于遮盖图片中的敏感数据的配置。

    如果您未包含 TransformationConfig 对象，Sensitive Data Protection 会将文本中的敏感数据替换为其 infoType。在图片中，它会使用黑框遮盖敏感数据。

  • TransformationDetailsStorageConfig 对象，用于指定 Sensitive Data Protection 必须 在其中存储有关每次转换的详细信息的 BigQuery 表。对于每次转换，详细信息包括说明、成功或错误代码、任何错误详细信息、转换的字节数、转换后的内容的位置，以及 Sensitive Data Protection 在其中进行转换的检查作业的名称。此表不会存储实际的去标识化内容。

  当 数据写入 BigQuery 表时，结算和配额用量将记在 包含目标表的项目名下。

复制的内容经过去标识化处理后，去标识化作业即会完成。该作业包含有关指定 转换应用次数的摘要，您可以使用 projects.dlpJobs.get方法在DlpJob上检索该摘要。返回的 DlpJob 同时包含 一个 DeidentifyDataSourceDetails 对象和 一个 InspectDataSourceDetails对象。这些对象分别包含 Deidentify 操作和检查作业的结果。

如果您在 DlpJob 中包含 TransformationDetailsStorageConfig 对象，系统会创建一个 BigQuery 表，其中包含有关转换详情的元数据。对于发生的每次转换，Sensitive Data Protection 都会向该表中写入一行元数据。如需详细了解该表的内容， 请参阅转换详情参考文档。

代码示例

以下示例演示了如何使用 DLP API 创建 Cloud Storage 文件的去标识化副本。

**注意**： 以下示例需要 OAuth 2.0 访问令牌。

HTTP 方法和网址

POST https://dlp.googleapis.com/v2/projects/PROJECT_ID/dlpJobs

C#

如需了解如何安装和使用 Sensitive Data Protection 客户端库，请参阅 Sensitive Data Protection 客户端库。

如需向 Sensitive Data Protection 进行身份验证，请设置应用默认凭证。如需了解详情，请参阅为本地开发环境设置身份验证。

using Google.Api.Gax.ResourceNames;
using Google.Cloud.Dlp.V2;
using System.Linq;

public class DeidentifyDataStoredInCloudStorage
{
    public static DlpJob Deidentify(
        string projectId,
        string gcsInputPath,
        string unstructuredDeidentifyTemplatePath,
        string structuredDeidentifyTemplatePath,
        string imageRedactionTemplatePath,
        string gcsOutputPath,
        string datasetId,
        string tableId)
    {
        // Instantiate the client.
        var dlp = DlpServiceClient.Create();

        //Construct the storage config by specifying the input directory.
        var storageConfig = new StorageConfig
        {
            CloudStorageOptions = new CloudStorageOptions
            {
                FileSet = new CloudStorageOptions.Types.FileSet
                {
                    Url = gcsInputPath
                }
            }
        };

        // Construct the inspect config by specifying the type of info to be inspected.
        var inspectConfig = new InspectConfig
        {
            InfoTypes =
            {
                new InfoType[]
                {
                    new InfoType { Name = "PERSON_NAME" },
                    new InfoType { Name = "EMAIL_ADDRESS" }
                }
            },
            IncludeQuote = true
        };

        // Construct the actions to take after the inspection portion of the job is completed.
        // Specify how Cloud DLP must de-identify sensitive data in structured files, unstructured files and images
        // using Transformation config.
        // The de-identified files will be written to the the GCS bucket path specified in gcsOutputPath and the details of 
        // transformations performed will be written to BigQuery table specified in datasetId and tableId.
        var actions = new Action[]
        {
            new Action
            {
                Deidentify = new Action.Types.Deidentify
                {
                    CloudStorageOutput = gcsOutputPath,
                    TransformationConfig = new TransformationConfig
                    {
                        DeidentifyTemplate = unstructuredDeidentifyTemplatePath,
                        ImageRedactTemplate = imageRedactionTemplatePath,
                        StructuredDeidentifyTemplate = structuredDeidentifyTemplatePath,
                    },
                    TransformationDetailsStorageConfig = new TransformationDetailsStorageConfig
                    {
                        Table = new BigQueryTable
                        {
                            ProjectId = projectId,
                            DatasetId = datasetId,
                            TableId = tableId
                        }
                    }
                }
            }
        };

        // Construct the inspect job config using created storage config, inspect config and actions.
        var inspectJob = new InspectJobConfig
        {
            StorageConfig = storageConfig,
            InspectConfig = inspectConfig,
            Actions = { actions }
        };

        // Create the dlp job and call the API.
        DlpJob response = dlp.CreateDlpJob(new CreateDlpJobRequest
        {
            ParentAsLocationName = new LocationName(projectId, "global"),
            InspectJob = inspectJob
        });

        return response;
    }
}

Go

如需了解如何安装和使用 Sensitive Data Protection 客户端库，请参阅 Sensitive Data Protection 客户端库。

如需向 Sensitive Data Protection 进行身份验证，请设置应用默认凭证。如需了解详情，请参阅为本地开发环境设置身份验证。

import (
	"context"
	"fmt"
	"io"

	dlp "cloud.google.com/go/dlp/apiv2"
	"cloud.google.com/go/dlp/apiv2/dlppb"
)

func deidentifyCloudStorage(w io.Writer, projectID, gcsUri, tableId, datasetId, outputDirectory, deidentifyTemplateId, structuredDeidentifyTemplateId, imageRedactTemplateId string) error {
	// projectId := "my-project-id"
	// gcsUri := "gs://" + "your-bucket-name" + "/path/to/your/file.txt"
	// tableId := "your-bigquery-table-id"
	// datasetId := "your-bigquery-dataset-id"
	// outputDirectory := "your-output-directory"
	// deidentifyTemplateId := "your-deidentify-template-id"
	// structuredDeidentifyTemplateId := "your-structured-deidentify-template-id"
	// imageRedactTemplateId := "your-image-redact-template-id"

	ctx := context.Background()

	// Initialize a client once and reuse it to send multiple requests. Clients
	// are safe to use across goroutines. When the client is no longer needed,
	// call the Close method to cleanup its resources.
	client, err := dlp.NewClient(ctx)
	if err != nil {
		return err
	}

	// Closing the client safely cleans up background resources.
	defer client.Close()

	// Set path in Cloud Storage.
	cloudStorageOptions := &dlppb.CloudStorageOptions{
		FileSet: &dlppb.CloudStorageOptions_FileSet{
			Url: gcsUri,
		},
	}

	// Define the storage config options for cloud storage options.
	storageConfig := &dlppb.StorageConfig{
		Type: &dlppb.StorageConfig_CloudStorageOptions{
			CloudStorageOptions: cloudStorageOptions,
		},
	}

	// Specify the type of info the inspection will look for.
	// See https://cloud.google.com/dlp/docs/infotypes-reference for complete list of info types
	infoTypes := []*dlppb.InfoType{
		{Name: "PERSON_NAME"},
		{Name: "EMAIL_ADDRESS"},
	}

	// inspectConfig holds the configuration settings for data inspection and analysis
	// within the context of the Google Cloud Data Loss Prevention (DLP) API.
	inspectConfig := &dlppb.InspectConfig{
		InfoTypes:    infoTypes,
		IncludeQuote: true,
	}

	// Types of files to include for de-identification.
	fileTypesToTransform := []dlppb.FileType{
		dlppb.FileType_CSV,
		dlppb.FileType_IMAGE,
		dlppb.FileType_TEXT_FILE,
	}

	// Specify the BigQuery table to be inspected.
	table := &dlppb.BigQueryTable{
		ProjectId: projectID,
		DatasetId: datasetId,
		TableId:   tableId,
	}

	// transformationDetailsStorageConfig holds configuration settings for storing transformation
	// details in the context of the Google Cloud Data Loss Prevention (DLP) API.
	transformationDetailsStorageConfig := &dlppb.TransformationDetailsStorageConfig{
		Type: &dlppb.TransformationDetailsStorageConfig_Table{
			Table: table,
		},
	}

	transformationConfig := &dlppb.TransformationConfig{
		DeidentifyTemplate:           deidentifyTemplateId,
		ImageRedactTemplate:          imageRedactTemplateId,
		StructuredDeidentifyTemplate: structuredDeidentifyTemplateId,
	}

	// Action to execute on the completion of a job.
	deidentify := &dlppb.Action_Deidentify{
		TransformationConfig:               transformationConfig,
		TransformationDetailsStorageConfig: transformationDetailsStorageConfig,
		Output: &dlppb.Action_Deidentify_CloudStorageOutput{
			CloudStorageOutput: outputDirectory,
		},
		FileTypesToTransform: fileTypesToTransform,
	}

	action := &dlppb.Action{
		Action: &dlppb.Action_Deidentify_{
			Deidentify: deidentify,
		},
	}

	// Configure the inspection job we want the service to perform.
	inspectJobConfig := &dlppb.InspectJobConfig{
		StorageConfig: storageConfig,
		InspectConfig: inspectConfig,
		Actions: []*dlppb.Action{
			action,
		},
	}

	// Construct the job creation request to be sent by the client.
	req := &dlppb.CreateDlpJobRequest{
		Parent: fmt.Sprintf("projects/%s/locations/global", projectID),
		Job: &dlppb.CreateDlpJobRequest_InspectJob{
			InspectJob: inspectJobConfig,
		},
	}

	// Send the request.
	resp, err := client.CreateDlpJob(ctx, req)
	if err != nil {
		fmt.Fprintf(w, "error after resp: %v", err)
		return err
	}

	// Print the results.
	fmt.Fprint(w, "Job created successfully: ", resp.Name)
	return nil

}

Java

如需了解如何安装和使用 Sensitive Data Protection 客户端库，请参阅 Sensitive Data Protection 客户端库。

如需向 Sensitive Data Protection 进行身份验证，请设置应用默认凭证。如需了解详情，请参阅为本地开发环境设置身份验证。

import com.google.cloud.dlp.v2.DlpServiceClient;
import com.google.privacy.dlp.v2.Action;
import com.google.privacy.dlp.v2.BigQueryTable;
import com.google.privacy.dlp.v2.CloudStorageOptions;
import com.google.privacy.dlp.v2.CreateDlpJobRequest;
import com.google.privacy.dlp.v2.DlpJob;
import com.google.privacy.dlp.v2.FileType;
import com.google.privacy.dlp.v2.InfoType;
import com.google.privacy.dlp.v2.InfoTypeStats;
import com.google.privacy.dlp.v2.InspectConfig;
import com.google.privacy.dlp.v2.InspectDataSourceDetails;
import com.google.privacy.dlp.v2.InspectJobConfig;
import com.google.privacy.dlp.v2.LocationName;
import com.google.privacy.dlp.v2.ProjectDeidentifyTemplateName;
import com.google.privacy.dlp.v2.StorageConfig;
import com.google.privacy.dlp.v2.TransformationConfig;
import com.google.privacy.dlp.v2.TransformationDetailsStorageConfig;
import java.io.IOException;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.List;
import java.util.concurrent.TimeUnit;

public class DeidentifyCloudStorage {

  // Set the timeout duration in minutes.
  private static final int TIMEOUT_MINUTES = 15;

  public static void main(String[] args) throws IOException, InterruptedException {
    // TODO(developer): Replace these variables before running the sample.
    // The Google Cloud project id to use as a parent resource.
    String projectId = "your-project-id";
    // Specify the cloud storage directory that you want to inspect.
    String gcsPath = "gs://" + "your-bucket-name" + "/path/to/your/file.txt";
    // Specify the big query dataset id to store the transformation details.
    String datasetId = "your-bigquery-dataset-id";
    // Specify the big query table id to store the transformation details.
    String tableId = "your-bigquery-table-id";
    // Specify the cloud storage directory to store the de-identified files.
    String outputDirectory = "your-output-directory";
    // Specify the de-identify template ID for unstructured files.
    String deidentifyTemplateId = "your-deidentify-template-id";
    // Specify the de-identify template ID for structured files.
    String structuredDeidentifyTemplateId = "your-structured-deidentify-template-id";
    // Specify the de-identify template ID for images.
    String imageRedactTemplateId = "your-image-redact-template-id";
    deidentifyCloudStorage(
        projectId,
        gcsPath,
        tableId,
        datasetId,
        outputDirectory,
        deidentifyTemplateId,
        structuredDeidentifyTemplateId,
        imageRedactTemplateId);
  }

  public static void deidentifyCloudStorage(
      String projectId,
      String gcsPath,
      String tableId,
      String datasetId,
      String outputDirectory,
      String deidentifyTemplateId,
      String structuredDeidentifyTemplateId,
      String imageRedactTemplateId)
      throws IOException, InterruptedException {

    try (DlpServiceClient dlp = DlpServiceClient.create()) {
      // Set path in Cloud Storage.
      CloudStorageOptions cloudStorageOptions =
          CloudStorageOptions.newBuilder()
              .setFileSet(CloudStorageOptions.FileSet.newBuilder().setUrl(gcsPath))
              .build();

      // Set storage config indicating the type of cloud storage.
      StorageConfig storageConfig =
          StorageConfig.newBuilder().setCloudStorageOptions(cloudStorageOptions).build();

      // Specify the type of info the inspection will look for.
      // See https://cloud.google.com/dlp/docs/infotypes-reference for complete list of info types
      List<InfoType> infoTypes = new ArrayList<>();
      for (String typeName : new String[] {"PERSON_NAME", "EMAIL_ADDRESS"}) {
        infoTypes.add(InfoType.newBuilder().setName(typeName).build());
      }

      InspectConfig inspectConfig =
          InspectConfig.newBuilder().addAllInfoTypes(infoTypes).setIncludeQuote(true).build();

      // Types of files to include for de-identification.
      List<FileType> fileTypesToTransform =
          Arrays.asList(
              FileType.valueOf("IMAGE"), FileType.valueOf("CSV"), FileType.valueOf("TEXT_FILE"));

      // Specify the big query table to store the transformation details.
      BigQueryTable table =
          BigQueryTable.newBuilder()
              .setProjectId(projectId)
              .setTableId(tableId)
              .setDatasetId(datasetId)
              .build();

      TransformationDetailsStorageConfig transformationDetailsStorageConfig =
          TransformationDetailsStorageConfig.newBuilder().setTable(table).build();

      // Specify the de-identify template used for the transformation.
      TransformationConfig transformationConfig =
          TransformationConfig.newBuilder()
              .setDeidentifyTemplate(
                  ProjectDeidentifyTemplateName.of(projectId, deidentifyTemplateId).toString())
              .setImageRedactTemplate(
                  ProjectDeidentifyTemplateName.of(projectId, imageRedactTemplateId).toString())
              .setStructuredDeidentifyTemplate(
                  ProjectDeidentifyTemplateName.of(projectId, structuredDeidentifyTemplateId)
                      .toString())
              .build();

      Action.Deidentify deidentify =
          Action.Deidentify.newBuilder()
              .setCloudStorageOutput(outputDirectory)
              .setTransformationConfig(transformationConfig)
              .setTransformationDetailsStorageConfig(transformationDetailsStorageConfig)
              .addAllFileTypesToTransform(fileTypesToTransform)
              .build();

      Action action = Action.newBuilder().setDeidentify(deidentify).build();

      // Configure the long-running job we want the service to perform.
      InspectJobConfig inspectJobConfig =
          InspectJobConfig.newBuilder()
              .setInspectConfig(inspectConfig)
              .setStorageConfig(storageConfig)
              .addActions(action)
              .build();

      // Construct the job creation request to be sent by the client.
      CreateDlpJobRequest createDlpJobRequest =
          CreateDlpJobRequest.newBuilder()
              .setParent(LocationName.of(projectId, "global").toString())
              .setInspectJob(inspectJobConfig)
              .build();

      // Send the job creation request.
      DlpJob response = dlp.createDlpJob(createDlpJobRequest);

      // Get the current time.
      long startTime = System.currentTimeMillis();

      // Check if the job state is DONE.
      while (response.getState() != DlpJob.JobState.DONE) {
        // Sleep for 30 second.
        Thread.sleep(30000);

        // Get the updated job status.
        response = dlp.getDlpJob(response.getName());

        // Check if the timeout duration has exceeded.
        long elapsedTime = System.currentTimeMillis() - startTime;
        if (TimeUnit.MILLISECONDS.toMinutes(elapsedTime) >= TIMEOUT_MINUTES) {
          System.out.printf("Job did not complete within %d minutes.%n", TIMEOUT_MINUTES);
          break;
        }
      }
      // Print the results.
      System.out.println("Job status: " + response.getState());
      System.out.println("Job name: " + response.getName());
      InspectDataSourceDetails.Result result = response.getInspectDetails().getResult();
      System.out.println("Findings: ");
      for (InfoTypeStats infoTypeStat : result.getInfoTypeStatsList()) {
        System.out.print("\tInfo type: " + infoTypeStat.getInfoType().getName());
        System.out.println("\tCount: " + infoTypeStat.getCount());
      }
    }
  }
}

Node.js

如需了解如何安装和使用 Sensitive Data Protection 客户端库，请参阅 Sensitive Data Protection 客户端库。

如需向 Sensitive Data Protection 进行身份验证，请设置应用默认凭证。如需了解详情，请参阅为本地开发环境设置身份验证。

// Imports the Google Cloud client library
const DLP = require('@google-cloud/dlp');
// Instantiates a client
const dlp = new DLP.DlpServiceClient();

// The project ID to run the API call under
// const projectId = 'my-project';

// The Cloud Storage directory that needs to be inspected
// const inputDirectory = 'your-google-cloud-storage-path';

// The ID of the dataset to inspect, e.g. 'my_dataset'
// const datasetId = 'my_dataset';

// The ID of the table to inspect, e.g. 'my_table'
// const tableId = 'my_table';

// The Cloud Storage directory that will be used to store the de-identified files
// const outputDirectory = 'your-output-directory';

// The full resource name of the default de-identify template
// const deidentifyTemplateId = 'your-deidentify-template-id';

// The full resource name of the de-identify template for structured files
// const structuredDeidentifyTemplateId = 'your-structured-deidentify-template-id';

// The full resource name of the image redaction template for images
// const imageRedactTemplateId = 'your-image-redact-template-id';

async function deidentifyCloudStorage() {
  // Specify storage configuration that uses file set.
  const storageConfig = {
    cloudStorageOptions: {
      fileSet: {
        url: inputDirectory,
      },
    },
  };

  // Specify the type of info the inspection will look for.
  const infoTypes = [{name: 'PERSON_NAME'}, {name: 'EMAIL_ADDRESS'}];

  // Construct inspect configuration
  const inspectConfig = {
    infoTypes: infoTypes,
    includeQuote: true,
  };

  // Types of files to include for de-identification.
  const fileTypesToTransform = [
    {fileType: 'IMAGE'},
    {fileType: 'CSV'},
    {fileType: 'TEXT_FILE'},
  ];

  // Specify the big query table to store the transformation details.
  const transformationDetailsStorageConfig = {
    table: {
      projectId: projectId,
      tableId: tableId,
      datasetId: datasetId,
    },
  };

  // Specify the de-identify template used for the transformation.
  const transformationConfig = {
    deidentifyTemplate: deidentifyTemplateId,
    structuredDeidentifyTemplate: structuredDeidentifyTemplateId,
    imageRedactTemplate: imageRedactTemplateId,
  };

  // Construct action to de-identify sensitive data.
  const action = {
    deidentify: {
      cloudStorageOutput: outputDirectory,
      transformationConfig: transformationConfig,
      transformationDetailsStorageConfig: transformationDetailsStorageConfig,
      fileTypes: fileTypesToTransform,
    },
  };

  // Construct the inspect job configuration.
  const inspectJobConfig = {
    inspectConfig: inspectConfig,
    storageConfig: storageConfig,
    actions: [action],
  };

  // Construct the job creation request to be sent by the client.
  const createDlpJobRequest = {
    parent: `projects/${projectId}/locations/global`,
    inspectJob: inspectJobConfig,
  };
  // Send the job creation request and process the response.
  const [response] = await dlp.createDlpJob(createDlpJobRequest);
  const jobName = response.name;

  // Waiting for a maximum of 15 minutes for the job to get complete.
  let job;
  let numOfAttempts = 30;
  while (numOfAttempts > 0) {
    // Fetch DLP Job status
    [job] = await dlp.getDlpJob({name: jobName});

    // Check if the job has completed.
    if (job.state === 'DONE') {
      break;
    }
    if (job.state === 'FAILED') {
      console.log('Job Failed, Please check the configuration.');
      return;
    }
    // Sleep for a short duration before checking the job status again.
    await new Promise(resolve => {
      setTimeout(() => resolve(), 30000);
    });
    numOfAttempts -= 1;
  }

  // Print out the results.
  const infoTypeStats = job.inspectDetails.result.infoTypeStats;
  if (infoTypeStats.length > 0) {
    infoTypeStats.forEach(infoTypeStat => {
      console.log(
        `  Found ${infoTypeStat.count} instance(s) of infoType ${infoTypeStat.infoType.name}.`
      );
    });
  } else {
    console.log('No findings.');
  }
}
await deidentifyCloudStorage();

PHP

如需了解如何安装和使用 Sensitive Data Protection 客户端库，请参阅 Sensitive Data Protection 客户端库。

如需向 Sensitive Data Protection 进行身份验证，请设置应用默认凭证。如需了解详情，请参阅为本地开发环境设置身份验证。

use Google\Cloud\Dlp\V2\Action;
use Google\Cloud\Dlp\V2\Action\Deidentify;
use Google\Cloud\Dlp\V2\BigQueryTable;
use Google\Cloud\Dlp\V2\Client\DlpServiceClient;
use Google\Cloud\Dlp\V2\CloudStorageOptions;
use Google\Cloud\Dlp\V2\CloudStorageOptions\FileSet;
use Google\Cloud\Dlp\V2\CreateDlpJobRequest;
use Google\Cloud\Dlp\V2\DlpJob\JobState;
use Google\Cloud\Dlp\V2\FileType;
use Google\Cloud\Dlp\V2\GetDlpJobRequest;
use Google\Cloud\Dlp\V2\InfoType;
use Google\Cloud\Dlp\V2\InspectConfig;
use Google\Cloud\Dlp\V2\InspectJobConfig;
use Google\Cloud\Dlp\V2\StorageConfig;
use Google\Cloud\Dlp\V2\TransformationConfig;
use Google\Cloud\Dlp\V2\TransformationDetailsStorageConfig;

/**
 * De-identify sensitive data stored in Cloud Storage using the API.
 * Create an inspection job that has a de-identification action.
 *
 * @param string $callingProjectId                  The project ID to run the API call under.
 * @param string $inputgcsPath                       The Cloud Storage directory that you want to de-identify.
 * @param string $outgcsPath                        The Cloud Storage directory where you want to store the
 *                                                  de-identified files.
 * @param string $deidentifyTemplateName            The full resource name of the default de-identify template — for
 *                                                  unstructured and structured files — if you created one. This value
 *                                                  must be in the format
 *                                                  `projects/projectName/(locations/locationId)/deidentifyTemplates/templateName`.
 * @param string $structuredDeidentifyTemplateName  The full resource name of the de-identify template for structured
 *                                                  files if you created one. This value must be in the format
 *                                                  `projects/projectName/(locations/locationId)/deidentifyTemplates/templateName`.
 * @param string $imageRedactTemplateName           The full resource name of the image redaction template for images if
 *                                                  you created one. This value must be in the format
 *                                                  `projects/projectName/(locations/locationId)/deidentifyTemplates/templateName`.
 * @param string $datasetId                         The ID of the BigQuery dataset where you want to store
 *                                                  the transformation details. If you don't provide a table ID, the
 *                                                  system automatically creates one.
 * @param string $tableId                           The ID of the BigQuery table where you want to store the
 *                                                  transformation details.
 */
function deidentify_cloud_storage(
    // TODO(developer): Replace sample parameters before running the code.
    string $callingProjectId,
    string $inputgcsPath = 'gs://YOUR_GOOGLE_STORAGE_BUCKET',
    string $outgcsPath = 'gs://YOUR_GOOGLE_STORAGE_BUCKET',
    string $deidentifyTemplateName = 'YOUR_DEIDENTIFY_TEMPLATE_NAME',
    string $structuredDeidentifyTemplateName = 'YOUR_STRUCTURED_DEIDENTIFY_TEMPLATE_NAME',
    string $imageRedactTemplateName = 'YOUR_IMAGE_REDACT_DEIDENTIFY_TEMPLATE_NAME',
    string $datasetId = 'YOUR_DATASET_ID',
    string $tableId = 'YOUR_TABLE_ID'
): void {
    // Instantiate a client.
    $dlp = new DlpServiceClient();

    $parent = "projects/$callingProjectId/locations/global";

    // Specify the GCS Path to be de-identify.
    $cloudStorageOptions = (new CloudStorageOptions())
        ->setFileSet((new FileSet())
            ->setUrl($inputgcsPath));
    $storageConfig = (new StorageConfig())
        ->setCloudStorageOptions(($cloudStorageOptions));

    // Specify the type of info the inspection will look for.
    $inspectConfig = (new InspectConfig())
        ->setInfoTypes([
            (new InfoType())->setName('PERSON_NAME'),
            (new InfoType())->setName('EMAIL_ADDRESS')
        ]);

    // Specify the big query table to store the transformation details.
    $transformationDetailsStorageConfig = (new TransformationDetailsStorageConfig())
        ->setTable((new BigQueryTable())
            ->setProjectId($callingProjectId)
            ->setDatasetId($datasetId)
            ->setTableId($tableId));

    // Specify the de-identify template used for the transformation.
    $transformationConfig = (new TransformationConfig())
        ->setDeidentifyTemplate(
            DlpServiceClient::projectDeidentifyTemplateName($callingProjectId, $deidentifyTemplateName)
        )
        ->setStructuredDeidentifyTemplate(
            DlpServiceClient::projectDeidentifyTemplateName($callingProjectId, $structuredDeidentifyTemplateName)
        )
        ->setImageRedactTemplate(
            DlpServiceClient::projectDeidentifyTemplateName($callingProjectId, $imageRedactTemplateName)
        );

    $deidentify = (new Deidentify())
        ->setCloudStorageOutput($outgcsPath)
        ->setTransformationConfig($transformationConfig)
        ->setTransformationDetailsStorageConfig($transformationDetailsStorageConfig)
        ->setFileTypesToTransform([FileType::TEXT_FILE, FileType::IMAGE, FileType::CSV]);

    $action = (new Action())
        ->setDeidentify($deidentify);

    // Configure the inspection job we want the service to perform.
    $inspectJobConfig = (new InspectJobConfig())
        ->setInspectConfig($inspectConfig)
        ->setStorageConfig($storageConfig)
        ->setActions([$action]);

    // Send the job creation request and process the response.
    $createDlpJobRequest = (new CreateDlpJobRequest())
        ->setParent($parent)
        ->setInspectJob($inspectJobConfig);
    $job = $dlp->createDlpJob($createDlpJobRequest);

    $numOfAttempts = 10;
    do {
        printf('Waiting for job to complete' . PHP_EOL);
        sleep(30);
        $getDlpJobRequest = (new GetDlpJobRequest())
            ->setName($job->getName());
        $job = $dlp->getDlpJob($getDlpJobRequest);
        if ($job->getState() == JobState::DONE) {
            break;
        }
        $numOfAttempts--;
    } while ($numOfAttempts > 0);

    // Print finding counts.
    printf('Job %s status: %s' . PHP_EOL, $job->getName(), JobState::name($job->getState()));
    switch ($job->getState()) {
        case JobState::DONE:
            $infoTypeStats = $job->getInspectDetails()->getResult()->getInfoTypeStats();
            if (count($infoTypeStats) === 0) {
                printf('No findings.' . PHP_EOL);
            } else {
                foreach ($infoTypeStats as $infoTypeStat) {
                    printf(
                        '  Found %s instance(s) of infoType %s' . PHP_EOL,
                        $infoTypeStat->getCount(),
                        $infoTypeStat->getInfoType()->getName()
                    );
                }
            }
            break;
        case JobState::FAILED:
            printf('Job %s had errors:' . PHP_EOL, $job->getName());
            $errors = $job->getErrors();
            foreach ($errors as $error) {
                var_dump($error->getDetails());
            }
            break;
        case JobState::PENDING:
            printf('Job has not completed. Consider a longer timeout or an asynchronous execution model' . PHP_EOL);
            break;
        default:
            printf('Unexpected job state. Most likely, the job is either running or has not yet started.');
    }
}

Python

如需了解如何安装和使用 Sensitive Data Protection 客户端库，请参阅 Sensitive Data Protection 客户端库。

如需向 Sensitive Data Protection 进行身份验证，请设置应用默认凭证。如需了解详情，请参阅为本地开发环境设置身份验证。

import time
from typing import List

import google.cloud.dlp


def deidentify_cloud_storage(
    project: str,
    input_gcs_bucket: str,
    output_gcs_bucket: str,
    info_types: List[str],
    deid_template_id: str,
    structured_deid_template_id: str,
    image_redact_template_id: str,
    dataset_id: str,
    table_id: str,
    timeout: int = 300,
) -> None:
    """
    Uses the Data Loss Prevention API to de-identify files in a Google Cloud
    Storage directory.
    Args:
        project: The Google Cloud project id to use as a parent resource.
        input_gcs_bucket: The name of google cloud storage bucket to inspect.
        output_gcs_bucket: The name of google cloud storage bucket where
            de-identified files would be stored.
        info_types: A list of strings representing info types to look for.
            A full list of info type categories can be fetched from the API.
        deid_template_id: The name of the de-identify template for
            unstructured and structured files.
        structured_deid_template_id: The name of the de-identify template
            for structured files.
        image_redact_template_id: The name of the image redaction template
            for images.
        dataset_id: The identifier of the BigQuery dataset where transformation
            details would be stored.
        table_id: The identifier of the BigQuery table where transformation
            details would be stored.
        timeout: The number of seconds to wait for a response from the API.
    """

    # Instantiate a client.
    dlp = google.cloud.dlp_v2.DlpServiceClient()

    # Construct the configuration dictionary.
    # Specify the type of info the inspection will look for.
    # See https://cloud.google.com/dlp/docs/infotypes-reference for complete list of info types.
    inspect_config = {"info_types": [{"name": info_type} for info_type in info_types]}

    # Construct cloud_storage_options dictionary with the bucket's URL.
    storage_config = {
        "cloud_storage_options": {"file_set": {"url": f"gs://{input_gcs_bucket}"}}
    }

    # Specify the big query table to store the transformation details.
    big_query_table = {
        "project_id": project,
        "dataset_id": dataset_id,
        "table_id": table_id,
    }

    # Convert the project id into a full resource id.
    parent = f"projects/{project}/locations/global"

    # Construct Transformation Configuration with de-identify Templates used
    # for transformation.
    transformation_config = {
        "deidentify_template": f"{parent}/deidentifyTemplates/{deid_template_id}",
        "structured_deidentify_template": f"{parent}/deidentifyTemplates/{structured_deid_template_id}",
        "image_redact_template": f"{parent}/deidentifyTemplates/{image_redact_template_id}",
    }

    # Tell the API where to send notification when the job is completed.
    actions = [
        {
            "deidentify": {
                "cloud_storage_output": f"gs://{output_gcs_bucket}",
                "transformation_config": transformation_config,
                "transformation_details_storage_config": {"table": big_query_table},
                "file_types_to_transform": ["IMAGE", "CSV", "TEXT_FILE"],
            }
        }
    ]

    # Construct the job definition.
    inspect_job = {
        "inspect_config": inspect_config,
        "storage_config": storage_config,
        "actions": actions,
    }

    # Call the API.
    response = dlp.create_dlp_job(
        request={
            "parent": parent,
            "inspect_job": inspect_job,
        }
    )

    job_name = response.name
    print(f"Inspection Job started : {job_name}")

    # Waiting for the job to get completed.
    job = dlp.get_dlp_job(request={"name": job_name})
    # Since the sleep time is kept as 30s, number of calls would be timeout/30.
    no_of_attempts = timeout // 30
    while no_of_attempts != 0:
        # Check if the job has completed.
        if job.state == google.cloud.dlp_v2.DlpJob.JobState.DONE:
            break
        if job.state == google.cloud.dlp_v2.DlpJob.JobState.FAILED:
            print("Job Failed, Please check the configuration.")
            break

        # Sleep for a short duration before checking the job status again.
        time.sleep(30)
        no_of_attempts -= 1

        # Get DLP job status.
        job = dlp.get_dlp_job(request={"name": job_name})

    if job.state != google.cloud.dlp_v2.DlpJob.JobState.DONE:
        print(f"Job did not complete within {timeout} minutes.")
        return

    # Print out the results.
    print(f"Job name: {job.name}")
    result = job.inspect_details.result
    print(f"Processed Bytes: {result.processed_bytes}")
    if result.info_type_stats:
        for stats in result.info_type_stats:
            print(f"Info type: {stats.info_type.name}")
            print(f"Count: {stats.count}")
    else:
        print("No findings.")

REST

JSON 输入

{
   "inspect_job": {
     "storage_config": {
       "cloud_storage_options": {
         "file_set": {
           "url": "INPUT_DIRECTORY"
         }
       }
     },
     "inspect_config": {
       "info_types": [
         {
           "name": "PERSON_NAME"
         }
       ]
     },
     "actions": {
       "deidentify": {
         "cloud_storage_output": "OUTPUT_DIRECTORY",
         "transformation_config": {
           "deidentify_template": "DEIDENTIFY_TEMPLATE_NAME",
           "structured_deidentify_template": "STRUCTURED_DEIDENTIFY_TEMPLATE_NAME",
           "image_redact_template": "IMAGE_REDACTION_TEMPLATE_NAME"
         },
         "transformation_details_storage_config": {
           "table": {
             "project_id": "TRANSFORMATION_DETAILS_PROJECT_ID",
             "dataset_id": "TRANSFORMATION_DETAILS_DATASET_ID",
             "table_id": "TRANSFORMATION_DETAILS_TABLE_ID"
           }
         },
         "fileTypesToTransform": ["IMAGE","CSV", "TEXT_FILE"]
       }
     }
   }
 }

替换以下内容：

• PROJECT_ID：您要在其中存储检查作业的项目的 ID 。
• INPUT_DIRECTORY：您要检查的 Cloud Storage 目录，例如 gs://input-bucket/folder1/folder1a。 如果网址以斜杠结尾，则不会扫描 INPUT_DIRECTORY 中的任何子目录。
• OUTPUT_DIRECTORY：您要在其中存储去标识化文件的 Cloud Storage 目录。此目录不得与 INPUT_DIRECTORY 位于同一 Cloud Storage 存储桶中。
• DEIDENTIFY_TEMPLATE_NAME：默认去标识化模板（适用于非结构化和结构化文件）的完整资源名称（如果您创建了该模板）。此值必须采用 projects/projectName/(locations/locationId)/deidentifyTemplates/templateName 格式。
• STRUCTURED_DEIDENTIFY_TEMPLATE_NAME：结构化文件的去标识化模板的完整资源名称（如果您创建了该模板）。 此值必须采用 projects/projectName/(locations/locationId)/deidentifyTemplates/templateName 格式。
• IMAGE_REDACTION_TEMPLATE_NAME：图片的图片遮盖模板的完整资源名称（如果您创建了该模板）。 此值必须采用 projects/projectName/(locations/locationId)/deidentifyTemplates/templateName 格式。
• TRANSFORMATION_DETAILS_PROJECT_ID：您要在其中存储转换详情的项目的 ID。
• TRANSFORMATION_DETAILS_DATASET_ID：您要在其中存储转换详情的 BigQuery 数据集的 ID。如果您未提供表 ID，系统会自动创建一个。
• TRANSFORMATION_DETAILS_TABLE_ID：您要在其中存储转换详情的 BigQuery 表的 ID。

请注意以下对象：

• inspectJob：作业的配置对象 (DlpJob)。此对象包含 检查和去标识化阶段的配置。
• storageConfig：要检查的内容的位置 (StorageConfig)。此示例指定了 Cloud Storage 存储桶 CloudStorageOptions。
• inspectConfig：有关您要检查的敏感数据的信息 （InspectConfig）。此示例检查 与内置 infoType PERSON_NAME 匹配的内容。
• actions：在作业的检查部分完成后要执行的操作 (Action)。
• deidentify：指定此操作会告知 Sensitive Data Protection 根据内部指定的配置对匹配的敏感数据进行去标识化处理 (Deidentify)。
• cloud_storage_output：指定您要检查的 Cloud Storage 目录的网址。

• transformation_config：指定 Sensitive Data Protection 必须 如何对结构化文件、非结构化文件和 图片中的敏感数据进行去标识化处理 (TransformationConfig)。

  如果您未包含 TransformationConfig 对象，Sensitive Data Protection 会将文本中的敏感数据替换为其 infoType。在图片中，它会使用黑框遮盖敏感数据。

• transformation_details_storage_config：指定 Sensitive Data Protection 必须存储有关其为此作业执行的每次转换的元数据。 此外，它还指定了 Sensitive Data Protection 必须在其中存储该元数据的表的位置和名称 (TransformationDetailsStorageConfig)。

• fileTypesToTransform：将去标识化操作限制为仅对 文件类型执行。如果您未设置此字段，则检查操作中包含的所有受支持 文件类型也会包含在 去标识化操作中。在此示例中，即使 您将 DlpJob 配置为检查所有受支持的文件类型，Sensitive Data Protection 也只会对图片、CSV 和文本文件进行去标识化处理。

通过 REST API 创建检查作业

如需创建检查作业 (DlpJob)，请发送 projects.dlpJobs.create 请求。如需使用 c网址 发送请求，请将之前的 REST 示例另存为 JSON 文件，然后运行 以下命令：

curl -s \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $(gcloud auth print-access-token)" \
-H "X-Goog-User-Project: PROJECT_ID" \
https://dlp.googleapis.com/v2/projects/PROJECT_ID/dlpJobs \
-d @PATH_TO_JSON_FILE

替换以下内容：

• PROJECT_ID：您在其中存储 DlpJob 的项目的 ID。
• PATH_TO_JSON_FILE：包含请求正文的 JSON 文件的路径。

Sensitive Data Protection 会返回新创建的 DlpJob 的标识符、其状态以及您设置的检查配置的快照。

{
  "name": "projects/PROJECT_ID/dlpJobs/JOB_ID",
  "type": "INSPECT_JOB",
  "state": "PENDING",
  ...
}

检索检查作业的结果

如需检索 DlpJob 的结果，请发送 projects.dlpJobs.get 请求：

curl -s \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $(gcloud auth print-access-token)" \
-H "X-Goog-User-Project: PROJECT_ID" \
https://dlp.googleapis.com/v2/projects/PROJECT_ID/dlpJobs/JOB_ID

替换以下内容：

• PROJECT_ID：您在其中存储 DlpJob 的项目的 ID。
• JOB_ID：创建 DlpJob 时返回的作业的 ID。

如果操作已完成，您会收到类似于以下内容的响应：

{
  "name": "projects/PROJECT_ID/dlpJobs/JOB_ID",
  "type": "INSPECT_JOB",
  "state": "DONE",
  "inspectDetails": {
    "requestedOptions": {
      "snapshotInspectTemplate": {},
      "jobConfig": {
        "storageConfig": {
          "cloudStorageOptions": {
            "fileSet": {
              "url": "INPUT_DIRECTORY"
            }
          }
        },
        "inspectConfig": {
          "infoTypes": [
            {
              "name": "PERSON_NAME"
            }
          ],
          "limits": {}
        },
        "actions": [
          {
            "deidentify": {
              "transformationDetailsStorageConfig": {
                "table": {
                  "projectId": "TRANSFORMATION_DETAILS_PROJECT_ID",
                  "datasetId": "TRANSFORMATION_DETAILS_DATASET_ID",
                  "tableId": "TRANSFORMATION_DETAILS_TABLE_ID"
                }
              },
              "transformationConfig": {
                "deidentifyTemplate": "DEIDENTIFY_TEMPLATE_NAME",
                "structuredDeidentifyTemplate": "STRUCTURED_DEIDENTIFY_TEMPLATE_NAME",
                "imageRedactTemplate": "IMAGE_REDACTION_TEMPLATE_NAME"
              },
              "fileTypesToTransform": [
                "IMAGE",
                "CSV",
                "TEXT_FILE"
              ],
              "cloudStorageOutput": "OUTPUT_DIRECTORY"
            }
          }
        ]
      }
    },
    "result": {
      "processedBytes": "25242",
      "totalEstimatedBytes": "25242",
      "infoTypeStats": [
        {
          "infoType": {
            "name": "PERSON_NAME"
          },
          "count": "114"
        }
      ]
    }
  },
  "createTime": "2022-06-09T23:00:53.380Z",
  "startTime": "2022-06-09T23:01:27.986383Z",
  "endTime": "2022-06-09T23:02:00.443536Z",
  "actionDetails": [
    {
      "deidentifyDetails": {
        "requestedOptions": {
          "snapshotDeidentifyTemplate": {
            "name": "DEIDENTIFY_TEMPLATE_NAME",
            "createTime": "2022-06-09T17:46:34.208923Z",
            "updateTime": "2022-06-09T17:46:34.208923Z",
            "deidentifyConfig": {
              "infoTypeTransformations": {
                "transformations": [
                  {
                    "primitiveTransformation": {
                      "characterMaskConfig": {
                        "maskingCharacter": "*",
                        "numberToMask": 25
                      }
                    }
                  }
                ]
              }
            },
            "locationId": "global"
          },
          "snapshotStructuredDeidentifyTemplate": {
            "name": "STRUCTURED_DEIDENTIFY_TEMPLATE_NAME",
            "createTime": "2022-06-09T20:51:12.411456Z",
            "updateTime": "2022-06-09T21:07:53.633149Z",
            "deidentifyConfig": {
              "recordTransformations": {
                "fieldTransformations": [
                  {
                    "fields": [
                      {
                        "name": "Name"
                      }
                    ],
                    "primitiveTransformation": {
                      "replaceConfig": {
                        "newValue": {
                          "stringValue": "[redacted]"
                        }
                      }
                    }
                  }
                ]
              }
            },
            "locationId": "global"
          },
          "snapshotImageRedactTemplate": {
            "name": "IMAGE_REDACTION_TEMPLATE_NAME",
            "createTime": "2022-06-09T20:52:25.453564Z",
            "updateTime": "2022-06-09T20:52:25.453564Z",
            "deidentifyConfig": {},
            "locationId": "global"
          }
        },
        "deidentifyStats": {
          "transformedBytes": "3972",
          "transformationCount": "110"
        }
      }
    }
  ],
  "locationId": "global"
}

后续步骤

• 详细了解存储空间中数据的去标识化过程。
• 了解如何使用控制台对存储空间中的数据进行去标识化处理 Google Cloud 。
• 完成在 Cloud Storage中创建去标识化的数据副本 Codelab。
• 详细了解去标识化转换。
• 了解如何检查存储空间中是否存在敏感数据。