创建数据产品模块

如需定义您自己的业务逻辑和分析模型，请创建自定义数据产品模块。这样，您就可以对基础表或上游数据产品运行计算，并将结果打包到可部署的数据集中。

注意： 您可以按照手动配置说明操作，也可以使用预打包的 AI 技能来自动搭建和验证模块。如需详细了解 AI 辅助工作流程，请参阅数据产品的自主创建。

前提条件

我们建议在专用自定义命名空间中创建自定义数据产品模块，以便更好地管理生命周期。此外，请确保您计划使用的源表存在于数据基础数据集中。

创建数据产品模块

数据产品模块定义需要执行以下步骤：

• 通过添加条目来扩展 data.modules.products 列表，从而在 config/config.yaml 文件中注册数据产品模块：

data:
  # Configuration for data foundation and product modules.
  modules:
    # List of data product modules.
    products:
        # Recommended naming for product_module_id:
        # custom_namespace_product_name
      - moduleId:  product_module_id
        # Path of the data product (namespaced).
        modulePath:  custom_namespace.system_type.products.product_name
        # Map of module dependencies.
        dependencyBindings:
          sapModule: erp
          sapModuleCustNS:  foundation_module_id
        # Reference to the target dataset ID.
        dataTargetId: product_target
        # Whether the module is enabled.
        # enabled: true
        # Whether the foundation is external (does not create target dataset).
        # external: false
        # Custom table settings file, relative to 'config/' file directory
        # Recommended path: '{custom_namespace}/{system_type}/products/{product_name}/table_settings.yaml'
        # If omitted, defaults to '../src/data_modules/{custom_namespace}/{system_type}/products/{product_name}/table_settings.default.yaml'
        # tableSettings: "{custom_namespace}/{system_type}/products/{product_name}/table_settings.yaml"
        

• 创建默认 tableSettings 文件（例如 src/data_modules/custom_namespace/system_type/products/product_name/table_settings.default.yaml）。

此 YAML 控制表配置，例如具体化和 BigQuery 优化详细信息：

common:
  custom_sales_summary:
    materializationType: "table"
    bigQueryLabels:
      - key: data_class
        value: transactional
    dataformTags: ["custom", "sales", "reporting"]
    partitionDetails:
      column: "created_date"
      partitionType: "date"
      timeGrain: "day"
    clusterDetails:
      columns:
        - "customer_id"

• 创建注解文件

系统会为每个数据产品输出制品（表、视图）创建注解文件 tablename.yaml，并以 YAML 格式描述列和字段。在编译期间，构建器会自动搜索产品 annotations/ 文件夹（例如 src/data_modules/custom_namespace/system_type/products/product_name/annotations/custom_sales_summary.yaml）中的注释，并将这些字符串直接合并到输出 Dataform 架构定义中，以便将它们保留在 BigQuery 表元数据中。

注解 src/data_modules/custom_namespace/system_type/products/product_name/annotations/tablename.yaml 文件的格式如下：

description: "Description of the table or view purpose"
fields:
  - name: "customer_id"                     # column name
    description: "Customer identifier"      # column description
  - name: "column2"
    description: "Description of Column 2"
  - name: "column3"
    description: "Description of Column 3"

• 在数据产品文件夹 src/data_modules/custom_namespace/system_type/products/product_name/ 中创建一个 manifest.yaml 文件，并保持类型、类别、表和模块依赖项不变。清单文件遵循以下格式：

displayName: Sales Performance Summary
description: Sales performance analytical data product.
category: product
type: generic
builder: sap_product     # Automatically resolves to the global SapProductBuilder fallback
dependencies:
  sapModule:
    modulePath: cortex.sap.foundations.sap
    supportedVersions:
      - ecc
      - s4

数据产品模块示例

在 flights 示例的命名空间 sap_bookingdatamodel 中实现 flights_usd 数据产品的步骤如下：

• 通过添加条目来扩展 data.modules.products 列表，从而在 config/config.yaml 文件中注册数据产品模块：

data:
  modules:
    products:
      - moduleId: sap_bookingdatamodel_flights_usd
        modulePath: sap_bookingdatamodel.sap.products.flights_usd
        dependencyBindings:
          sapModule: erp
          sapModuleCustNS: sap_bookingdatamodel
        dataTargetId: product_target

• 接下来，创建包含以下内容的 src/data_modules/sap_bookingdatamodel/sap/products/flights_usd/manifest.yaml

displayName: Flights USD
description: Flight scheduling and pricing USD data product.
category: product
type: generic
dependencies:
  sapModule:
    modulePath: cortex.sap.foundations.sap
    supportedVersions:
      - ecc
      - s4
    tables:
      common:
        - tcurr
  sapModuleCustNS:
    # Type of the dependent Module.
    # use cortex.sap.foundations.sap if you followed "Configure multiple instances of a data foundation module"
    # https://docs.cloud.google.com/cortex/docs/deployment-configuration#multiple-data-foundation-instances
    modulePath: cortex.sap.foundations.sap
    # use sap_bookingdatamodel.sap.foundations.sap if you are connecting to custom-data foundation module:
    # https://docs.cloud.google.com/cortex/docs/extensibility-guide-data-foundation
    #modulePath: sap_bookingdatamodel.sap.foundations.sap
    supportedVersions:
      - ecc
      - s4
    tables:
      common:
        - sflight
builder: sap_product

• 在下一步中，创建引用的表格设置文件，以配置 BigQuery 中输出表格或视图的架构和元数据。

在所用示例中，创建 src/data_modules/sap_bookingdatamodel/sap/products/flights_usd/table_settings.default.yaml，其中包含以下内容：

ecc:
  flights_usd:
    materializationType: incremental
    bigQueryLabels:
      - key: data_class
        value: transactional
    dataformTags: [sap, dataproduct, masterdata]
s4:
  flights_usd:
    materializationType: incremental
    bigQueryLabels:
      - key: data_class
        value: transactional
    dataformTags: [sap, dataproduct, masterdata]

• 为数据产品表创建注释，以使用说明丰富存储架构。

在所用示例中，创建文件：src/data_modules/sap_bookingdatamodel/sap/products/flights_usd/annotations/flights_usd.yaml，其中包含以下内容：

description: "Flight scheduling and pricing information, including currency conversion to USD."
fields:
  - name: "client_mandt"
    description: "Client (Mandant), PK"
  - name: "airline_code_carrid"
    description: "Airline Carrier ID, PK"
  - name: "flight_connection_number_connid"
    description: "Flight Number, PK"
  - name: "flight_date_fldate"
    description: "Flight Date"
  - name: "price_usd"
    description: "Price in USD"
  - name: "price"
    description: "Price in local currency"
  - name: "currency"
    description: "Local currency"

• 数据产品的业务逻辑存储在 js 或 sqlx 文件中。

注意：对于不同方言（例如 SAP ECC 和 S/4HANA）的逻辑存在差异的数据产品，请使用 definitions 文件夹的子文件夹 ecc 和 s4。

注意：在本地模块的 definitions/ 文件夹内创建的文件由 cortex-build 编译，并在部署期间同步到远程 Dataform 工作区。如果您需要使用 Google Cloud 控制台直接在远程 Dataform 工作区内添加自定义代码，而无需在源代码控制工具中创建模块，请将文件放置在 Dataform 远程代码库中的 definitions/custom/ 内。Dataform 远程仓库中 definitions/custom/ 之外的所有非受管文件将在下次部署运行 (cortex-deploy) 时被覆盖或删除。

在给定的示例中，创建一个包含以下内容的 src/data_modules/sap_bookingdatamodel/sap/products/flights_usd/definitions/flights_usd.js 文件：

// ___MODULE_CONTEXT___
// ___TABLE_CONFIG___

const moduleConfig = config.product[moduleContext.moduleId];
const sapModuleConfigDatasetId = moduleConfig.sources.sapModule.datasetId;
const sapModuleCustNSConfigDatasetId = moduleConfig.sources.sapModuleCustNS.datasetId;

const materializationType = tableConfig.materializationType || "incremental";

const incremental = require("includes/cortex/incremental.js");
const publish_config = require("includes/cortex/publish_config.js");

const publishConfig = publish_config.getPublishConfig(
   materializationType,
   tableConfig,
   moduleConfig,
   [
       "client_mandt",
       "airline_code_carrid",
       "flight_connection_number_connid",
       "flight_date_fldate"
   ]
);

publish("flight_usd", publishConfig).query(
   (ctx) => `
WITH flight_base AS (
   SELECT
       mandt,
       carrid,
       connid,
       fldate,
       price,
       currency,
       -- Convert flight date string (YYYYMMDD) to an integer to calculate SAP's inverted date key
       CAST(99999999 - CAST(fldate AS INT64) AS STRING) AS inverted_fldate
   FROM   ${ctx.ref(sapModuleCustNSConfigDatasetId, 'sflight')} AS flight
),
ranked_exchange_rates AS (
   SELECT
       f.mandt,
       f.carrid,
       f.connid,
       f.fldate,
       f.price,
       f.currency,
       t.ukurs,
       -- Window function to grab the closest historical exchange rate
       ROW_NUMBER() OVER (
           PARTITION BY f.mandt, f.carrid, f.connid, f.fldate
           ORDER BY t.gdatu ASC
       ) AS latest_rate_rank
   FROM flight_base f
   LEFT JOIN ${ctx.ref(sapModuleConfigDatasetId, 'tcurr')} AS t
     ON f.mandt = t.mandt
    AND t.kurst = 'M'       -- 'M' is the standard SAP default for average exchange rates
    AND t.fcurr = f.currency
    AND t.tcurr = 'USD'
    -- Chronological (rate_date <= flight_date) translates to (t.gdatu >= inverted_fldate)
    AND t.gdatu >= f.inverted_fldate
)

SELECT
   client_mandt,
   airline_code_carrid,
   flight_connection_number_connid,
   flight_date_fldate,
   price,
   currency,
   price_usd,
   CURRENT_TIMESTAMP() AS bq_loaded_at
FROM (
  SELECT
    mandt              AS client_mandt,
    carrid             AS airline_code_carrid,
    connid             AS flight_connection_number_connid,
    PARSE_TIMESTAMP('%Y%m%d', fldate) AS flight_date_fldate,
    price              AS price,
    currency           AS currency,
    -- Currency Conversion Logic
    CASE
       WHEN currency = 'USD' THEN price
       WHEN ukurs IS NULL   THEN NULL -- Handles cases where no exchange rate is found
       -- If UKURS is negative, it's an indirect quotation (1 USD = X Local) -> Divide
       WHEN ukurs < 0       THEN ROUND(price / ABS(ukurs), 2)
       -- If UKURS is positive, it's a direct quotation (1 Local = X USD) -> Multiply
       ELSE ROUND(price * ukurs, 2)
     END AS price_usd
  FROM ranked_exchange_rates
  WHERE latest_rate_rank = 1
)
${incremental.getWhere(ctx, ["flight_date_fldate"])}
`
);

自定义命名空间扩展的验证

如需验证 Google Cloud Cortex Framework 数据产品模块是否已成功创建，请按以下步骤操作：

• 执行引用更新后的 config.yaml 文件的 Cortex Framework 部署。
• 按照部署后步骤执行 Dataform 操作，并在 BigQuery 中验证结果

---

• 上一步：创建数据基础模块
• 下一步：部署后步骤
• 返回概览