Setting up a custom namespace

Custom namespaces serve as logical environments to package and isolate your data foundation and analytical data product modules. Running inside a dedicated custom namespace lets you manage configuration and deployment options independently, providing isolation for your environment. A key benefit of this separation is that you can seamlessly pull future updates and enhancements from the cortex namespace in Google Cloud Cortex Framework without the risk of overriding or corrupting your custom modules. We highly recommend creating your custom modules within a dedicated namespace.

Custom namespace folder structure

The Cortex Framework namespace folders are used to package and isolate custom module artifacts. The folder structure of a custom namespace is defined in the table:

Directory path Purpose and description Typical subfolder structure
config/ Cortex deployment configuration (Required)
Creating new namespaces requires their setup within config.yaml. 
config/config.yaml
src/data_modules/custom_namespace_path/data_foundation/ Data foundation modules (Optional)
Raw-to-foundation datasets transformations. 
-
src/data_modules/custom_namespace_path/data_product/ Data product definitions (Optional)
Contains business logic and analytical models. Each product is separated in a subdirectory (for example, your_product_name) and consists of:
  • Readme: Documentation of the module
  • manifest.yaml: Declares the builder type, dependencies, and configuration.
  • annotations/: Column and field-level descriptions.
  • definitions/: Dataform SQLX or JS source files.
 
data_product/
└── your_product_name/
    ├── README.md
    ├── manifest.yaml
    ├── annotations/
    └── definitions/
src/data_modules/custom_namespace_path/includes/ JavaScript helpers (Optional)
Any JavaScript files placed here are automatically packaged and made available to your Dataform models during compilation under the namespace path (for example, includes/custom_namespace/). 
-
src/data_modules/custom_namespace_path/common/ Namespace shared tools (Optional)
Namespace-scoped custom builders, deployers and other tools. 
-

Custom namespace configuration

A new namespace: custom_namespace can be defined by extending the corresponding section of the config.yaml file, creating a new directory under src/data_modules/ and adding the new module code assets.

To make the Google Cloud Cortex Framework compiler aware of your custom namespace and modules, you must register them in your global configuration file (e.g., config/config.yaml).

Step A: Register the namespace

Under the data.namespaces block, add your namespace metadata:

data:
  namespaces:
    - name: cortex
      path: cortex/
    - name:  custom_namespace    # <-- Name of custom namespace
      path:  custom_namespace_path/  # <-- Folder name that is used for custom namespace, points to subdirectory of 'src/data_modules/'

Step B: Configure custom modules

Register your custom modules under data.modules.foundation or data.modules.product. Use the dot-separated format custom_namespace.module_type for the type field:

data:
  modules:
    foundation:
      - moduleId:  custom_namespace_foundation_module_id
        type: custom_namespace.sap   # Format: <namespace>.<module_type>
        dataSourceId: sap_raw_s4
        dataTargetId: data_foundation_sap_custom_namespace
        moduleSettings:
          sapVersion: s4
          mandt: "100"
        tableSettings: "config/custom_namespace_path/data_foundation/sap/table_settings.yaml"

    product:
      - moduleId: custom_namespace_data_product_id
        type: custom_namespace.data_product_id  # Format: <namespace>.<module_type>
        dependsOn:
          sapModule: erp_ecc            # References an existing foundation moduleId
        dataTargetId: product_target
        tableSettings: "config/custom_namespace_path/data_product/data_product_id/table_settings.yaml"

Custom namespace shared code

The custom namespace supports sharing code artifacts between its modules. The supported artifact types are:

  • includes - Contains any JavaScript helpers that will be made available to your Dataform artifacts. They can be placed in the folder: src/data_modules/custom_namespace_path/includes/.
  • builders - Any custom generator classes that can compile your module source code into Dataform artifacts. They can be placed in src/data_modules/custom_namespace_path/common/.

Understanding and referencing builders

Builders are Python generator classes that compile your module source files into physical Dataform outputs. Google Cloud Cortex Framework compiles modules using a flexible three-tier builder resolution sequence:

Google Cloud Cortex Framework extensibility builders resolution sequence

Figure 1. Cortex Framework extensibility builders resolution sequence.

Tier 1: Global fallback builders

The framework includes highly robust builders located in src/common/builders/. These builders are registered globally and can be used by any custom namespace.

For example, to use the standard SAP data product builders for your custom product, reference sap_product in your manifest.yaml:

# src/data_modules/custom_namespace_path/data_product/data_product_id/manifest.yaml
type: data_product_id
builder: sap_product     # Automatically resolves to the global SapProductBuilder fallback
dependencies:
  sapModule:
    type: sap
    supportedVersions:
      - ecc
      - s4

Tier 2: Namespace-scoped builders

If modules inside your custom namespace require a specialized compiler, you can register a custom builder for that namespace.

  1. Create a builder file under src/data_modules/custom_namespace_path/common/builders/my_builder.py.

  2. Decorate your builder class with @builder_registry.register("my_builder") :

   from common.builders.base import ProductBuilder
   from common.registry import builder_registry

   @builder_registry.register("my_builder")
   class MyCustomProductBuilder(ProductBuilder):
       # Implement build logic here...
  1. Reference it in your product's manifest.yaml using builder: my_builder.

Tier 3: Individual product-level builders

For highly unique pipelines, you can define a builder scoped exclusively to a single data product.

  1. Simply create a file named builder.py directly inside your data product folder (e.g., src/data_modules/custom_namespace_path/data_product/data_product_id/builder.py).

  2. Define a subclass of BaseBuilder or ProductBuilder inside it:

   from common.builders.base import ProductBuilder

   class UniqueSalesPerformanceBuilder(ProductBuilder):
       # Implement unique build logic here...
  1. The compiler will automatically dynamically import and execute this class without requiring any registration or decorators.

Custom namespace example

In the config/config.yaml file, add under the data.namespaces this block:

data:
  namespaces:
    - name: cortex
      path: cortex/
    - name: sap_bookingdatamodel # <-- Name of custom namespace: sap_bookingdatamodel
      path: sap_bookingdatamodel/  # <-- Folder name that is used for custom namespace:sap_bookingdatamodel, points to subdirectory of 'src/data_modules/'

In the next steps: Data foundation module creation and Data product module creation, you will learn how to add data modules to the namespace.