Cortex Framework extensibility guide

Google Cloud Cortex Framework provides data product accelerators for advanced analytics and AI. This documentation explains how to customize and extend Cortex Framework.

For proper isolation and lifecycle management, custom foundation and data product modules should be packaged within a custom namespace. This guide covers two major areas of extensibility:

• Data foundation module creation: Register custom data modules to define custom compilation behaviors or ingest tables from source systems not supported out-of-the-box (e.g., Salesforce).
• Data product module creation: Create new analytical data products with custom logic using local SQLX or JS files.

Note: You do not need to create a custom data foundation module to ingest and transform additional standard or custom tables. You can add such extra tables directly to the table configuration of the standard modules, or configure multiple instances of the existing modules in config/config.yaml as described in Deployment configuration.

Example scenario

To illustrate an end-to-end example of extending Google Cloud Cortex Framework, this guide provides practical examples using SAP flight scheduling data:

1. Namespace definition: To package and isolate custom modules, a dedicated custom namespace sap_bookingdatamodel is introduced. See Custom namespace setup for details.
2. Data foundation extension: While adding tables typically only requires modifying table settings, this walkthrough demonstrates how to create a custom data foundation module to illustrate packaging and compiler customization. See Data foundation module creation for details.
3. Custom data product creation: This data product consumes the sflight foundation table, joins it with the standard SAP currency exchange rates table (tcurr), and converts the flight pricing from local currency into US dollars (USD). See Data product module creation for details.