SAP ERP data source

Important: The steps outlined on this page are required for the Cortex Framework deployment. For the demo deployment, you can skip this page.

Google Cloud Cortex Framework data foundation layer for SAP ERP requires connectivity to the source system raw data. Both SAP ECC and SAP S/4HANA are supported.

Prior to deploying Cortex Framework content, relevant SAP ERP tables must be replicated into BigQuery. You can achieve this by landing data in a dedicated raw layer dataset for Change Data Capture (CDC) processing or by using established CDC pipelines to feed the data foundation layer directly. For more information, see Technical requirements for replicating SAP ERP data.

You may use any replication tool of your choice, provided it can replicate data in raw table format to BigQuery. For example, Google Cloud solutions include the BigQuery Connector for SAP (requires SAP SLT), and the BigQuery Toolkit for SAP.

To ensure compatibility between the replicated raw datasets from SAP ERP and Cortex Framework data foundation layer, ensure you meet the following requirements.

Technical requirements for replicating SAP ERP data

Make sure to review and complete the following technical requirements for replicating SAP data into Cortex Framework in BigQuery.

1. Raw data structure: Data from ECC or S/4HANA should land in BigQuery with the same structure as the base tables in SAP and without business transformations. Tables must be replicated with the required field names, types, and granularity as they exist in SAP.

2. Table configuration: The list of tables to be transformed is defined in the table_settings.yaml file (located under config/cortex/data_foundation/sap). If a required table is missing during deployment, the specific data products depending on it will fail.

3. Metadata requirements: You must replicate table DD03L from your SAP source. This table is critical for the dependency resolver as it contains field metadata and keys.

4. Casing: Names of the replicated SAP tables in BigQuery must be in lowercase for Cortex Framework data model compatibility (for example, the SAP table MARA becomes mara in BigQuery).

5. Object names (columns) and special characters: For object names (columns) containing special characters (like /, -, or leading underscores _), Cortex expects a generic sanitization pattern:

   • All non-alphanumeric characters are replaced with an underscore _.
   • Leading underscores and digits are not allowed. For example, /GOOG/TEST becomes goog_test, and _DATAAGING becomes dataaging. If your replication tool lands data with leading underscores preserved, a normalization step (aliasing) in the Data Foundation layer is required.

6. Data propagation fields: To support CDC (Change Data Capture) and data propagation, replicated SAP tables need to have:

   • An operation flag named operation_flag (L = initial load, I = insert, U = update, D = delete).
   • A timestamp named recordstamp (populated with the current timestamp at the time of the load).
   • Optional: An additional field is_deleted (BOOLEAN) is chosen in replicated _DS_RAW tables (defaulting to false on initial load). Runtime views generated by Cortex reference this column, but it can be removed from the CDC and view templates prior to execution if the replication tool doesn't produce it.

7. Data types: Necessary mapping of SAP data types with BigQuery data types for compatibility:

   Necessary for standard operations:

   Data type SAP	Data type BigQuery	Description
   DATS	DATE	Date data type
   TIMS	TIME	Time data type

8. Highly Recommended for precision and compatibility:

   • CURR (Currency) and QUAN (Quantity) mapped to NUMERIC or BIGNUMERIC (Avoid FLOAT64 to prevent rounding errors in financial calculations).
   • NUMC (Numeric Character) mapped to STRING (To preserve leading zeros for document numbers and item numbers, ensuring successful joins).
   Note: Generally, you can assume the default mapping as long as the same operations on the target field can be applied. If the mappings are changed, customers can manually CAST datatypes in the templates.

9. Payload Compression: To prevent empty SAP columns (initial values like spaces or zeros) from being filled with NULL in BigQuery, ensure that payload compression is deactivated in the connector configuration (or "Send Uncompressed" is enabled). This ensures that empty strings or zeros are preserved as such in the target rather than defaulting to NULL.