Overview
Google Cloud Cortex Framework provides data product accelerators designed to transform raw data from strategic enterprise systems into trusted, high-quality data assets ready for advanced analytics and AI.
Delivered as customizable open-source code deployable securely within your own Google Cloud environment, Google Cloud Cortex Framework offers a serverless, BigQuery-native execution model powered by Google Cloud Dataform.
By streamlining the way teams build, orchestrate, and deploy data pipelines, Cortex Framework accelerates time-to-value and provides a high-fidelity foundation for the next generation of enterprise AI agents.
Data architecture
Cortex Framework standardizes data processing inside BigQuery into three distinct architectural layers using an Extract, Load, and Transform (ELT) methodology. This standardized data layering strategy aligns with enterprise data mesh concepts.
Source system
Source systems are the origins of the data ingested. Source systems can include various enterprise applications, databases, or platforms from which data is extracted. For all supported source systems, see supported source systems.
Raw layer
The raw layer represents the immutable landing zone dataset in BigQuery for source data, either Change Data Capture (CDC) logs or batch extracts. While it frequently stores CDC logs (for example, from SAP ECC or S/4HANA using replication tools like BigQuery Connector for SAP or BigQuery Toolkit for SAP), it is designed to represent any raw format. For sources that don't provide CDC logs, such as Salesforce or external API feeds, this layer represents the full batch extracts or raw event payloads exactly as they arrive, with minimal to no structural alterations. This layer feeds the data foundation layer.
Data foundation layer
The data foundation layer is a standardized, clean representation of the latest records of the source data and feeds the data product layer. This layer is updated in an incremental way for CDC-enabled sources and uses views for non-CDC-enabled as well as externally implemented CDC sources. The implementation adapts to the source system's capabilities. For more information, see Data foundation.
Data product layer
The data product layer provides aggregations, KPI-calculations, business logic and cross application connection logic. The views and tables exposed by the data product layer are designed for direct consumption by BigQuery Conversational Analytics Agents, Gemini Enterprise, Machine Learning Models, BI dashboards and reports, as well as application integrations. For more information, see Data products.
Data management
Cortex Framework uses Dataform to manage the lifecycle of data. Dataform lets you manage data transformation for data integration. It provides a service for data analysts to develop, test, control versions, and schedule complex workflows for data transformation in BigQuery. To adapt dynamically to customizations and extensions, Cortex Framework creates the Dataform code during the build phase, and stages it to the Dataform repository in Google Cloud.
By triggering an execution of the pipelines in the Dataform UI, the BigQuery tables and views of the data foundation and data product layers are created and filled with data.
Next steps
Ready to build and deploy? Explore the following guides to get your environment up and running:
- Demo deployment: Deploy the sample solution content in minutes to see Cortex Framework in action.
- Deployment: Follow step-by-step instructions to configure and deploy Cortex Framework for your enterprise data.