Deployment guide

This page guides you through the Cortex Framework deployment steps for production-path deployments.

Before you begin

Ensure you have completed the prerequisites outlined on the Prerequisites page before continuing with the deployment.

Prior to deploying Cortex Framework, choose and prepare the Source system integration.

Configure IAM roles

Configure the required Identity and Access Management (IAM) permissions to deploy Cortex Framework content within the required Google Cloud projects.

Roles for build project

To get the permissions that you need to build Cortex Framework, ask your administrator to grant you the BigQuery Data JobUser (roles/bigquery.jobUser) IAM role on your build project. For more information about granting roles, see Manage access to projects, folders, and organizations.

You might also be able to get the required permissions through custom roles or other predefined roles.

To grant the requested roles to a user, you can use the script:

      gcloud projects add-iam-policy-binding BUILD_PROJECT_ID \
        --member="user:USER_EMAIL" \
        --role="roles/bigquery.jobUser"

Roles for source project

To get the permissions that you need to deploy Cortex Framework, ask your administrator to grant you the BigQuery Data Viewer (roles/bigquery.dataViewer) IAM role on your source project. For more information about granting roles, see Manage access to projects, folders, and organizations.

You might also be able to get the required permissions through custom roles or other predefined roles. 

      gcloud projects add-iam-policy-binding SOURCE_PROJECT_ID \
        --member="user:USER_EMAIL" \
        --role="roles/bigquery.dataViewer"

Roles for target projects

To get the permissions that you need to deploy Cortex Framework, ask your administrator to grant you the following IAM roles on your target projects:

For more information about granting roles, see Manage access to projects, folders, and organizations.

You might also be able to get the required permissions through custom roles or other predefined roles.

To grant the requested roles to a user, you can use the script:

gcloud projects add-iam-policy-binding TARGET_PROJECT_ID --member="user:USER_EMAIL" \
        --role="roles/bigquery.dataEditor"
gcloud projects add-iam-policy-binding TARGET_PROJECT_ID --member="user:USER_EMAIL" \
        --role="roles/dataform.admin"
gcloud projects add-iam-policy-binding TARGET_PROJECT_ID --member="user:USER_EMAIL" \
        --role="roles/serviceusage.serviceUsageConsumer"

Deployment Configuration

Before deployment, the data foundation and data product layers are configured through a centralized, configuration-as-code approach. Cortex Framework uses a config/config.yaml file to configure the Cortex Framework deployment. We recommend that you make a copy of config/config.yaml.example as a starting point. This file contains the essential mapping between your source data and the target Google Cloud environment. Proper configuration ensures that Cortex Framework knows exactly where to find your raw data and where to materialize the resulting layers.

For a successful Cortex Framework deployment, these mandatory values need to be added:

  • YOUR_BUILD_PROJECT_ID: The project where the deployment orchestration and uv scripts are executed.
  • YOUR_SOURCE_PROJECT_ID: The project containing your raw source data.
  • YOUR_TARGET_PROJECT_ID: The destination project where the processed data products and BigQuery datasets will reside.
  • YOUR_REPO_PROJECT_ID: The project hosting the Dataform repository and deployment artifacts.

For detailed information on the configuration file, see Deployment configuration.

Execute deployment

Run the following command to trigger the Cortex Framework deployment. This process will perform the following actions:

  • Verify that all prerequisites are complete.
  • Connect to the raw dataset, and based on the schema information, build and compile the .sqlx scripts.
  • Create the Dataform repository and workspace, then sync the compiled artefacts with the repository.
uv run cortex-build-and-deploy --config config/config.yaml

uv run cortex-build-and-deploy CLI parameters: you can optionally display a list of possible parameters supported by the cortex-demo script by running the uv run cortex-build-and-deploy -h command. For more information, see CLI Cortex Framework uv run cortex-build-and-deploy.

Command line options

The uv run command line interface (CLI) support these actions:

  • uv run cortex-build: Executes compilation of the Dataform workspace, translating YAML configs and SQLX/JS templates into deployable Dataform assets. For more information, see CLI Cortex build.
  • uv run cortex-deploy: Pushes the compiled assets to the target Dataform workspace. For more information, see CLI Cortex deploy.
  • uv run cortex-build-and-deploy: Combines the compilation and pushing of the build assets into Dataform workspace. For more information, see CLI Cortex build and deploy.

Next steps