Cortex Framework Observability

To successfully run and operate enterprise-grade data platforms, having visibility into pipeline execution, data quality, and operational errors is critical.

Cortex Framework categorizes observability across its two distinct life cycles:

  1. Deploy-Time Observability: Tracking configuration load, template compilation, validation checks, deployment actions, and API telemetry during the execution of CLI tools.
  2. Run-Time Observability: Tracking execution, pipeline progress, query performance, data quality assertions, and automated alerting for the deployed data pipelines running inside Google Cloud.

Deploy-Time Observability

Deploy-time observability focuses on the execution of the CLI tools (e.g., uv run cortex-build, uv run cortex-deploy, uv run cortex-build-and-deploy, uv run cortex-demo, uv run cortex-kc-sync).

Local Console Logging

When you run CLI commands, Cortex Framework logs progress directly to the console (stdout).

  • Log Level: By default, logs are output at the INFO level.
  • Visual Highlights: Messages are color-coded to highlight errors and warnings dynamically:
    • ❌ Errors (Red) indicating critical failures that halt execution.
    • ⚠️ Warnings (Orange) indicating potential configuration anomalies or non-blocking issues.
  • Timestamp and Source: Every log line displays execution time and the active Python class or module names for precise tracing.

Persistent Local Log Files

On every command execution, the Python orchestrator automatically streams the complete execution log to a temporary log file in your system's temp directory:

/tmp/cortex-framework-logs-<YYYYMMDD-HHMM>.log

The precise path is printed to the console during startup of the CLI tools. These files contain comprehensive log details (including stack traces for unexpected errors) and are invaluable for debugging issues encountered during the execution of the CLI tools or when you are attaching to support requests.

Google Cloud Environment Validation

Prior to performing build, deploy, or sync actions, the orchestration engine executes the GcpEnvironmentChecker utility. This check validates:

  1. Required APIs: Confirms that crucial Google Cloud APIs are enabled (e.g. bigquery.googleapis.com, dataform.googleapis.com).
  2. Dataset Existence: Verifies that the required raw and target datasets exist or can be created.
  3. Locations & Regions: Ensures target datasets match the geographical regions of the source datasets.
  4. Capacity and Settings: Validates reservation settings and catalog configurations.

Any mismatch is logged as an error with recommended hints on how to resolve them before making Google Cloud service calls.

Telemetry

During deployment and synchronization processes, Cortex Framework logs anonymous adoption, variant, and framework version telemetry to Google Cloud. For details on how this works and instructions on how to opt out, see Telemetry.

Run-Time Observability

Once built and deployed, the Cortex Framework conformed data layers and data products run entirely within Dataform and BigQuery. As a result, runtime observability integrates directly with Google Cloud operational suites.

Pipeline Execution Logging

All deployed pipelines are tracked using Cloud Logging and execution tools:

  • Dataform Execution Logs: Dataform logs every compilation and execution event. These details can be accessed in the Google Cloud console or programmatically using the Dataform API.
  • BigQuery Job History: Each table and view materialized by your Dataform pipelines executes SQL queries inside BigQuery. Detailed resource usage, query performance, processed bytes, and execution timestamps are logged in the BigQuery Job History.

Pipeline Monitoring

You can monitor pipeline health, release configurations, and run history visually or programmatically:

  1. Dataform Web UI: Navigate to the Dataform console to:
    • Inspect compiled data models and visualize the compiled graph.
    • Verify the status of release configurations, compiled templates, and active environments.
    • Monitor the history and details of current and past workflow executions.
  2. Cloud Monitoring integration: Track Dataform pipeline metrics, such as execution durations, active compilations, and workflow job failure rates, through custom dashboard panels.

Alerting & Data Quality

To ensure data integrity and flag pipeline failures automatically, configure alerting using the following mechanisms:

Data Quality Assertions

You can define custom data validation rules (e.g., ensuring a column is never null, checking that primary keys are unique, or validating numeric ranges) by creating .sqlx assertion files.

  • You can supply a custom assertions file using the --assertions parameter: bash uv run cortex-deploy --config config/config.yaml --assertions config/assertions.sqlx
  • During pipeline execution, Dataform runs these validation queries. If an assertion query returns one or more rows, the validation fails, and the pipeline execution is immediately marked as failed.
  • For more information on writing data validation rules, see the official Dataform assertions documentation.

Example Assertion File (assertions.sqlx)

Example of a Dataform assertion query that checks for NULL values and duplicate customer records. If this query returns any rows, the assertion will fail and stop the execution workflow:

config {
  type: "assertion",
  description: "Ensure customer_number_kunnr is not null and unique"
}

-- Check for NULL values
(
  SELECT
    "customer_number_kunnr is NULL" AS error_message
  FROM
    ${ref("customers")}
  WHERE
    customer_number_kunnr IS NULL
)
UNION ALL
-- Check for duplicate keys
(
  SELECT
    CONCAT("Duplicate customer number found: ", customer_number_kunnr) AS error_message
  FROM
    ${ref("customers")}
  GROUP BY
    customer_number_kunnr,
    client_mandt
  HAVING
    COUNT(*) > 1
)

Cloud Alerting Policies

Set up standard Google Cloud Alerting Policies to notify your engineering or operations teams when issues arise:

  • Log-based Alerts: Create alerts in Cloud Logging that trigger when error events, failed workflow executions, or compiler issues are detected in your logs.
  • Metric-based Alerts: Define thresholds in Cloud Monitoring based on execution duration or compilation failures.
  • Notification Channels: Configure these alerts to route issues to your team's preferred communication channels