Run data and AI pipelines and provision resources for them with Orchestration Pipelines

Managed Airflow (Gen 3) | Managed Airflow (Gen 2) | Managed Airflow (Legacy Gen 1)

This page describes the relationship between Orchestration Pipelines and Managed Airflow.

About Orchestration Pipelines

Orchestration Pipelines is a unified, declarative orchestration and automated deployment framework optimized for seamlessly managing data and AI pipelines on Google Cloud.

With Orchestration Pipelines, you can define your pipelines and their deployment configurations using a declarative YAML-based Domain Specific Language (DSL). This framework abstracts the underlying infrastructure, allowing you to focus on the logic of your data and AI workflows while Orchestration Pipelines handles the deployment, versioning, and orchestration.

About Managed Airflow environments in Orchestration Pipelines

Managed Airflow is the orchestration engine that runs your pipelines after they're deployed. You assign a Managed Airflow environment as a part of a pipeline definition. This environment is called a runner environment in Orchestration Pipelines.

Benefits that Orchestration Pipelines:

  1. You can empower teams in your organizations that aren't Airflow experts to create and run their workflows without writing DAGs or configuring Airflow. For example, you can run a Notebook in an ephemeral Managed Service for Apache Spark cluster and specify all schedule, resources, and configuration parameters for its execution in YAML without writing any DAG code.

  2. All configuration and deployment in Orchestration Pipelines is based on YAML and gcloud CLI commands. All Airflow DAGs are autogenerated and interacting with Airflow, Managed Airflow environments, or environment buckets isn't required. You can develop and deploy your workflows in a regular Git repository where the asset files are located.

  3. Your YAML definitions are working on all versions of Airflow. You don't need to adjust your code for changes between Airflow versions or differences in installed packages. For example, it's not required to migrate your pipelines if you move from Airflow 2 to Airflow 3.

  4. Orchestration Pipelines are integrated with Google Cloud Data Agent Kit extension, so you can use agentic authoring and troubleshooting. You can use the agent to write your pipelines, ease deployment, and observe your pipelines status through the same extension.

How Orchestration Pipelines work

Orchestration Pipelines support a variety of actions and Google Cloud services, such as:

  • Running PySpark scripts in Managed Service for Apache Spark.
  • Running notebook files in Managed Service for Apache Spark.
  • Executing SQL queries in BigQuery or Managed Service for Apache Spark.
  • Executing data processing pipelines in Dataform or dbt framework.
  • Running Python scripts.

A typical workflow for Orchestration Pipelines is:

  1. You define your pipeline as a sequence of actions that must be executed using one of the Google Cloud services.
  2. You define resource configurations for your pipeline actions. For example, you might specify that a particular action must be executed on an ephemeral Managed Service for Apache Spark cluster that has specific configuration.
  3. (Optional) You define resources that must be provisioned automatically if they don't already exist through the provisioned resources mechanism. For example, you might specify that a static Managed Service for Apache Spark cluster in a specific configuration must be created.
  4. You add your pipeline definition file with actions you want executed to a Git repository.
  5. You add assets for individual pipeline actions (such as script or notebook files) to a Git repository.

  6. You deploy your pipelines to your Managed Airflow environment with a gcloud CLI command. Orchestration Pipelines automatically creates a few DAG files that run your pipeline.

    Compared to standalone Airflow DAGs, these DAGs are generated automatically and you don't need to manage them in any way. You can check the status of your pipeline execution and manage pipelines with gcloud CLI commands.

  7. Your environment executes the pipeline on a schedule.

What's next