<br />

<br />


**Managed Airflow (Gen 3)** \| [Managed Airflow (Gen 2)](https://docs.cloud.google.com/composer/docs/composer-2/run-orchestration-pipelines "View this page for Managed Airflow (Gen 2)") \| Managed Airflow (Legacy Gen 1)

<br />

<br />

<br />

<br />

<br />

<br />

This page describes the relationship between Orchestration Pipelines
and Managed Airflow.

## About Orchestration Pipelines

[Orchestration Pipelines](https://docs.cloud.google.com/orchestration-pipelines) is a unified, declarative orchestration and
automated deployment framework optimized for seamlessly managing data and AI
pipelines on Google Cloud.

With Orchestration Pipelines, you can define your pipelines
and their deployment configurations using a declarative YAML-based Domain
Specific Language (DSL). This framework abstracts the underlying
infrastructure, allowing you to focus on the logic of your data and AI
workflows while Orchestration Pipelines handles the deployment,
versioning, and orchestration.

## About Managed Airflow environments in Orchestration Pipelines

Managed Airflow is the orchestration engine that runs your pipelines
after they're deployed. You assign a Managed Airflow environment as a
part of a pipeline definition. This environment is called a *runner environment*
in Orchestration Pipelines.

Benefits that Orchestration Pipelines:

1. You can empower teams in your organizations that aren't Airflow experts to
   create and run their workflows without writing DAGs or configuring Airflow.
   For example, you can run a Notebook in an ephemeral Managed Service for Apache Spark
   cluster and specify all schedule, resources, and configuration parameters
   for its execution in YAML without writing any DAG code.

2. All configuration and deployment in Orchestration Pipelines is
   based on YAML and gcloud CLI commands. All Airflow DAGs are
   autogenerated and interacting with Airflow, Managed Airflow
   environments, or environment buckets isn't required. You can develop and
   deploy your workflows in a regular Git repository where the asset files are
   located.

3. Your YAML definitions are working on all versions of Airflow. You don't need
   to adjust your code for changes between Airflow versions or differences in
   installed packages. For example, it's not required to migrate your
   pipelines if you move from Airflow 2 to Airflow 3.

4. Orchestration Pipelines are integrated with
   [Google Cloud Data Agent Kit](https://docs.cloud.google.com/data-agent-kit), so you can
   use agentic authoring and troubleshooting. You can use the agent to
   [write your pipelines](https://docs.cloud.google.com/data-agent-kit/build-pipelines), ease
   deployment, and observe your pipelines status using your preferred IDE or
   CLI.

## How Orchestration Pipelines work

Orchestration Pipelines support a variety of actions and
Google Cloud services, such as:

- Running PySpark scripts in Managed Service for Apache Spark.
- Running notebook files in Managed Service for Apache Spark.
- Executing SQL queries in BigQuery or Managed Service for Apache Spark.
- Executing data processing pipelines in Dataform or dbt framework.
- Running Python scripts.

A typical workflow for Orchestration Pipelines is:

1. You define your pipeline as a [sequence of actions](https://docs.cloud.google.com/orchestration-pipelines/reference/orchestration-pipelines-dsl-reference#actions) that must be executed using one of the Google Cloud services.
2. You define resource configurations for your pipeline actions. For example, you might specify that a particular action must be executed on an ephemeral Managed Service for Apache Spark cluster that has specific configuration.
3. (Optional) You define [resources](https://docs.cloud.google.com/orchestration-pipelines/reference/resource-types) that must be provisioned automatically if they don't already exist through the [provisioned resources](https://docs.cloud.google.com/orchestration-pipelines/provision-resources) mechanism. For example, you might specify that a static Managed Service for Apache Spark cluster in a specific configuration must be created.
4. You add your pipeline definition file with actions you want executed to a Git repository.
5. You add assets for individual pipeline actions (such as script or notebook files) to a Git repository.
6. You deploy your pipelines to your Managed Airflow environment with
   a gcloud CLI command. Orchestration Pipelines
   automatically creates a few DAG files that run your pipeline.

   Compared to standalone Airflow DAGs, these DAGs are generated automatically
   and you don't need to manage them in any way. You can check the status of
   your pipeline execution and manage pipelines with gcloud CLI
   commands.
7. Your environment executes the pipeline on a schedule.

## Monitor pipeline status

You can view the status of all pipelines that run in your environment and
pipeline run history in Google Cloud console and Google Cloud CLI. For more
information, see
[Manage orchestration pipelines](https://docs.cloud.google.com/orchestration-pipelines/manage-orchestration-pipelines).

## What's next

- [Build data engineering pipelines with Google Cloud Data Agent Kit extension for VS Code](https://docs.cloud.google.com/data-cloud-extension/vs-code/build-pipelines)
- [Deploy orchestration pipelines](https://docs.cloud.google.com/orchestration-pipelines/deploy-orchestration-pipelines)
- [Provision resources in Orchestration Pipelines](https://docs.cloud.google.com/orchestration-pipelines/provision-resources)