Enable and disable the Managed Airflow service

Managed Airflow (Gen 3) | Managed Airflow (Gen 2) | Managed Airflow (Legacy Gen 1)

This page explains how to enable and disable the Managed Service for Apache Airflow service in your Google Cloud project.

For information about services management on Google Cloud, see Enabling and Disabling Services.

For a list of services required by Managed Airflow in VPC Service Controls configurations, see Configure VPC Service Controls.

Services required by Managed Service for Apache Airflow

Upcoming deprecation: We're phasing out the APIs that aren't required by Managed Airflow (Gen 3).

This section describes services that are required by Managed Airflow (Gen 2). In some cases, organization or project administrators can restrict which Google services can be used in their projects.

The following services are required by Managed Airflow (Gen 2):

• composer.googleapis.com
• compute.googleapis.com
• logging.googleapis.com
• monitoring.googleapis.com
• storage.googleapis.com
• cloudresourcemanager.googleapis.com
• servicedirectory.googleapis.com
• cloudkms.googleapis.com, if you are using Cloud KMS or CMEK keys

• secretmanager.googleapis.com, if you are using Secret Manager as a secret backend

• artifactregistry.googleapis.com

• cloudbuild.googleapis.com

• container.googleapis.com

• pubsub.googleapis.com

• (Only in Managed Airflow versions 2.0.*) deploymentmanager.googleapis.com

Google services like Cloud DNS or IAM are already allowed by default in most projects.

Enable the Managed Airflow service

This section describes how to enable the Managed Airflow service in your Google Cloud project.

Before you enable the service

• To enable the Managed Airflow service, you must have the correct Identity and Access Management permissions. To learn about the IAM requirements for Service Usage, see Service Usage page for Access Control.

• Make sure that billing is enabled in your project.

Enable Cloud Composer API

Console

Enable the Cloud Composer API:

Roles required to enable APIs

To enable APIs, you need the serviceusage.services.enable permission. If you created the project, then you likely already have this permission through the Owner role (roles/owner). Otherwise, you can get this permission through the Service Usage Admin role (roles/serviceusage.serviceUsageAdmin). Learn how to grant roles.

Enable the API

gcloud

Enable the composer.googleapis.com API:

Roles required to enable APIs

To enable APIs, you need the serviceusage.services.enable permission. If you created the project, then you likely already have this permission through the Owner role (roles/owner). Otherwise, you can get this permission through the Service Usage Admin role (roles/serviceusage.serviceUsageAdmin). Learn how to grant roles.

gcloud services enable composer.googleapis.com

API

Use the services.enable method to enable the composer.googleapis.com API.

Terraform

Use the google_project_service resource to configure the composer.googleapis.com API.

Important: If you disable the BigQuery API, your Managed Airflow environment stops working. This happens because Google Kubernetes Engine depends on BigQuery and Managed Service for Apache Airflow inherits this dependency from the Kubernetes Engine API.

Enable Managed Airflow (Gen 2) dependencies

Note: For information about why these APIs must be enabled separately, see Upcoming deprecation of services that aren't required by Managed Airflow (Gen 3).

To enable all dependencies required by Managed Airflow (Gen 2), you can enable the services/composer.googleapis.com/groups/dependencies@composer2 service group with the following Google Cloud CLI command:

gcloud

Run the gcloud beta services enable command:

gcloud beta services enable composer.googleapis.com \
    --group dependencies@composer2

Alternatively, you can manually enable the additional dependencies required by Managed Airflow (Gen 2), for example through Google Cloud console or Terraform:

• artifactregistry.googleapis.com
• cloudbuild.googleapis.com
• container.googleapis.com
• pubsub.googleapis.com
• sqladmin.googleapis.com

Enable services required by your DAGs

In addition to services required by Managed Airflow, enable all services that your DAGs use, for example:

• bigquery.googleapis.com
• dataflow.googleapis.com
• datacatalog.googleapis.com

You can find the *.googleapis.com names of the services in API pages linked from the API Explorer page.

Disable the Managed Airflow service

This section describes how to disable the Managed Airflow service in your Google Cloud project.

Before you disable the service

Warning: You cannot recover environments in your project after you disable the Cloud Composer API. After the API is disabled, you still have access to data stored in your environment's buckets, but the environments themselves are no longer usable.

• To disable the Managed Airflow service, you must have the correct Identity and Access Management permissions. To learn about the IAM requirements for Service Usage, see the Service Usage page for Access Control.

• Make sure that all Managed Airflow environments in your project are deleted. Disabling the API irreversibly deletes tenant project parts of Managed Airflow environments, and these environments become unusable.

• To avoid additional charges, complete the following steps for each environment before you disable the Cloud Composer API:

  1. Pause the DAGs.
  2. If required, export the environment's data, such as DAGs from the environment's bucket.
  3. Delete your environment, including the resources that are not deleted automatically.

Disable Cloud Composer API

To disable the Managed Airflow service:

Console

1. In the Google Cloud console, go to the Cloud Composer API page.

   Go to Cloud Composer API

2. Click Manage.

3. Click Disable API.

gcloud

Run the gcloud services disable command:

gcloud services disable composer.googleapis.com

API

Use the services.disable method to disable the composer.googleapis.com API.

Terraform

Remove the configuration for the composer.googleapis.com API. You usually use the google_project_service resource to configure Google Cloud services.

Deprecation of services that aren't required by Managed Airflow (Gen 3)

We're phasing out the APIs that aren't required by Managed Airflow (Gen 3).

The following API dependencies aren't detached yet and are scheduled to be detached from the Cloud Composer API in the future:

• sqladmin.googleapis.com

The following APIs became fully detachable. Deactivating these APIs won't cause the deactivation of the Cloud Composer API:

Caution: Deactivating these APIs will cause Managed Airflow (Gen 2) and Managed Airflow (Legacy Gen 1) environments to stop working and become non-recoverable.

• artifactregistry.googleapis.com
• cloudbuild.googleapis.com
• container.googleapis.com
• pubsub.googleapis.com

All APIs listed as detached are no longer enabled automatically when you enable the Cloud Composer API. To create Managed Airflow (Gen 2) environments in new projects, enable the group of detached APIs in addition to the Cloud Composer API.

Existing Managed Airflow (Gen 3) and Managed Airflow (Gen 2) environments in projects where the Cloud Composer API is already enabled aren't affected by this change. You can do the following:

• If your project has only Managed Airflow (Gen 3) environments, then you can manually disable the detached APIs.

• If your project has Managed Airflow (Gen 2) environments, then we recommend keeping these APIs enabled because disabling them might lead to environment malfunction.

• If you use automation scripts to provision Managed Airflow (Gen 2) environments, then make sure that the listed APIs are enabled in addition to the Cloud Composer API. You can enable them manually or with a gcloud CLI command.

What's next

• Create environments
• Access control