To more strongly embrace the success and growing customer preference for OSS solutions, Cloud Composer is evolving to become Managed Service for Apache Airflow. This name change provides improved customer understanding of our portfolio while reinforcing our commitment to being the most open cloud ecosystem.

Configure environments with Restrict Endpoint Usage policy

Managed Airflow (Gen 3) | Managed Airflow (Gen 2) | Managed Airflow (Legacy Gen 1)

This page explains how to create Managed Airflow environments when the Restrict Endpoint Usage organization policy is enforced.

The constraints/gcp.restrictEndpointUsage organization policy restricts which Google API endpoints can be used to access resources. This allows organizations to enforce the use of locational or regional endpoints, for example, to satisfy requirements for data to remain within certain geographical areas.

About Restrict Endpoint Usage policy in Managed Airflow

When you create a Managed Airflow environment in a project where the Restrict Endpoint Usage policy denies access to global endpoints for services like Cloud Storage API, Cloud Logging API, Pub/Sub API, or Data Lineage API, Managed Airflow configures the internal components of the environment to use the corresponding Regional Endpoints (REPs) for these services.

The specific endpoints an environment uses are determined by the Restrict Endpoint Usage policy only during environment creation. These endpoints don't change even if the policy is reconfigured later.

Restricting services required by Managed Airflow environments

When the Restrict Endpoint Usage policy restricts some of the services required by Managed Airflow, the following limitations apply:

Airflow task logs in the Airflow UI

Airflow UI relies on the global Cloud Logging endpoint (logging.googleapis.com) to display task logs. If you deny logging.googleapis.com in your Restrict Endpoint Usage policy, then task logs won't be available in the Airflow UI.

This limitation applies only to the Airflow UI. You will still be able to view task logs in DAG UI and Airflow worker logs in Google Cloud console and Cloud Logging.
Environment metrics

Managed Airflow uses global endpoints for Cloud Monitoring (monitoring.googleapis.com) because some Monitoring APIs don't fully support regional endpoints. Restricting monitoring.googleapis.com might impact some environment metrics.
Airflow operators in your DAGs

Airflow operators that connect to a global endpoint denied by the Restrict Endpoint Usage policy will fail. Make sure that any Airflow operators in your DAGs are configured to use regional endpoints, if the operator supports it.

Before you begin

Environments with Restrict Endpoint Usage policy are supported in Managed Airflow (Gen 2) versions 2.16.7 and later.

Configure Restrict Endpoint Usage policy

For more information and examples of how to configure the Restrict Endpoint Usage policy to deny access to specific global endpoints, see Restricting Endpoint Usage.

Create a Public IP environment with Restrict Endpoint Usage policy

Public IP Managed Airflow environments don't require any additional setup. Do the following:

Check that the services you want to regionalize have regional endpoints available in the selected regions.
Make sure that the Restrict Endpoint Usage policy is already configured and denies global endpoints for the services you intend to regionalize.
Create a new Managed Airflow environment.

The environment will use regional endpoints according to the Restrict Endpoint Usage policy.

Create a Private IP environment with Restrict Endpoint Usage policy

To create an environment with Restrict Endpoint Usage policy in the Private IP configuration, you must configure your VPC network to resolve and route traffic to the required Regional Endpoints (REPs) using Private Service Connect and Cloud DNS.

To create a Private IP environment with Restrict Endpoint Usage policy:

Check that the services you want to regionalize have regional endpoints available in the selected regions.
Make sure that the Restrict Endpoint Usage policy is already configured and denies global endpoints for the services you intend to regionalize.
Make sure that you already have a VPC network and subnetwork that you want to use for your Private IP environment.
Repeat the networking configuration steps for each of the following services that are restricted by the Restrict Endpoint Usage policy:
- storage.googleapis.com
- logging.googleapis.com
- pubsub.googleapis.com
- datalineage.googleapis.com
- Google API endpoints for all other services used in your DAGs.

Configure Private IP networking for a service restricted with the policy

Apply the following steps to the corresponding Regional Endpoints (REPs) for any Google services in your project where access to their Global Endpoints is restricted.

You must create Private Service Connect endpoints to target regional variations of these services, specifying the region for your Managed Airflow environment. For example, if your environment is located in us-central1, you target the storage.us-central1.rep.googleapis.com REP for Cloud Storage.

For more information on how to use Private Service Connect endpoints to connect to regional endpoints of supported Google APIs, see Access regional Google APIs through endpoints.

Create a Private Service Connect endpoint

In the Google Cloud console, go to the Private Service Connect page.

Go to Private Service Connect
Click Connect endpoint.
Configure a Private Service Connect endpoint as follows:
- Target: Select Google APIs.
- Target Details:
  1. Scope: Select Regional.
  2. Region: Select the region of your Managed Airflow environment.
  3. Target service: Select the service you're configuring. This selects a REP in the selected region.
  Example:
  
  If your environment is located in us-central1 and you're configuring Cloud Storage, then select Iowa (us-central1), and Cloud Storage (storage.us-central1.rep.googleapis.com).
- Endpoint Details:
  1. Endpoint name: A name for the endpoint.
    
    Example: psc-storage-endpoint-us-central1.
  2. Network and Subnetwork: Select the VPC network and subnetwork of your Managed Airflow environment.
  3. IP address: Choose an existing IP address, or click Create IP address to reserve a new static internal IP address from the selected subnetwork.
    
    Give this IP address a descriptive name. You will use this IP address later to configure a DNS record.
    
    Example: psc-storage-us-central1.
Click Add endpoint.

Create a private DNS zone

In the Google Cloud console, go to the Cloud DNS page.

Go to Cloud DNS
Click Create zone.
In Zone type, select Private.
Specify the following configuration:
- Zone name: A name for the zone.
  
  Example: storage-us-central1
- DNS name: A DNS name. Include the trailing dot.
  
  Example: storage.us-central1.rep.googleapis.com.
- Networks: Select the VPC network for your Managed Airflow environment.
Click Create.

Add a DNS record in the created DNS zone

In the Google Cloud console, go to the Cloud DNS page.

Go to Cloud DNS
Click the name of the zone you've created.
Click Add standard.
Configure the DNS record as follows:
- DNS name: Specify the subdomain of the DNS zone.
  
  Example: storage.us-central1.rep.googleapis.com
- Resource record type: Select A.
- IPv4 Address: Select the IP address that you assigned to the Private Service Connect endpoint.
Click Create.

Create a Private IP Managed Airflow environment

After you've configured all services that will be used by your environment, proceed to create a Private IP environment in the selected VPC network and subnetwork.