[Managed Airflow (Gen 3)](https://docs.cloud.google.com/composer/docs/composer-3/cloud-storage "View this page for Managed Airflow (Gen 3)") \| **Managed Airflow (Gen 2)** \| [Managed Airflow (Legacy Gen 1)](https://docs.cloud.google.com/composer/docs/composer-1/cloud-storage "View this page for Managed Airflow (Legacy Gen 1)")





This page describes what data Managed Service for Apache Airflow stores for your environment in Cloud Storage. When you create an environment, Managed Airflow creates a [Cloud Storage](https://docs.cloud.google.com/storage) bucket and associates the bucket with your environment. The name of the bucket is based on the environment region, name, and a random ID such as `us-central1-b1-6efabcde-bucket`. Managed Airflow synchronizes specific folders in your environment's bucket to Airflow components that run in your environment. For example, when you update a file with the code of your Airflow DAG in the environment's bucket, Airflow components also receive the updated version. Managed Airflow uses [Cloud Storage FUSE](https://docs.cloud.google.com/storage/docs/gcs-fuse) for synchronization. > [!NOTE] > **Note:** Deleting your environment does not delete the environment's bucket. To avoid incurring charges to your Cloud Billing account, download your data and then [delete the Cloud Storage bucket](https://docs.cloud.google.com/storage/docs/deleting-buckets). ## Folders in the Cloud Storage bucket | Folder | Storage path | Mapped directory | Description | |---|---|---|---| | DAG | `gs://bucket-name/dags` | `/home/airflow/gcs/dags` | Stores the DAGs for your environment. | | Plugins | `gs://bucket-name/plugins` | `/home/airflow/gcs/plugins` | Stores your custom plugins, such as custom in-house Airflow operators, hooks, sensors, or interfaces. | | Data | `gs://bucket-name/data` | `/home/airflow/gcs/data` | Stores the data that tasks produce and use. | | Logs | `gs://bucket-name/logs` | `/home/airflow/gcs/logs` | Stores the Airflow logs for tasks. Logs are also available in the [Airflow UI](https://docs.cloud.google.com/composer/docs/composer-2/access-airflow-web-interface) and in **Logs** tab in Google Cloud console. | Managed Airflow synchronizes folders uni-directionally: from the bucket to Airflow components. Unidirectional synchronizing means that local changes in these folders on an Airflow component are overwritten. > [!IMPORTANT] > **Important:** The content of the `data/` folder is synchronized to Airflow schedulers and workers. This folder **is not** synchronized to Airflow triggerers or the web server. Apply suggestions from the related [known limitations](https://docs.cloud.google.com/composer/docs/composer-2/known-issues#sharing-files-with-webserver) to synchronize files to the Airflow web server. Data synchronization is eventually consistent. To send messages from one operator to another, use [XComs](https://airflow.apache.org/docs/apache-airflow/stable/concepts/xcoms.html). ## Capacity considerations Data from `dags/`, `plugins/` and `data/` folders are synchronized to Airflow scheduler(s) and workers. The contents of the `plugins/` folder is synchronized to the Airflow web server. > [!CAUTION] > **Caution:** Don't put more than 100 MB of data into `/dags` or `/plugins` folder. Avoid storing additional files generated by Airflow components in these folders. The more data is put into these folders, the more space is occupied in the local storage of Airflow components. Saving too much data in `dags/` and `plugins/` can disrupt your operations and lead to issues such as: - A worker or a scheduler runs out of local storage and is evicted because of insufficient space on the local disk of the component. - Synchronization of files from `dags/` and `plugins/` folders to workers and schedulers takes a long time. - Synchronizing files from `dags/` and `plugins/` folders to workers and schedulers becomes impossible. For example, you store a 2 GB file in the `dags/` folder, but the local disk of an Airflow worker can only accommodate 1 GB. During the synchronization, the worker runs out of local storage and synchronization can't be completed. ## DAGs and plugins folders To avoid DAG run failures, store your DAGs, plugins, and Python modules in the `dags/` or `plugins/` folders, even if your Python modules don't contain DAGs or plugins. For example, you use a `DataFlowPythonOperator` that references a `py_file` Dataflow pipeline. That `py_file` doesn't contain DAGs or plugins, but you must still store it in the `dags/` or `plugins/` folder. ## Data folder > [!IMPORTANT] > **Important:** In general, the contents of the `data/` folder are not synchronized to Airflow components by default. There are scenarios when certain files from the `data/` folder are synchronized to a specific Airflow component. For example, when Managed Service for Apache Airflow attempts to read a given file for the first time during: - DAG parsing: When a file is read for the first time during DAG parsing, Managed Service for Apache Airflow synchronizes it to the scheduler that parses the DAG. - DAG execution: When a file is read for the first time during DAG execution, Managed Service for Apache Airflow synchronizes it to the worker running the execution. Airflow components have limited local storage, so consider deleting downloaded files to free disk space in your components. Notice that local storage usage can also temporarily go up if you have concurrent tasks that download the same file to a single Airflow worker. > [!NOTE] > **Note:** Do not put Python modules required during the DAG parsing into the `data/` folder. It can have a negative impact on your DAG parsing and task scheduling latency. ## Logs folder The `logs/` folder is synchronized from Airflow workers to the environment's bucket using the Cloud Storage API. Cloud Storage API quota is calculated by the amount of data moved, so the number of Airflow tasks your system runs can increase your Cloud Storage API usage: the more tasks you run, the bigger your log files. > [!NOTE] > **Note:** In rare cases you might be increasing log levels from `INFO` to `DEBUG`, as described in [Troubleshooting DAGs](https://docs.cloud.google.com/composer/docs/composer-2/troubleshooting-dags). Note that such operations produce additional logs and could increase your syncing and storing costs. ## Synchronization with the web server The `plugins/` folder is automatically synchronized to the web server so that plugins can be loaded by Airflow UI. > [!IMPORTANT] > **Important:** The contents of the `data/` folder aren't synchronized to the web server. ## What's next - [View logs](https://docs.cloud.google.com/composer/docs/composer-2/view-logs) - [Manage DAGs](https://docs.cloud.google.com/composer/docs/composer-2/manage-dags) - [Write DAGs](https://docs.cloud.google.com/composer/docs/composer-2/write-dags)