These release notes apply to the Cloud Composer service. Announcements and information about new or updated features, bug fixes and deprecated functionality is published on this page.
You can also:
- Follow Cloud Composer release notes in the Apache Airflow Community workspace on Slack.
- See Cloud Composer version list for more information about Cloud Composer images and support dates.
- Check the Cloud Composer known issues page to learn more about known issues and possible workarounds.
You can see the latest product updates for all of Google Cloud on the Google Cloud page, browse and filter all release notes in the Google Cloud console, or programmatically access release notes in BigQuery.
To get the latest product updates delivered to you, add the URL of this page to your feed reader, or add the feed URL directly.
November 05, 2025
New Airflow builds are available in Cloud Composer 3:
Fixed the CVE-2025-49844 vulnerability. The updated version of the environment component that hosts a Redis-based task queue is available:
- To apply the fix in Cloud Composer 3, upgrade the Airflow build of your environment to airflow-2.10.5-build.19 or airflow-2.9.3-build.39, or later.
- To apply the fix in Cloud Composer 2, upgrade your environment to Cloud Composer version 2.15.3, or later.
New images are available in Cloud Composer 2:
A new Cloud Composer release has started on November 05, 2025. Get ready for upcoming changes and features as we roll out the new release to all regions. This release is in progress at the moment. Listed changes and features might not be available in some regions yet.
We strongly recommend to use highly resilient environments for production use cases. Highly resilient environments provide high availability and use built-in redundancy and failover mechanisms to reduce the environment's susceptibility to zonal failures and single point of failure outages.
October 30, 2025
The following Cloud Composer versions and builds have reached their end of support period: composer-3-airflow-2.7.3-build.21 and composer-2.9.9-*.
New images are available in Cloud Composer 2:
(Cloud Composer 3) Airflow workers are restarted when they exceed their storage limit.
Fixed an error where Airflow components failed to start up when multiple
objects with the same name were present in the /dags or /plugins folder in
the environment's bucket.
October 27, 2025
New images are available in Cloud Composer 2:
October 22, 2025
The default value of the
NO_PROXY
environment variable now includes the IP address of the environment's cluster
Compute Engine metadata server.
(Airflow 2.10.5) The apache-airflow-providers-cncf-kubernetes package was
upgraded to version 10.8.2.
For changes in other packages, see the
preinstalled packages changelog.
New Airflow builds are available in Cloud Composer 3:
The following Cloud Composer versions and builds have reached their end of support period: composer-3-airflow-2.7.3-build.20, composer-3-airflow-2.7.3-build.19, composer-2.9.8-*, composer-2.9.7-*.
New images are available in Cloud Composer 2:
(Airflow 2.10.5) The apache-airflow-providers-google package was upgraded to version 18.0.0 in Cloud Composer 2 images and Cloud Composer 3 builds.
For more information about changes, see the apache-airflow-providers-google changelog from version 17.2.0 to version 18.0.0.
October 10, 2025
The following Cloud Composer versions and builds have reached their end of support period: composer-2.9.6-*, composer-3-airflow-2.7.3-build.17, composer-3-airflow-2.7.3-build.18, composer-3-airflow-2.9.1-build.8, and composer-3-airflow-2.9.1-build.9.
New images are available in Cloud Composer 2:
New Airflow builds are available in Cloud Composer 3:
(Cloud Composer 2 versions from 2.12.0 to 2.12.3) Fixed a problem where Airflow components in the environment's cluster were running out of ephemeral storage. This change rolls out gradually over several releases to all regions supported by Cloud Composer 2.
October 09, 2025
Upgrade checks are now generally available (GA) in Cloud Composer 3 and Cloud Composer 2.
September 30, 2025
DAG UI now correctly generates error messages about malformed serialized DAG.
The following Cloud Composer versions and builds have reached their end of support period: composer-2.9.4-*, composer-3-airflow-2.9.1 builds from build.0 to build.7, and composer-3-airflow-2.7.3-build.16.
New images are available in Cloud Composer 2:
(Airflow 2.10.5) CloudComposerDAGRunSensor is broken in the
apache-airflow-providers-google package version 17.2.0. This package is used
by Cloud Composer versions and builds with Airflow 2.10.5 available in this
release. If your DAGs use this sensor, we recommend you postpone upgrading
until the issue is resolved.
New Airflow builds are available in Cloud Composer 3:
The GCE_METADATA_TIMEOUT environment variable is changed to reserved. This
change addresses an issue where setting a low timeout value disrupted the
environment's operations that relied on the metadata server.
(Airflow 2.10.5) The apache-airflow-providers-cncf-kubernetes package was
upgraded to version 10.8.0
from version 10.7.0. For changes in other packages, see the
preinstalled packages changelog.
(Airflow 2.10.5) The apache-airflow-providers-google package was upgraded
to version 17.2.0 in Cloud Composer 2 images and Cloud Composer 3 builds.
For more information about changes, see the apache-airflow-providers-google changelog from version 17.1.0 to version 17.2.0.
September 11, 2025
Fixed Airflow logs not exporting to Cloud Logging because of a GKE version mismatch between Airflow worker and GKE Control plane nodes.
New Airflow builds are available in Cloud Composer 3:
New images are available in Cloud Composer 2:
Fixed an issue where values of Airflow configuration options were evaluated before being set. As a result, the actual value was set to the evaluated result.
September 03, 2025
You can now check if a Cloud Composer 2 environment's configuration is compatible with Cloud Composer 3. We recommend doing this check before migrating to Cloud Composer 3.
The following Cloud Composer versions and builds have reached their end of support period: composer-2.9.2-*, composer-2.9.3-*, and composer-3-airflow-2.7.3-build.15.
(Available without upgrading) Cloud Composer 3 now supports DNS resolution for regional service endpoints. You can now reach regional service endpoints from DAGs in your environment. This change is available in Public IP environments without additional configuration. For Private IP environments, an environment must be connected to a VPC network where private endpoints are configured.
New Airflow builds are available in Cloud Composer 3:
(Cloud Composer 2) Cloud Composer's high availability infrastructure was enhanced to provide greater resilience against zonal outages. This change rolls out gradually over several releases to all regions supported by Cloud Composer 2.
August 21, 2025
(Airflow 2.10.5) The apache-airflow-providers-cncf-kubernetes package was upgraded to version 10.7.0 from version 10.6.1. For changes in other packages, see the preinstalled packages changelog.
Airflow UI now shows the correct value of the [core]dags_folder Airflow
configuration option.
The following Cloud Composer versions and builds have reached their end of support period: composer-2.9.0, composer-2.9.1, and composer-3-airflow-2.7.3 builds from build.5 to build.14.
Improved the startup times of Airflow workers for environments that have a large number of custom PyPI packages installed.
This feature was announced previously and has finished gradually rolling out to all regions supported by Cloud Composer.
(Airflow 2.10.5) The apache-airflow-providers-google package was upgraded to version 17.1.0 in Cloud Composer 2 images and Cloud Composer 3 builds.
For more information about changes, see the apache-airflow-providers-google changelog from version 15.1.0 to version 17.1.0.
New Airflow builds are available in Cloud Composer 3:
New images are available in Cloud Composer 2:
August 05, 2025
New Airflow builds are available in Cloud Composer 3:
Cloud Composer versions 2.8.7 and 2.8.8 have reached their end of support period.
New images are available in Cloud Composer 2:
(Cloud Composer 2) Moved the update_fab_perms option from [webserver] to
[fab] in the Airflow configuration. The corresponding deprecation warning is
no longer generated in Airflow web server logs.
(Airflow 2.10.5 only) Added task-level resource consumption Airflow metrics to Cloud Composer.
composer.googleapis.com/workflow/task/cpu_usage: percentage of CPU used by a task.composer.googleapis.com/workflow/task/cpu_usage: percentage of memory used by a task.
July 28, 2025
Fixed an issue that caused unexpected restarts of Airflow component workloads in the environment's cluster.
(Cloud Composer 3) The DAGS_FOLDER reserved environment variable now
correctly points to the local directory where DAG files are stored.
New Airflow builds are available in Cloud Composer 3:
New images are available in Cloud Composer 2:
(Cloud Composer 3) The DAGS_FOLDER reserved environment variable now
correctly points to the local directory where DAG files are stored.
July 24, 2025
Web server restarting is is now generally available (GA) in Cloud Composer 2 and Cloud Composer 3.
July 23, 2025
If your environment uses dag-factory package version 0.22, then you might
experience DAG failures in Cloud Composer versions that have
apache-airflow-providers-cncf-kubernetes package version 10.4.2 or later. At
the same time, upgrading the dag-factory package to version 0.23 might require
you to update your DAG code to make it compatible.
If your environment uses dag-factory version 0.22, we recommend to do the
following:
- Temporarily postpone upgrading your environment until you're ready to switch
to
dag-factoryversion 0.23. Last versions of Cloud Composer that support version 0.22 are composer-3-airflow-2.10.5-build.3, composer-3-airflow-2.9.3-build.23, composer-2.13.1-airflow-2.10.5, and composer-2.13.1-airflow-2.9.3 released on May 14, 2025. - When you are ready to upgrade, update your DAGs for compatibility with 0.23.
We recommend to do this in a development environment first.
Install
dag-factoryversion 0.23, then check that your DAGs are parsed and are working correctly, and update them if needed. After your DAGs are compatible, installdag-factoryversion 0.23 in your production environment and transfer the updated DAGs. Your environment can now be upgraded to a later version of Cloud Composer or Airflow. - If your environment is already upgraded to a later version of Cloud Composer
and you experience problems, then update
dag-factoryto version 0.23 and update your DAGs for compatibility with 0.23.
July 17, 2025
Highly resilient environments are now generally available (GA) in Cloud Composer 3.
July 14, 2025
We're changing the way we provide support dates for Airflow builds in Cloud Composer 3. Before this change, some Airflow builds had their end of support date listed as "To be announced" until a later Airflow version became available. We're deprecating this approach for all builds that are released after July 01, 2025.
We are now providing support dates that depend on a date when a particular Airflow build was released:
- We are introducing the standard support period of 12 months after the release date.
- All Airflow builds that didn't have an end of support date when this change was introduced are supported until July 1, 2026. Because of this change, some Airflow builds released before July 1, 2025 are supported for longer than the standard support period.
- All Airflow builds that had their support date shorter than 12 months are now supported for 12 months since the date of their initial release.
- All builds released after July 1, 2025 will use the standard support period.
July 09, 2025
(Available without upgrading) During Cloud Composer 2 environment operations, a more informative error message is returned when an environment's web server has connectivity issues.
New images are available in Cloud Composer 2:
Cloud Composer versions 2.8.4 and 2.8.5 have reached their end of support period.
New Airflow builds are available in Cloud Composer 3:
July 01, 2025
New images are available in Cloud Composer 2:
New Airflow builds are available in Cloud Composer 3:
June 23, 2025
Improved the startup times of Airflow workers for environments that have a large number of custom PyPI packages installed.
This feature was announced previously and is gradually rolling out over several releases. In this release, it's available in asia-east2, asia-northeast3, europe-central2, europe-west9, me-central1, me-west1, northamerica-northeast1, northamerica-northeast2, northamerica-south1, and us-west1 re
(Available without upgrading) Fixed an issue where deleting a Cloud Composer 2 environment could fail when the environment's cluster was in the process of creating a node pool.
(Airflow 2.10.5) The apache-airflow-providers-cncf-kubernetes package was upgraded to version 10.5.0 from version 10.4.2. For changes in other packages, see the preinstalled packages changelog.
New images are available in Cloud Composer 2:
Cloud Composer version 2.8.3 has reached its end of support period.
New Airflow builds are available in Cloud Composer 3:
June 16, 2025
We're planning to phase out the APIs that aren't required by Cloud Composer 3.
Starting February 27, 2026, the following APIs will become fully detachable. Deactivating these APIs won't cause the deactivation of the Cloud Composer API:
- artifactregistry.googleapis.com
- cloudbuild.googleapis.com
- container.googleapis.com
- pubsub.googleapis.com
- sqladmin.googleapis.com
Starting May 27, 2026, these APIs will no longer be enabled automatically when you enable the Cloud Composer API. To create Cloud Composer 2 environments in new projects, the group of detached APIs must be enabled manually.
Existing Cloud Composer 3 and Cloud Composer 2 environments in projects where the Cloud Composer API is already enabled will not be impacted. You can do the following:
- After February 27, 2026, if your project has only Cloud Composer 3 environments, then you can manually disable the detached APIs.
- After February 27, 2026, if your project has Cloud Composer 2 environments, then we recommend keeping these APIs enabled because disabling them might lead to environment's malfunction.
- After May 27, 2026, if you use automation scripts to provision Cloud Composer 2 environments, then make sure that the listed APIs are enabled in addition to the Cloud Composer API.
June 10, 2025
Fixed an issue that caused Airflow worker and scheduler Pods to be evicted when a large number of tasks was executed.
Cloud Composer version 2.8.2 has reached its end of support period.
June 02, 2025
(Cloud Composer 3) All newly created private DNS zones are immediately visible to a Cloud Composer environment. Previously, re-attaching a VPC network was required.
(Cloud Composer 3) It's now possible to use zones with cross-project binding. Before this change, cross-project bound zones weren't supported in Cloud Composer 3.
Improved the startup times of Airflow workers for environments that have a large number of custom PyPI packages installed.
This change rolls out gradually. In this release, it's available in asia-east1, asia-northeast2, asia-south1, europe-north1, europe-west3, us-east1, us-south1, and us-west2 regions.
(Cloud Composer 2) In Cloud Composer versions 2.11.5 and later, log processing is switching to using OpenTelemetry instead of Fluentd.
This change was announced previously and is gradually rolling out over several releases. In this release, it's available in the following regions: asia-east1, asia-east2, asia-northeast2, asia-northeast3, asia-south2, asia-southeast2, australia-southeast1, australia-southeast2, europe-central2, europe-north1, europe-north2, europe-southwest1, europe-west, europe-west10, europe-west12, europe-west6, europe-west8, europe-west9, me-central1, me-central2, me-west1, northamerica-northeast2, northamerica-south1, southamerica-east1, southamerica-west1, us-east5, us-south1, us-west, and us-west3.
Cloud Composer version 2.8.1 has reached its end of support period.
New Airflow builds are available in Cloud Composer 3:
May 29, 2025
The Composer Local Development CLI tool is now available in Cloud Composer 3. This tool helps to streamline testing and developing by providing local Airflow environments based on Airflow builds used by Cloud Composer 3.
May 26, 2025
(Cloud Composer 3) New metrics that show the number of active Airflow components are now available for Cloud Composer 3 environments:
- The number of active schedulers
- The number of active DAG processors
- The number of active triggerers
- The number of active web servers
(Cloud Composer 3) It's now possible to override the default scopes of access tokens in all regions supported by Cloud Composer 3. This fix was announced previously and has finished gradually rolling out.
(Airflow 2.10.5 and 2.9.3) The apache-airflow-providers-cncf-kubernetes package was upgraded to version 10.4.2 from version 10.1.0.
New Airflow builds are available in Cloud Composer 3:
May 14, 2025
It is now possible to migrate from Cloud Composer 1 to Cloud Composer 3 using snapshots in all regions supported by Cloud Composer 3. The gradual rollout of this feature is finished.
It is now possible to restart the Airflow web server through Cloud Composer API by using the RestartWebServer method.
(Cloud Composer 3) Fixed an issue where the Airflow scheduler failed to initialize and work properly if a custom email backend was enabled. The Airflow scheduler now supports the following email backends:
- airflow.providers.sendgrid.utils.emailer.send_email
- airflow.utils.email.send_email_smtp
This issue affected only the Airflow scheduler. Airflow workers already support these and other backends. If you use an email backend that isn't supported by the scheduler, you can still send email from Airflow workers.
(Cloud Composer 2) In Cloud Composer versions 2.11.5 and later, log processing is switching to using OpenTelemetry instead of Fluentd.
This change is gradually rolled out to the asia-south1 and africa-south1 regions. It will be rolled out to more regions in future releases. New and upgraded Cloud Composer 2 environments in the listed regions will get this change.
(Cloud Composer 3) Improved the computation of metrics related to snapshots. This change reduces the number of calls made by Cloud Composer to the Cloud Storage API.
The default value of the [scheduler]max_tis_per_query Airflow configuration option is set to 128. This change fixes the issue with DAGs remaining in the scheduled state in some cases.
(Airflow 2.9.3) The apache-airflow-providers-google package was upgraded to version 15.1.0 in Cloud Composer 2 images and Cloud Composer 3 builds.
For more information about changes, see the apache-airflow-providers-google changelog from version 14.0.0 to version 15.1.0.
(Airflow 2.9.3) Changes in preinstalled packages:
apache-airflow-providers-standardwas upgraded to 1.0.0 from 0.4.0.aiosqlitewas removed from preinstalled packages.json-merge-patchwas removed from preinstalled packages.time-machinewas removed from preinstalled packages.pyjwtwas downgraded to 2.9.0 from 2.10.1.
The following Cloud Composer versions have an extended upgrade timeline:
- composer-3-airflow-2.10.5-build.3
- composer-3-airflow-2.9.3-build.23
- composer-2.13.1-airflow-2.10.5
- composer-2.13.1-airflow-2.9.3
New Airflow builds are available in Cloud Composer 3:
- composer-3-airflow-2.10.5-build.3 (default)
- composer-3-airflow-2.9.3-build.23
New images are available in Cloud Composer 2:
- composer-2.13.1-airflow-2.10.5 (default)
- composer-2.13.1-airflow-2.9.3
Cloud Composer version 2.8.0 has reached its end of support period.
May 07, 2025
(Airflow 2.10.5) The apache-airflow-providers-google package was upgraded to version 15.1.0 in Cloud Composer 2 images and Cloud Composer 3 builds.
For more information about changes, see the apache-airflow-providers-google changelog from version 14.0.0 to version 15.1.0.
For newly created environments, database retention policy is now enabled by default in Google Cloud CLI, API, and Terraform. Before this change, it was enabled by default only in Google Cloud Console.
For newly created Cloud Composer 3 environments, the minimum amount of memory is changed to 2 GB.
Improved the environment liveness monitoring. This change addresses some cases of transient failures that caused "Liveness probe failed" warnings in the environment's logs.
(Airflow 2.10.5) Changes in preinstalled packages:
apache-airflow-providers-standardwas upgraded to 1.0.0 from 0.4.0.aiosqlitewas removed from preinstalled packages.json-merge-patchwas removed from preinstalled packages.time-machinewas removed from preinstalled packages.
The default version of Airflow is changed to 2.10.5.
Airflow 2.10.2 is no longer included in Cloud Composer images and builds.
Cloud Composer versions 2.7.0 and 2.7.1 have reached their end of support period.
New Airflow builds are available in Cloud Composer 3:
- composer-3-airflow-2.10.5-build.2 (default)
- composer-3-airflow-2.9.3-build.22
New images are available in Cloud Composer 2:
- composer-2.13.0-airflow-2.10.5 (default)
- composer-2.13.0-airflow-2.9.3
May 06, 2025
The Deployment Manager API is no longer automatically enabled when you enable Cloud Composer API because this API isn't used by the Cloud Composer service.
Environments with Cloud Composer versions 2.0.* still rely on the Deployment Manager API for updates, upgrades, and environment deletion. It won't be possible to perform these operations if this API is disabled. We recommend to upgrade your 2.0.* environments to a later version to remove this dependency.
May 05, 2025
It is now possible to migrate from Cloud Composer 1 to Cloud Composer 3 using snapshots. For more information, see the new migration guide.
This feature will gradually roll out to all regions supported by Cloud Composer 3. At the moment it is available in the africa-south1, asia-south1, me-central1, me-central2, me-west1, southamerica-east1, and southamerica-west1 regions.
April 30, 2025
Starting from 5 May, 2025, new Cloud Composer 3 environments will use 1 CPU and 4 GB of memory for the Airflow web server by default. The minimum and maximum values for these parameters will not change.
April 29, 2025
A script for migrating from Cloud Composer 2 to Cloud Composer 3 is now available on GitHub.
For instructions about migrating with the script, see the new migration guide in Cloud Composer documentation.
April 24, 2025
Starting from June 2025, the default version for new Cloud Composer environments changes from Cloud Composer 2 to Cloud Composer 3. New environments will use the latest default Airflow build (composer-3-airflow-2). Currently, the default version is composer-2-airflow-2.
April 17, 2025
The default environment's service account setting is gradually removed in Cloud Composer. After the change, you'll need to explicitly specify a service account when you create a new Cloud Composer environment. For more information about addressing the change, see the eariler announcement of this change.
In this release, the change is rolling out to the following regions: africa-south1, asia-northeast2, asia-south2, australia-southeast2, europe-north2, europe-southwest1, europe-west8, europe-west10, europe-west12, me-central1, me-central2, me-west1, northamerica-northeast2, northamerica-south1, southamerica-west1, us-east7, and us-south1. It will be rolled out to more regions in future releases.
Cloud Composer 2 environments now always use the environment's service account for performing PyPI packages installations:
- Existing Cloud Composer 2 environments that previously used the default Cloud Build service account now use the environment's service account instead.
- Cloud Composer 2 environments created in versions 2.10.2 and later already have this change.
- Cloud Composer 3 environments already use the environment's service account, and are not affected by this change.
- This change is gradually rolled out to all regions supported by Cloud Composer 2.
Airflow 2.10.5 is available in Cloud Composer.
Cloud Composer now detects situations when asynchronous tasks are blocked in Airflow triggerers. If a trigger's execution is blocked for more than five minutes, Cloud Composer restarts the triggerer, which solves this transient issue.
(Cloud Composer 3) Key Access Justifications now correctly works for Customer Managed Encryption Keys (CMEK).
(Cloud Composer 3) It's now possible to override the default scopes of access tokens. Before the fix, the scope always defaulted to https://www.googleapis.com/auth/cloud-platform and https://www.googleapis.com/auth/userinfo.email. This resulted in authentication failures when accessing non-Google Cloud services.
The change is gradually rolled out to the following regions: africa-south1, asia-south2, australia-southeast2, europe-north2, europe-west3, europe-west10, europe-west12, northamerica-south1, southamerica-west1, us-east7, and us-south1. It will be rolled out to more regions in future releases.
The bucket synchronization process doesn't fail if the /plugins folder isn't available in the environment's bucket.
New Airflow builds are available in Cloud Composer 3:
- composer-3-airflow-2.10.5-build.0
- composer-3-airflow-2.10.2-build.13 (default)
- composer-3-airflow-2.9.3-build.20
New images are available in Cloud Composer 2:
- composer-2.12.1-airflow-2.10.5
- composer-2.12.1-airflow-2.10.2 (default)
- composer-2.12.1-airflow-2.9.3
Support dates for previous Cloud Composer 3 builds are available. All Cloud Composer 3 builds with Airflow 2.10.2 are supported until April 17, 2026.
April 09, 2025
The Airflow web server in Cloud Composer 3 requires at least 2 GB of memory when an environment is created or updated. This might lead to longer operation times or failures to perform these operations.
As a workaround, when you create a new Cloud Composer 3 environment or upgrade an existing environment, provide at least 2 GB of memory (default value) to the Airflow web server.
April 07, 2025
All Cloud Composer environment's GKE clusters are set up with maintenance exclusions from March 27, 2025 to April 12, 2025. For more information, see Maintenance exclusions.
April 03, 2025
The unification of Cloud Composer 3 billing with BigQuery is paused until further notice. The change was previously scheduled for April 13, 2025.
March 26, 2025
Data lineage in Cloud Composer now uses OpenLineage.
Data lineage support for a specific Airflow operator is now provided by the provider package where the operator is located. See Supported classes in the apache-airflow-providers-openlineage documentation for a list of latest supported operators.
For more information about data lineage in Cloud Composer, see Data lineage with Dataplex.
This feature is gradually rolled out. It will be available in us-west1, us-south1, europe-north1, me-west1, asia-northeast2, asia-southeast2, and africa-south1 regions. We plan to provide this feature in other regions in future releases.
(Available without upgrading) Fixed an issue with updating maintenance windows when there is an upcoming Cloud Composer 3 infrastructure operation.
(Airflow 2.10.2 and 2.9.3) The apache-airflow-providers-google package was upgraded to version 14.0.0 in Cloud Composer 2 images and Cloud Composer 3 builds.
This package is a new major version where many previously deprecated Airflow operators are removed. It is not possible to use these operators in your DAGs.
Make sure that you update your DAGs to use up-to-date alternatives of the removed operators. For more information about removed and deprecated Airflow operators and their up-to-date alternatives, see Deprecated and removed Airflow operators.
For more information about changes, see the apache-airflow-providers-google changelog from version 10.26.0 to version 14.0.0.
(Airflow 2.10.2 and 2.9.3) The apache-airflow-providers-cncf-kubernetes package was upgraded to version 10.3.0 in Cloud Composer 2 images and Cloud Composer 3 builds. For more information about changes, see the apache-airflow-providers-cncf-kubernetes changelog from version 10.1.0 to version 10.3.0.
Cloud Composer versions 2.6.4, 2.6.5, and 2.6.6 have reached their end of support period.
New Airflow builds are available in Cloud Composer 3:
- composer-3-airflow-2.10.2-build.12 (default)
- composer-3-airflow-2.9.3-build.19
New images are available in Cloud Composer 2:
- composer-2.12.0-airflow-2.10.2 (default)
- composer-2.12.0-airflow-2.9.3
March 18, 2025
The issue with Cloud Composer 2 upgrade operations is now resolved. The upgrade operations are unblocked in all regions.
March 15, 2025
The Custom constraints with Organization Policy feature is now generally available (GA).
This feature provides more granular control over Cloud Composer environment configuration fields. You can use custom organization policies to allow or deny specific configuration values for Cloud Composer environments.
March 14, 2025
March 18, 2025 update: The issue is resolved.
Some upgrade operations for Cloud Composer 2 might lead to unhealthy environments. That is why upgrades for Cloud Composer 2 versions will be blocked until the issue is fully resolved.
March 11, 2025
(Cloud Composer 3) Validation error messages now use the correct format for image versions.
New Airflow builds are available in Cloud Composer 3:
- composer-3-airflow-2.10.2-build.11 (default)
- composer-3-airflow-2.9.3-build.18
New images are available in Cloud Composer 2:
- composer-2.11.5-airflow-2.10.2 (default)
- composer-2.11.5-airflow-2.9.3
March 05, 2025
You can now specify an order in which Airflow searches for secrets by overriding the
[secrets]backends_order Airflow configuration option.
Fixed an issue in Cloud Composer REST API that allowed some environment.patch operations to succeed when multiple update masks that aren't related to each other were passed in a request. Now operations with such masks fail with an error.
New images are available in Cloud Composer 2:
- composer-2.11.4-airflow-2.10.2 (default)
- composer-2.11.4-airflow-2.9.3
Cloud Composer versions 2.6.2 and 2.6.3 have reached their end of support period.
New Airflow builds are available in Cloud Composer 3:
- composer-3-airflow-2.10.2-build.10 (default)
- composer-3-airflow-2.9.3-build.17
February 27, 2025
Database retention policy is available in Cloud Composer 3. You can use this feature to automatically delete older records from the Airflow database, which helps to maintain the Airflow database's size.
February 25, 2025
Cloud Composer 3 is now available in Stockholm (europe-north2). The change is gradually rolling out.
February 20, 2025
(Cloud Composer 3) Fixed a problem with configuring access to SMTP servers that don't support user and password authentication. The [smtp]smtp_user and [smtp]smtp_password Airflow configuration options are no longer preconfigured with placeholder values.
(Airflow 2.10.2 and 2.9.3) Preinstalled packages were changed:
virtualenvwas downgraded from 20.29.1 to 20.28.1
New Airflow builds are available in Cloud Composer 3:
- composer-3-airflow-2.10.2-build.9 (default)
- composer-3-airflow-2.9.3-build.16
New images are available in Cloud Composer 2:
- composer-2.11.3-airflow-2.10.2 (default)
- composer-2.11.3-airflow-2.9.3
February 13, 2025
(New Cloud Composer environments only) Cloud Composer now enforces SSL connections to Cloud SQL instances.
This change implements a security compliance standard in Cloud Composer and enforces SSL usage. You can adopt other security standards by following recommendations listed in Vulnerability findings. Previously, it was possible to create a non-SSL connection to the Airflow database. After the change, the connection will be refused.
(Available without upgrading) Fixed a problem that caused Terraform to unnecessarily force the replacement of cloud_composer_network_ipv4_cidr_block, web_server_ipv4_cidr_block, and cloud_sql_ipv4_cidr_block fields in some configurations of Cloud Composer 2 and Cloud Composer 1 environments.
(Cloud Composer 2) The dependency_permissions_check_count metric now correctly reports the number of missing permissions.
(Cloud Composer 2) The list of permissions reported by dependency_permissions_check_count metric was updated. Now the metric reports more permissions that are required for a functioning environment. Permissions that weren't necessary were replaced or removed.
(Cloud Composer 3) The GOOGLE_CLOUD_PROJECT environment variable is changed to reserved for Cloud Composer 3. This fixes an issue where creating an environment with this variable failed with a non-specific error.
Cloud Composer versions 2.6.0 and 2.6.1 have reached their end of support period.
New Airflow builds are available in Cloud Composer 3:
- composer-3-airflow-2.10.2-build.8 (default)
- composer-3-airflow-2.9.3-build.15
New images are available in Cloud Composer 2:
- composer-2.11.2-airflow-2.10.2 (default)
- composer-2.11.2-airflow-2.9.3
February 07, 2025
Starting March 2025, the default version for new Cloud Composer environments changes from Cloud Composer 2 to Cloud Composer 3.
New environments that are created without a specified version will use the default Airflow build of Cloud Composer 3, composer-3-airflow-2. Currently, the default version is composer-2-airflow-2.
February 06, 2025
Cloud Composer 2 is no longer available in Mexico (northamerica-south1). It is not possible to select this region when creating a new Cloud Composer 2 environment. Existing Cloud Composer 2 environments and both new and existing Cloud Composer 3 environments are not affected by this change.
February 04, 2025
All Cloud Composer environment's GKE clusters are set up with maintenance exclusions from January 21, 2025 to February 10, 2025. For more information, see Maintenance exclusions.
January 31, 2025
(Cloud Composer 3) The issue with upgrading Airflow builds is now resolved. You can now upgrade Airflow builds for Cloud Composer 3 in the asia-south1, asia-northeast2, europe-west1, europe-west3, and europe-north1 regions.
January 28, 2025
(Available without upgrading) Starting from Cloud Composer version 2.11.0 with Airflow 2.10.2 and 2.9.3, the enable_task_context_logger Airflow configuration option is no longer blocked.
(Cloud Composer 3) The zone in the Environment.config.node_config.location field is no longer populated in the Cloud Composer API.
Terminating Airflow workers now generate log messages with information about still running tasks.
(Cloud Composer 3) Fixed an issue with KubernetesPodOperator failing to run properly when do_xcom_push was set to True.
(Available without upgrading) In Cloud Composer 3, fixed an issue where deleting an environment with an attached VPC network failed if the project accept list of the network attachment was cleared manually.
Fixed an issue where log messages were incorrectly formatted for non-string content, such as dictionaries and numbers.
Fixed a problem where synchronization between the environment's bucket and Airflow components failed because incorrect objects in the bucket were not ignored.
(Cloud Composer 3) Fixed an issue with KubernetesPodOperator failing to run properly when do_xcom_push was set to True.
(Airflow 2.10.2 and 2.9.3) The following preinstalled packages were downgraded:
certifidowngraded from 2024.12.14 to 2024.8.30.charset-normalizerdowngraded from 3.4.1 to 3.4.0.cryptographydowngraded from 44.0.0 to 43.0.3.filelockdowngraded from 3.16.1 to 3.15.4.google-authdowngraded from 2.37.0 to 2.35.0.keyringdowngraded from 25.6.0 to 25.5.0.urllib3downgraded from 2.3.0 to 2.2.3.zippdowngraded from 3.21.0 to 3.20.2.
(Airflow 2.10.2 and 2.9.3) The apache-airflow-providers-cncf-kubernetes package was upgraded to version 10.1.0 in Cloud Composer 2 images and Cloud Composer 3 builds. For more information about changes, see the apache-airflow-providers-cncf-kubernetes changelog from version 10.0.1 to version 10.1.0.
(Airflow 2.10.2 and 2.9.3) The following preinstalled provider packages were upgraded to new major versions:
apache-airflow-providers-apache-beamupgraded from 5.9.1 to 6.0.0apache-airflow-providers-dbt-cloudupgraded from 3.11.2 to 4.0.0apache-airflow-providers-hashicorpupgraded from 3.8.0 to 4.0.0apache-airflow-providers-httpupgraded from 4.13.3 to 5.0.0apache-airflow-providers-mysqlupgraded from 5.7.4 to 6.0.0apache-airflow-providers-sendgridupgraded from 3.6.0 to 4.0.0apache-airflow-providers-sqliteupgraded from 3.9.1 to 4.0.0apache-airflow-providers-sshupgraded from 3.14.0 to 4.0.0
Cloud Composer version 2.5.5 has reached its end of support period.
New Airflow builds are available in Cloud Composer 3:
- composer-3-airflow-2.10.2-build.7 (default)
- composer-3-airflow-2.9.3-build.14
January 27, 2025
January 31, 2025 update: The issue is resolved.
(Cloud Composer 3 only) We are currently experiencing an issue with upgrading Airflow builds for Cloud Composer 3 in asia-south1, asia-northeast2, europe-west1, europe-west3, europe-north1 regions.
The upgrades are temporarily disabled as we continue our work to restore the listed functionalities. We will release an additional announcement after the issue is resolved.
January 24, 2025
The following recently released Cloud Composer 3 Airflow builds and Cloud Composer 2 versions are rolled back and aren't available for creating and upgrading existing environments. We will roll out new builds in the next release.
- composer-2.11.0-airflow-2.10.2
- composer-2.11.0-airflow-2.9.3
- composer-3-airflow-2.10.2-build.6
- composer-3-airflow-2.9.3-build.13
January 15, 2025
Starting April 13, 2025, we are removing the default environment's service account setting. This change enhances security and provides greater control over your Cloud Composer environments.
- Previously, the default Compute Engine service account was used by default when a user didn't specify a service account during Cloud Composer creation.
- After the change, you'll need to explicitly specify a service account when you create a new Cloud Composer environment.
- Existing Cloud Composer environments will not be affected by this change.
To address this change:
- We recommend to create one or more user-managed service accounts for Cloud Composer environments in your project and grant them the minimum of required permissions. For more information and instructions, see Grant roles to an environment's service account.
- If you use Terraform, scripts or other automation and configuration management tools, then make sure to update them, so that an environment's service account is specified when you create an environment.
In April 2025, Cloud Composer 2 environments will always use the environment's service account for performing PyPI packages installations:
- The environment's service account will be used instead.
- Existing Cloud Composer 2 environments that previously used the default Cloud Build service account will change to using the environment's service account instead.
- Cloud Composer 2 environments created in versions 2.10.2 and later already have this change.
- Cloud Composer 3 environments already use the environment's service account, and are not impacted by this change.
January 12, 2025
April 3, 2025 update: This change is postponed.
Starting after April 13, 2025, Cloud Composer 3 will unify its billing with BigQuery. The Cloud Composer 3 standard milli DCU-hours SKU will be replaced with the new BigQuery Engine for Apache Airflow SKU, which will be calculated based on the number of BigQuery slot hours that your Cloud Composer 3 environments consume. Other SKUs will be renamed and moved under the BigQuery hierarchy. The change will be rolled out gradually to all regions supported by Cloud Composer 3. For more information about the change, see Transition to BigQuery slot-hour-based pricing.
January 10, 2025
(Cloud Composer 3) New metrics are available for Cloud Composer 3 environments:
- CPU quota limit for Cloud Composer workloads
- CPU quota usage for Cloud Composer workloads
(Cloud Composer 3) You can now use custom certificates when installing packages from your private repository. This change is gradually rolled out to all Cloud Composer 3 environments. To obtain this change earlier, upgrade the Airflow build of your environment.
The Redis persistent disk is now automatically deleted together with the environment. This persistent disk is used by the Redis queue and stores only technical data.
The issue with automatic environment upgrades and upgrading Airflow builds in Cloud Composer 3 is resolved and these operations are working. If you think that your environment is still impacted by this issue, please reach out to the Cloud Support team.
(New Cloud Composer 2 environments only) Cloud Composer 2 environments in versions 2.10.2 and later always use the environment's service account for performing PyPI packages installations. This change applies only to newly created environments, existing environments that are upgraded to 2.10.2 and later versions will not get this change.
(Cloud Composer 2) Cloud Composer 2 environments that use PSC interfaces will no longer try to allocate IP ranges for VPC peerings in the tenant project. Environments that use PSC instead of VPC peerings do not use these ranges. This fixes a problem where these ranges overlapped with ranges used for the PSC subnetwork.
(Cloud Composer 3 only) The /data folder is now synchronized with Airflow triggerers.
Fixed an issue where the user-defined Cloud DNS configuration for Google API domains would break Cloud Composer 3 environment creation and attachment of VPC networks.
(Available without upgrading) Improved the error message generated when the Cloud Composer Service Agent service account is missing permissions on the project or on the environment's service account.
Fixed a problem with the interpolation of pip.conf file. Now the pip.conf file is not interpolated and can contain unescaped % characters.
The worker_autoscale Airflow configuration option is blocked in Cloud Composer 2. Previously, it was blocked only in Cloud Composer 3.
Removed the warning log message about in-memory storage because it doesn't apply to Cloud Composer.
The importlib-resources package was removed from preinstalled packages.
Cloud Composer 2.10.1 is a version with an extended upgrade timeline.
New Airflow builds are available in Cloud Composer 3:
- composer-3-airflow-2.10.2-build.5 (default)
- composer-3-airflow-2.9.3-build.12
New images are available in Cloud Composer 2:
- composer-2.10.2-airflow-2.10.2 (default)
- composer-2.10.2-airflow-2.9.3
Cloud Composer version 2.5.4 has reached its end of support period.
January 09, 2025
In January 2025, we will delete inactive Cloud Composer 1 environments that are non-recoverable. Environments that have both of the following problems present at the same time will be deleted:
- The environment's underlying GKE cluster is deleted.
- The environment is in the ERROR state for at least 60 days because of a disabled billing account or because the Cloud Composer API service was deactivated in its project.
This change doesn't affect buckets of these environments. You can still recover your DAGs and other data from the environment's bucket and then delete the bucket manually. See Delete environments for information about data that is not deleted automatically together with the environment.
January 10, 2025 update: The issue is resolved.
(Cloud Composer 3 only) We are currently experiencing an issue with automatic environment upgrades and upgrading Airflow builds for Cloud Composer 3 in asia-south1, europe-west1, and asia-northeast2 regions.
The upgrades are temporarily disabled as we continue our work to restore the listed functionalities. We will release an additional announcement after the issue is resolved.
December 20, 2024
Cloud Composer 3 is now available in Mexico (northamerica-south1).
December 16, 2024
Cloud Composer 3 is now generally available (GA):
- All Airflow builds starting from airflow-2.9.3-build.11 and airflow-2.10.2-build.4 are supported at the GA level.
- If your environment uses an earlier Airflow build, then upgrade it to airflow-2.9.3-build.11, airflow-2.10.2-build.4, or a later build to use Cloud Composer 3 on the GA level.
December 15, 2024
When creating new environments in Google Cloud console, it's now required to explicitly select a service account for the environment. We recommend to create a user-managed service account and use it for Cloud Composer environments.
December 11, 2024
In January 2025, we plan to release Cloud Composer 2 versions that will always use the environment's service account for performing PyPI packages installations:
- New Cloud Composer 2 environments created in versions 2.10.2 and later will have this change. This change applies only to newly created environments, existing environments that are upgraded to 2.10.2 and later versions will not get this change.
- Currently, Cloud Composer 2 environments use the default Cloud Build service account if it exists (and the environment's service account if it doesn't). Depending on the way Cloud Build is configured in your project, this might mean that the default Cloud Compute service account or the legacy Cloud Build service account might be used by your environment. We recommend to configure Cloud Build to adhere to the principle of least privilege.
- Make sure to check the Cloud Build default service account change page for information about changes to the default Cloud Build service account.
- Cloud Composer 3 environments already use the environment's service account, and are not impacted by this change.
Cloud Composer no longer adds any missing IAM permissions to the Cloud Storage bucket when it is used to create an environment with a custom environment's bucket. Make sure that the environment's service account has permissions from the Composer Worker role on the bucket.
The COMPOSER_AGENT_BUILD_SERVICE_ACCOUNT environment variable is changed to reserved. This change improves the security of Cloud Composer environments.
(Cloud Composer 2) Airflow worker liveness check configuration was changed to be consistent with the configuration used in Cloud Composer 3. In particular, this change increases the timeout, giving the liveness check more time to detect unhealthy Airflow workers. This change is gradually rolled out to all regions supported by Cloud Composer.
The COMPOSER_AGENT_BUILD_SERVICE_ACCOUNT environment variable is changed to reserved. This change improves the security of Cloud Composer environments.
(Cloud Composer 3) It is now possible to upgrade an environment if the [sentry]sentry_on Airflow configuration option is set to true.
New Airflow builds are available in Cloud Composer 3:
- composer-3-airflow-2.10.2-build.4 (default)
- composer-3-airflow-2.9.3-build.11
New images are available in Cloud Composer 2:
- composer-2.10.1-airflow-2.10.2 (default)
- composer-2.10.1-airflow-2.9.3
Cloud Composer version 2.5.3 has reached its end of support period.
December 05, 2024
New Cloud Composer 3 environments can now be created in VPC SC. This feature is gradually rolled out to all regions supported by Cloud Composer.
The maximum limit on the database size during upgrades in Cloud Composer 3 is now the same as the limit for snapshots (20 GB).
(New Cloud Composer 3 environments) Increased the maximum number of internet connections that each Airflow worker can support at the same time.
(Cloud Composer 2 only) It is now possible to upgrade an environment if the [sentry]sentry_on Airflow configuration option is set to true.
(Cloud Composer 2) If an upgrade operation fails, Cloud Composer 2 now restores the environment with the correct number of triggers.
(Cloud Composer 3) KubernetesPodOperator now works when the do_xcom_push parameter is set to True.
Fixed the issue in the environment's component responsible for uploading the logs of Airflow components to Cloud Logging. This bug sometimes lead to a situation where Cloud Composer-level log might be missing for an Airflow component. The same log was still available on the Kubernetes-component level.
(Airflow 2.10.2 and 2.9.3) The apache-airflow-providers-cncf-kubernetes package was upgraded to version 10.0.1 in Cloud Composer 2 images and Cloud Composer 3 builds. For more information about changes, see the apache-airflow-providers-cncf-kubernetes changelog from version 9.0.1 to version 10.0.1.
(Available without upgrading) Fixed an issue where Airflow workers sometimes generated incomplete or unreadable output.
(Airflow 2.10.2 and 2.9.3) The apache-airflow-providers-google package was upgraded to version 10.26.0 in Cloud Composer 2 images and Cloud Composer 3 builds. For more information about changes, see the apache-airflow-providers-google changelog from version 10.25.0 to version 10.26.0.
The aiohttp package was downgraded from 3.11.0 to 3.10.11.
Airflow 2.7.3 is no longer included in Cloud Composer images and builds.
Airflow 2.7.3 is no longer included in Cloud Composer images and builds.
The default version of Airflow is changed to 2.10.2.
New Airflow builds are available in Cloud Composer 3:
- composer-3-airflow-2.10.2-build.3 (default)
- composer-3-airflow-2.9.3-build.10
New images are available in Cloud Composer 2:
- composer-2.10.0-airflow-2.10.2 (default)
- composer-2.10.0-airflow-2.9.3
December 04, 2024
Scheduled snapshots are available in Cloud Composer 3. This feature will be gradually rolled out to all regions supported by Cloud Composer 3.
Cloud Composer 2 is now available in Mexico (northamerica-south1).
Cloud Composer 2 is now available in Mexico (northamerica-south1).
November 27, 2024
In December 2024, Google will remove the following previously deprecated Airflow operators from the apache-airflow-providers-google package.
The new version of this package will be included in one of the future releases of Cloud Composer and the change will be announced in the Release Notes. After this change, it will not be possible to use these operators in your DAGs.
Make sure that you use up-to-date alternatives of the removed operators instead. For more information about removed and deprecated Airflow operators and their up-to-date alternatives, see Deprecated and removed Airflow operators.
Operators that will be removed in December 2024: DataPipelineHook, CreateDataPipelineOperator, RunDataPipelineOperator, AutoMLDatasetLink, AutoMLDatasetListLink, AutoMLModelLink, AutoMLModelTrainLink, AutoMLModelPredictLink.
November 13, 2024
Airflow 2.10.2 is available in Cloud Composer.
Improved the error message generated when a Cloud Composer 3 environment creation fails because of missing permissions.
(Airflow 2.7.3) Backported #35887 to fix an issue that occurred during the DST transition. The issue affected DAGs with timezone-aware cron schedule and caused infinite loops in the Airflow scheduler.
(Airflow 2.10.2 and 2.9.3) The apache-airflow-providers-google package was upgraded to version 10.25.0 in Cloud Composer 2 images and Cloud Composer 3 builds. For more information about changes, see the apache-airflow-providers-google changelog from version 10.24.0 to version 10.25.0.
(Airflow 2.10.2 and 2.9.3) The apache-airflow-providers-cncf-kubernetes package was upgraded to version 9.0.1 in Cloud Composer 2 images and Cloud Composer 3 builds. For more information about changes, see the apache-airflow-providers-cncf-kubernetes changelog from version 9.0.0 to version 9.0.1.
Support dates for previous Cloud Composer 3 builds are available. All Cloud Composer 3 builds with Airflow 2.9.3 are supported until November 13, 2025.
New Airflow builds are available in Cloud Composer 3:
- composer-3-airflow-2.10.2-build.0
- composer-3-airflow-2.9.3-build.7 (default)
- composer-3-airflow-2.7.3-build.23
New images are available in Cloud Composer 2:
- composer-2.9.11-airflow-2.10.2
- composer-2.9.11-airflow-2.9.3 (default)
- composer-2.9.11-airflow-2.7.3
November 06, 2024
(Cloud Composer 3) Fixed an issue that affected the speed of PyPI package installation. PyPI packages are now installed slightly faster.
(Airflow 2.9.3 and 2.7.3) The docutils package was removed from preinstalled packages.
Cloud Composer version 2.5.1 has reached its end of support period.
Cloud Composer 2.9.7 is a version with an extended upgrade timeline.
New Airflow builds are available in Cloud Composer 3:
- composer-3-airflow-2.9.3-build.6 (default)
- composer-3-airflow-2.7.3-build.22
October 30, 2024
(Airflow 2.9.3 and 2.7.3) The dbt-common package was downgraded from 1.11.0 to 1.10.0.
(Cloud Composer 3) Airflow workers now generate a proper OpenID Connect (OIDC) token.
Cloud Composer 2.9.9 images are available:
- composer-2.9.9-airflow-2.9.3 (default)
- composer-2.9.9-airflow-2.7.3
New Airflow builds are available in Cloud Composer 3:
- composer-3-airflow-2.9.3-build.5 (default)
- composer-3-airflow-2.7.3-build.21
Cloud Composer version 2.5.0 has reached its end of support period.
October 22, 2024
(Only new Cloud Composer 2 environments, all versions) If a GKE Control Plane IP range is specified for an environment, GKE creates a new subnetwork in this range to provision the IP address for communication with the GKE Control Plane. Otherwise, the subnetwork specified in the Cloud Composer connection subnetwork range is used (this range defaults to the environment's subnetwork). For more information about IP ranges used by Cloud Composer environments, see Configure private IP networking.
(Airflow 2.9.3 and 2.7.3) Changes in preinstalled packages:
- The
grpciopackage was downgraded from 1.66.2 to 1.65.5. - The
js2pypackage was removed from dependencies. - The
pyjsparserpackage was removed from dependencies. - The
tzlocalpackage was removed from dependencies.
(Airflow 2.9.3) The apache-airflow-providers-cncf-kubernetes package was upgraded to version 9.0.0 in Cloud Composer 2 images and Cloud Composer 3 builds. For more information about changes, see the apache-airflow-providers-cncf-kubernetes changelog from version 8.4.2 to version 9.0.0.
(Airflow 2.9.3) The apache-airflow-providers-google package was upgraded to version 10.24.0 in Cloud Composer 2 images and Cloud Composer 3 builds. For more information about changes, see the apache-airflow-providers-google changelog from version 10.23.0 to version 10.24.0.
Cloud Composer version 2.4.6 has reached its end of support period.
Cloud Composer 2.9.8 images are available:
- composer-2.9.8-airflow-2.9.3 (default)
- composer-2.9.8-airflow-2.7.3
New Airflow builds are available in Cloud Composer 3:
- composer-3-airflow-2.9.3-build.4 (default)
- composer-3-airflow-2.7.3-build.20
October 16, 2024
Airflow 2.9.1 is no longer included in Cloud Composer images and builds.
The default version of Airflow is changed to 2.9.3.
Fixed a bug where upgrading a private IP environment could fail because of an invalid CIDR range.
New Airflow builds are available in Cloud Composer 3:
- composer-3-airflow-2.9.3-build.3 (default)
- composer-3-airflow-2.7.3-build.19
Cloud Composer 2.9.7 images are available:
- composer-2.9.7-airflow-2.9.3 (default)
- composer-2.9.7-airflow-2.7.3
October 08, 2024
Cloud Composer version 2.4.5 has reached its end of support period.
New Airflow builds are available in Cloud Composer 3:
- composer-3-airflow-2.9.3-build.2
- composer-3-airflow-2.9.1-build.9 (default)
- composer-3-airflow-2.7.3-build.18
Cloud Composer 2.9.6 images are available:
- composer-2.9.6-airflow-2.9.3
- composer-2.9.6-airflow-2.9.1 (default)
- composer-2.9.6-airflow-2.7.3
October 01, 2024
(Available without upgrading) Fixed the cause of DAG run failures for runs created from the Cloud Console when the [scheduler]allowed_run_id_pattern Airflow configuration option is set to a custom value.
(Airflow 2.9.3 and 2.9.1) The apache-airflow-providers-google package was upgraded to version 10.23.0 in Cloud Composer 2 images and Cloud Composer 3 builds. For more information about changes, see the apache-airflow-providers-google changelog from version 10.22.0 to version 10.23.0.
(Airflow 2.9.3 and 2.9.1) The apache-airflow-providers-cncf-kubernetes package was upgraded to version 8.4.2 in Cloud Composer 2 images and Cloud Composer 3 builds. For more information about changes, see the apache-airflow-providers-cncf-kubernetes changelog from version 8.4.1 to version 8.4.2.
New Airflow builds are available in Cloud Composer 3:
- composer-3-airflow-2.9.3-build.1
- composer-3-airflow-2.9.1-build.8 (default)
- composer-3-airflow-2.7.3-build.17
Cloud Composer version 2.4.4 has reached its end of support period.
September 18, 2024
Airflow 2.9.3 is available in Cloud Composer images.
(Cloud Composer 3) Fixed the issue that caused KubernetesPodOperator tasks to fail if they ran for longer than 15 minutes.
Support dates for previous Cloud Composer 3 builds are available. All Cloud Composer 3 builds with Airflow 2.9.1 are supported until September 18, 2025.
New Airflow builds are available in Cloud Composer 3:
- composer-3-airflow-2.9.3-build.0
- composer-3-airflow-2.9.1-build.7 (default)
- composer-3-airflow-2.7.3-build.16
Cloud Composer 2.9.4 images are available:
- composer-2.9.4-airflow-2.9.3
- composer-2.9.4-airflow-2.9.1 (default)
- composer-2.9.4-airflow-2.7.3
Cloud Composer versions 2.4.2 and 2.4.3 have reached their end of support period.
September 02, 2024
Added a new metric: composer.googleapis.com/workflow/task_instance/queued_duration. This metric is based on the dag.<dag_id>.<task_id>.queued_duration Airflow metric.
(Cloud Composer 3) Fixed the cause of false-positive failures of Airflow Celery workers reported by the liveness health check.
The apache-airflow-providers-cncf-kubernetes package was upgraded to version 8.4.1 in Cloud Composer 2 images and Cloud Composer 3 builds with Airflow 2.9.1. For more information about changes, see the apache-airflow-providers-cncf-kubernetes changelog from version 8.3.4 to version 8.4.1.
New Airflow builds are available in Cloud Composer 3:
- composer-3-airflow-2.9.1-build.6 (default)
- composer-3-airflow-2.7.3-build.15
Cloud Composer 2.9.3 images are available:
- composer-2.9.3-airflow-2.9.1 (default)
- composer-2.9.3-airflow-2.7.3
August 27, 2024
(Cloud Composer 2) Starting on September 26, 2024, all Cloud Composer environments within a VPC SC perimeter will have no access to public PyPI repositories by default. For details about installing PyPI packages in the VPC SC mode, see Installing PyPI packages.
August 22, 2024
Cloud Composer version 2.4.1 has reached its end of support period.
August 13, 2024
Cloud Composer 2.9.1 images are available:
- composer-2.9.1-airflow-2.9.1 (default)
- composer-2.9.1-airflow-2.7.3
August 08, 2024
The kubelet read-only port in GKE clusters (TCP port 10255) is not used by Cloud Composer. You can define appropriate firewall rules to block external traffic over TCP 10255 in your customer project as described in Modify VPC firewall rules.
The default version of Airflow is changed to 2.9.1.
New Airflow builds are available in Cloud Composer 3:
- composer-3-airflow-2.9.1-build.3
- composer-3-airflow-2.7.3-build.12
Cloud Composer version 2.4.0 has reached its end of support period.
August 01, 2024
In July and August 2024, new Cloud Composer 2 environments keep switching to using GKE 1.29 and PSC as a connectivity channel to the GKE control plane. Creating new Cloud Composer 2 private IP environments will require using PSC SERVICE_PRODUCER endpoints, which might be blocked by the Disable Private Service Connect for Consumers organization policy constraint. Previously, this org policy limitation could be addressed by using VPC peerings instead of PSC, but starting from GKE 1.29, this workaround is no longer available.
(Cloud Composer 3 only) The July 24th issue with Airflow upgrade operations for Cloud Composer 3 environments is now fixed. Airflow upgrade operations are now being re-enabled, and should reach all Cloud Composer 3 environments within the next couple of days.
The initial synchronization of DAGs and plugins in Airflow workloads is performed in a separate container and utilizes full resources of the Pod.
(Cloud Composer 3) Fixed an error that happened when values in certain formats were used for Airflow configuration options.
(Cloud Composer 3) Fixed an error that happened when values in certain formats were used for Airflow configuration options.
Cloud Composer 2.8.8 images are available:
- composer-2.8.8-airflow-2.9.1
- composer-2.8.8-airflow-2.7.3 (default)
Airflow 2.6.3 is no longer included in Cloud Composer 2 images.
New Airflow builds are available in Cloud Composer 3:
- composer-3-airflow-2.9.1-build.2
- composer-3-airflow-2.7.3-build.11
July 25, 2024
Cloud Composer now verifies that the environment's service account is enabled before starting an environment creation operation. As a result, this type of error is reported sooner and the operation does not start.
Cloud Composer 2.8.7 images are available:
- composer-2.8.7-airflow-2.9.1
- composer-2.8.7-airflow-2.7.3 (default)
- composer-2.8.7-airflow-2.6.3
July 24, 2024
2024-08-01 Update: This issue has been resolved. For more information, see the release note from August 1st 2024.
(Cloud Composer 3 only) We are currently experiencing an issue with the Airflow upgrade operations for Cloud Composer 3 environments. Upgrading Airflow builds for Cloud Composer 3 is temporarily disabled as we continue our work to restore Airflow upgrade functionalities.
July 18, 2024
Information about excluded Cloud Storage objects in the environment's bucket is no longer logged. This change reduces the usage of the Storage API during the synchronization of DAG files, which improves the performance of Airflow components and results in fewer Airflow component restarts. The change will gradually become available in all Cloud Composer environments.
July 17, 2024
(Airflow 2.9.1) Raw HTML code in DAG docs and DAG parameter descriptions is disabled by default.
(Airflow 2.9.1) Task context logging is disabled, and it is not possible to enable it.
The apache-airflow-providers-google package was upgraded to version 10.21.0 in Cloud Composer 2 images with Airflow 2.7.3 and 2.9.1, and in all latest Cloud Composer 3 builds. For more information about changes, see the apache-airflow-providers-google changelog from version 10.18.0 to version 10.21.0.
The upgraded apache-airflow-providers-google package provides new operators and sensors for triggering DAGs across Cloud Composer environments:
- CloudComposerRunAirflowCLICommandOperator makes a call to the Airflow REST API of a Cloud Composer environment.
- CloudComposerDAGRunSensor checks if a DAG Run is completed.
Support dates for previous Cloud Composer 3 builds are available. All Cloud Composer 3 builds with Airflow 2.7.3 are supported until July 17, 2025.
Cloud Composer 2.8.6 images are available:
- composer-2.8.6-airflow-2.9.1
- composer-2.8.6-airflow-2.7.3 (default)
- composer-2.8.6-airflow-2.6.3
Cloud Composer version 2.3.4 has reached its end of support period.
When installing PyPI packages, if you want your builds to run with a custom service account, you can override the
COMPOSER_AGENT_BUILD_SERVICE_ACCOUNT environment variable with the chosen service account. For more information, see Install Python dependencies.
July 15, 2024
It is no longer possible to create Cloud Composer 1 environments in Google Cloud console. It's still possible to create Cloud Composer 1 environments through Google Cloud SDK, Terraform, and API in projects that support creating new Cloud Composer 1 environments.
July 09, 2024
A new Airflow build is available in Cloud Composer 3:
- composer-3-airflow-2.7.3-build.8
July 02, 2024
Cloud Composer version 2.3.3 has reached its end of support period.
March 19, 2024
In BigQueryInsertJobOperator, fixed the handling of parsing errors during Lineage emission when the query is too long or deeply nested.
Fixed an issue where Airflow task logs for the first try of a task might not be visible in Airflow UI.
February 05, 2024
Cloud Composer 2.6.0 is a version with an extended upgrade timeline.
January 08, 2024
(Available without upgrading) Cloud Composer now creates an audit log on a successful completion of create and update environment operations.
August 16, 2023
Fixed an issue where data lineage for BigQueryInsertJobOperator was reported to the incorrect target object in cases where Airflow tasks were running in parallel.
April 25, 2023
(Cloud Composer 2) Airflow 2.5.1 is available in Cloud Composer images.
March 18, 2023
(Composer 2 only) The instance_name field for Airflow Webservers in new environments is now by default set to the Composer Environment name.
February 24, 2023
The default Composer version is changed to Cloud Composer 2.
The default Cloud Composer image used for operations is now the latest version of Cloud Composer 2, unless the image-version parameter is explicitly specified.
The composer-latest-airflow-* version aliases now point to Cloud Composer 2.
December 13, 2022
Fixed an issue where a failed upgrade to the latest Cloud Composer version caused further upgrade attempts to fail.
November 18, 2022
Airflow 2.3.4 is available in Cloud Composer images.
(Cloud Composer 2 with Private Service Connect) SQL proxy now gets restarted faster when connection issues are detected.
(Cloud Composer 2) You can now create Composer environments with labels that contain only digits.
Airflow 2.3.3 is no longer included in Cloud Composer images.
September 29, 2022
Cloud Composer 1.19.11 and 2.0.28 images are available:
- composer-1.19.11-airflow-1.10.15 (default)
- composer-1.19.11-airflow-2.1.4
- composer-1.19.11-airflow-2.2.5
- composer-1.19.11-airflow-2.3.3
- composer-2.0.28-airflow-2.1.4
- composer-2.0.28-airflow-2.2.5
- composer-2.0.28-airflow-2.3.3
September 12, 2022
(New environments only) Encryption with customer-managed encryption keys (CMEK) now applies to the persistent disk of the environment's Redis queue.
Fixed a potential race condition in Airflow workers that could cause new tasks to be executed on a worker that is scheduled to be scaled down. This fix prevents Airflow tasks from being stuck in the running state.
August 29, 2022
Cloud Composer now generates an error message if an environment label matches internal environment labels used by Cloud Composer.
July 28, 2022
(Available without upgrading) Fixed a problem where DAG import errors were not displayed on the Environment details page, if the error messages did not have a creation time set.
June 29, 2022
The Per-folder Roles Registration feature is rolled out to all regions.
June 23, 2022
(Cloud Composer 2) Incremental task logs are now correctly refreshed and displayed in Airflow UI.
Cloud Composer 1.18.12 is a version with an extended upgrade timeline.
June 20, 2022
In July 2022, Cloud Composer 1 environments created in Cloud console will use the latest available version of Airflow 2 by default.
June 15, 2022
Logs in Cloud Logging now have Airflow DAG and task annotations for multilined output.
(Airflow 2) Enabled User Stats Chart view in Airflow UI for users with the Admin role.
(Airflow 2) Fix processor cleanup on DagFileProcessorManager #22685
(Airflow 2) Enabled User Stats Chart view in Airflow UI for users with the Admin role.
June 14, 2022
Authorized networks support is now generally available (GA).
June 01, 2022
Web server restarting is available in Preview in Cloud Composer 2.
(Cloud Composer 2) Environment's size can now be updated for environments with Private Service Connect.
(Cloud Composer 2) The amount of memory available to Redis queue now scales with the environment's size.
Fixed a problem where an upgrade operation could fail when deleting the previous Cloud Composer namespace.
Airflow 2.2.3 is no longer included in Cloud Composer images.
May 26, 2022
(Cloud Composer 2) Several false error log messages are no longer generated after an environment is created.
May 19, 2022
(Cloud Composer 2) Increased the memory limit for the Redis queue and made it scale with the environment's size.
May 13, 2022
Cloud Composer performs several retries when checking pip connectivity.
(Cloud Composer 2) Workers and schedulers generate a warning log message when storage usage is close to the limit.
(Airflow 2) The default value for the [webserver]worker_refresh_interval Airflow configuration option is changed to 600 seconds.
(Available without upgrading) The domain prefix for Private Service Connect subnetwork (connection_subnetwork) is now omitted in environment details.
(Airflow 2.2.3) Web server log messages in Airflow UI now have a correct time zone.
Fixed a problem where DAG import errors were not displayed in Cloud Console for Private IP environments in certain versions of Cloud Composer.
(Airflow 1.10.15) Fixed the variables set command. Now it correctly sets values for specified variables.
April 22, 2022
Cloud Composer automatically recreates and unpauses the Airflow monitoring DAG if it was deleted or paused.
(Cloud Composer 2) Cloud Composer Service Agent account cannot be used as a service account for an environment.
(Available without upgrading) Fixed a problem that caused Cloud Composer 1 environments with IP Masquerade Agent to break after an upgrade.
April 13, 2022
Cloud Composer now supports CMEK encryption using keys stored in External Key Managers.
(Airflow 2) Exception traces from Airflow task executions are now properly annotated with labels in Cloud Logging.
(Cloud Composer 2) Fixed a problem where some info log messages were logged as errors during environment operations.
(Airflow 1.10.15) Airflow Upgrade Checker updated to version 1.4.0.
(Airflow 1.10.15) Backported the fix for KubernetesPodOperator. KubernetesPodOperator now retries log tailing in in long-living tasks.
April 04, 2022
(Cloud Composer 2) Fixed an issue which caused failures when creating environments with Private Service Connect in a Shared VPC configuration.
Improved the validation when installing PyPI packages. Fixed a problem where DNS resolution errors in the cluster caused PyPI package installation failures.
(Airflow 2) Fixed the logging level of some messages generated by Airflow workers. These messages now have the correct INFO logging level.
Improved the error message generated when a PyPI package installation times out.
Fixed the severity level for Airflow Worker Set Controller logs
March 28, 2022
Improved the validation of custom IP ranges that are specified when an environment is created. The validation is more extensive and redundant error messages were removed.
(Cloud Composer 2) Fixed a problem when an unhealthy web server is not restarted.
(New Cloud Composer 1 environments) The minimum disk size for environment nodes is changed from 20 GB to 30 GB.
(Cloud Composer 1) Fixed problems with upgrading to Cloud Composer 1.18.* from earlier versions of Cloud Composer.
Fixed a problem with "Environment health" and "Worker Pod eviction" metrics occasionally not reporting new time-series points.
March 17, 2022
The Logs tab in Cloud Console is now generally available (GA).
Environment labels are now propagated to the environment's bucket.
Improved the syncing of DAGs and plugins to Airflow components. Objects named "." are ignored when syncing.
(Cloud Composer 1) It is no longer possible to select the e2-micro machine type.
March 08, 2022
(Cloud Composer 1) Fixed a problem with web server metrics not being reported or being reported partially.
(Available without upgrading) DAG UI now correctly handles invalid serialized DAG data.
Improved the handling of errors generated during update and upgrade operations because of an invalid pip.conf configuration file.
(Airflow 1) The apache-airflow-backport-providers-google package is updated to version 2022.2.22.
March 01, 2022
Cloud Composer 2 supports Customer Managed Encryption Keys (CMEK).
February 28, 2022
(New environments only) Temporary Pub/Sub topics created during environment operations are now labeled.
(Airflow 2) The google-cloud-datastore package was added to the list of preinstalled packages.
(Airflow 2) Fix the problem with task logs not being exported to Cloud Logging.
(Airflow 1) The apache-airflow-backport-providers-google package is updated to version 2022.2.11.
(Cloud Composer 2) Fixed the problem with an environment having 0 workers after an unsuccessful upgrade operation is rolled back.
(Available without upgrading) Improved the handling of errors in DAG UI for tasks without a set operator.
February 17, 2022
DAG UI is available in Preview.
February 15, 2022
(Cloud Composer 2) Environment creation no longer fails if the default Compute Engine service account is deleted in a project.
Airflow 2.0.2 is no longer included in Cloud Composer images.
February 07, 2022
Airflow 2.2.3 is available in Cloud Composer images.
January 28, 2022
The DAG UI feature was rolled back. We plan to make it available with the next release of Cloud Composer.
January 20, 2022
Reliability of writing to and reading from /logs and /data folders in Airflow components is improved in Cloud Composer 2 environments.
January 10, 2022
Fixed a problem with rollback of Cloud Composer system workloads after a failed environment upgrade attempt.
Fixed the "First DAG run for an uploaded DAG file has several failed tasks" known issue for Airflow 2 environments.
Fixed levels of Airflow log messages in Cloud Composer 2 environments.
Logs from SQL proxy are now correctly passed to the customer project in environments with enabled Private Service Connect support.
Improved error handling when configuring workload identity in Cloud Composer 2 environments.
December 16, 2021
Cloud Composer 2 is now generally available (GA).
Authorized networks support is available in Preview.
December 09, 2021
Fixed the issue with continuous web server reloading when syncing plugins.
(Cloud Composer 2) Messages in web server logs now have correctly assigned severity.
December 01, 2021
(Cloud Composer 2) Clusters that remain after failed environment creation operations are now deleted.
Airflow versions 2.1.2, 1.10.14, and 1.10.12 are no longer included in Cloud Composer images.
November 16, 2021
(Cloud Composer 1) GCSfuse version updated to 37.0. Cloud Composer uses GCSfuse to sync files between the environment bucket and worker pods.
(New environments only) Default values for maintenance windows are from 00:00:00 to 04:00:00 (GMT) on Sunday, Friday, and Saturday every week.
November 04, 2021
Airflow 2.1.4 is available in Cloud Composer images.
The scheduler heartbeat probe no longer checks for the availability of logs in Cloud Logging. This change reduces the chance of false positives.
November 02, 2021
Cloud Composer is now available in Singapore (asia-southeast1).
October 18, 2021
Cloud Composer 2 supports Airflow web server plugins.
Added the google-cloud-aiplatform package to Cloud Composer images with Airflow versions 2.1.2, 2.0.2, and 1.10.15.
Fixed an issue with the Airflow web server availability in Cloud Composer 2.
(New environments only) Shielded Nodes and Secure Boot features are enabled for Cloud Composer 1 environment clusters.
(New environments only) Cloud Composer 1 environment creation no longer fails when the constraints/compute.requireShieldedVm policy is turned on.
Fixed a bug that caused the __pycache__ folder to sometimes appear in an environment's bucket.
September 23, 2021
Cloud Composer images with Airflow 2 now use Python 3.8.12.
(New environments only) The FluentD environment component (composer-fluentd) no longer breaks if it was manually turned off for more than 30 days.
(New environments only) The airflow-monitoring pod is restarted instead of being marked as unhealthy when the GKE control plane IP changes.
New versions of Cloud Composer images:
- composer-1.17.1-airflow-2.1.2
- composer-1.17.1-airflow-2.0.2
- composer-1.17.1-airflow-1.10.15 (default)
- composer-1.17.1-airflow-1.10.14
- composer-1.17.1-airflow-1.10.12
- composer-2.0.0-preview.2-airflow-2.1.2
- composer-2.0.0-preview.2-airflow-2.0.2
Cloud Composer 1.12.0 has reached its end of full support period.
September 15, 2021
Airflow 2 in Cloud Composer is now generally available (GA).
HA Scheduler in Cloud Composer is now generally available (GA).
We plan to switch new versions of Cloud Composer with Airflow 1.10.15 to Python 3.8. Starting from November 2021, new Cloud Composer images with Airflow 1.10.15 will use Python 3.8 instead of Python 3.6. Please check for PyPI package compatibility before upgrading your environment to a version with Python 3.8.
Java Runtime in Airflow workers and schedulers is updated from version 8 to version 11.
If an Airflow configuration option is blocked in the image version specified for an upgrade operation, and this option has an override in your environment, the upgrade operation is rejected.
In Airflow UI, menu items for Configuration and DAG dependencies pages are now correctly displayed for users with the Op role.
Airflow 2.1.1 is no longer included in Cloud Composer images.
September 09, 2021
Cloud Composer 2 uses the following Cloud Composer images:
- composer-2.0.0-preview.0-airflow-2.1.2
- composer-2.0.0-preview.0-airflow-2.1.1
- composer-2.0.0-preview.0-airflow-2.0.2
August 30, 2021
(Airflow 2) Cloud Composer now supports the stable Airflow REST API. The stable Airflow REST API is enabled by default.
August 23, 2021
Non-error output of CloudSQL Proxy instances is routed to stdout instead of stderr. Regular CloudSQL Proxy output is no longer marked as errors in container logs.
(Airflow 2.1.1) Airflow configuration options that are deprecated in Airflow 2 are now blocked. In Airflow 2, these options are located in a different configuration section, but Airflow still uses the deprecated location as a fallback.
(Airflow 2) The airflow_db connection used by Airflow to connect to the Airflow database now correctly uses PostgreSQL connection type for Airflow 2 environments.
August 12, 2021
(Airflow 2.1.1) Downgraded colorlog dependency from 5.0.1 to 4.8.0
July 29, 2021
Airflow 2.1.1 is available in Cloud Composer images.
Fixed the problem with environment health metrics not being exported in VPC Service Controls environments because of airflow-monitoring component restarts.
July 22, 2021
Airflow 2.0.1 is no longer included in Cloud Composer images.
July 14, 2021
Airflow 2.0.2 is available in Cloud Composer images.
Fixed memory issues on the FluentD environment component.
July 08, 2021
New versions of Cloud Composer images:
- composer-1.17.0-preview.5-airflow-2.0.1
- composer-1.16.9-airflow-1.10.15 (default)
- composer-1.16.9-airflow-1.10.14
- composer-1.16.9-airflow-1.10.12
June 30, 2021
New versions of Cloud Composer images:
- composer-1.17.0-preview.4-airflow-2.0.1
- composer-1.16.8-airflow-1.10.15 (default)
- composer-1.16.8-airflow-1.10.14
- composer-1.16.8-airflow-1.10.12
Fixed remaining memory issues that occurred while syncing files on machine types with more than 8 vCPUs.
Cloud Composer 1.10.5 has reached its end of full support period.
June 17, 2021
Cloud Composer is now available in Warsaw (europe-central2).
New versions of Cloud Composer images:
- composer-1.17.0-preview.3-airflow-2.0.1
- composer-1.16.7-airflow-1.10.15
- composer-1.16.7-airflow-1.10.14 (default)
- composer-1.16.7-airflow-1.10.12
For DAG runs that have long-running tasks with certain operators, task level logs are now periodically updated in the Airflow UI. Before this change, logs for such tasks were only available in Airflow UI after the task was completed.
Fixed the cause of Liveness probe errored events that appeared in the scheduler and worker pod logs.
Fixed a problem where the Airflow worker health was calculated incorrectly because of leftover queued tasks without DAGs being present in the Airflow database. This led to problems with task execution because Airflow workers were constantly restarted in healthy environments.
June 07, 2021
You can now store values for the smtp_password Airflow configuration option in Secret Manager.
Fixed memory issues that occurred while syncing files on machine types with more than 8 vCPUs.
(New environments only) Some log messages related to Airflow web server access were previously missing in Cloud Logging. This problem is fixed and these messages now appear in Cloud Logging.
May 21, 2021
Error messages about PyPI package conflicts now contain links to corresponding cluster build logs.
May 13, 2021
Preview: Cloud Composer supports Airflow 2. For more information about transferring from environments with Airflow 1 to Airflow 2, see Migrate environments to Airflow 2.
Airflow 2.0.1 is available in Cloud Composer images.
You can now break down costs associated with particular Cloud Composer environments. User labels that you assign to your environments now appear in billing reports.
New versions of Cloud Composer images:
- composer-1.17.0-preview.0-airflow-2.0.1
- composer-1.16.4-airflow-1.10.15
- composer-1.16.4-airflow-1.10.14 (default)
- composer-1.16.4-airflow-1.10.12
Improved the error message that is generated when the specified service account does not have enough permissions to run Airflow workloads.
Added troubleshooting information to error messages generated on Airflow web server deployment failures.
Kerberos client (krb5-user) package is pre-installed in Cloud Composer container images.
Some environment operations that failed because of networking problems are now retried instead of failing.
Database passwords are now redacted in error messages that appear in Composer Agent logs.
Error messages about dependency conflicts that happen when installing Python packages are now correctly reported.
April 29, 2021
New versions of Cloud Composer images:
- composer-1.16.3-airflow-1.10.15
- composer-1.16.3-airflow-1.10.14 (default)
- composer-1.16.3-airflow-1.10.12
When Airflow configuration is updated, an erroneous log message about a web server update failure no longer appears in logs.
Fixed problems with execution date in environment health monitoring when Airflow uses a custom time zone.
Cloud Composer versions 1.8.3 to 1.10.2 have reached their end of full support period.
April 22, 2021
New versions of Cloud Composer images:
- composer-1.16.2-airflow-1.10.15
- composer-1.16.2-airflow-1.10.14 (default)
- composer-1.16.2-airflow-1.10.12
Airflow 1.10.10 is no longer included in Cloud Composer images.
When a GKE authorization error occurs during an environment operation, the GKE error message is reported and the operation fails immediately.
A deprecation message is now displayed for the xcom_push argument of KubernetesPodOperator.
April 15, 2021
New versions of Cloud Composer images:
- composer-1.16.1-airflow-1.10.15
- composer-1.16.1-airflow-1.10.14 (default)
- composer-1.16.1-airflow-1.10.12
- composer-1.16.1-airflow-1.10.10
If an environment's service account does not have required permissions for a requested operation, Cloud Composer generates an actionable error message. The operation fails faster in this case.
April 08, 2021
Airflow 1.10.15 is available in Cloud Composer images.
New versions of Cloud Composer images:
- composer-1.16.0-airflow-1.10.15
- composer-1.16.0-airflow-1.10.14 (default)
- composer-1.16.0-airflow-1.10.12
- composer-1.16.0-airflow-1.10.10
In Airflow 1.10.14, PythonVirtualenvOperator now uses the Python version of the environment when a Python version is not specified.
Environments with already deleted GKE clusters can now be deleted as usual. Deleting such environments no longer requires a workaround.
March 31, 2021
New versions of Cloud Composer images:
- composer-1.15.2-airflow-1.10.14 (default)
- composer-1.15.2-airflow-1.10.12
- composer-1.15.2-airflow-1.10.10
Irrelevant warnings about asynchronous DAG loading parameters no longer show up in the Airflow logs.
Corrected the validation of custom Cloud SQL and Airflow web server IP ranges that are specified during the environment creation. Changed the error code and the message that are returned when a specified CIDR range is not valid.
Fixed an Airflow web UI bug that caused the DAG Tree View page to crash in rare cases.
March 23, 2021
Invalid resource names in API requests now cause a 4xx response. The invalid resource name is reported in the error message.
Removed the "@-@" workload info prefixes from Airflow task logs in the Airflow UI and Cloud Logging. This change is available for Airflow 1.10.14.
For Cloud Composer versions 1.13.2 and later, regional base images are used for Airflow web server and worker-scheduler builds. This improves customized image build times.
Improved error handling when creating node pools during upgrade operations. In some cases, the error was not reported when an upgrade operation failed on a timeout.
Improved the validation procedure for custom IP ranges that are specified during the environment creation. Changed the error code and the message that are returned when a specified CIDR range is not valid.
Fixed the documentation link in the Airflow UI.
March 11, 2021
DAG serialization is enabled by default in new environments created for Cloud Composer versions 1.15.0 and later. Upgrading an existing environment to 1.15.0 does not change the existing DAG serialization settings.
When creating new environments, enabling asynchronous DAG loading disables DAG serialization.
Before creating or updating an environment, Cloud Composer checks that required APIs are enabled in a project and that CIDR blocks specified for VPC Native and Private IP are valid. If these requirements are not met, Cloud Composer reports an error and the operation does not start. This change is available only for new Cloud Composer environments.
Environment deletion operations no longer fail when Artifact Registry API is disabled.
Fixed a bug that caused upgrades to fail during an in-cluster build in public IP environments.
Environment creation operations no longer fail in Private IP configurations that use Customer Managed Encryption Keys (CMEK).
The environment creation process now aborts early on any web server deployment failure.
Improved error reporting for web server deployment failures. App Engine errors that occur during the deployment are now marked as web server deployment errors.
March 05, 2021
New versions of Cloud Composer images:
- composer-1.14.5-airflow-1.10.14
- composer-1.14.5-airflow-1.10.12 (default)
- composer-1.14.5-airflow-1.10.10
Improved the logging of Airflow exceptions. Full Python tracebacks for Airflow exceptions are reported and marked as errors in the logs.
Fixed a potential infinite loop in the airflow-monitoring pod. Environment health checks no longer get stuck after certain types of environment update operations.
Fixed the cause of failures when creating Qwiklabs environments.
When creating environments, unmet network requirements for pods and services cause the operation to fail immediately. Previously, the operation failed when a timeout was reached.
Cloud Composer acquires existing environment resources if they are available during an upgrade operation. Before, the operation could fail with the "ALREADY_EXISTS" error in some cases.
Fixed a problem with airflow-monitoring not having logs after changing the machine type for GKE Cluster in a Cloud Composer environment.
PyPI packages can now be installed in Cloud Composer versions 1.11.0 and 1.11.1.
March 02, 2021
GA: Support for Resource location restrictions and Data Residency is now generally available.
February 25, 2021
When an environment update operation cannot start, an error message that lists possible causes for the error is generated.
Improved the syncing of DAGs and plugins to the Airflow web server. DAG parsing is now less likely to break because of race conditions.
Added FreeTDS system package to Cloud Composer images.
Upgraded apache-airflow-backport-providers-google package to version 2021.2.5 in Airflow 1.10.12 and 1.10.14. This is potentially a breaking change because the package contains updates of Python Google Cloud libraries. For a list of new operators and for more information about breaking changes, see the 2021.2.5 release notes.
The google-cloud-pubsublite package is installed by default in Composer images for Airflow 1.10.12 and 1.10.14.
Improved the reliability of environment upgrade operations. Added new retrying procedures and enhanced existing ones.
When an environment deletion operation fails, a correct GKE error is displayed. Before, a different GKE error was displayed in some cases.
Fixed an error when some MsSQL operators were not working with Azure instances.
Fixed an error in GKE cluster builds. In private IP environments, an additional nodepool no longer remains after an update.
February 24, 2021
GA: Support for Customer Managed Encryption Keys (CMEK) is now generally available.
February 16, 2021
GA: Setting and updating machine types for CloudSQL/Web Server is now generally available.
GA: Support for Domain restricted sharing is now generally available.
February 03, 2021
Timeouts for environment upgrade operations are increased.
Fix for CVE-2021-3156 vulnerability.
On a failed environment upgrade operation, the created CloudSQL database is now correctly rolled back.
Create and update operations for environments no longer fail if your account doesn't have the serviceusage.services.get permission.
Fixed SQL operation conflicts that were occurring during environment upgrade operations.
You can now enable and disable RBAC in environments with installed custom PyPI packages.
January 31, 2021
Preview: You can now configure Cloud Composer to use Artifact Registry instead of Container Registry.
January 28, 2021
The default Airflow version for Composer images changes from 1.10.10 to 1.10.12.
Airflow 1.10.9 is no longer included in Cloud Composer images.
January 22, 2021
- You can now specify maintenance windows for your environments. GKE cluster and SQL database are updated only during the specified period. This feature is available in Google Cloud SDK and Beta API.
- The maximum number of pods per node in the GKE cluster changes from 100 to 32. This change affects all newly created environments. You can use the
max-pods-per-nodeparameter when creating an environment to increase or decrease the number of pods.
January 12, 2021
- GA: Network ACL support for the Airflow web server is now generally available.
- Preview: Composer can now be configured to use Customer-managed encryption keys (CMEK).
- New versions of Cloud Composer images:
composer-1.13.4-airflow-1.10.9,composer-1.13.4-airflow-1.10.10, andcomposer-1.13.4-airflow-1.10.12. The default iscomposer-1.13.4-airflow-1.10.10. Upgrade your Cloud SDK to use features in this release. - If environment creation succeeds when the requester did not have the
iam.serviceAccounts.actAspermission on the service account, Composer will now generate a warning in the audit log entry. - Error handling for files synchronization between buckets has been improved.
- Fixed an issue where connections were inserted into the Airflow database twice.
- The Airflow UI will now always send requests to HTTPS addresses.
January 05, 2021
- In Cloud Composer 1.15.0, DAG serialization will be enabled by default when creating new Cloud Composer environments. Starting from this version, if you do not want to use DAG serialization, you must explicitly disable it.
December 16, 2020
- Preview: A new Logs tab has been added to the Environment details page.
December 14, 2020
- New versions of Cloud Composer images:
composer-1.13.3-airflow-1.10.9,composer-1.13.3-airflow-1.10.10, andcomposer-1.13.3-airflow-1.10.12. The default iscomposer-1.13.3-airflow-1.10.10. Upgrade your Cloud SDK to use features in this release.
- Composer will now fail faster when the network settings in Private IP environments prohibit the download of publicly stored Python packages.
- Composer Agent error messages are now more descriptive.
- Composer will now check whether the Artifact Registry API is enabled during updates (if it is required).
December 03, 2020
- New versions of Cloud Composer images:
composer-1.13.2-airflow-1.10.9,composer-1.13.2-airflow-1.10.10, andcomposer-1.13.2-airflow-1.10.12. The default iscomposer-1.13.2-airflow-1.10.10. Upgrade your Cloud SDK to use features in this release.
- Airflow worker pods are now forced to be scheduled on different nodes. Each node runs only a single Airflow worker pod.
- You can now set web server network access control using the v1 Composer API.
- New metrics have been added to monitor web server CPU and memory usage:
- CPU usage time
- CPU reserved cores
- Memory bytes used
- Memory quota
- During environment creation and updates, Composer will now verify whether you have chosen a region compliant with any location restriction organization policies. Error reporting has also been improved in cases where location restrictions cause environment updates to fail.
- Composer versions 1.8.1 and 1.8.2 have been deprecated.
November 23, 2020
- Cloud Composer is now available in Los Angeles (
us-west2).
November 20, 2020
- Composer version 1.12.5 or newer: File synchronization between buckets in Domain restricted sharing compliant environments has been improved. If your environment was created before November 20, 2020 you must upgrade your environment to access this feature
November 09, 2020
- New versions of Cloud Composer images:
composer-1.13.0-airflow-1.10.6,composer-1.13.0-airflow-1.10.9,composer-1.13.0-airflow-1.10.10, andcomposer-1.13.0-airflow-1.10.12. The default iscomposer-1.13.0-airflow-1.10.10. Upgrade your Cloud SDK to use features in this release. - Airflow 1.10.12:
api-auth_backendis now set toairflow.api.auth.backend.deny_allby default. To use the experimental REST API, change theapi-auth_backendconfiguration override toairflow.api.auth.backend.default.
- Airflow 1.10.12 is now available for Cloud Composer.
- The GKE release channel is set to STABLE for new and upgraded Composer environments.
- The following Cloud Composer versions have been deprecated:
- Composer-1.7.2
- Composer-1.7.3
- Composer-1.7.4
- Composer-1.7.5
- Composer-1.7.6
- Composer-1.7.7
- Composer-1.8.0
October 30, 2020
- New versions of Cloud Composer images:
composer-1.12.5-airflow-1.10.6,composer-1.12.5-airflow-1.10.9, andcomposer-1.12.5-airflow-1.10.10. The default iscomposer-1.12.5-airflow-1.10.10. Upgrade your Cloud SDK to use features in this release.
- More detailed error messages are now provided for errors during Python package installation.
- Fixed an issue where DAG definition files that exceeded 64kb were not working with
[core] store_dag_codemode enabled. Now these files can be up to 16 MB. - Fixed an issue where updating Airflow configurations with value interpolation caused update requests to timeout; these requests are not supported and will now be rejected.
- Improved GCSfuse stability to resolve intermittent issues where the mounted directory was unavailable.
October 23, 2020
- The following Composer environment database metrics are now available in Cloud Monitoring: CPU usage, CPU cores, CPU utilization, memory usage, memory quota, memory utilization, disk usage, disk quota, disk utilization.
- Worker liveness checks now also check whether the log syncing process is running.
- Improved configuration synchronization with the Airflow web server, fixing an issue with loading logs.
- The Airflow database will no longer disconnect after 30 seconds of query runtime.
- Worker health checks will no longer overload the Airflow database.
October 15, 2020
- New versions of Cloud Composer images:
composer-1.12.3-airflow-1.10.6,composer-1.12.3-airflow-1.10.9, andcomposer-1.12.3-airflow-1.10.10. The default iscomposer-1.12.3-airflow-1.10.9. Upgrade your Cloud SDK to use features in this release.
- Cloud Build logs from the tenant project are now published in the Composer logs. They are available under the log name
build-log-webserver. - Airflow DAG processor manager logs are now published in the Composer logs. They are available under the log name
dag-processor-manager. - If an update operation fails, links to the specific Cloud Build log will now be included in the error message.
- Compatibility with Domain Restricted sharing has been improved. Upgrading your environment to the newest version of Composer can now enable or disable its compatibility with Domain Restricted Sharing based on your organization policy.
October 07, 2020
- New versions of Cloud Composer images:
composer-1.12.2-airflow-1.10.6,composer-1.12.2-airflow-1.10.9, andcomposer-1.12.2-airflow-1.10.10. The default iscomposer-1.12.2-airflow-1.10.9. Upgrade your Cloud SDK to use features in this release.
- Added the log entry labels
version_idandinstance_idto differentiate the logs of different Airflow web server instances. - Airflow database upgrade logs are now published in the Composer logs under a separate log name.
- Cloud Storage syncing logs are now published together in the Composer logs under a separate log name. They can be separated based further on
pod_id.
- Fixed upgrade rollback failures due to a
mismatch in API versionserror. - Improved handling of errors caused by a missing
plugins/directory in the Cloud Storage bucket. - Backported an Airflow change that fixes an issue with Airflow 1.10.9 and 1.10.10 that causes MySQL to deadlock on the
rendered_task_instance_fieldstable when using DAG serialization. - Fixed a bug that undercharged customers for App Engine storage.
- Backfilled GKE API versions to make older Composer environments GKE 1.16-proof.
- Fixed an issue that caused
bq loadcommands for files larger than 100MB to fail with aRedirectMissingLocationerror.
September 28, 2020
- New versions of Cloud Composer images:
composer-1.12.1-airflow-1.10.6,composer-1.12.1-airflow-1.10.9, andcomposer-1.12.1-airflow-1.10.10. The default iscomposer-1.12.1-airflow-1.10.6. Upgrade your Cloud SDK to use features in this release.
- In-cluster image build logs will now appear in Monitoring under separate log names
build-log-*, instead of in the Composer Agent logs. - You can now set or update machine types for the Airflow web server and Cloud SQL instance for any environment running a Composer version 1.7.2 or newer, regardless of Airflow version.
- Airflow 1.10.6, 1.10.9, and 1.10.9: An Airflow change has been backported that lowers the network cost of DAG serialization.
- Airflow 1.10.9: An Airflow change has been backported that improves GKE cluster resource usage, improving overall Airflow performance.
September 17, 2020
- New versions of Cloud Composer images:
composer-1.12.0-airflow-1.10.3,composer-1.12.0-airflow-1.10.6,composer-1.12.0-airflow-1.10.9, andcomposer-1.12.0-airflow-1.10.10. The default iscomposer-1.12.0-airflow-1.10.6. Upgrade your Cloud SDK to use features in this release.
- Airflow 1.10.10 is now available.
- You can now securely store Airflow connections and secrets by configuring Composer environments with Secret Manager.
- All new Environments now use Zonal DNS names.
August 21, 2020
- New versions of Cloud Composer images:
composer-1.11.3-airflow-1.10.3,composer-1.11.3-airflow-1.10.6, andcomposer-1.11.3-airflow-1.10.9. The default iscomposer-1.11.3-airflow-1.10.6. Upgrade your Cloud SDK to use features in this release. - The max size for serialized DAGs has been increased to 128 MB.
- PyPi dependency conflicts will now cause environment updates to fail by default. This can now be changed by setting the environmental variable
IGNORE_PYPI_DEPENDENCY_CONFLICTS=True. This is the default behavior For older environments, but you can setIGNORE_PYPI_DEPENDENCY_CONFLICTS=Falseto force conflicts to fail.
- The Airflow configuration
[core]load_examplescan no longer be modified. - Domain Restricted Sharing setup with buckets in the tenant project is now applied during environment creation using the Beta API only when it is required.
Airflow 1.10.3, 1.10.6 and 1.10.9:
- Disabled editing DAG properties that will be overwritten when the DAG is reloaded.
- Fixed the following security vulnerabilities: CVE-2020-11981, CVE-2020-11982, CVE-2020-11978
Airflow 1.10.9:
- Fixed an issue that caused BigQuery operators from backported providers to fail when DAG serialization was turned on.
- The Airflow web server UI will now correctly reload when the plugin folder is modified.
- Rendered
template_fieldscan now be displayed without accessing DAG files.
Airflow 1.10.6: Fixed intermittent issues template rendering when DAG serialization is turned on.
August 10, 2020
- New versions of Cloud Composer images:
composer-1.11.2-airflow-1.10.3,composer-1.11.2-airflow-1.10.6, andcomposer-1.11.2-airflow-1.10.9. The default iscomposer-1.11.2-airflow-1.10.6. Upgrade your Cloud SDK to use features in this release.
- Airflow 1.10.6 and 1.10.9: You can now specify a location argument when creating a
BigQueryCheckOperatorto use it in a different region from the Composer environment.
- Fixed GKE setting incompatibilities that broke environment creation for Composer versions between 1.7.2 and 1.8.3.
- When DAG serialization is on, plugins and DAGs are no longer synced when the Airflow web server starts up. This fixes web server failures when plugins use custom PyPI packages.
- Fixed intermittent failures when triggering a DAG from the Airflow Web UI with DAG serialization turned on.
- Fixed update operations (installing Python dependencies and upgrading environments) for domain-scoped projects.
- Fixed a broken link to the Airflow documentation in Airflow 1.10.9.
August 03, 2020
- New versions of Cloud Composer images:
composer-1.11.1-airflow-1.10.3,composer-1.11.1-airflow-1.10.6, andcomposer-1.11.1-airflow-1.10.9. The default iscomposer-1.11.1-airflow-1.10.6. Upgrade your Cloud SDK to use features in this release. - Composer now enforces
iam.serviceAccounts.actAspermission checks on the service account specified during Composer environment creation. See Creating environments for details.
- Private IP environments can now be creating using non-rfc 1918 CGN ranges (100.64.0.0/10)
- New PyPi packages have been added for Composer version composer-1.11.0-airflow-1.10.6. These make it possible to install apache-airflow-backport-providers-google with no additional package upgrades.
- The PyPi package
google-cloud-datacatalogcan now be installed on Composer environments running Airflow 1.10.6 and Python 3. - Cloud Composer 1.11.1+: Backport providers are installed by default for Airflow 1.10.6 and 1.10.9.
- You can now use the
label.worker_idfilter in Cloud Monitoring logs to see logs sent out of a specific Airflow worker Pod. - With the Composer Beta API, you can now upgrade an environment to any of the three latest Composer versions (instead of just the latest).
- You can now modify these previously blocked Airflow configurations:
[scheduler] scheduler_heartbeat_sec,[scheduler] job_heartbeat_sec,[scheduler] run_duration
- A more informative error message was added for environment creation failures caused by issues with Cloud SQL instance creation.
- Improved error reporting has been added for update operations that change the web server image in cases where the error occurs before the new web server image is created.
- The Airflow-worker liveness check has been changed so that a task just added to a queue will not fire an alert.
- Reduced the amount of non-informative logs thrown by the environment in Composer 1.10.6.
- Improved the syncing procedure for env_var.json in Airflow 1.10.9 (it should no longer throw "missing file:" errors).
- Airflow-worker and airflow-scheduler will no longer throw "missing env_var.json" errors in Airflow 1.10.6.
July 30, 2020
Cloud Composer is now available in Osaka (asia-northeast2).
July 24, 2020
- New versions of Cloud Composer images:
composer-1.11.0-airflow-1.10.2,composer-1.11.0-airflow-1.10.3,composer-1.11.0-airflow-1.10.6, andcomposer-1.11.0-airflow-1.10.9. The default iscomposer-1.11.0-airflow-1.10.3. Upgrade your Cloud SDK to use features in this release.
- Airflow 1.10.9 is now supported.
- Environment upgrades have been enabled for the latest two Composer versions (1.11.0 and 1.10.6).
- Added a retry feature to the Airflow CeleryExecutor (disabled by default). You can configure the number of times Celery will attempt to execute a task by setting the
[celery] max_command_attemptsproperty. The delay between each retry can also be adjusted with[celery] command_retry_wait_duration(default: 5 seconds).
- New PyPi packages have been added for Composer version
composer-1.11.0-airflow-1.10.6. These make it possible to installapache-airflow-backport-providers-googlewith no additional package upgrades. - The PyPi package
google-cloud-datacatalogcan now be installed on Composer environments running Airflow 1.10.6 and Python 3. - Fixed synchronization of environment variables to the web server.
- Improved error reporting when PyPI package installation fails.
- Composer versions 1.6.1, 1.7.0, and 1.7.1 are now deprecated.
July 07, 2020
New versions of Cloud Composer images:
composer-1.10.6-airflow-1.10.2,composer-1.10.6-airflow-1.10.3andcomposer-1.10.6-airflow-1.10.6. The default iscomposer-1.10.6-airflow-1.10.3. Upgrade your Cloud SDK to use features in this release.For Airflow 1.10.6 and later: The Airflow config property
[celery] poolis now blocked.The
[core]sql_alchemy_pool_recycleAirflow setting has been modified to improve SQL connection reliability.
- Fixed an issue with Airflow 1.10.6 environments where task logs were not visible in the UI when DAG serialization was enabled.
- It is now possible to upgrade from Composer versions 1.1.1, 1.2.0, 1.3.0, 1.4.0, 1.4.1, 1.4.2, 1.5.0, and 1.5.2 to the newest version.
- Fixed a misleading red environment health status bar when the
[core]logging_levelAirflow property is set toDEBUG.
June 30, 2020
Cloud Composer support for VPC Service Controls is now in Beta.
June 24, 2020
- New versions of Cloud Composer images:
composer-1.10.5-airflow-1.10.2,composer-1.10.5-airflow-1.10.3andcomposer-1.10.5-airflow-1.10.6. The default iscomposer-1.10.5-airflow-1.10.3. Upgrade your Cloud SDK to use features in this release.
- Composer now uses the Kubernetes v1 API, and is compatible with GKE 1.16
- An updated haproxy configuration for Composer increases the maximum number of connections to 2000, and changes load balancing to be based on the number of connections. These settings can be configured with environment variables.
- Error messages for
TP_APP_ENGINE_CREATINGtimeout and RPC delivery issues have been expanded. - Airflow Providers can now be installed inside Cloud Composer.
- Error handling for rendering templates in the Airflow web server UI has been improved.
- Fixed an issue with rendering task instance details (logs, task instance template, params) in the Airflow web server UI when DAG serialization is enabled.
- Fixed an issue with
DataFlowJavaOperator, so it can now be used with Apache Beam 2.20. - Improved error reporting for failing operations.
- Memory consumption of the
gcs-syncdcontainer is now constrained to prevent system instability.
May 31, 2020
Cloud Composer is now available in Seoul (asia-northeast3).
May 26, 2020
- New versions of Cloud Composer images:
composer-1.10.4-airflow-1.10.2,composer-1.10.4-airflow-1.10.3andcomposer-1.10.4-airflow-1.10.6. The default iscomposer-1.10.4-airflow-1.10.3. Upgrade your Cloud SDK to use features in this release. - For Airflow 1.10.6 and later: The Airflow config property
[celery] poolis now blocked.
- Fixed an issue with Airflow 1.10.6 environments where task logs were not visible in the UI when DAG serialization was enabled.
May 15, 2020
- New versions of Cloud Composer images:
composer-1.10.3-airflow-1.10.2,composer-1.10.3-airflow-1.10.3andcomposer-1.10.3-airflow-1.10.6. The default iscomposer-1.10.3-airflow-1.10.3. Upgrade your Cloud SDK to use features in this release.
- Resource quota limits have been updated, allowing environment administrators to set quotas with more granularity. The default quotas for read and write operations have also changed; see Cloud Composer resource quotas for details. The old limits are deprecated, but will not be removed from the Cloud Console Quotas page until a future release.
- The machine type of the Airflow web server will now be preserved during Composer environment updates, including cases like new PyPi module installations, or adding new environment variables.
- Synchronization of log files between the Airflow scheduler, web server and workers has been improved.
- More useful error messages have been added for Composer environment upgrade failures.
- Future change: Airflow 1.10.6 will become the default Airflow version for Composer environments in an upcoming release.
- Composer version 1.6.1 has been deprecated.
May 08, 2020
Cloud Composer is now available in Hong Kong (asia-east2).
Cloud Composer is now available in Las Vegas (us-west4).
April 27, 2020
- New versions of Cloud Composer images:
composer-1.10.2-airflow-1.10.2,composer-1.10.2-airflow-1.10.3andcomposer-1.10.2-airflow-1.10.6. The default iscomposer-1.10.2-airflow-1.10.3. Upgrade your Cloud SDK to use features in this release.
- Fixed an issue with the CloudSQL Proxy HealthCheck that caused the Proxy Pod to restart repeatedly.
- The fluentd spec for in-cluster build log exporting now correctly points to the production fluentd image from
cloud-airflow-releaser. This fix is required for Composer to correctly perform in-cluster builds for VPC SC configuration. - Adjusted ImageBuilder to fix PyPI package installation issues when using VPC SC.
- Fixed intermittent issues with
airflow-monitoringduring the initialization phase. - Fixed an issue that caused the Airflow scheduler and worker pods to take ~10 minutes to terminate.
- Fixed an issue with upgrading the image version and improved error handling during Composer environment upgrades.
- The oldest supported version of Composer is now
composer-1.6.0-airflow-x.x.x
April 17, 2020
Composer version 1.10.1 has been rolled back. If you created an environment with composer-1.10.1-airflow-*, you can retrieve and delete the environment, but not update it. We recommend that you delete the environment and create a new environment with the latest image version. Refer to the March 20, 2020 release notes for default version.
April 10, 2020
Private IP Composer environments are now generally available (GA). See Configuring private IP to learn how to use this feature.
Support for Shared VPC networks is now generally available (GA).
April 05, 2020
Cloud Composer is now available in Salt Lake City (us-west3).
March 31, 2020
The new Composer monitoring dashboard is now in beta.
March 23, 2020
Cloud Composer is now available in Sao Paulo (southamerica-east1).
March 20, 2020
- New versions of Cloud Composer images:
composer-1.10.0-airflow-1.10.2,composer-1.10.0-airflow-1.10.3andcomposer-1.10.0-airflow-1.10.6. The default iscomposer-1.10.0-airflow-1.10.2. Upgrade your Cloud SDK to use features in this release.
- Improved Composer logs: Composer Agent adds more detailed logs in Cloud Monitoring to describe the current stage of environment creation and provide better error messages if an operation fails.
- MySQL passwords are now stored in Kubernetes Secrets instead of the config map.
- You can now set the
core.store_serialized_dagsproperty toFalseafter setting it toTrue. (Previously it was not possible to switch back.) - Composer now uses Pip version 19.0.2 for both Python 2 and Python 3 Airflow environments.
- Added the Airflow property
core.store_dag_code, which lets you see DAG code in the web UI while VPC-SC is enabled. - New metrics have been added to the monitoring API. See the full list here.
- Fixed an issue where Airflow database connection errors did not propagate correctly.
- Logs from the Composer Agent will now show the correct severity level in Cloud Monitoring.
- Fixed an issue where network tags added to environments did not propagate to the node pools created during the in-cluster image building.
- The Composer Agent will now wait for
env_varsto be sent before starting the web server. - Backported a fix for SQL proxy, which improves the stability of SQL connections inside workers.
- Added new scripts to handle synchronization of files from Customer Project buckets with tenant project buckets in DRS mode, so that DAG synchronization is not affected by long-running logs synchronization.
- Fixed an issue that prevented the creation of environments in the same VPC but different regions.
- Fixed an issue with updating the node count for Composer environments running versions 1.6.0 to 1.8.2.
- Installing PyPI packages from private repositories in the public Internet (specified in the pip.conf file) now works in private IP Composer environments without having to configure Cloud NAT.
March 09, 2020
You can now control access to the Airflow web server, either allowing access from any IP address (default), or specifying which IP ranges have access. For details, see Creating environments.
February 28, 2020
- New versions of Cloud Composer images:
composer-1.9.2-airflow-1.10.1,composer-1.9.2-airflow-1.10.2,composer-1.9.2-airflow-1.10.3andcomposer-1.9.2-airflow-1.10.6. The default iscomposer-1.9.2-airflow-1.10.2. Upgrade your Cloud SDK to use features in this release.
- (Beta) You can now create private IP Cloud Composer environments that are configured for Shared VPC.
Fixed an issue with updating Composer environments running versions 1.6.0 to 1.8.2. Updating the node count for an environment is still broken, and will be fixed in a future version.
- Workaround: To resize the cluster for an environment using version 1.6.0 - 1.8.2, resize the default node pool in the cluster directly through GKE interface, or delete the Composer environment and recreate it with a different node count.
Fixed an issue with the Dataflow Python hook for Airflow 1.10.6 that restricted it to using Python 2. Dataflow will now use the same Python version as the Composer environment.
February 25, 2020
- New versions of Cloud Composer images:
composer-1.9.1-airflow-1.10.1,composer-1.9.1-airflow-1.10.2,composer-1.9.1-airflow-1.10.3andcomposer-1.9.1-airflow-1.10.6. The default iscomposer-1.9.1-airflow-1.10.2. Upgrade your Cloud SDK to use features in this release.
- Fixed an issue where updating environment variables cleared the stored Fernet key.
- Fixed an issue with running DAGs via the Airflow 1.10.2 REST API when
dag_serializationis turned on. - Fixed an issue with environment creation for Composer versions 1.6.0 to 1.8.2 when using GKE version 1.14.
- Improved the health check functionality for SQLProxy.
February 13, 2020
- New versions of Cloud Composer images:
composer-1.9.0-airflow-1.10.1,composer-1.9.0-airflow-1.10.2,composer-1.9.0-airflow-1.10.3andcomposer-1.9.0-airflow-1.10.6. The default iscomposer-1.9.0-airflow-1.10.2. Upgrade your Cloud SDK to use features in this release.
- Airflow 1.10.6 is now available for Composer.
- Python 3 is now the default version for new Composer environments.
- Airflow 1.9.0 is no longer supported for new Composer environments.
- Fixed BigQuery operators for Python 3 in Airflow 1.10.6.
- Fixed a bug where some deserialized tasks had no
start_datein Airflow 1.10.6. - The Fernet Key is now stored in Kubernetes Secrets instead of the Config Map.
- Workers now wait until they have synced with the
dagsfolder before executing tasks. - New items have been added to the Airflow security properties blocklist in airflow.cfg:
webserver-auth_backendandscheduler-auth_backend. - Fixed a rare bug where Airflow workers tried to execute tasks before the DAG code was synced to the Airflow worker volume.
- Composer is planning to deprecate support for Airflow 1.10.1.
- In the near future it will not be possible to create Composer environments for Python 2 from the user interface. You will still be able to create Python 2 environments using gcloud.
January 31, 2020
- New versions of Cloud Composer images:
composer-1.8.4-airflow-1.9.0,composer-1.8.4-airflow-1.10.1,composer-1.8.4-airflow-1.10.2, andcomposer-1.8.4-airflow-1.10.3. The default iscomposer-1.8.4-airflow-1.9.0. Upgrade your Cloud SDK to use features in this release.
- Deprecation policy update: It is no longer possible to create environments with image versions older than 1 year. Currently the oldest version allowed is composer-1.4.1.
- The Python version for Dataflow Operators is no longer hard-coded. Composer users can now use Dataflow Operators with Python 2 and 3. (Airflow 1.10.3 only.)
- Fixed an issue with App Engine health check errors when creating environments with Composer version 1.7.5 or lower.
- The
airflow_home configurationparameter is no longer added toairflow.cfgif using Airflow version 1.10.3 or greater. - Airflow 1.10.2 is now the default version used for new Composer environments when Airflow version is not specified.
December 16, 2019
- New versions of Cloud Composer images:
composer-1.8.3-airflow-1.9.0,composer-1.8.3-airflow-1.10.1,composer-1.8.3-airflow-1.10.2, andcomposer-1.8.3-airflow-1.10.3. The default iscomposer-1.8.3-airflow-1.9.0. Upgrade your Cloud SDK to use features in this release.
- Composer now uses OpenAPI to manage components running on GKE clusters. This ensures Composer will be compatible with future versions of GKE.
- Fixed the formatting for validation errors returned by the
projects.locations.environments.createAPI method.
November 25, 2019
- New versions of Cloud Composer images:
composer-1.8.2-airflow-1.9.0,composer-1.8.2-airflow-1.10.1,composer-1.8.2-airflow-1.10.2, andcomposer-1.8.2-airflow-1.10.3. The default iscomposer-1.8.2-airflow-1.9.0. Upgrade your Cloud SDK to use features in this release.
- Fixed an issue with triggering DAGs from the Web UI while DAG serialization is turned on (for Airflow 1.10.3).
- CloudSQL Proxy should now use less CPU power.
November 18, 2019
- New versions of Cloud Composer images:
composer-1.8.1-airflow-1.9.0,composer-1.8.1-airflow-1.10.1,composer-1.8.1-airflow-1.10.2, andcomposer-1.8.1-airflow-1.10.3. The default iscomposer-1.8.1-airflow-1.9.0. Upgrade your Cloud SDK to use features in this release.
- Fixed an issue that prevented upgrades from Airflow 1.10.2 to 1.10.3.
- Fixed an issue with triggering DAGs from the Web UI while DAG serialization is turned on.
October 31, 2019
Cloud Composer is now available in Frankfurt (europe-west3).
October 30, 2019
- New versions of Cloud Composer images:
composer-1.8.0-airflow-1.9.0,composer-1.8.0-airflow-1.10.1,composer-1.8.0-airflow-1.10.2, andcomposer-1.8.0-airflow-1.10.3. The default iscomposer-1.8.0-airflow-1.9.0. Upgrade your Cloud SDK to use features in this release.
- Added support for Apache Airflow 1.10.3.
- Fixed a DAG serialization issue in Airflow 1.10.2.
- Fixed an issue with domain restricted sharing support.
October 18, 2019
- New versions of Cloud Composer images:
composer-1.7.9-airflow-1.9.0,composer-1.7.9-airflow-1.10.1, andcomposer-1.7.9-airflow-1.10.2. The default iscomposer-1.7.9-airflow-1.9.0. Upgrade your Cloud SDK to use features in this release.
- Bucket Policy Only is now supported, and you will no longer need to disable it during Cloud Composer environment creation.
- Support for Domain restricted sharing is now in Beta. See Creating environments to learn how to enable this feature.
- Improved automatic zone selection during environment creation.
- Fixed an issue with Private IP setup during environment creation.
October 08, 2019
- New versions of Cloud Composer images:
composer-1.7.7-airflow-1.9.0,composer-1.7.7-airflow-1.10.1, andcomposer-1.7.7-airflow-1.10.2. The default iscomposer-1.7.7-airflow-1.9.0. Upgrade your Cloud SDK to use features in this release.
- The ODBC Linux driver (
unixodbc-dev) is now included by default in the Cloud Composer images. jsonschemais now available as a Python dependency for DAG serialization.- Composer now ensures that you can only change the Airflow config
[core] store_serialized_dagsone way (from false to true). - Added default Airflow config and environment variables for DAG serialization.
- Environment update timeout has been extended to facilitate migrating large databases.
- New environments will use regional buckets instead of multi-regional. Previously-created environments will continue to use multi-regional buckets.
- Fixed an issue that sporadically caused Cloud Storage bucket synchronization to hang.
- Fixed an issue where service account permission errors would overwrite other errors.
- Fixed an issue that caused extraneous disk usage in the Airflow webserver.
September 20, 2019
Cloud Composer is now available in Zurich (europe-west6).
September 12, 2019
- The ODBC Linux driver (
unixodbc-dev) is now included by default in the Cloud Composer images. jsonschemais now available as a Python dependency for DAG serialization.
- New versions of Cloud Composer images:
composer-1.7.6-airflow-1.9.0,composer-1.7.6-airflow-1.10.1, andcomposer-1.7.6-airflow-1.10.2. The default iscomposer-1.7.6-airflow-1.9.0. Upgrade your Cloud SDK to use features in this release.
- New environments will use regional buckets instead of multi-regional. Previously-created environments will continue to use multi-regional buckets.
- Fixed an issue that sporadically caused Cloud Storage bucket synchronization to hang.
- Fixed an issue where service account permission errors would overwrite other errors.
- Fixed an issue that caused extraneous disk usage in the Airflow webserver.
August 30, 2019
Cloud Composer is now available in Sydney (australia-southeast1) and Montreal (northamerica-northeast1).
August 28, 2019
New versions of Cloud Composer images: composer-1.7.5-airflow-1.9.0, composer-1.7.5-airflow-1.10.1, and composer-1.7.5-airflow-1.10.2. The default is composer-1.7.5-airflow-1.9.0. Upgrade your Cloud SDK to use features in this release.
- Fixed unwanted error output from scheduler pod in Private IP Composer environments.
- Improved reliability of Private IP environment creation in crowded IP address spaces.
- Fixed an issue with asynchronous DAG loading when handling Unicode strings.
- Fixed an issue that caused some environment deletion operations to fail.
- Fixed an issue that sometimes prevented the liveness DAG from running when there were failed scheduler pods.
- Improved error messages when corrupted Cloud Storage bucket permissions prevent operations from succeeding.
- Reduced the latency of failed Python package updates when invalid packages are selected.
- Backported AIRFLOW-4015 to support the
GET dag_runsendpoint in Airflow 1.10.2. - Fixed an issue that caused image version upgrades to fail when the Airflow database was too large.
August 22, 2019
New versions of Cloud Composer images: composer-1.7.4-airflow-1.9.0, composer-1.7.4-airflow-1.10.1, and composer-1.7.4-airflow-1.10.2. The default is composer-1.7.4-airflow-1.9.0. Upgrade your Cloud SDK to use features in this release.
- Fixed unwanted error output from scheduler pod in Private IP Composer environments.
- Improved reliability of Private IP environment creation in crowded IP address spaces.
- Fixed an issue with asynchronous DAG loading when handling Unicode strings.
- Fixed an issue that caused some environment deletion operations to fail.
- Fixed an issue that sometimes prevented the liveness DAG from running when there were failed scheduler pods.
- Improved error messages when corrupted Cloud Storage bucket permissions prevent operations from succeeding.
- Reduced the latency of failed Python package updates when invalid packages are selected.
- Backported AIRFLOW-4015 to support the
GET dag_runsendpoint in Airflow 1.10.2. - Fixed an issue that caused image version upgrades to fail when the Airflow database was too large.
August 14, 2019
New versions of Cloud Composer images: composer-1.7.3-airflow-1.9.0, composer-1.7.3-airflow-1.10.1, and composer-1.7.3-airflow-1.10.2. The default is composer-1.7.3-airflow-1.9.0. Upgrade your Cloud SDK to use features in this release.
This release contains only internal reliability improvements.
June 21, 2019
Cloud Composer is now available in London (europe-west2).
June 14, 2019
- New versions of Cloud Composer images:
composer-1.7.2-airflow-1.9.0,composer-1.7.2-airflow-1.10.1, andcomposer-1.7.2-airflow-1.10.2. The default iscomposer-1.7.2-airflow-1.9.0. Upgrade your Cloud SDK to use features in this release. - The Airflow configuration
webserver-workersis now updatable. Available only in Airflow 1.10.2 and later.
- Fixed an asynchronous DAG loading issue that prevented the Airflow web server from loading some DAGs. Available only in
composer-1.7.2-airflow-1.10.2versions and later. - Fixed an issue that caused image version upgrades to fail when migrating particularly large Airflow databases.
- Fixed an issue that sometimes prevented data plane monitoring metrics from being reported in Stackdriver.
- Fixed an issue that caused PyPI package installation to sometimes time out and fail.
May 28, 2019
To display DAGs faster and reduce potential downtime due to heavy DAG processing, Cloud Composer now supports asynchronous DAG loading in the Airflow web server. To enable, set the following Airflow configurations: [webserver]async_dagbag_loader=True and [webserver]worker_refresh_interval=3600. Available only in composer-1.7.1-Airflow-1.10.2 versions and later.
New versions of Cloud Composer images: composer-1.7.1-airflow-1.9.0, composer-1.7.1-airflow-1.10.1, and composer-1.7.1-airflow-1.10.2. The default is composer-1.7.1-airflow-1.9.0. Upgrade your Cloud SDK to use features in this release.
- Backported AIRFLOW-3143 to support auto-zone in DataprocClusterCreateOperator. Available only in
composer-1.7.1-airflow-1.10.2versions and later. - Added back the list of blocked Airflow configurations for Airflow 1.10.2.
May 16, 2019
- Cloud Composer now supports Private Python package repositories. Available only in
composer-1.7.0-*and later versions. - Added support for Apache Airflow 1.10.2.
- Added new health metrics in Stackdriver to monitor your environment:
composer.googleapis.com/environment/healthyandcomposer.googleapis.com/environment/database_health - VPC Native support is now in beta.
- Backported AIRFLOW-2747 to support explicit rescheduling of Airflow sensors. Available only in Airflow 1.10.2 versions and later.
- New versions of Cloud Composer images:
composer-1.7.0-airflow-1.9.0,composer-1.7.0-airflow-1.10.1, andcomposer-1.7.0-airflow-1.10.2. The default iscomposer-1.7.0-airflow-1.9.0. Upgrade your Cloud SDK to use features in this release. - To prevent co-locating workers on the same node, improved Airflow worker pods scheduling.
- When creating a Private IP Cloud Composer environment, the GKE master IP CIDR block no longer needs to be specified.
Starting from
composer-1.7.0-*, Apache Airflow version 1.10.0 is no longer available for environment creation or in-place upgrade.
- Added back the
pingcommand-line utility to the default Cloud Composer managed Docker images. - Backported AIRFLOW-2715 AIRFLOW-2715 to fix the
DataflowTemplateOperatorregion support issue. Available only in Airflow 1.10.2 versions and later. - Fixed an issue that prevented in-place upgrades when the environment service account had only the
roles/composer.workerrole. - Backported fixes for CVE-2019-0216 and CVE-2019-0229.
- Fixed an issue that caused some environment deletion operations to fail.
- Fixed an issue that could cause Cloud Composer workloads to leak into the node pools added for
KubernetesPodOperatorworkloads. - Fixed an issue in which the
airflow_dbconnection did not work correctly for some environments. - Fixed an issue that prevented upgrading a Private IP Cloud Composer environment.
April 04, 2019
Stackdriver metrics for Cloud Composer is in beta. You can now use Stackdriver Monitoring to understand the performance and health of your Cloud Composer environments and examine Airflow metrics. Stackdriver is available for composer-1.6.0-airflow-1.9.0 and later versions.
New versions of Cloud Composer images: composer-1.6.1-airflow-1.9.0, composer-1.6.1-airflow-1.10.0, and composer-1.6.1-airflow-1.10.1. The default is composer-1.6.1-airflow-1.9.0. Upgrade your Cloud SDK to use features in this release.
Stackdriver Monitoring: In rare cases, a TRANSIENT_FAILURE connection error occurs when Cloud Composer writes metrics to Stackdriver Monitoring. Typically, the automatic retry is successful, but occasionally, the failure state persists for an extended period of time.
April 03, 2019
- Private IP Cloud Composer environment is in beta. To isolate your workflows from the public internet, you can now assign only private IP (RFC 1918) addresses to the managed Google Kubernetes Engine and Cloud SQL VMs in your Cloud Composer environment.
- Image version upgrade is in beta. You can now perform an in-place upgrade on the Airflow version or Cloud Composer version that your environment runs.
New versions of Cloud Composer images: composer-1.6.0-airflow-1.9.0, composer-1.6.0-airflow-1.10.0, and composer-1.6.0-airflow-1.10.1. The default is composer-1.6.0-airflow-1.9.0. Upgrade your Cloud SDK to use features in this release.
Increased the operation timeout from 20 to 30 minutes to fix the sporadic, premature PyPI dependency update timeout issue.
- Currently, you cannot perform an image version upgrade.
- Currently, you cannot simultaneously create or delete two Cloud Composer environments on the same subnetwork if one of the two environments is a Private IP Cloud Composer environment.
- You must wait until the first Cloud Composer environment is created before creating the second environment to avoid environment creation failures.
- You must wait until the first Cloud Composer environment is deleted before deleting the second environment to avoid a GKE cluster resource leak. If the leak occurs, manually delete the GKE cluster for the Cloud Composer environment.
airflow_db connection: The airflow_db connection works only for the Airflow webserver. Workarounds are as follows:
- Update the
airflow_dbconnection's Host component toairflow-sqlproxy-service.default. This update breaks the ability to use the connection in the Airflow webserver's Ad Hoc Query page but is preferred for DAGs that rely on theairflow_dbconnection. - Create a new connection that mirrors the
airflow_dbconnection but usesairflow-sqlproxy-service.defaultfor the Host.
New versions of Cloud Composer images: composer-1.4.2-airflow-1.9.0 and composer-1.4.2-airflow-1.10.0.
- Fixed the issue where
max_active_runsandconcurrencyare not updated in the Airflow web UI DAG Details page when configurations are updated. - Fixed the failed
KubernetesPodOperatorandGkePodOperatorwhen a task a runs longer than 1 hour due to authentication issues in the Kuberentes client.
March 20, 2019
You can now configure Shared VPC in the Google Cloud Console.
March 11, 2019
You can now access the Airflow web interface logs in Stackdriver Logging for Cloud Composer under airflow-webserver.
- New versions of Cloud Composer images:
composer-1.5.2-airflow-1.9.0,composer-1.5.2-airflow-1.10.0, andcomposer-1.5.2-airflow-1.10.1. The default iscomposer-1.5.2-airflow-1.9.0. - For the Cloud SQL instance for Cloud Composer, the
wait_timeoutandinteractive_timeoutare reduced to 1800 seconds to enable a faster deadlock recovery if an idle SQL connection holds a database lock. - Added back the openssh-client package to the default Cloud Composer managed docker images.
- Backported the Apache Airflow fix for the SubDag failed by scheduler deadlock issue.
- Fixed an issue where a DAG backfill could not be disabled on a per-DAG basis.
March 07, 2019
You can now access the Airflow web interface logs in Stackdriver Logging for Cloud Composer under airflow-webserver.
- New versions of Cloud Composer images:
composer-1.5.1-airflow-1.9.0,composer-1.5.1-airflow-1.10.0, andcomposer-1.5.1-airflow-1.10.1. The default iscomposer-1.5.1-airflow-1.9.0. - For the Cloud SQL instance for Cloud Composer, the
wait_timeoutandinteractive_timeoutare reduced to 1800 seconds to enable a faster deadlock recovery if an idle SQL connection holds a database lock. - Added back the openssh-client package to the default Cloud Composer managed docker images.
- Backported the Apache Airflow fix for the SubDag failed by scheduler deadlock issue.
- Fixed an issue where a DAG backfill could not be disabled on a per-DAG basis.
February 06, 2019
- New versions of Cloud Composer images:
composer-1.5.0-airflow-1.9.0,composer-1.5.0-airflow-1.10.0, andcomposer-1.5.0-airflow-1.10.1 - Increased the disk size for the Apache Airflow webserver from 10 to 20 GB.
We now install Airflow dependencies of "mysql" instead of "devel".
If your DAG uses dependencies that are available only in "devel", add the dependencies to your DAG as PyPI dependencies.
- Backported Airflow upstream fix for the broken Google Cloud connection in the
DataflowJavaOperatorandDataflowPythonOperatorAIRFLOW-2009 in all Cloud Composer managed Airflow versions. Backported Airflow upstream fixes for CVE-2018-20244: Stored XSS in Apache Airflow in all Cloud Composer managed Airflow versions (1.9.0, 1.10.0, 1.10.1).
Due to this vulnerability, we recommend that you upgrade to the latest Cloud Composer version.
- Multi-version Apache Airflow support is now General Availability. You can select the Cloud Composer image version when you create a new environment without enabling Beta feature support.
- You can now view available Cloud Composer image versions in all Cloud Composer interfaces.
- Added support for Apache Airflow 1.10.1.
January 10, 2019
Added a Kubernetes pod liveness checker for the airflow-sqlproxy service to recover from the cloudsqlproxy process error.
December 19, 2018
Cloud Composer is now available in the Northern Virginia (us-east4) region.
December 17, 2018
- New versions of Cloud Composer images:
composer-1.4.1-airflow-1.9.0andcomposer-1.4.1-airflow-1.10.0. - The Airflow workers restart when a Celery concurrency configuration change is requested so that the change takes effect automatically.
- The
logs/anddata/directories in Cloud Storage are mounted with file mode 755, enabling binaries to be placed in these directories and executed from Airflow. - The GCP_PROJECT environment variable is set to the Google Cloud Platform project ID in the Airflow webserver.
- Fixed the broken Airflow webserver when DAG parsing times out on the webserver side.
- Pinned the Airflow base Docker image tag for initial environment creation and future PyPI dependency installations.
November 26, 2018
GA launch of Python 3 support. Currently, the v1 API and the Google Cloud Console support GA Python 3 environment creation.
New versions of Cloud Composer images: composer-1.4.0-airflow-1.9.0 and composer-1.4.0-airflow-1.10.0.
- Fixed the Stackdriver logging severity level for
airflow-sqlproxyand improved the labeling of logs. - Fixed boundless HTTP header growth in the
set_user_agent googleapiclientmethods in 1.10.0 images.
October 24, 2018
- Multi-version Apache Airflow support is now in Beta. You can choose from multiple versions of Airflow when creating a new Cloud Composer environment. Multiple versions of Airflow can run in the same Google Cloud project simultaneously.
- Cloud Composer is now available in the Mumbai (
asia-south1) region. - The Cloud Composer now displays in the top-right banner of the Airflow webserver UI.
New versions of Cloud Composer images: composer-1.3.0-airflow-1.9.0 and composer-1.3.0-airflow-1.10.0.
October 02, 2018
Support for Python 3 is now in Beta You can create Python 3 Cloud Composer environments and run Python 2 and Python 3 environments in the same Google Cloud project simultaneously. Currently, the v1beta1 API and Google Cloud Console support Python 3 environment creation.
For composer-1.2.0 environments, some commands in Python 2 Composer environments might run under the Python 3 runtime. Operators should explicitly specify python2 to ensure that commands run under Python 2. For more info, see PEP 394.
- New version of the Cloud Composer image:
composer-1.2.0-airflow-1.9.0. - To reduce potential scheduling delays due to DAG processing, the scheduler restart behavior is now time based (every 600 seconds), not run based.
September 17, 2018
- New version of the Cloud Composer image:
composer-1.1.1-airflow-1.9.0. - The following Airflow configurations are now updatable: core-dag_concurrency, core-parallelism, core-max_active_runs_per_dag, and scheduler-max_threads.
August 17, 2018
- Support for GKE Shared VPC is now in Beta. For information about Cloud Composer configuration, see Creating Environments.
- Added the following Airflow updates:
- Backported network, subnetwork, and tags support in
DataprocClusterCreateOperatorto create Cloud Dataproc clusters in user-specified subnetworks. - Backported
GKEPodOperatorto enable launching Kubernetes pods in GKE clusters that are accessible from Google Cloud connections.
- Backported network, subnetwork, and tags support in
- New version of the Cloud Composer image:
composer-1.1.0-airflow-1.9.0. - Cloud Composer environments running composer-1.0.0 or later now have GKE cluster auto-upgrade enabled by default.
- Introduced additional liveness monitoring for the Cloud Composer Stackdriver fluentd agent.
Fixed the following Airflow issues:
- Fixed the issue where the
DataFlowJavaOperatorand theDataFlowPythonOperatormight run continually due to a job name mismatch. - Fixed a rare scheduler bug where the Airflow scheduler freezes by restarting the scheduler if its Stackdriver logs are old.
July 19, 2018
- GA launch of Cloud Composer.
- Cloud Composer is now available in the following regions:
asia-northeast1us-east1
- Revamped Stackdriver Logging as follows:
- Stackdriver Logging now based on fluentd and now includes
composer-agent,airflow-redis, andairflow-sqlproxylogs. - Exposed the Airflow worker and scheduler container logs in Stackdriver Logging, including exceptions and stderr.
- Disabled Logging for Google Kubernetes Engine to avoid duplicate logging in Stackdriver Logging.
- The log format for some raw Airflow logs stored in Cloud Storage now include the delimiter string, dag-id, task-id, and execution date.
- Stackdriver Logging now based on fluentd and now includes
- Added new Identity and Access Management roles.
- The
composer.environmentAndStorageObjectAdminandcomposer.environmentAndStorageObjectViewerroles provide users the permissions necessary to access environments and objects in Cloud Storage buckets. - The
composer.workerrole provides service accounts the permissions necessary to run a Cloud Composer VM.
- The
- Added the following Airflow updates:
- Backported the
KubernetesPodOperatorto enable launching Docker containers in Cloud Composer. - Added
PARQUETsource format support inbigquery_hook. - Backported the JDBC/DBAPI autocommit issue fix.
- Fixed the Airflow documentation link in the Airflow web interface.
- Backported the
- New version of the Cloud Composer image:
composer-1.0.0-airflow-1.9.0. - Cloud Composer environments running composer-1.0.0 or later can now launch Cloud Dataflow jobs in supported Cloud Dataflow regions.
- Added the option to modify some celery configurations.
Fixed the following Airflow issues:
- Fixed the issue where Airflow tasks were stuck in queued or running states for extended periods of time before failing.
- Fixed the Airflow web interface crashlooping issue that resulted from statsd exceptions.
- Revised the Airflow web interface message when the TaskInstance RUN button is clicked. Note that the task instance restarts automatically if the DAG is running and its dependencies are met.
June 27, 2018
- New version of the Cloud Composer image:
composer-0.5.3-airflow-1.9.0. - Added requirement to include
https://www.googleapis.com/auth/cloud-platformwhen specifying OAuth scopes during environment creation. - Improved health checking on Airflow to restart scheduler and worker pods if scheduled tasks are not queued or executed.
- Improved messaging for create and update environment failures.
June 15, 2018
Audit logging is now available in Beta.
- New version of the Cloud Composer image:
composer-0.5.2-airflow-1.9.0. - Added OAuth scope configuration on the Google Cloud Platform Console Environment creation page
- Increased the minimum disk size from 10 GB to 20 GB.
- Improved error reporting.
- Improved stability of the CeleryExecutor Redis message backend.
Fixed the BAD REQUEST error message for environment creation or deletion failures to provide information about the failures.
May 17, 2018
New version of the Cloud Composer image: composer-0.5.1-airflow-1.9.0.
- Fixed permanent environment
DELETEerror that occurred when the Google Kubernetes Engine cluster is deleted or that occurred when in an error state prior to environment deletion. - Removed inapplicable "Failed to update environment from json" error message from worker/scheduler logs.
May 01, 2018
Beta launch of Cloud Composer (composer-0.5.0-airflow-1.9.0). This release includes Apache Airflow 1.9.0.