Use Queue Management or queue.yaml

This page explains the differences between using the Cloud Tasks API to manage queues and using the upload of a Cloud Tasks queue.yaml file to accomplish the same ends. It also discusses some of the pitfalls of mixing mechanisms and how to deal with common problems.

Introduction

The Cloud Tasks API provides an App Engine-independent interface to the App Engine Task Queue service. As part of that interface, it provides the ability to manage queues, including doing so through the console or thegcloud command. Queues that are created by Cloud Tasks API are accessible from the App Engine SDK and vice versa. To maintain compatibility, it is possible to use the configuration file used by the App Engine SDK, queue.yaml, to also create and configure queues to be used through the Cloud Tasks API. However, mixing configuration via file with configuration through the Cloud Tasks API can produce unexpected consequences.

Pitfalls of mixing queue.yaml with Cloud Tasks queue management methods

For the underlying service,queue.yaml files are definitive. Uploading a queue.yaml that omits existing queues in your project, no matter how they were created, causes those queues to be disabled, or paused. Thus if you use the Cloud Tasks API to call CreateQueue or UpdateQueue and then upload a queue.yaml file that omits them, the queues that were created in the Cloud Tasks calls are disabled.

Consider the following scenario:

1. Call CreateQueue to create a queue named "cloud-tasks-queue".

2. Upload a queue.yaml file with the following contents:

   queue:
   - name: queue-yaml-queue
   

What is the current state of queues in this project? The queue named "cloud-tasks-queue" and any other queues that existed prior are in DISABLED state, and the queue named "queue-yaml-queue" is in RUNNING state.

Note: The default queue is an exception. It is never disabled by App Engine. See Cloud Tasks and the default App Engine queue for more details.

This behavior might be surprising if you create queues through the Cloud Tasks API. The instructions below explain how to resume a disabled queue.

Similarly, if a queue is disabled in the Cloud Tasks API but later appears in an uploaded queue.yaml file, that queue is resumed.

If a queue is deleted with the DeleteQueue method and later appears in a queue.yaml file, the queue.yaml upload can fail because queue names are not allowed to be reused for several days after deletion.

Best practices

Caution: It is strongly recommended that you use either the configuration file method or the Cloud Tasks API to configure your queues, but not both.

If you are new to Cloud Tasks or App Engine, use the Cloud Tasks API exclusively to manage your queues and avoid the use of queue.yaml altogether. Cloud Tasks queue management methods give users more choice in creating, updating, and deleting queues.

If, however, you are an existing queue.yaml user, you should only consider switching to queue management methods if you understand the pitfalls of mixing queue.yaml with Cloud Tasks queue management methods.

To prevent users from mixing task management methods, one option is to create a web app or command line tool that all users must use to create, update, and delete queues. Whether that tool uses Cloud Tasks queue management methods or queue.yaml is an implementation detail of the tool that users do not need to worry about. If users are required to use the tool, then you can guarantee that there is no inadvertent mixing of Cloud Tasks queue management methods and queue.yaml use. To help enforce use of such a tool, you can grant queue admin roles to the tool and require users to authenticate to use the tool. To learn more about access management, see Secure queue configuration.

Debugging

You can inspect your project's Admin Activity audit logs to retrieve the history of queue configuration changes including queue creations, updates, and deletions:

    gcloud logging read \
      'protoPayload.methodName=
       (com.google.appengine.legacy.queue_created OR
        com.google.appengine.legacy.queue_updated OR
        google.cloud.tasks.v2.CloudTasks.CreateQueue OR
        google.cloud.tasks.v2.CloudTasks.UpdateQueue OR
        google.cloud.tasks.v2.CloudTasks.DeleteQueue)'

For example, if an existing queue is disabled by a queue.yaml upload, a "Disabled queue '[QUEUE_NAME]'" message would appear in the audit log through the com.google.appengine.legacy.queue_updated method.

How to resume a queue disabled by a queue.yaml upload

If you mix queue.yaml with Cloud Tasks queue management methods, uploading a queue.yaml file might accidentally disable a queue created through the Cloud Tasks API.

To resume the queue, you can either call ResumeQueue on the queue or add it to queue.yaml and upload. Be aware that if you had previously set a custom processing rate in the queue.yaml configuration for the queue, ResumeQueue resets the queue to the default rate. This is reflected in the maxDispatchesPerSecond field of the response to ResumeQueue.

Warning: Resuming many high-QPS queues at the same time can lead to target overloading. If you are resuming high-QPS queues, follow the 500/50/5 pattern described in Manage scaling risks.

Quotas

If you use queue.yaml to create your queues, by default you can create a maximum of 100 queues. Queues created using the Cloud Tasks API have a default maximum of 1,000 queues. As in other cases, mixing queue.yaml and Cloud Tasks API methods can produce unexpected results. For example, suppose you create some queues using queue.yaml, and then get a quota increase to, for example, 2,000. If you then subsequently use the Cloud Tasks API method of creating further queues, you will get out of quota errors. To remedy this, file a request using Edit Quotas from the Quotas page of the Google Cloud console.

Additional information about Cloud Tasks queue management methods

Queue configuration and queue startup delay

Changes to queue configuration can require several minutes to take effect. For example, after calling CreateQueue or UpdateQueue, several minutes might pass before you can successfully call CreateTask on that queue.

Cloud Tasks and the default App Engine queue

The App Engine queue named "default" is given special treatment in the App Engine SDK and in the Cloud Tasks API.

If the default queue does not already exist, it is created in the following situations:

1. When a task is first added to the default queue using the App Engine SDK.
2. When a queue.yaml file that specifies a default queue is uploaded.
3. When CreateQueue or UpdateQueue is called to create the default queue.

To preserve compatibility with App Engine, Cloud Tasks enforces the following restrictions:

1. If a queue named "default" is created, it must be a queue using App Engine tasks.
2. Once created, users cannot delete the default queue.

In the Cloud Tasks API, the following also applies to the default queue:

1. The Cloud Tasks API does not automatically create the default queue or any other queues.
2. Just like any other queue, calling GetQueue on the default queue results in a not found error if the call is made before the queue is created.
3. Similarly, the default queue does not appear in the output of ListQueues before it is created.
4. The configuration of the default queue can be changed with the UpdateQueue call.

What's next

• See the methods available in the RPC Cloud Tasks API in the reference documents.
• See the methods available in the REST Cloud Tasks API in the reference documents.
• Read about queue.yaml.