Use a Cloud Tasks queue to buffer your workflow executions

This tutorial shows you how to create a Cloud Tasks queue that can regulate the rate of workflow executions.

There is a maximum number of active workflow executions that can happen concurrently. Once this quota is exhausted, and if execution backlogging is disabled, or if the quota for backlogged executions is reached, any new executions fail with an HTTP 429 Too many requests status code. By enabling a Cloud Tasks queue to execute child workflows at a rate that you define, you can avoid Workflows quota-related issues and achieve a better execution rate.

Note that Cloud Tasks is designed to provide "at least once" delivery; however, Workflows doesn't ensure exactly-once processing of duplicate requests from Cloud Tasks.

In the following diagram, a parent workflow invokes child workflows that are regulated by a Cloud Tasks queue that has a dispatch rate applied.

Parent workflow invoking iterations of a child workflow through Cloud Tasks Queue

Create a Cloud Tasks queue

Create a Cloud Tasks queue that you can use in the parent workflow and that lets you regulate the rate of workflow executions.

Console

  1. In the Google Cloud console, go to the Cloud Tasks page:

    Go to Cloud Tasks

  2. Click Create push queue.

  3. Enter the Queue name, queue-workflow-child.

  4. In the Region list, select us-central1 (Iowa).

  5. Click Create.

gcloud

QUEUE=queue-workflow-child
LOCATION=us-central1
gcloud tasks queues create $QUEUE --location=$LOCATION

Create and deploy a child workflow

A child workflow can receive and process data from a parent workflow. Create and deploy a child workflow that does the following:

  • Receives an iteration as an argument
  • Sleeps for 10 seconds to simulate some processing

  • Returns a string upon successful execution

Console

  1. In the Google Cloud console, go to the Workflows page.

    Go to Workflows

  2. Click Create.

  3. Enter the name, workflow-child, for the new workflow.

  4. In the Region list, select us-central1 (Iowa).

  5. In the Service account list, select the Compute Engine default service account.

  6. Click Next.

  7. In the workflow editor, enter the following definition for your workflow:

    main:
  params: [args]
  steps:
    - init:
        assign:
          - iteration : ${args.iteration}
    - wait:
        call: sys.sleep
        args:
            seconds: 10
    - return_message:
        return: ${"Hello world"+iteration}

  8. Click Deploy.

gcloud

  1. Create a source code file for your workflow:

    touch workflow-child.yaml

  2. Open your source code file in a text editor and copy the following workflow to the file.

    main:
  params: [args]
  steps:
    - init:
        assign:
          - iteration : ${args.iteration}
    - wait:
        call: sys.sleep
        args:
            seconds: 10
    - return_message:
        return: ${"Hello world"+iteration}

  3. Deploy the workflow:

    gcloud workflows deploy workflow-child \
    --source=workflow-child.yaml \
    --location=us-central1 \
    --service-account=PROJECT_NUMBER-compute@developer.gserviceaccount.com

Create and deploy the parent workflow

The parent workflow executes multiple branches of the child workflow using a for loop.

  1. Copy the source code that defines the parent workflow:

    main:
  steps:
    - init:
        assign:
          - project_id: ${sys.get_env("GOOGLE_CLOUD_PROJECT_ID")}
          - project_number: ${sys.get_env("GOOGLE_CLOUD_PROJECT_NUMBER")}
          - location: ${sys.get_env("GOOGLE_CLOUD_LOCATION")}
          - workflow_child_name: "workflow-child"
          - queue_name: "queue-workflow-child"
    - enqueue_tasks_to_execute_child_workflow:
        for:
          value: iteration
          range: [1, 100]
          steps:
              - iterate:
                  assign:
                    - data:
                        iteration: ${iteration}
                    - exec:
                        # Encode object to JSON string in expression for workflow argument
                        argument: ${json.encode_to_string(data)}
              - create_task_to_execute_child_workflow:
                  call: googleapis.cloudtasks.v2.projects.locations.queues.tasks.create
                  args:
                      parent: ${"projects/" + project_id + "/locations/" + location + "/queues/" + queue_name}
                      body:
                        task:
                          httpRequest:
                            body: ${base64.encode(json.encode(exec))}
                            url: ${"https://workflowexecutions.googleapis.com/v1/projects/" + project_id + "/locations/" + location + "/workflows/" + workflow_child_name + "/executions"}
                            oauthToken:
                              serviceAccountEmail: ${project_number + "-compute@developer.gserviceaccount.com"}

    The workflow consists of the following parts:

    • A map that is used to assign constants referring to the child workflow and the Cloud Tasks queue name. For more information, see Maps.

    • A for loop that is executed to invoke the child workflow, iteratively. For more information, see Iteration.

    • A workflow step that creates and adds a large number of tasks to the Cloud Tasks queue to execute the child workflow. For more information, see Cloud Tasks API connector.

  2. Deploy the workflow:

    Console

    1. In the Google Cloud console, go to the Workflows page:

      Go to Workflows

    2. Click Create.

    3. Enter the name, workflow-parent, for the new workflow.

    4. In the Region list, select us-central1 (Iowa).

    5. In the Service account list, select the Compute Engine default service account.

    6. Click Next.

    7. In the workflow editor, paste the definition for the parent workflow.

    8. Click Deploy.

    gcloud

    1. Create a source code file for your workflow:

      touch workflow-parent.yaml

    2. Open your source code file in a text editor and paste the definition for the parent workflow.

    3. Deploy the workflow:

      gcloud workflows deploy workflow-parent \
    --source=workflow-parent.yaml \
    --location=us-central1 \
    --service-account=PROJECT_NUMBER-compute@developer.gserviceaccount.com

Execute the parent workflow without rate limits

Execute the parent workflow to invoke the child workflows through the Cloud Tasks queue. The executions should take about 10 seconds to complete.

Console

  1. In the Google Cloud console, go to the Workflows page:

    Go to Workflows

  2. On the Workflows page, click the workflow-parent workflow to go to its details page.

  3. On the Workflow details page, click Execute.

  4. Click Execute again.

  5. As the parent workflow is running, return to the Workflows page, and click the workflow-child workflow to go to its details page.

  6. Click the Executions tab.

    You should see executions of the child workflow, running around the same time, similar to the following:

    Details of the child workflow executions running around the same time.

gcloud

  1. Execute the workflow:

    gcloud workflows run workflow-parent \
     --location=us-central1

  2. To verify that a workflow execution was triggered, list the last four executions:

    gcloud workflows executions list workflow-child --limit=4

    Since the number of executions (100 ) is below the Workflows concurrency limit, the results should be similar to the following. Quota issues might arise if you submit thousands of executions at the same time. 

    NAME: projects/620278351741/locations/us-central1/workflows/workflow-child/executions/1570d06e-d133-4536-a859-b7b6a1a85524
STATE: ACTIVE
START_TIME: 2023-07-27T00:56:15.093934448Z
END_TIME:
NAME: projects/620278351741/locations/us-central1/workflows/workflow-child/executions/82724960-7d92-4961-aa2c-a0f0be46212c
STATE: ACTIVE
START_TIME: 2023-07-27T00:56:14.903007626Z
END_TIME:
NAME: projects/620278351741/locations/us-central1/workflows/workflow-child/executions/598126fb-37f9-45bc-91d8-aea7d795d702
STATE: ACTIVE
START_TIME: 2023-07-27T00:56:14.698260524Z
END_TIME:
NAME: projects/620278351741/locations/us-central1/workflows/workflow-child/executions/d2e9960b-f93f-4df4-a594-3e7e5c2be53f
STATE: ACTIVE
START_TIME: 2023-07-27T00:56:14.503818840Z
END_TIME:

You have created and deployed a workflow that invokes 100 iterations of the child workflow.

Execute the parent workflow with rate limits

Apply a rate limit of one dispatch per second to the Cloud Tasks queue and then execute the parent workflow.

Console

  1. In the Google Cloud console, go to the Cloud Tasks page:

    Go to Cloud Tasks

  2. Click queue-workflow-child, the Cloud Tasks queue you created, and click Edit queue.

  3. In the Rate limits for task dispatches section, for the Max dispatches field, type 1.

  4. Click Save.

  5. Go to the Workflows page:

    Go to Workflows

  6. Click the workflow-parent workflow to go to its details page.

  7. On the Workflow details page, click Execute.

  8. Click Execute again.

  9. As the parent workflow is running, return to the Workflows page, and click the workflow-child workflow to go to its details page.

  10. Click the Executions tab.

    You should see executions of the child workflow, running at one request per second, similar to the following:

    Details of the child workflow executing at request per second.

gcloud

  1. Update the Cloud Tasks queue to apply a rate limit of one dispatch per second:

    gcloud tasks queues update $QUEUE \
    --max-dispatches-per-second=1 \
    --location=us-central1

  2. Execute the workflow:

    gcloud workflows run workflow-parent \
   --location=us-central1

  3. To verify that a workflow execution was triggered, list the last four executions:

    gcloud workflows executions list workflow-child --limit=4

    The results should be similar to the following, with one workflow executed per second: 

    NAME: projects/620278351741/locations/us-central1/workflows/workflow-child/executions/becf4957-9fb2-40d9-835d-0ff2dd0c1249
STATE: ACTIVE
START_TIME: 2023-07-27T01:07:24.446361457Z
END_TIME:
NAME: projects/620278351741/locations/us-central1/workflows/workflow-child/executions/6c1e7c4b-7ac6-4121-b351-1e2d56d10903
STATE: ACTIVE
START_TIME: 2023-07-27T01:07:23.448213989Z
END_TIME:
NAME: projects/620278351741/locations/us-central1/workflows/workflow-child/executions/f2ba5027-af40-4cd3-8cd0-b8033bcc6211
STATE: ACTIVE
START_TIME: 2023-07-27T01:07:22.431485914Z
END_TIME:
NAME: projects/620278351741/locations/us-central1/workflows/workflow-child/executions/ecc61ee5-fe87-49eb-8803-89dba929f6c8
STATE: ACTIVE
START_TIME: 2023-07-27T01:07:21.443466369Z
END_TIME:

You have successfully deployed a workflow that invokes 100 iterations of the child workflow with a dispatch rate of one execution per second.