Run a workflow that executes other workflows in parallel

This tutorial shows you how to create and run a parent workflow that executes multiple child workflows in parallel.

In the following diagram, four parallel executions of the child workflow are invoked. This allows the parent workflow to process data in parallel branches, and reduces the overall execution time. The parent workflow waits for all the child workflow executions to finish before returning a summary of successful and failed executions, simplifying any error detection.

Parent workflow invoking parallel iterations of a child workflow

Create and deploy a child workflow

A child workflow can receive and process data from a parent workflow. The child workflow demonstrates this by doing the following:

  • Receives an integer as an argument
  • Sleeps for 10 seconds to simulate some processing

  • Returns an indicator (based on whether the integer is an odd or even number) to simulate success or failure of the workflow execution

Console

  1. In the Google Cloud console, go to the Workflows page.

    Go to Workflows

  2. Click Create.

  3. Enter the name, workflow-child, for the new workflow.

  4. In the Region list, select us-central1.

  5. Select the Service account you previously created.

  6. Click Next.

  7. In the workflow editor, enter the following definition for your workflow:

    main:
  params: [args]
  steps:
    - init:
        assign:
          - iteration : ${args.iteration}
    - wait:
        call: sys.sleep
        args:
            seconds: 10
    - check_iteration_even_or_odd:
        switch:
          - condition: ${iteration % 2 == 0}
            next: raise_error
    - return_message:
        return: ${"Hello world"+iteration}
    - raise_error:
        raise: ${"Error with iteration "+iteration}

  8. Click Deploy.

gcloud

  1. Create a source code file for your workflow:

    touch workflow-child.yaml

  2. Open your source code file in a text editor and copy the following workflow to the file.

    main:
  params: [args]
  steps:
    - init:
        assign:
          - iteration : ${args.iteration}
    - wait:
        call: sys.sleep
        args:
            seconds: 10
    - check_iteration_even_or_odd:
        switch:
          - condition: ${iteration % 2 == 0}
            next: raise_error
    - return_message:
        return: ${"Hello world"+iteration}
    - raise_error:
        raise: ${"Error with iteration "+iteration}

  3. Deploy the workflow:

    gcloud workflows deploy workflow-child \
    --source=workflow-child.yaml \
    --location=us-central1 \
    --service-account=SERVICE_ACCOUNT_NAME@PROJECT_ID.iam.gserviceaccount.com

    Replace SERVICE_ACCOUNT_NAME with the name of the service account you previously created.

Create and deploy the parent workflow

The parent workflow executes multiple branches of the child workflow using a parallel for loop.

  1. Copy the source code for the workflow definition. It consists of the following parts:

    1. A map is used to store the results of the child workflow executions. For more information, see Maps.

      main:
  steps:
    - init:
        assign:
          - execution_results: {} # results from each execution
          - execution_results.success: {} # successful executions saved under 'success' key
          - execution_results.failure: {} # failed executions saved under 'failure' key

    2. A for loop is executed in parallel to invoke the child workflow. For more information, see Parallel steps and Iteration.

      - execute_child_workflows:
    parallel:
      shared: [execution_results]
      for:
        value: iteration
        in: [1, 2, 3, 4]
        steps:
            - iterate:

    3. The child workflow is invoked using a connector. Each iteration of the child workflow is passed the iteration argument. The parent workflow waits for and stores the result of each child workflow execution. For more information, see Workflows Executions API connector and Runtime arguments.

      try:
  steps:
    - execute_child_workflow:
        call: googleapis.workflowexecutions.v1.projects.locations.workflows.executions.run
        args:
          workflow_id: workflow-child
          #location: ...
          #project_id: ...
          argument:
            iteration: ${iteration}
        result: execution_result
    - save_successful_execution:
        assign:
          - execution_results.success[string(iteration)]: ${execution_result}
except:
    as: e
    steps:
      - save_failed_execution:
          assign:
            - execution_results.failure[string(iteration)]: ${e}

    4. The execution results are returned. For more information, see Complete the execution of a workflow.

      - return_execution_results:
    return: ${execution_results}

  2. Deploy the workflow:

    Console

    1. In the Google Cloud console, go to the Workflows page:

      Go to Workflows

    2. Click Create.

    3. Enter the name, workflow-parent, for the new workflow.

    4. In the Region list, select us-central1.

    5. Select the Service account you previously created.

    6. Click Next.

    7. In the workflow editor, paste the definition for the parent workflow.

    8. Click Deploy.

    gcloud

    1. Create a source code file for your workflow:

      touch workflow-parent.yaml

    2. Open your source code file in a text editor and paste the definition for the parent workflow.

    3. Deploy the workflow:

      gcloud workflows deploy workflow-parent \
    --source=workflow-parent.yaml \
    --location=us-central1 \
    --service-account=SERVICE_ACCOUNT_NAME@PROJECT_ID.iam.gserviceaccount.com

      Replace SERVICE_ACCOUNT_NAME with the name of the service account you previously created.

Execute the parent workflow

Execute the parent workflow so that invocations of the child workflow run in parallel. The executions should take about about 10 seconds to complete.

Console

  1. In the Google Cloud console, go to the Workflows page:

    Go to Workflows

  2. On the Workflows page, click the workflow-parent workflow to go to its details page.

  3. On the Workflow details page, click Execute.

  4. Click Execute again.

  5. View the results of the workflow in the Output pane.

    The results should be similar to the following, indicating errors with iterations 2 and 4, and success with iterations 1 and 3.

    "failure": {
  "2": {
    "message": "Execution failed or cancelled.",
    "operation": {
      "argument": "{\"iteration\":2}",
      "duration": "10.157992541s",
      "endTime": "2023-07-11T13:13:13.028424329Z",
      "error": {
        "context": "RuntimeError: \"Error with iteration 2\"\nin step \"raise_error\", routine \"main\", line: 18",
        "payload": "\"Error with iteration 2\"",
...
  "4": {
    "message": "Execution failed or cancelled.",
    "operation": {
      "argument": "{\"iteration\":4}",
      "duration": "10.157929734s",
      "endTime": "2023-07-11T13:13:13.061289142Z",
      "error": {
        "context": "RuntimeError: \"Error with iteration 4\"\nin step \"raise_error\", routine \"main\", line: 18",
        "payload": "\"Error with iteration 4\"",
...
"success": {
  "1": "Hello world1",
  "3": "Hello world3"

gcloud

Execute the workflow:

gcloud workflows run workflow-parent \
    --location=us-central1

The results should be similar to the following, indicating errors with iterations 2 and 4, and success with iterations 1 and 3.

Waiting for execution [06c753e4-6947-4c62-ac0b-2a9d53fb1b8f] to complete...done.
argument: 'null'
duration: 14.065415004s
endTime: '2023-07-11T12:50:43.929023883Z'
name: projects/386837416586/locations/us-central1/workflows/workflow-parent/executions/06c753e4-6947-4c62-ac0b-2a9d53fb1b8f
result: '{"failure":{"2":{"message":"Execution failed or cancelled.","operation":{"argument":"{\"iteration\":2}","duration":"10.143718070s","endTime":"2023-07-11T12:50:40.673209821Z","error":{"context":"RuntimeError:
...
"Error with iteration 2\"\nin step \"raise_error\", routine \"main\", line: 18","payload":"\"Error
...
"Error with iteration 4\"\nin step \"raise_error\", routine \"main\", line: 18","payload":"\"Error
...
"success":{"1":"Hello world1","3":"Hello world3"}}'
startTime: '2023-07-11T12:50:29.863608879Z'
state: SUCCEEDED

You have successfully created and deployed a workflow that invokes a child workflow, executes four iterations of the child workflow in parallel branches, and returns an indicator of success or failure for each child workflow execution.