Create and run Batch jobs using Terraform and Cloud Scheduler

This tutorial explains how you can use Terraform to create and run Batch jobs by using a Cloud Scheduler cron job.

Terraform is an open-source tool that lets you provision and manage infrastructure by specifying the desired state in configuration files. These files that can be treated as code and stored in version control systems like GitHub.

Although Terraform doesn't have resources for Batch, this tutorial shows how you can use Terraform to create Batch jobs. Specifically, you can use Terraform to schedule and run a Cloud Scheduler cron job that targets the Batch API to create and run Batch jobs. Cloud Scheduler is a Google Cloud service that allows you to automatically schedule cron jobs and supports Terraform.

This tutorial is intended for Batch users who already manage infrastructure with Terraform and want to incorporate Batch jobs into Terraform.

Create the Terraform directory and configuration file

Create a directory for Terraform and a configuration file that defines the resources that you want create or update using Terraform. The example configuration file for this tutorial defines a Cloud Scheduler cron job named batch-job-invoker. When it is enabled, the batch-job-invoker cron job runs every 5 minutes to create a new instance of the defined Batch job.

1. To create a directory and a new Terraform configuration (.tf) file within that directory, type the following command, and then press Enter:

   mkdir terraform && cd terraform && cat > main.tf
   

   This command creates the terraform directory, navigates you to it, and starts defining a new main.tf configuration file on the next line.

2. Copy and paste the following Terraform configuration:

   # define variables
   variable "project_id" {
     type        = string
     description = "The project name to use."
     default = "PROJECT_ID"
   }
   
   variable "project_number" {
     type        = string
     description = "The project number to use."
     default = "PROJECT_NUMBER"
   }
   
   variable "region" {
     type        = string
     description = "The region where resources are created."
     default = "us-central1"
   }
   
   variable "cloud_scheduler_service_account_email" {
     type        = string
     description = "The service account email."
     default = "CLOUD_SCHEDULER_SERVICE_ACCOUNT_EMAIL"
   }
   
   variable "batch_service_account_email" {
     type        = string
     description = "The service account email."
     default = "BATCH_SERVICE_ACCOUNT_EMAIL"
   }
   
   # define a Cloud Scheduler cron job which triggers Batch jobs
   resource "google_cloud_scheduler_job" "batch-job-invoker" {
     paused           = false # this cron job is enabled
     name             = "batch-job-invoker"
     project          = var.project_id
     region           = var.region
     schedule         = "*/5 * * * *" # when enabled, run every 5 minutes
     time_zone        = "America/Los_Angeles"
     attempt_deadline = "180s"
   
     retry_config {
       max_doublings        = 5
       max_retry_duration   = "0s"
       max_backoff_duration = "3600s"
       min_backoff_duration = "5s"
     }
   
     # when this cron job runs, create and run a Batch job
     http_target {
       http_method = "POST"
       uri = "https://batch.googleapis.com/v1/projects/${var.project_id}/locations/${var.region}/jobs"
       headers = {
         "Content-Type" = "application/json"
         "User-Agent"   = "Google-Cloud-Scheduler"
       }
       # Batch job definition
       body = base64encode(<<EOT
       {
         "taskGroups":[
           {
             "taskSpec": {
               "runnables":{
                 "script": {
                   "text": "echo Hello world! This job was created using Terraform and Cloud Scheduler."
                 }
               }
             }
           }
         ],
         "allocationPolicy": {
           "serviceAccount": {
             "email": "${var.batch_service_account_email}"
           }
         },
         "labels": {
           "source": "terraform_and_cloud_scheduler_tutorial"
         },
         "logsPolicy": {
           "destination": "CLOUD_LOGGING"
         }
       }
       EOT
       )
       oauth_token {
         scope                 = "https://www.googleapis.com/auth/cloud-platform"
         service_account_email = var.cloud_scheduler_service_account_email
       }
     }
   }
   
   

   Replace the following:

   • PROJECT_ID: the project ID of your project.
   • PROJECT_NUMBER: the project number of your project.

   • CLOUD_SCHEDULER_SERVICE_ACCOUNT_EMAIL: the email address of the service account that you prepared for the Cloud Scheduler cron job.

     For example, to use the Compute Engine default service account, specify the following:

     [e-mail]
     

   • BATCH_SERVICE_ACCOUNT_EMAIL: the email address of the service account that you prepared for Batch jobs.

     For example, to use the Compute Engine default service account, specify the following:

     [e-mail]
     

   This Terraform configuration defines some input variables and a cron job that contacts the API method for creating a Batch job.

   Objective: If you want to learn more about how this Terraform configuration is defined and how to write your own, see the following pages:

   • To learn more about the fields that define variables, see the documentation for Terraform input variables.
   • To learn more about the fields that define the cron job, see the documentation for the google_cloud_scheduler_job Terraform resource.
   • To learn more about the fields that request to create a Batch job, see the documentation for the jobs.create Batch API method.

3. To save and close the file, press Ctrl+D (or Command+D on macOS).

Deploy the Terraform configuration to create the cron job

Deploy the Terraform configuration by initializing Terraform, generating the planned changes, and applying these changes. After deploying the Terraform configuration, you can describe the resources in your project to verify that Terraform successfully created the batch-job-invoker cron job.

1. Initialize Terraform in the directory:

   terraform init
   

   The output is similar to the following:

   ...
   Terraform has been successfully initialized!
   
   You may now begin working with Terraform. Try running "terraform plan" to see
   any changes that are required for your infrastructure. All Terraform commands
   should now work.
   
   If you ever set or change modules or backend configuration for Terraform,
   rerun this command to reinitialize your working directory. If you forget, other
   commands will detect it and remind you to do so if necessary.
   

2. Generate the Terraform execution plan based on the current state of your project and the configuration file:

   terraform plan
   

   The output is similar to the following, which shows that the plan is to create the batch-job-invoker cron job:

   Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
     + create
   
   Terraform will perform the following actions:
   
     # google_cloud_scheduler_job.batch-job-invoker will be created
     + resource "google_cloud_scheduler_job" "batch-job-invoker" {
         + id        = (known after apply)
         + name      = "batch-job-invoker"
         + paused    = false
         + project   = "PROJECT_ID"
         + region    = "us-central1"
         + schedule  = "*/5 * * * *"
         + state     = (known after apply)
         + time_zone = "America/Los_Angeles"
   
         + http_target {
             + body        = "..."
             + headers     = {
                 + "Content-Type" = "application/json"
                 + "User-Agent"   = "Google-Cloud-Scheduler"
               }
             + http_method = "POST"
             + uri         = "https://batch.googleapis.com/v1/projects/PROJECT_ID/locations/us-central1/jobs"
   
             + oauth_token {
                 + scope                 = "https://www.googleapis.com/auth/cloud-platform"
                 + service_account_email = "CLOUD_SCHEDULER_SERVICE_ACCOUNT_EMAIL"
               }
           }
   
         + retry_config {
             + max_backoff_duration = "3600s"
             + max_doublings        = 5
             + max_retry_duration   = "0s"
             + min_backoff_duration = "5s"
             + retry_count          = (known after apply)
           }
       }
   
   Plan: 1 to add, 0 to change, 0 to destroy.
   
   ───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
   
   Note: You didn't use the -out option to save this plan, so Terraform can't guarantee to take exactly these actions if you run "terraform apply" now.
   

3. To apply the plan to create the batch-job-invoker cron job, follow these steps:

   1. Enter the following command:

      terraform apply
      

      The output is similar to the previous terraform plan command except that it ends with a confirmation prompt.

   2. To confirm and apply the plan, enter yes.

      The output is similar to the following:

      google_cloud_scheduler_job.batch-job-invoker: Creating...
      google_cloud_scheduler_job.batch-job-invoker: Creation complete after 0s [id=projects/PROJECT_ID/locations/us-central1/jobs/batch-job-invoker]
      
      Apply complete! Resources: 1 added, 0 changed, 0 destroyed.
      

4. To verify that the batch-job-invoker cron job exists and is enabled, describe it:

   gcloud scheduler jobs describe batch-job-invoker --location us-central1
   

   The output is similar to the following:

   attemptDeadline: 180s
   httpTarget:
     body: ...
     headers:
       Content-Type: application/json
       User-Agent: Google-Cloud-Scheduler
     httpMethod: POST
     oauthToken:
       scope: https://www.googleapis.com/auth/cloud-platform
       serviceAccountEmail: CLOUD_SCHEDULER_SERVICE_ACCOUNT_EMAIL
     uri: https://batch.googleapis.com/v1/projects/PROJECT_ID/locations/us-central1/jobs
   lastAttemptTime: '...'
   name: projects/PROJECT_ID/locations/us-central1/jobs/batch-job-invoker
   retryConfig:
     maxBackoffDuration: 3600s
     maxDoublings: 5
     maxRetryDuration: 0s
     minBackoffDuration: 5s
   schedule: '*/5 * * * *'
   scheduleTime: '...'
   state: ENABLED
   status: {}
   timeZone: America/Los_Angeles
   userUpdateTime: '...'
   

   In the output, verify that the state field is set to ENABLED.

Verify that the cron job creates a Batch job

Verify that the batch-job-invoker cron job is correctly creating Batch jobs.

1. Either wait 5 minutes for the cron job to run automatically or trigger the cron job to run immediately:

   gcloud scheduler jobs run batch-job-invoker --location us-central1
   

2. List the Batch jobs that have been created by the batch-job-invoker cron job:

   gcloud batch jobs list \
   --filter labels.source=\"terraform_and_cloud_scheduler_tutorial\" \
   --sort-by ~createTime
   

   • The --filter labels.source=\"terraform_and_cloud_scheduler_tutorial\" flag filters the list to only include Batch jobs that have a label with the key source and the value terraform_and_cloud_scheduler_tutorial.
   • The --sort-by ~createTime flag sorts the list from newest to oldest.

Update the Terraform configuration to pause the cron job

After you have the desired number of Batch jobs, update and deploy the Terraform configuration to pause the batch-job-invoker cron job. If you want to update other properties of the cron job or future Batch jobs, this same process applies.

Note: For recurring workloads, you don't need to pause the Cloud Scheduler cron job if you want to continue to automatically create Batch jobs. For example, you could configure the cron job schedule to run a Batch job on the first day of each month.

1. Update the Terraform configuration file to pause the cron job by setting the paused field to true:

   sed -i 's/paused           = false # this cron job is enabled/paused           = true # this cron job is paused/g' main.tf
   

2. Generate the Terraform execution plan based on the current state of your project and the configuration file:

   terraform plan
   

   The output is similar to the following, which shows that the plan is to update the value of paused field from false to true:

   google_cloud_scheduler_job.batch-job-invoker: Refreshing state... [id=projects/PROJECT_ID/locations/us-central1/jobs/batch-job-invoker]
   
   Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
     ~ update in-place
   
   Terraform will perform the following actions:
   
     # google_cloud_scheduler_job.batch-job-invoker will be updated in-place
     ~ resource "google_cloud_scheduler_job" "batch-job-invoker" {
           id               = "projects/PROJECT_ID/locations/us-central1/jobs/batch-job-invoker"
           name             = "batch-job-invoker"
         ~ paused           = false -> true
           # (6 unchanged attributes hidden)
   
         ~ http_target {
             ~ headers     = {
                 + "User-Agent"   = "Google-Cloud-Scheduler"
                   # (1 unchanged element hidden)
               }
               # (3 unchanged attributes hidden)
   
               # (1 unchanged block hidden)
           }
   
           # (1 unchanged block hidden)
       }
   
   Plan: 0 to add, 1 to change, 0 to destroy.
   
   ────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
   
   Note: You didn't use the -out option to save this plan, so Terraform can't guarantee to take exactly these actions if you run "terraform apply" now.
   

3. To apply the plan to update the batch-job-invoker cron job, follow these steps:

   1. Enter the following command:

      terraform apply
      

      The output is similar to the previous terraform plan command except that it ends with a confirmation prompt.

   2. To confirm and apply the plan, enter yes.

      The output is similar to the following:

      google_cloud_scheduler_job.batch-job-invoker: Modifying... [id=projects/PROJECT_ID/locations/us-central1/jobs/batch-job-invoker]
      google_cloud_scheduler_job.batch-job-invoker: Modifications complete after 1s [id=projects/PROJECT_ID/locations/us-central1/jobs/batch-job-invoker]
      
      Apply complete! Resources: 0 added, 1 changed, 0 destroyed.
      

4. To verify that the batch-job-invoker cron job is paused, describe it:

   gcloud scheduler jobs describe batch-job-invoker --location us-central1
   

   The output is similar to the following:

   attemptDeadline: 180s
   httpTarget:
     body: ...
     headers:
       Content-Type: application/json
       User-Agent: Google-Cloud-Scheduler
     httpMethod: POST
     oauthToken:
       scope: https://www.googleapis.com/auth/cloud-platform
       serviceAccountEmail: CLOUD_SCHEDULER_SERVICE_ACCOUNT_EMAIL
     uri: https://batch.googleapis.com/v1/projects/PROJECT_ID/locations/us-central1/jobs
   lastAttemptTime: '...'
   name: projects/PROJECT_ID/locations/us-central1/jobs/batch-job-invoker
   retryConfig:
     maxBackoffDuration: 3600s
     maxDoublings: 5
     maxRetryDuration: 0s
     minBackoffDuration: 5s
   schedule: '*/5 * * * *'
   scheduleTime: '...'
   state: PAUSED
   status: {}
   timeZone: America/Los_Angeles
   userUpdateTime: '...'
   

   In the output, verify that the state field is set to PAUSED.