Improve resource obtainability for jobs

This document explains resource obtainability and recommends how to improve obtainability when you create and run Batch jobs.

What is obtainability?

Key Point: Higher obtainability isn't always better, even when your priority is reliability. The level of obtainability that you want varies based on competing priorities like cost and performance.

Obtainability is the probability that resources are available to run your workload. When resources aren't available, your workload fails due to resource availability errors. Obtainability is related to but distinct from reliability, the probability that your workload is available for its users. (Availability might mean either or both.)

The relationship between obtainability and reliability is complex. Your workload needs to obtain resources to be reliable. But, while any flexibility across location, time, and hardware increases obtainability, reliability only increases with flexibility that meets your workload's requirements. For example, note the following complexities:

• Higher-performance machine types might be less obtainable due to higher demand, but reliability decreases when your workload doesn't have enough performance.
• Location flexibility increases obtainability, but densely allocated machines improve network performance.
• Time flexibility increases obtainability, but excessive waiting can cause unnecessary delays.

Consequently, higher obtainability isn't always better. The optimal level of obtainability varies based on your priorities and requirements. For example, you might want lower obtainability to prioritize reducing costs. Even if you want to prioritize reliability, then you might need to balance the benefits of higher obtainability with the risks of lower performance or increased delays.

Recommendations to improve obtainability for jobs

Review the following sections to improve obtainability for Batch jobs:

• Recommendations for all consumption options: Review these recommendations for every job.
• Recommendations based on consumption option: Choose a recommendation based on the consumption option for your job.

Recommendations for all consumption options

• Minimize your needs for large and high-performance machine types by separating your workload. For example, first separate compute-intensive operations from low-demand operations by using dependent jobs. Then, reduce as much as possible the minimum compute resources that each job requires by maximizing parallel processing and minimizing tightly coupled tasks.

  Learn more about dependent jobs and compute resources and parallelism.

• Maximize the number of locations that are allowed for your job's Compute Engine resources. This is true by default; unless you specify the allowedLocations[] field for a job, the job's Compute Engine resources are allowed to run in every zone in the job's location.

  Learn more about locations.

• Consider using fail-fast strategies before waiting for better obtainability. To try avoiding delays, consider fail-fast strategies like the following:

  • Although you can troubleshoot resource availability errors by trying your request at another time, it's often faster to try different locations or hardware first.
  • If you're open to using multiple consumption options, it's often faster to first try the consumption options without delays: on-demand reservations, on-demand, and Spot VMs.
  • For new workloads that you plan to repeat, you might prefer to expedite initial testing and troubleshooting by temporarily using options that are faster to obtain.

Recommendations based on consumption option

The consumption option that you select is a primary factor for a job's obtainability. This section briefly summarizes each consumption option and recommends how to optimize obtainability for each consumption option.

Important: Obtainability isn't the only factor to consider when choosing a consumption option. For example, a consumption option also needs to be compatible with your requirements for pricing, run time, and machine type support. For additional information about how to choose a consumption option for a job, see Select the consumption option.

The following consumption options are listed from highest to lowest obtainability:

• For workloads with high obtainability requirements, use reservations. Reservations provide a higher level of assurance in obtaining capacity than all other consumption options. To use reserved capacity, first create or identify existing reservations that you can use. Then, configure instances to consume that reserved capacity. Unlike other consumption options, reserved instances incur costs even when they aren't being used until you delete them, due to their capacity assurance. When considering obtainability, choose between the following types of reservations:

  • To require reserved capacity, consume a specifically targeted reservation. These reservations make it easiest to verify that reserved capacity is available and consumable by your workload. Whenever reserved capacity isn't available and compatible, your workload fails and returns an error. A specifically targeted reservation includes all reservations for the RESERVATION_BOUND provisioning model and some reservations for the STANDARD provisioning model.

    Learn how to create a job that uses reservations.

  • For optional reserved capacity, consume an automatically consumed reservation and use instance flexibility (Preview). These reservation options provide high levels of assurance in obtaining capacity but only when reserved capacity that your workload can consume is available. For automatically consumed reservations, you must specify the STANDARD provisioning model and manually verify that reserved capacity is available and matches the instance configuration of your workload. Whenever reserved capacity isn't available or doesn't match, your workload attempts to use on-demand capacity instead. But, unlike specifically targeted reservations, you can use instance flexibility with automatically consumed reservations to let your workload attempt to consume multiple automatically consumed reservations.

    Learn how to create a job that uses reservations and instance flexibility (Preview).

• For workloads with medium obtainability requirements, either use on-demand capacity with instance flexibility (Preview) or Flex-start VMs. Choose between these options as follows:

  • Unless your machine type is in the A3 machine series or unsupported, consider using on-demand capacity with instance flexibility (Preview). The on-demand consumption option requires the STANDARD (default) provisioning model. Unless you explicitly block reservation consumption, your workload also attempts to consume automatically consumed reservations by default. Notably, we don't recommend on-demand for the A3 machine series.

    Learn how to create a job that uses the standard provisioning model and instance flexibility (Preview).

  • Otherwise or if on-demand capacity is repeatedly unavailable, consider using Flex-start VMs. To use the Flex-start VMs consumption option, specify the FLEX_START provisioning model. Flex-start VMs require your workload to withstand a wait time and limited run duration, but also provide discounts through Dynamic Workload Scheduler pricing.

    Learn how to create a job that uses Flex-start VMs.

• For workloads with low obtainability requirements, use Spot VMs with multiple obtainability features. To use the Spot VMs consumption option, specify the SPOT provisioning model. Spot VMs might not always be available and can be preempted at any time, so they are only recommended for fault tolerant workloads, but might provide the largest discounts through Spot VMs pricing.

  We generally recommend that you view the availability of Spot VMs before creation and verify that your workload can handle preemption. Then, to improve obtainability of Spot VMs in Batch jobs specifically, we recommend that you use instance flexibility with multiple machine types in the same rank (Preview) and use automatic retries to automatically attempt to recreate Spot VMs after preemption.

  Learn how to view the availability of Spot VMs and manage preemption of Spot VMs in the Compute Engine documentation. Then, learn how to create a job that uses the Spot VMs provisioning model, instance flexibility (Preview), and automatic retries for preemption.

What's next

• Learn about more job creation options.