Maximize cluster and job flexibility and efficiency

This page provides best practices and scheduling strategies to help you design resilient Managed Service for Apache Spark clusters that maximize resource obtainability across Google Cloud regions, reduce provisioning delays, and take advantage of performance and cost optimizations.

Overview of resource obtainability

Running jobs on rigid Managed Service for Apache Spark clusters creates single points of failure that can lead to job interruptions and cluster and job failures due to localized unavailability or provisioning delays.

The risk of resource unavailability is high for clusters configured with any of the following rigid anti-patterns:

  • Fixed zone placement: Hardcoding a single zone using the zone flag or field instead of allowing Managed Service for Apache Spark AutoZone to dynamically place the cluster in the optimal zone. This anti-pattern prevents cluster creation from using compute resources available in adjacent zones.
  • Single machine type or generation per node role: Restricting node roles to a single machine generation or type. This anti-pattern prevents the cluster from falling back to alternatives.
  • Vertical scaling with fewer large VMs: Scaling clusters vertically to rely on a small number of large VM shapes. This anti-pattern restricts scheduling flexibility.
  • All-or-nothing cluster creation: Requiring all worker nodes to provision simultaneously rather than using partial cluster creation paired with autoscaling. This anti-pattern causes cluster creation to fail if a worker node can't be allocated due to temporary resource unavailability.
  • Peak scheduling spikes: Triggering large batch pipelines during peak-use hours, which increases regional contention.

Resource obtainability is the practice of designing jobs to be hardware-flexible, multi-zonal, and scalable. By decoupling clusters from rigid configurations and enabling multi-machine family fallbacks, you can increase creation success rates, minimize startup latency, and consistently achieve SLAs.

Resource obtainability recommendations

Adopt the following optimizations to improve resource availability and job stability:

Use flexible VMs

The Flexible VM feature lets you specify a ranked list of VM types for master, primary, and secondary worker nodes. This increases creation success rates by evaluating listed VM types and automatically selecting zones with available capacity.

Recommendation: Use flexible VMs with disk overrides, which lets you specify different disk types, such as hyperdisk and persistent disk, for different candidate machine families in the same cluster policy. You can also blend Gen2 and Gen4 machine families within a single policy to scale jobs across a broader pool and bypass capacity limits.

For jobs using N2 and N2D machines, consider the following rankings:

  • Rank 0: N2, N2D
  • Rank 1: N4, N4D (with hyperdisk balanced)
  • Rank 2: C4, C4D, C3, C3D (with hyperdisk balanced). Local SSDs can be used with C4 and C4D, but typically 8 or 16 cores support only 1–2 local SSDs.
  • Rank 3: E2 (lower performance; use only if needed)

For example, for jobs using n2d-standard-16, the rankings are:

  • Rank 0: n2d-standard-16, n2-standard-16
  • Rank 1: n4-standard-16, n4d-standard-16
  • Rank 2: c4-standard-16, c4d-standard-16, c3-standard-22, c3d-standard-16
  • Rank 3: e2-standard-16

When using Flexible VMs, consider the following factors:

  • Mixed disk type support: Gen3 and Gen4 machine types only support Hyperdisk disk types and don't support Persistent Disk types. When blending Gen2 and Gen4, specify a diskConfig for each instance selection in the instanceFlexibilityPolicy. For more information, see Disk overrides.

  • Resource quotas: When you define a Flexible VM policy with fallback types, Compute Engine checks quotas for all candidate types and disks in the region. Ensure that your project has sufficient compute and disk quotas allocated for all configured items.

  • Compute Engine discounts: Take advantage of flexible committed use discounts (CUDs) to apply spend-based savings across multiple VM families and regions.

  • Pricing: Use the Google Cloud Pricing Calculator to compare costs for each rank in your policy.

For configuration templates and deployment examples, see:

Use autozone placement

Use AutoZone placement to let Managed Service for Apache Spark select the best zone to provision resources. If you use custom Virtual Private Cloud (VPC) networks, ensure that the subnet has sufficient IP addresses across all regional zones.

Use smaller machine shapes

Design jobs to scale horizontally across smaller machine types (4, 8, or 16 cores) instead of scaling vertically with larger VMs. Smaller VM sizes have higher availability across zones, helping prevent creation delays.

  • Review and re-architect jobs that use large machine types for driver nodes.
  • Review jobs that run only driver nodes without worker nodes (single-node clusters). Jobs running on single-node clusters can't scale up dynamically and bind execution to a single physical host.

Use cluster autoscaling

Use cluster autoscaling with a sufficient maximum number of instances to manage capacity for jobs with resource variability (spikes).

Use partial cluster creation with autoscaling

Use partial cluster creation, which lets you specify a minimum number of primary workers, with autoscaling. If the cluster starts with fewer workers than requested, autoscaling can add more workers dynamically once resources become available.

Schedule jobs during off-peak hours

Schedule jobs during off-peak hours, such as midday and weekends. Schedule at non-standard times, for example, at 10:07 instead of 10:00, to avoid scheduling spikes.

What's next