New Features and Improvements
- Flexible VM machine types: Starting with image version
4.0, when you create a cluster without specifying a machine type for a cluster node, Managed Service for Apache Spark specifies the node with a ranked list of flexible VM machine types, such as a ranked list of N4, N2, and E2 series machine types, with the list optimized for resource availability.
Limitations
Apache Hudi: Apache Hudi is not supported in image version
4.0.Metastores with Hive 4.2: Image version
4.0doesn't support BigQuery Metastore, BigLake Metastore, or Dataproc Metastore with Hive 4.2 (direct Hive queries using these metastores are not supported).Delta Lake with Hive: Delta Lake
4.x, which is included in Managed Service for Apache Spark image version4.0, doesn't support Hive (queries on Delta Lake 4.4 tables cannot be executed on Hive).
Notes:
Java versions: Image version
4.0includes Java 25 for Spark workloads. Java 21 is included for backward compatibility.Iceberg Spark properties filtering: On standard 4.0 clusters, if the optional
icebergcomponent is omitted, Iceberg session extensions and default catalogs are automatically filtered fromspark-defaults.confduring cluster activation.GPU clusters: All
4.0image versions except4.0-ml-ubuntuenablecgroups V2. Since YARN doesn't support GPU discovery whencgroups V2is enabled, you must use the4.0-ml-ubuntuimage when creating clusters with GPUs.Version
4.0is a lightweight image that contains only core components, reducing exposure to Common Vulnerabilities and Exposures (CVEs). For higher security compliance requirements, use image version2.3or later when creating a Managed Service for Apache Spark cluster.Pre-installed optional components: The following optional components are pre-installed in 4.0 images and are activated if you include them during cluster creation:
- Apache Iceberg
- Delta Lake
You must specify the optional components that you want to install when you create a cluster. For more information, see Add optional components.
The following example shows the Google Cloud CLI command for creating a cluster with optional components:gcloud dataproc clusters create CLUSTER_NAME --optional-components=COMPONENT_NAME \ ... other flags
The default resource calculator for YARN has been changed from DefaultResourceCalculator to DominantResourceCalculator, which uses the dominant-resource concept to determine resource allocation, such as Memory and CPU allocation. This change impacts Autoscaler, which scales based on the dominant resource usage of the cluster.