4.0.x release image versions

Component 4.0.0-debian13/
-ubuntu24/
-ml-ubuntu24/
-rocky9
2026/11/19




Apache Flink
optional component
2.2.0
Apache Gluten
installed
1.6
Apache Hadoop
installed
3.5.0
Apache Hive
installed
4.2.0
Apache Hive WebHCat
optional component
4.2.0
Apache Iceberg
optional component
1.12.0
Apache Kafka
initialization action
3.9.2
Apache Pig
optional component
0.18.0
Apache Ranger
optional component
2.8.0
Apache Spark
installed
4.2.0
Apache Solr
optional component
9.10.1
Apache Tez
installed
0.10.5
Apache Zeppelin Notebook
optional component
0.12.0
Apache Zookeeper
optional component
3.9.5
BigQuery Connector
installed
0.46.0
Cloud Storage Connector
installed
4.0.7
Conscrypt
installed
2.6
Delta Lake
optional component
4.4.0
Docker
optional component
28.1
Java
installed
25
JupyterLab Notebook
optional component
4.6.6
Python
installed
Pixi 0.76.0 with Python 3.13
R
installed
R 4.5
Scala
installed
2.13.17
Trino
optional component
480

New Features and Improvements

  • Flexible VM machine types: Starting with image version 4.0, when you create a cluster without specifying a machine type for a cluster node, Managed Service for Apache Spark specifies the node with a ranked list of flexible VM machine types, such as a ranked list of N4, N2, and E2 series machine types, with the list optimized for resource availability.

Limitations

  • Apache Hudi: Apache Hudi is not supported in image version 4.0.

  • Metastores with Hive 4.2: Image version 4.0 doesn't support BigQuery Metastore, BigLake Metastore, or Dataproc Metastore with Hive 4.2 (direct Hive queries using these metastores are not supported).

  • Delta Lake with Hive: Delta Lake 4.x, which is included in Managed Service for Apache Spark image version 4.0, doesn't support Hive (queries on Delta Lake 4.4 tables cannot be executed on Hive).

Notes:

  • Java versions: Image version 4.0 includes Java 25 for Spark workloads. Java 21 is included for backward compatibility.

  • Iceberg Spark properties filtering: On standard 4.0 clusters, if the optional iceberg component is omitted, Iceberg session extensions and default catalogs are automatically filtered from spark-defaults.conf during cluster activation.

  • GPU clusters: All 4.0 image versions except 4.0-ml-ubuntu enable cgroups V2. Since YARN doesn't support GPU discovery when cgroups V2 is enabled, you must use the 4.0-ml-ubuntu image when creating clusters with GPUs.

  • Version 4.0 is a lightweight image that contains only core components, reducing exposure to Common Vulnerabilities and Exposures (CVEs). For higher security compliance requirements, use image version 2.3 or later when creating a Managed Service for Apache Spark cluster.

  • Pre-installed optional components: The following optional components are pre-installed in 4.0 images and are activated if you include them during cluster creation:

    • Apache Iceberg
    • Delta Lake

    You must specify the optional components that you want to install when you create a cluster. For more information, see Add optional components.
    The following example shows the Google Cloud CLI command for creating a cluster with optional components:

    gcloud dataproc clusters create CLUSTER_NAME
        --optional-components=COMPONENT_NAME \
        ... other flags
    

  • The default resource calculator for YARN has been changed from DefaultResourceCalculator to DominantResourceCalculator, which uses the dominant-resource concept to determine resource allocation, such as Memory and CPU allocation. This change impacts Autoscaler, which scales based on the dominant resource usage of the cluster.