3.0.x release image versions

Component	3.0.0-debian13/ -ubuntu24/ -ml-ubuntu24/ -rocky9 2026/07/15	3.0.0-RC2-debian13/ -ubuntu24/ -ml-ubuntu24/ -rocky9 2026/05/03	3.0.0-RC1-debian12/ -ubuntu24/ -rocky9 2025/09/08
Apache Flink ^optional component	2.2.0	2.2.0
Apache Hadoop ^installed	3.5.0	3.5.0	3.4.1
Apache Hive ^installed	4.2.0	4.2.0	4.1.0
Apache Hive WebHCat ^optional component	4.2.0	4.2.0
Apache Iceberg ^optional component	1.11.0
Apache Kafka ^initialization action	3.9.2	3.9.2
Apache Pig ^optional component	0.18.0	0.18.0-SNAPSHOT
Apache Ranger ^optional component	2.8.0
Apache Spark ^installed	4.1.2	4.1.1	4.0.0
Apache Solr ^optional component	9.10.1	9.10.1	9.7.0
Apache Tez ^installed	0.10.5	0.10.5	0.10.5
Apache Zeppelin Notebook ^optional component	0.12.0	0.12.0
Apache Zookeeper ^optional component	3.9.5	3.9.5	3.9.3
BigQuery Connector ^installed	0.44.1-Preview	0.44.1-Preview
Cloud Storage Connector ^installed	3.1.13	3.1.13	3.1.6
Conscrypt ^installed	2.6	2.6	2.6
Delta Lake ^optional component	4.2.0
Docker ^optional component	28.1	28.1
Java ^installed	21	21	17
JupyterLab Notebook ^optional component	4.5.7	4.5.7
Python ^installed	Pixi 0.67.1 with Python 3.12	Pixi 0.67.1 with Python 3.11	micromamba 2.0.5 with Python 3.11
R ^installed	R 4.5	R 4.5	R 4.3
Scala ^installed	2.13.17	2.13.17	2.13.14
Trino ^optional component	480	480

Limitations

• Lightning Engine: Image version 3.0 doesn't support Lightning Engine.

• Metastores with Hive 4.2: Image version 3.0 doesn't support BigQuery Metastore, BigLake Metastore, or Dataproc Metastore with Hive 4.2 (direct Hive queries using these metastores are not supported).

• Delta Lake with Hive: Delta Lake 4.x, which is included in Managed Service for Apache Spark image version 3.0, doesn't support Hive (queries on Delta Lake 4.2 tables cannot be executed on Hive).

• Iceberg with Hive: Managed Service for Apache Spark image version 3.0 includes Hive 4.2, which supports Iceberg through the Iceberg REST catalog.

• GPU clusters: All 3.0 image versions except 3.0-ml-ubuntu enable cgroups V2. Since YARN doesn't support GPU discovery when cgroups V2 is enabled, you must use the 3.0-ml-ubuntu image when creating clusters with GPUs.

• Hudi: Image version 3.0 doesn't support Apache Hudi.

Notes:

• Version 3.0 is a lightweight image that contains only core components, reducing exposure to Common Vulnerabilities and Exposures (CVEs). For higher security compliance requirements, use the image version 2.3or later, when creating a Managed Service for Apache Spark cluster.

• If you choose to install optional components when creating a Managed Service for Apache Spark cluster with 3.0 image, they will be downloaded and installed during cluster creation. This might increase the cluster startup time. To avoid this delay, you can create a custom image with the optional components pre-installed. This is achieved by running generate_custom_image.py with the --optional-components flag.

  Note: You must specify the optional components that you want to install when you create the cluster. For more information, see Add optional components.
  The following example shows the Google Cloud CLI command for creating a cluster with optional components:

  gcloud dataproc clusters create CLUSTER_NAME
      --optional-components=COMPONENT_NAME \
      ... other flags
  

• Since the 3.0 image version Pixi is used to install python packages instead of conda.

• The following optional components are supported in 3.0 images:

  • Apache Flink
  • Apache Hive WebHCat
  • Apache Iceberg
  • Apache Pig
  • Apache Ranger
  • Apache Solr
  • Apache Zeppelin notebook
  • Apache Zookeeper
  • Delta Lake
  • Docker
  • JupyterLab Notebook
  • Trino

• yarn.nodemanager.recovery.enabled and HDFS Audit Logging are enabled by default in 3.0 images.

• pixi is installed as part of the Python installation.

• The default resource calculator for YARN has been changed from DefaultResourceCalculator to DominantResourceCalculator, which uses the dominant-resource concept to determine resource allocation, such as Memory and CPU allocation. This change impacts Autoscaler, which scales based on the dominant resource usage of the cluster.