New Managed Service for Apache Spark (formerly Dataproc on Compute Engine) subminor cluster image versions:

• 2.1.114-debian11, 2.1.114-rocky8, 2.1.114-ubuntu20, 2.1.114-ubuntu20-arm
• 2.2.82-debian12, 2.2.82-rocky9, 2.2.82-ubuntu22, 2.2.82-ubuntu22-arm
• 2.3.30-debian12, 2.3.30-ml-ubuntu22, 2.3.30-rocky9, 2.3.30-ubuntu22, 2.3.30-ubuntu22-arm

Breaking

Managed Service for Apache Spark (formerly Dataproc on Compute Engine): The configuration for Spark shuffle partitions (spark.sql.shuffle.partitions) has changed from an integer to a string type.

This change impacts image versions 2.3.30 and later in version 2.3, and 2.2.82 and later in version 2.2.

• Impact: This change only affects users who are programmatically setting the configuration in code using spark.conf.set() with an integer literal.
  • Impacted example: spark.conf.set("spark.sql.shuffle.partitions", 100)
• User action: Update your code to pass a string literal instead of an integer.
  • Example fix: spark.conf.set("spark.sql.shuffle.partitions", "100")
• Not impacted: Setting the configuration via command-line arguments (e.g., spark-submit --conf spark.sql.shuffle.partitions=100), properties files, or Spark SQL commands (spark.sql("SET spark.sql.shuffle.partitions=100")) remains unaffected, as these methods naturally parse the input as strings.

New Managed Service for Apache Spark (formerly Google Cloud Serverless for Apache Spark) subminor runtime versions:

• 1.2.80
• 2.2.80
• 2.3.33

New Managed Service for Apache Spark (formerly Dataproc on Compute Engine) subminor cluster image versions for 3.0 (Preview):

• 3.0.0-RC2-debian13, 3.0.0-RC2-ml-ubuntu24, 3.0.0-RC2-rocky9, 3.0.0-RC2-ubuntu24

Key updates in this release:

• The debian image for 3.0 is upgraded to Debian 13.
• The default Java runtime is upgraded to Java 21.
• Apache Hadoop upgraded to 3.5.0.
• Apache Spark upgraded to 4.1.1.
• Apache Hive upgraded to 4.2.0.
• Trino upgraded to 480.
• Apache Solr upgraded to 9.10.1.
• Apache Kafka upgraded to 3.9.2.
• Docker upgraded to 28.1.
• Apache Flink upgraded to 2.2.0.
• Scala upgraded to 2.13.17.
• Cloud Storage Connector upgraded to 3.1.13.
• Apache Zookeeper upgraded to 3.9.5.
• BigQuery Connector upgraded to 0.44.1-Preview.
• Zeppelin Notebook upgraded to 0.12.0.
• JupyterLab upgraded to 4.5.7.
• Pixi is used as python package manager instead of conda.

Announcement

New Managed Service for Apache Spark (formerly Google Cloud Serverless for Apache Spark) subminor runtime versions:

• 1.2.79
• 2.2.79
• 2.3.32

Key updates in this release:

• Upgraded Metastore Proxy version to v0.0.79.
• Upgraded Spark RAPIDS to version 26.04.0 version in the Managed Service for Apache Spark version 3.0 serverless runtime.

New Managed Service for Apache Spark (formerly Dataproc on Compute Engine) subminor cluster image versions:

• 2.1.113-debian11, 2.1.113-rocky8, 2.1.113-ubuntu20, 2.1.113-ubuntu20-arm
• 2.2.81-debian12, 2.2.81-rocky9, 2.2.81-ubuntu22, 2.2.81-ubuntu22-arm
• 2.3.29-debian12, 2.3.29-ml-ubuntu22, 2.3.29-rocky9, 2.3.29-ubuntu22, 2.3.29-ubuntu22-arm

Managed Service for Apache Spark (formerly Dataproc on Compute Engine): Announcing the following cluster features:

• Flexible VMs: Minimize stockouts and improve machine obtainability by defining prioritized lists for your master, primary, and secondary worker VM types. Managed Spark for Apache Spark utilizes these lists to create your cluster, and selects the optimal VM type based on current capacity, quotas, and existing reservations.
• Cluster Scheduled Stop: Optimize cost and maintain your cluster configuration by stopping clusters after a specified idle period, at a specified future time, or after a specified period from cluster creation or a cluster update request. All custom cluster configurations are restored once you restart the cluster.
• Zero-scale clusters: Reduce costs by creating only secondary workers, scaling them down to zero when they are not in use.

New Managed Service for Apache Spark (formerly Google Cloud Serverless for Apache Spark) subminor runtime versions:

• 1.2.78
• 2.2.78
• 2.3.31

Dataproc and Google Cloud Serverless for Apache Spark are now unified in the Google Cloud console under the Managed Service for Apache Spark brand. This change consolidates our managed Spark deployment options into a single umbrella brand that includes the full breadth of our Spark capabilities. No existing functionality is being removed as part of this change, and there will be no impact to the Dataproc API, client library, gcloud CLI, or Dataproc IAM role and permission names.

Managed Service for Apache Spark (formerly Dataproc on Compute Engine) subminor cluster image versions:

• 2.3.28-debian12, 2.3.28-ml-ubuntu22, 2.3.28-rocky9, 2.3.28-ubuntu22, 2.3.28-ubuntu22-arm

Change

• Managed Service for Apache Spark (formerly Dataproc on Compute Engine): Upgraded Apache Zookeeper to version 3.9.5 in image version 2.3.

Fixed

• Upgraded Dataproc Metastore Proxy to v0.0.79 to fix CVEs.
• Fixed CVEs CVE-2026-24308 and CVE-2026-24281.

New Serverless for Apache Spark runtime versions:

• 1.2.77
• 2.2.77
• 2.3.30
• 3.0.13

New Dataproc on Compute Engine subminor image versions:

• 2.1.112-debian11, 2.1.112-rocky8, 2.1.112-ubuntu20, 2.1.112-ubuntu20-arm
• 2.2.80-debian12, 2.2.80-rocky9, 2.2.80-ubuntu22, 2.2.80-ubuntu22-arm
• 2.3.27-debian12, 2.3.27-ml-ubuntu22, 2.3.27-rocky9, 2.3.27-ubuntu22, 2.3.27-ubuntu22-arm

New Serverless for Apache Spark runtime versions:

• 1.2.76
• 2.2.76
• 2.3.29
• 3.0.12

New Dataproc on Compute Engine subminor image versions:

• 2.1.111-debian11, 2.1.111-rocky8, 2.1.111-ubuntu20, 2.1.111-ubuntu20-arm
• 2.2.79-debian12, 2.2.79-rocky9, 2.2.79-ubuntu22, 2.2.79-ubuntu22-arm
• 2.3.26-debian12, 2.3.26-ml-ubuntu22, 2.3.26-rocky9, 2.3.26-ubuntu22, 2.3.26-ubuntu22-arm

Fixed

• Fixed CVEs CVE-2025-58057, CVE-2025-53864, CVE-2025-68161, CVE-2025-48924, and CVE-2025-33042.
• Upgraded Dataproc Metastore Proxy to v0.0.78 to fix CVEs.
• Default JDK is set to Temurin JDK - 11.0.30 in all 2.1, 2.2 and 2.3 images.

New Serverless for Apache Spark runtime versions:

• 1.2.75
• 2.2.75
• 2.3.28
• 3.0.11

Dataproc on Compute Engine: The following subminor image versions announced on March 08, 2026 have been rolled back:

• 2.1.110-debian11, 2.1.110-rocky8, 2.1.110-ubuntu20, 2.1.110-ubuntu20-arm
• 2.2.78-debian12, 2.2.78-rocky9, 2.2.78-ubuntu22, 2.2.78-ubuntu22-arm
• 2.3.25-debian12, 2.3.25-ml-ubuntu22, 2.3.25-rocky9, 2.3.25-ubuntu22, 2.3.25-ubuntu22-arm

New Dataproc on Compute Engine subminor image versions:

• 2.1.110-debian11, 2.1.110-rocky8, 2.1.110-ubuntu20, 2.1.110-ubuntu20-arm
• 2.2.78-debian12, 2.2.78-rocky9, 2.2.78-ubuntu22, 2.2.78-ubuntu22-arm
• 2.3.25-debian12, 2.3.25-ml-ubuntu22, 2.3.25-rocky9, 2.3.25-ubuntu22, 2.3.25-ubuntu22-arm

Rollback Notice: These image versions were rolled back on March 11, 2026.

Fixed

• Fixed CVEs CVE-2025-58057, CVE-2025-53864, CVE-2025-68161, CVE-2025-48924 (partial), and CVE-2025-33042.
• Upgraded Dataproc Metastore Proxy to v0.0.78 to fix CVEs.

New Serverless for Apache Spark runtime versions:

• 1.2.74
• 2.2.74
• 2.3.27
• 3.0.10

Announcement

Serverless for Apache Spark 3.0 runtime version:

• Updated Spark version to 4.0.1.
• Updated BigQuery Spark Connector version to 0.44.0.
• Applied patch for SPARK-52601.
• Applied patch for SPARK-52614.

Announcement

Serverless for Apache Spark 1.2, 2.2 and 2.3 runtime versions:

• Updated Delta Lake version to 3.2.1.

New Serverless for Apache Spark runtime versions:

• 1.2.73
• 2.2.73
• 2.3.26

New Serverless for Apache Spark runtime versions:

• 1.2.72
• 2.2.72
• 2.3.25
• 3.0.9

New Dataproc on Compute Engine subminor image versions:

• 2.0.160-debian10, 2.0.160-rocky8, 2.0.160-ubuntu18
• 2.1.109-debian11, 2.1.109-rocky8, 2.1.109-ubuntu20, 2.1.109-ubuntu20-arm
• 2.2.77-debian12, 2.2.77-rocky9, 2.2.77-ubuntu22, 2.2.77-ubuntu22-arm
• 2.3.24-debian12, 2.3.24-ml-ubuntu22, 2.3.24-rocky9, 2.3.24-ubuntu22, 2.3.24-ubuntu22-arm

Fixed

Fixed an issue that could cause gsutil failures on clusters using custom images built from new base images. This problem occurred when the gcloud core/universe_domain property was not set in the build environment, resulting in a malformed /etc/boto.cfg. The image build process now ensures a correct default configuration, restoring proper Google Cloud Storage connectivity.

New Serverless for Apache Spark runtime versions:

• 1.2.71
• 2.2.71
• 2.3.24
• 3.0.8

Hyperdisks can now be added to a Dataproc cluster.

New Dataproc on Compute Engine subminor image versions:

• 2.0.159-debian10, 2.0.159-rocky8, 2.0.159-ubuntu18
• 2.1.108-debian11, 2.1.108-rocky8, 2.1.108-ubuntu20, 2.1.108-ubuntu20-arm
• 2.2.76-debian12, 2.2.76-rocky9, 2.2.76-ubuntu22, 2.2.76-ubuntu22-arm
• 2.3.23-debian12, 2.3.23-ml-ubuntu22, 2.3.23-rocky9, 2.3.23-ubuntu22, 2.3.23-ubuntu22-arm

New Serverless for Apache Spark runtime versions:

• 1.2.70
• 2.2.70
• 2.3.23
• 3.0.7

New Serverless for Apache Spark runtime versions:

• 1.2.69
• 2.2.69
• 2.3.22
• 3.0.5

Feature

Serverless for Apache Spark: Added support for removing conscrypt from Serverless for Apache Spark 2.3+ runtimes using the dataproc.artifacts.remove property .

Feature

Dataproc on Compute Engine: Sharing snapshot diagnostic data: Setting the --tarball-access=GOOGLE_DATAPROC_DIAGNOSE flag with the gcloud dataproc clusters diagnose command shares all of the output Cloud Storage bucket contents with Google Cloud support if uniform bucket-level access is enabled on the output Cloud Storage bucket. If object-level access control is enabled on the output Cloud Storage bucket, only the generated diagnostic tar file is shared.

New Dataproc on Compute Engine subminor image versions:

• 2.0.158-debian10, 2.0.158-ubuntu18, 2.0.158-rocky8
• 2.1.107-debian11, 2.1.107-ubuntu20, 2.1.107-ubuntu20-arm, 2.1.107-rocky8
• 2.2.75-debian12, 2.2.75-ubuntu22, 2.2.75-ubuntu22-arm, 2.2.75-rocky9
• 2.3.22-debian12, 2.3.22-ml-ubuntu22, 2.3.22-rocky9, 2.3.22-ubuntu22, 2.3.22-ubuntu22-arm

Feature

Zookeeper version upgraded to 3.9.4 in Dataproc image 2.3.22.

Feature

Parquet CLI version upgraded to 1.15.2 in 2.1 and 2.2 images.

Feature

Delta subminor version upgraded to 3.2.1 in Dataproc on Compute Engine image 2.2 and 2.3.

Feature

Apache Pig is now available in ARM images.

Feature

Added a new dataproc:pypi.repository property to customize the PyPI repository used for pip. The value can be a URL, or google to use a Google-hosted cache of PyPI, accessible without public internet connectivity. Starting in image version 3.1, google will be the default; to opt out and return to public PyPI, use the value pypi.

Change

Removed use of deprecated Hadoop configuration properties fs.default.name and yarn.resourcemanager.system-metrics-publisher.enabled.

Fixed

Fixed a bug in the ARM image that prevented connecting to a Dataproc Metastore instance with a gRPC protocol endpoint.

Fixed

Fixed the spark.driver.extraClassPath delimiter for the Jupyter SparkMonitor Listener.

Upcoming Spark data lineage changes See the upcoming May, 2026 Dataproc and Serverless for Apache Spark release notes for an announcement of a change that will automatically enable Dataproc Spark data lineage and Serverless for Apache Spark data lineage when you enable the Data Lineage API (see Control lineage ingestion for a service) without requiring additional project, cluster, batch workload, or interactive session settings.

New Serverless for Apache Spark runtime versions:

• 3.0.4

Announcement

Dataproc on Compute Engine: The following subminor image versions announced on January 24, 2026 have been rolled back:

• 2.0.157-debian10, 2.0.157-ubuntu18, 2.0.157-rocky8
• 2.1.106-debian11, 2.1.106-ubuntu20, 2.1.106-ubuntu20-arm, 2.1.106-rocky8
• 2.2.74-debian12, 2.2.74-ubuntu22, 2.2.74-ubuntu22-arm, 2.2.74-rocky9
• 2.3.21-debian12, 2.3.21-ml-ubuntu22, 2.3.21-rocky9, 2.3.21-ubuntu22, 2.3.21-ubuntu22-arm

New Dataproc on Compute Engine subminor image versions:

• 2.0.157-debian10, 2.0.157-ubuntu18, 2.0.157-rocky8
• 2.1.106-debian11, 2.1.106-ubuntu20, 2.1.106-ubuntu20-arm, 2.1.106-rocky8
• 2.2.74-debian12, 2.2.74-ubuntu22, 2.2.74-ubuntu22-arm, 2.2.74-rocky9
• 2.3.21-debian12, 2.3.21-ml-ubuntu22, 2.3.21-rocky9, 2.3.21-ubuntu22, 2.3.21-ubuntu22-arm

Rollback Notice: These image versions were rolled back on January 30, 2026.

Feature

Upgraded the Delta subminor version to 3.2.1 in images 2.2 and 2.3.

Feature

Apache Pig is now available in ARM images.

Feature

Added a new dataproc:pypi.repository property to customize the PyPI repository that pip uses. The value can be a internet-accessible URL, or you can specify google to use a Google-hosted cache of PyPI, which is accessible without public internet connectivity. Starting with Managed Service for Apache Spark image version 3.1, google will be the default (you can specify the pypi property value to use public PyPI instead of the default google 3.1 value).

Change

Removed the use of deprecated Hadoop configuration properties fs.default.name and yarn.resourcemanager.system-metrics-publisher.enabled.

Fixed

Fixed a bug in the ARM image that prevented connecting to a Dataproc Metastore instance with a gRPC protocol endpoint.

Fixed

Fixed the spark.driver.extraClassPath delimiter for the Jupyter SparkMonitor Listener.

New Serverless for Apache Spark runtime versions:

• 1.2.67
• 2.2.67
• 2.3.20
• 3.0.3

Fixed

• Applied patch for SPARK-48292 in Serverless for Apache Spark 1.2 and 2.2 runtimes.

Dataproc on Compute Engine: The following subminor image versions announced on January 06, 2026 have been rolled back:

• 2.0.156-debian10, 2.0.156-ubuntu18, 2.0.156-rocky8
• 2.1.105-debian11, 2.1.105-ubuntu20, 2.1.105-ubuntu20-arm, 2.1.105-rocky8
• 2.2.73-debian12, 2.2.73-ubuntu22, 2.2.73-ubuntu22-arm, 2.2.73-rocky9
• 2.3.20-debian12, 2.3.20-ml-ubuntu22, 2.3.20-rocky9, 2.3.20-ubuntu22, 2.3.20-ubuntu22-arm