Key updates in these image versions include:
2.1.117 and 2.2.85 subminor image versions don't have preconfigured Conda channels, and are not mapped to default aliases (such as 2.1-debian11 and 2.2-debian12) until August 25, 2026.
2.1.117-debian11 or 2.2.85-debian12). Packages cannot be installed using Conda unless channels are manually configured during cluster initialization.2.1.116-debian11 or 2.2.84-debian12).2.1 and 2.2 will point to image versions without preconfigured Conda channels (the latest subminor versions). All workloads must transition to these new images after August 25, 2026 since the use of prior subminor versions with preconfigured Conda channels will be disallowed.You may need to delete and replace existing clusters After August 25, 2026, existing clusters created with images that have preconfigured Conda channels (even if cluster jobs don't use Conda to install packages) need to be deleted and replaced with new clusters created or recreated with images that don't have preconfigured Conda channels.
]]>Managed Service for Apache Spark (formerly Dataproc on Compute Engine):
The following new subminor image versions, 1.3.96, 1.4.81, 1.5.92, 2.0.161, and 2.3.32,
don't have preconfigured Conda channels, and are not mapped to default aliases
(such as 2.3-debian12 and 2.3-ubuntu22) until August 25, 2026.
The new 2.1 and 2.2 subminor images continue to have preconfigured Conda channels,
but note the following Default change schedule:, which will impact all supported
image versions.
2.3.32-debian12). Packages cannot be installed using Conda unless channels are manually configured during cluster initialization.1.3.96, 1.4.81, 1.5.92, and 2.0.161 will become default after August 25, 2026.
Additionally, newer subminor versions for 2.1, 2.2, and 2.3 released after August 25, 2026 will not have preconfigured Conda channels and will be mapped to default aliases.
All workloads must use new images after August 25, 2026
since the use of prior subminor versions with preconfigured Conda
channels will be disallowed.You may need to delete and replace existing clusters After August 25, 2026, existing clusters created with images that have preconfigured Conda channels (even if cluster jobs don't use Conda to install packages) need to be deleted and replaced with new clusters created or recreated with images that don't have preconfigured Conda channels.
Managed Service for Apache Spark (formerly Dataproc on Compute Engine): Key updates in these image versions include:
ICEBERG optional component. Users can opt-in to Iceberg 1.10 by setting the dataproc:dataproc.iceberg.version cluster property to 1.10 during cluster creation.spark.scheduler.listenerbus.exitTimeout to 30s for Compute Engine deployments.Managed Service for Apache Spark (formerly Dataproc on Compute Engine): Rollout of the new sub-minor versions without pre-configured channels will begin on June 22, 2026, delayed from the previously planned date of June 15, 2026 ETA.
]]>Key updates in these runtime versions include:
Managed Service for Apache Spark (formerly Dataproc on Compute Engine):
Added support for selecting specific Confidential Computing technologies (AMD SEV, AMD SEV-SNP, Intel TDX) when creating clusters using the new --confidential-compute-type flag in gcloud and the confidentialInstanceType field in the API. The boolean --enable-confidential-compute flag is now deprecated but will continue to function, defaulting to AMD SEV for backward compatibility.
confidentialInstanceType enum in the API.--enable-confidential-compute flag and enableConfidentialCompute field are deprecated in favor of the new type-specific flag/field.SEV.SEV, SEV-SNP, and TDX.Managed Service for Apache Spark (formerly Dataproc on Compute Engine): The following subminor image versions announced on May 19, 2026 have been rolled back:
Rollback Notice: The 2.2.82 image versions were rolled back on May 25, 2026.
Managed Service for Apache Spark (formerly Dataproc on Compute Engine):
The configuration for Spark shuffle partitions (spark.sql.shuffle.partitions) has changed from an integer to a string type.
This change impacts image versions 2.3.30 and later in version 2.3, and 2.2.82 and later in version 2.2.
spark.conf.set() with an integer literal.
spark.conf.set("spark.sql.shuffle.partitions", 100)spark.conf.set("spark.sql.shuffle.partitions", "100")spark-submit --conf spark.sql.shuffle.partitions=100), properties files, or Spark SQL commands (spark.sql("SET spark.sql.shuffle.partitions=100")) remains unaffected, as these methods naturally parse the input as strings.New Managed Service for Apache Spark (formerly Dataproc on Compute Engine) subminor cluster image versions for 3.0 (Preview):
Key updates in this release:
Key updates in this release:
v0.0.79.26.04.0 version in the Managed Service
for Apache Spark version 3.0 serverless runtime.Key updates in this release:
Managed Service for Apache Spark (formerly Dataproc on Compute Engine): Announcing the following cluster features:
Dataproc and Google Cloud Serverless for Apache Spark are now unified in the Google Cloud console under the Managed Service for Apache Spark brand. This change consolidates our managed Spark deployment options into a single umbrella brand that includes the full breadth of our Spark capabilities. No existing functionality is being removed as part of this change, and there will be no impact to the Dataproc API, client library, gcloud CLI, or Dataproc IAM role and permission names.
]]>3.9.5 in image version 2.3.New Serverless for Apache Spark runtime versions:
New Dataproc on Compute Engine subminor image versions:
New Serverless for Apache Spark runtime versions:
New Dataproc on Compute Engine subminor image versions:
2.1, 2.2 and 2.3 images.New Serverless for Apache Spark runtime versions:
Dataproc on Compute Engine: The following subminor image versions announced on March 08, 2026 have been rolled back:
New Dataproc on Compute Engine subminor image versions:
Rollback Notice: These image versions were rolled back on March 11, 2026.
New Serverless for Apache Spark runtime versions:
Serverless for Apache Spark 3.0 runtime version:
Serverless for Apache Spark 1.2, 2.2 and 2.3 runtime versions:
New Serverless for Apache Spark runtime versions:
New Serverless for Apache Spark runtime versions:
New Dataproc on Compute Engine subminor image versions:
Fixed an issue that could cause gsutil failures on clusters using custom images built from new base images. This problem occurred when the gcloud core/universe_domain property was not set in the build environment, resulting in a malformed /etc/boto.cfg. The image build process now ensures a correct default configuration, restoring proper Google Cloud Storage connectivity.
New Serverless for Apache Spark runtime versions:
Hyperdisks can now be added to a Dataproc cluster.
]]>New Dataproc on Compute Engine subminor image versions:
New Serverless for Apache Spark runtime versions:
New Serverless for Apache Spark runtime versions:
Serverless for Apache Spark: Added support for removing conscrypt
from Serverless for Apache Spark 2.3+ runtimes using the
dataproc.artifacts.remove
property .
Dataproc on Compute Engine: Sharing snapshot diagnostic data: Setting the --tarball-access=GOOGLE_DATAPROC_DIAGNOSE flag with the gcloud dataproc clusters diagnose command shares all of the output Cloud Storage bucket contents with Google Cloud support if uniform bucket-level access is enabled on the output Cloud Storage bucket. If object-level access control is enabled on the output Cloud Storage bucket, only the generated diagnostic tar file is shared.