| Component | [2.3.38-debian12/ -ubuntu22/ -ubuntu22-arm/ -ml-ubuntu22/ -rocky9 2026/11/02](https://docs.cloud.google.com/managed-spark/docs/release-notes.md.txt#November_02_2026) | [2.3.37-debian12/ -ubuntu22/ -ubuntu22-arm/ -ml-ubuntu22/ -rocky9 2026/09/30](https://docs.cloud.google.com/managed-spark/docs/release-notes.md.txt#September_30_2026) | [2.3.36-debian12/ -ubuntu22/ -ubuntu22-arm/ -ml-ubuntu22/ -rocky9 2026/09/04](https://docs.cloud.google.com/managed-spark/docs/release-notes.md.txt#September_04_2026) | [2.3.35-debian12/ -ubuntu22/ -ubuntu22-arm/ -ml-ubuntu22/ -rocky9 2026/08/19](https://docs.cloud.google.com/managed-spark/docs/release-notes.md.txt#August_19_2026) | [2.3.34-debian12/ -ubuntu22/ -ubuntu22-arm/ -ml-ubuntu22/ -rocky9 2026/07/15](https://docs.cloud.google.com/managed-spark/docs/release-notes.md.txt#July_15_2026) |
|---|---|---|---|---|---|
| [**Apache Atlas** ^initialization action^](https://github.com/GoogleCloudDataproc/initialization-actions/tree/master/atlas) | [2.2.0](https://atlas.apache.org/2.2.0/index.html) | [2.2.0](https://atlas.apache.org/2.2.0/index.html) | [2.2.0](https://atlas.apache.org/2.2.0/index.html) | [2.2.0](https://atlas.apache.org/2.2.0/index.html) | [2.2.0](https://atlas.apache.org/2.2.0/index.html) |
| [**Apache Flink** ^optional component^](https://docs.cloud.google.com/managed-spark/docs/concepts/components/flink.md.txt) | [1.17.0](https://flink.apache.org/2023/03/23/announcing-the-release-of-apache-flink-1.17/) | [1.17.0](https://flink.apache.org/2023/03/23/announcing-the-release-of-apache-flink-1.17/) | [1.17.0](https://flink.apache.org/2023/03/23/announcing-the-release-of-apache-flink-1.17/) | [1.17.0](https://flink.apache.org/2023/03/23/announcing-the-release-of-apache-flink-1.17/) | [1.17.0](https://flink.apache.org/2023/03/23/announcing-the-release-of-apache-flink-1.17/) |
| [**Apache Hadoop** ^installed^](https://hadoop.apache.org/) | [3.3.6](https://hadoop.apache.org/docs/r3.3.6/) | [3.3.6](https://hadoop.apache.org/docs/r3.3.6/) | [3.3.6](https://hadoop.apache.org/docs/r3.3.6/) | [3.3.6](https://hadoop.apache.org/docs/r3.3.6/) | [3.3.6](https://hadoop.apache.org/docs/r3.3.6/) |
| [**Apache Hive** ^installed^](https://hive.apache.org/) | [3.1.3](https://archive.apache.org/dist/hive/hive-3.1.3/) | [3.1.3](https://archive.apache.org/dist/hive/hive-3.1.3/) | [3.1.3](https://archive.apache.org/dist/hive/hive-3.1.3/) | [3.1.3](https://archive.apache.org/dist/hive/hive-3.1.3/) | [3.1.3](https://archive.apache.org/dist/hive/hive-3.1.3/) |
| [**Apache Hive WebHCat** ^optional component^](https://docs.cloud.google.com/managed-spark/docs/concepts/components/hivewebhcat.md.txt) | [3.1.3](https://cwiki.apache.org/confluence/display/Hive/WebHCat) | [3.1.3](https://cwiki.apache.org/confluence/display/Hive/WebHCat) | [3.1.3](https://cwiki.apache.org/confluence/display/Hive/WebHCat) | [3.1.3](https://cwiki.apache.org/confluence/display/Hive/WebHCat) | [3.1.3](https://cwiki.apache.org/confluence/display/Hive/WebHCat) |
| [**Apache Hudi** ^optional component^](https://docs.cloud.google.com/managed-spark/docs/concepts/components/hudi.md.txt) | [0.15.0](https://hudi.apache.org/releases/) | [0.15.0](https://hudi.apache.org/releases/) | [0.15.0](https://hudi.apache.org/releases/) | [0.15.0](https://hudi.apache.org/releases/) | [0.15.0](https://hudi.apache.org/releases/) |
| [**Apache Iceberg** ^optional component^](https://docs.cloud.google.com/managed-spark/docs/concepts/components/iceberg.md.txt) | [1.6.1](https://github.com/apache/iceberg/releases/tag/apache-iceberg-1.6.1) | [1.6.1](https://github.com/apache/iceberg/releases/tag/apache-iceberg-1.6.1) | [1.6.1](https://github.com/apache/iceberg/releases/tag/apache-iceberg-1.6.1) | [1.6.1](https://github.com/apache/iceberg/releases/tag/apache-iceberg-1.6.1) | [1.6.1](https://github.com/apache/iceberg/releases/tag/apache-iceberg-1.6.1) |
| [**Apache Kafka** ^initialization action^](https://github.com/GoogleCloudDataproc/initialization-actions/tree/master/kafka) | [3.1.0](https://archive.apache.org/dist/kafka/3.1.0/RELEASE_NOTES.html) | [3.1.0](https://archive.apache.org/dist/kafka/3.1.0/RELEASE_NOTES.html) | [3.1.0](https://archive.apache.org/dist/kafka/3.1.0/RELEASE_NOTES.html) | [3.1.0](https://archive.apache.org/dist/kafka/3.1.0/RELEASE_NOTES.html) | [3.1.0](https://archive.apache.org/dist/kafka/3.1.0/RELEASE_NOTES.html) |
| [**Apache Pig** ^optional component^](https://docs.cloud.google.com/managed-spark/docs/concepts/components/pig.md.txt) | [0.18.0-SNAPSHOT](https://pig.apache.org/docs/latest/) | [0.18.0-SNAPSHOT](https://pig.apache.org/docs/latest/) | [0.18.0-SNAPSHOT](https://pig.apache.org/docs/latest/) | [0.18.0-SNAPSHOT](https://pig.apache.org/docs/latest/) | [0.18.0-SNAPSHOT](https://pig.apache.org/docs/latest/) |
| [**Apache Spark** ^installed^](https://spark.apache.org/) | [3.5.3](https://spark.apache.org/releases/spark-release-3-5-3.html) | [3.5.3](https://spark.apache.org/releases/spark-release-3-5-3.html) | [3.5.3](https://spark.apache.org/releases/spark-release-3-5-3.html) | [3.5.3](https://spark.apache.org/releases/spark-release-3-5-3.html) | [3.5.3](https://spark.apache.org/releases/spark-release-3-5-3.html) |
| [**Apache Sqoop** ^initialization action^](https://github.com/GoogleCloudDataproc/initialization-actions/tree/master/sqoop) | [1.5.0-SNAPSHOT](https://sqoop.apache.org/) | [1.5.0-SNAPSHOT](https://sqoop.apache.org/) | [1.5.0-SNAPSHOT](https://sqoop.apache.org/) | [1.5.0-SNAPSHOT](https://sqoop.apache.org/) | [1.5.0-SNAPSHOT](https://sqoop.apache.org/) |
| [**Apache Tez** ^installed^](https://tez.apache.org/) | [0.10.2](https://tez.apache.org/releases/apache-tez-0-10-2.html) | [0.10.2](https://tez.apache.org/releases/apache-tez-0-10-2.html) | [0.10.2](https://tez.apache.org/releases/apache-tez-0-10-2.html) | [0.10.2](https://tez.apache.org/releases/apache-tez-0-10-2.html) | [0.10.2](https://tez.apache.org/releases/apache-tez-0-10-2.html) |
| [**BigQuery Connector** ^installed^](https://docs.cloud.google.com/managed-spark/docs/concepts/connectors/bigquery.md.txt) | [0.42.3](https://github.com/GoogleCloudDataproc/spark-bigquery-connector/releases/tag/0.42.3) | [0.42.3](https://github.com/GoogleCloudDataproc/spark-bigquery-connector/releases/tag/0.42.3) | [0.42.3](https://github.com/GoogleCloudDataproc/spark-bigquery-connector/releases/tag/0.42.3) | [0.42.3](https://github.com/GoogleCloudDataproc/spark-bigquery-connector/releases/tag/0.42.3) | [0.42.3](https://github.com/GoogleCloudDataproc/spark-bigquery-connector/releases/tag/0.42.3) |
| [**Cloud Storage Connector** ^installed^](https://docs.cloud.google.com/managed-spark/docs/concepts/connectors/cloud-storage.md.txt#clusters) | [3.1.13](https://github.com/GoogleCloudDataproc/hadoop-connectors/releases/tag/v3.1.13) | [3.1.13](https://github.com/GoogleCloudDataproc/hadoop-connectors/releases/tag/v3.1.13) | [3.1.13](https://github.com/GoogleCloudDataproc/hadoop-connectors/releases/tag/v3.1.13) | [3.1.13](https://github.com/GoogleCloudDataproc/hadoop-connectors/releases/tag/v3.1.13) | [3.1.13](https://github.com/GoogleCloudDataproc/hadoop-connectors/releases/tag/v3.1.13) |
| [**Conscrypt** ^installed^](https://conscrypt.org/) | [2.5.2](https://github.com/google/conscrypt/releases/tag/2.5.2) | [2.5.2](https://github.com/google/conscrypt/releases/tag/2.5.2) | [2.5.2](https://github.com/google/conscrypt/releases/tag/2.5.2) | [2.5.2](https://github.com/google/conscrypt/releases/tag/2.5.2) | [2.5.2](https://github.com/google/conscrypt/releases/tag/2.5.2) |
| [**Delta Lake** ^optional component^](https://docs.cloud.google.com/managed-spark/docs/concepts/components/delta.md.txt) | [3.2.1](https://delta.io/blog/delta-lake-3-2/) | [3.2.1](https://delta.io/blog/delta-lake-3-2/) | [3.2.1](https://delta.io/blog/delta-lake-3-2/) | [3.2.1](https://delta.io/blog/delta-lake-3-2/) | [3.2.1](https://delta.io/blog/delta-lake-3-2/) |
| [**Docker** ^optional component^](https://docs.cloud.google.com/managed-spark/docs/concepts/components/docker.md.txt) | [28.1](https://docs.docker.com/engine/release-notes/#version-2010) | [28.1](https://docs.docker.com/engine/release-notes/#version-2010) | [28.1](https://docs.docker.com/engine/release-notes/#version-2010) | [28.1](https://docs.docker.com/engine/release-notes/#version-2010) | [28.1](https://docs.docker.com/engine/release-notes/#version-2010) |
| [**Hue** ^initialization action^](https://github.com/GoogleCloudDataproc/initialization-actions/tree/master/hue) | [4.11.0](https://github.com/cloudera/hue/releases/tag/release-4.11.0) | [4.11.0](https://github.com/cloudera/hue/releases/tag/release-4.11.0) | [4.11.0](https://github.com/cloudera/hue/releases/tag/release-4.11.0) | [4.11.0](https://github.com/cloudera/hue/releases/tag/release-4.11.0) | [4.11.0](https://github.com/cloudera/hue/releases/tag/release-4.11.0) |
| [**Java** ^installed^](https://www.java.com/) | [11](https://jdk.java.net/11/) | [11](https://jdk.java.net/11/) | [11](https://jdk.java.net/11/) | [11](https://jdk.java.net/11/) | [11](https://jdk.java.net/11/) |
| [**JupyterLab Notebook** ^optional component^](https://docs.cloud.google.com/managed-spark/docs/concepts/components/jupyter.md.txt) | [3.6](https://jupyterlab.readthedocs.io/en/stable/getting_started/changelog.html#v3-6) | [3.6](https://jupyterlab.readthedocs.io/en/stable/getting_started/changelog.html#v3-6) | [3.6](https://jupyterlab.readthedocs.io/en/stable/getting_started/changelog.html#v3-6) | [3.6](https://jupyterlab.readthedocs.io/en/stable/getting_started/changelog.html#v3-6) | [3.6](https://jupyterlab.readthedocs.io/en/stable/getting_started/changelog.html#v3-6) |
| [**Oozie** ^initialization action^](https://github.com/GoogleCloudDataproc/initialization-actions/tree/master/oozie) | [5.2.1](https://oozie.apache.org/docs/5.2.1/index.html) | [5.2.1](https://oozie.apache.org/docs/5.2.1/index.html) | [5.2.1](https://oozie.apache.org/docs/5.2.1/index.html) | [5.2.1](https://oozie.apache.org/docs/5.2.1/index.html) | [5.2.1](https://oozie.apache.org/docs/5.2.1/index.html) |
| [**Python** ^installed^](https://www.python.org/) | [micromamba 2.0.5](https://github.com/mamba-org/micromamba-releases/releases/tag/2.0.5-0) with [Python 3.11](https://docs.python.org/3/whatsnew/3.11.html) | [micromamba 2.0.5](https://github.com/mamba-org/micromamba-releases/releases/tag/2.0.5-0) with [Python 3.11](https://docs.python.org/3/whatsnew/3.11.html) | [micromamba 2.0.5](https://github.com/mamba-org/micromamba-releases/releases/tag/2.0.5-0) with [Python 3.11](https://docs.python.org/3/whatsnew/3.11.html) | [micromamba 2.0.5](https://github.com/mamba-org/micromamba-releases/releases/tag/2.0.5-0) with [Python 3.11](https://docs.python.org/3/whatsnew/3.11.html) | [micromamba 2.0.5](https://github.com/mamba-org/micromamba-releases/releases/tag/2.0.5-0) with [Python 3.11](https://docs.python.org/3/whatsnew/3.11.html) |
| [**R** ^installed^](https://www.r-project.org/about.html) | [R 4.3](https://cran.r-project.org/src/base/R-4/) | [R 4.3](https://cran.r-project.org/src/base/R-4/) | [R 4.3](https://cran.r-project.org/src/base/R-4/) | [R 4.3](https://cran.r-project.org/src/base/R-4/) | [R 4.3](https://cran.r-project.org/src/base/R-4/) |
| [**Ranger** ^optional component^](https://docs.cloud.google.com/managed-spark/docs/concepts/components/ranger.md.txt) | [2.4.0](https://ranger.apache.org/index.html) | [2.4.0](https://ranger.apache.org/index.html) | [2.4.0](https://ranger.apache.org/index.html) | [2.4.0](https://ranger.apache.org/index.html) | [2.4.0](https://ranger.apache.org/index.html) |
| [**Scala** ^installed^](https://www.scala-lang.org/) | [2.12.18](https://www.scala-lang.org/news/2.12.18) | [2.12.18](https://www.scala-lang.org/news/2.12.18) | [2.12.18](https://www.scala-lang.org/news/2.12.18) | [2.12.18](https://www.scala-lang.org/news/2.12.18) | [2.12.18](https://www.scala-lang.org/news/2.12.18) |
| [**Solr** ^optional component^](https://docs.cloud.google.com/managed-spark/docs/concepts/components/solr.md.txt) | [9.4.1](https://solr.apache.org/docs/9_4_1/changes/Changes.html) | [9.4.1](https://solr.apache.org/docs/9_4_1/changes/Changes.html) | [9.4.1](https://solr.apache.org/docs/9_4_1/changes/Changes.html) | [9.4.1](https://solr.apache.org/docs/9_4_1/changes/Changes.html) | [9.4.1](https://solr.apache.org/docs/9_4_1/changes/Changes.html) |
| [**Trino** ^optional component^](https://docs.cloud.google.com/managed-spark/docs/concepts/components/trino.md.txt) | [432](https://trino.io/docs/current/release/release-432.html) | [432](https://trino.io/docs/current/release/release-432.html) | [432](https://trino.io/docs/current/release/release-432.html) | [432](https://trino.io/docs/current/release/release-432.html) | [432](https://trino.io/docs/current/release/release-432.html) |
| [**Zeppelin Notebook** ^optional component^](https://docs.cloud.google.com/managed-spark/docs/concepts/components/zeppelin.md.txt) | [0.10.1](https://zeppelin.apache.org/docs/0.10.1/) | [0.10.1](https://zeppelin.apache.org/docs/0.10.1/) | [0.10.1](https://zeppelin.apache.org/docs/0.10.1/) | [0.10.1](https://zeppelin.apache.org/docs/0.10.1/) | [0.10.1](https://zeppelin.apache.org/docs/0.10.1/) |
| [**Zookeeper** ^optional component^](https://docs.cloud.google.com/managed-spark/docs/concepts/components/zookeeper.md.txt) | [3.9.5](https://zookeeper.apache.org/doc/r3.9.5/releasenotes.html) | [3.9.5](https://zookeeper.apache.org/doc/r3.9.5/releasenotes.html) | [3.9.5](https://zookeeper.apache.org/doc/r3.9.5/releasenotes.html) | [3.9.5](https://zookeeper.apache.org/doc/r3.9.5/releasenotes.html) | [3.9.5](https://zookeeper.apache.org/doc/r3.9.5/releasenotes.html) |

## Notes:

- Version `2.3` is a lightweight image that contains only core components,
  reducing exposure to Common Vulnerabilities and Exposures (CVEs). For higher
  security compliance requirements, use the image version `2.3`or later, when
  creating a Managed Service for Apache Spark cluster.

- If you choose to install
  [optional components](https://docs.cloud.google.com/managed-spark/docs/concepts/components/overview.md.txt) when
  creating a Managed Service for Apache Spark cluster with `2.3` image, they will be
  downloaded and installed during cluster creation. This might increase the
  cluster startup time. To avoid this delay, you can create a
  [custom image](https://docs.cloud.google.com/managed-spark/docs/guides/images.md.txt#generate_a_custom_image)
  with the optional components pre-installed. This is achieved by running
  [`generate_custom_image.py`](https://github.com/GoogleCloudDataproc/custom-images?tab=readme-ov-file#generate-custom-image)
  with the
  [`--optional-components`](https://docs.cloud.google.com/managed-spark/docs/guides/images.md.txt#run_the_code)
  flag.

  > [!NOTE]
  > **Note:** You must specify the optional components that you want to install when you create the cluster. For more information, see [Add optional components](https://docs.cloud.google.com/managed-spark/docs/concepts/components/overview.md.txt#add_optional_components).   
  > The following example shows the Google Cloud CLI command for creating a cluster with optional components:
  >
  > ```
  > gcloud dataproc clusters create CLUSTER_NAME
  >     --optional-components=COMPONENT_NAME \
  >     ... other flags
  > ```

- The `2.3.32`, `2.3.36` and later image versions
  [do not have preconfigured Conda channels](https://docs.cloud.google.com/managed-spark/docs/release-notes.md.txt#June_22_2026).
  Packages can't be installed using Conda unless channels are manually
  configured during cluster initialization.
  Also, `2.3` image version clusters that have preconfigured Conda channels,
  including `2.3.35` and earlier image versions but excluding `2.3.32`,
  can't be used after August 25, 2026.

  - **You may need to delete and replace existing clusters** . After August 25, 2026, existing clusters that were created with images that have pre-configured Conda channels, even if cluster jobs don't use Conda to install packages, must be deleted and replaced with new clusters [created](https://docs.cloud.google.com/managed-spark/docs/guides/create-cluster.md.txt#creating_a_cloud_dataproc_cluster) or [recreated](https://docs.cloud.google.com/managed-spark/docs/guides/recreate-cluster.md.txt#recreate_and_update_a_cluster_2) with images that don't have preconfigured Conda channels.
- The following optional components are supported in non-arm 2.3 images:

  - Apache Flink
  - Apache Hive WebHCat
  - Apache Hudi
  - Apache Iceberg
  - Apache Pig
  - Delta Lake
  - Docker
  - JupyterLab Notebook
  - Ranger
  - Solr
  - Trino
  - Zeppelin notebook
  - Zookeeper
- [`2.3.x-*-arm` images](https://docs.cloud.google.com/managed-spark/docs/concepts/compute/supported-machine-types.md.txt#arm_machine_types)
  support only the pre-installed components and the following optional
  components. The other 2.3 optional components and all initialization actions
  aren't supported:

  - Apache Hive WebHCat
  - Apache Pig (starting with `2.3.22-ubuntu22-arm`)
  - Docker
  - Zeppelin notebook
  - Zookeeper (installed in [high availability clusters](https://docs.cloud.google.com/managed-spark/docs/concepts/configuring-clusters/high-availability.md.txt); optional component in other clusters)
- [Lightning Engine and Native Query Execution on ARM](https://docs.cloud.google.com/managed-spark/docs/guides/lightning-engine.md.txt#native_query_execution_on_arm): Supported on `2.3-ubuntu22-arm` images.

- `yarn.nodemanager.recovery.enabled` and HDFS Audit Logging
  are enabled by default in 2.3 images.

- `micromamba`, instead of `conda` in previous image versions, is installed as part
  of the Python installation.

- Apache Iceberg version: In 2.3 images, Apache Iceberg is available in versions `1.6.1` and `1.10`. The default version is `1.6.1`. Starting with image version `2.3.35`, 2.3 clusters created with Lightning Engine enabled use Iceberg version `1.10` by default. To specify the Iceberg version on any 2.3 cluster, set the `dataproc:dataproc.iceberg.version` cluster property to `1.10` (or `1.6`) during cluster creation.

- Docker and Zeppelin installation issues:

  - Installation fails if the cluster has no public internet access. As a workaround, create a cluster that uses a custom image with optional components pre-installed. You can do this by running [`generate_custom_image.py`](https://github.com/GoogleCloudDataproc/custom-images) with the [`--optional-components` flag](https://docs.cloud.google.com/managed-spark/docs/guides/images.md.txt#run_the_code).
  - Installation can fail if the cluster is pinned to an older sub-minor image version: Packages are installed on demand from public OSS repositories, and a package might not be available upstream to support the installation. As a workaround, create a cluster that uses a custom image with optional components pre-installed in the custom image. To do this, run [`generate_custom_image.py`](https://github.com/GoogleCloudDataproc/custom-images) with the [`--optional-components` flag](https://docs.cloud.google.com/managed-spark/docs/guides/images.md.txt#run_the_code).
- The default resource calculator for YARN has been changed from
  [DefaultResourceCalculator](https://hadoop.apache.org/docs/r3.3.6/hadoop-yarn/hadoop-yarn-common/apidocs/org/apache/hadoop/yarn/util/resource/DefaultResourceCalculator.html)
  to [DominantResourceCalculator](https://hadoop.apache.org/docs/r3.3.6/hadoop-yarn/hadoop-yarn-common/apidocs/org/apache/hadoop/yarn/util/resource/DominantResourceCalculator.html),
  which uses the dominant-resource concept to determine resource allocation,
  such as Memory and CPU allocation. This change impacts
  [Autoscaler](https://docs.cloud.google.com/managed-spark/docs/concepts/configuring-clusters/autoscaling.md.txt#how_autoscaling_works),
  which scales based on the dominant resource usage of the cluster.

## Image version 2.3 machine learning (ML) components

The Managed Service for Apache Spark `2.3-ml-ubuntu` image extends the 2.3 base image
with ML-specific software. It supports 2.3 image optional components and other
2.3 features, and adds the component versions listed in the following sections.

### GPU-specific libraries

For Managed Service for Apache Spark jobs that use GPU VMs,
the following NVIDIA driver and libraries are available in the
`2.3-ml-ubuntu` image. You can use them to accomplish the following
tasks:

- Accelerate Spark batch workloads with the [NVIDIA cuDF for Apache Spark library](https://docs.nvidia.com/cudf-spark/latest/index.html)
- Train machine learning workloads
- Run distributed batch inference using Spark

| Package Name | Version |
|---|---|
| Spark cuDF | 25.04.0 |
| NVIDIA Driver | Ubuntu 22.04 LTS Accelerated with NVIDIA driver version 570 |
| CUDA | 12.6.3 |
| cublas | 12.6.4 |
| cusolver | 11.7.1 |
| cupti | 12.6.80 |
| cusparse | 12.5.4 |
| cuDNN | 9.10.1 |
| NCCL | 2.27.5 |

### XGBoost libraries

The following [Maven package versions](https://mvnrepository.com/artifact/ml.dmlc)
are available in `2.3-ml-ubuntu` image to let you use
[XGBoost](https://www.nvidia.com/en-us/glossary/xgboost/) with Spark in Java or
Scala.

| Group ID | Package Name | Version |
|---|---|---|
| ml.dmlc | xgboost4j-gpu_2.12 | 2.1.1 |
| ml.dmlc | xgboost4j-spark-gpu_2.12 | 2.1.1 |

> [!NOTE]
> **Note:** You cannot use distributed Spark XGBoost on a Managed Service for Apache Spark job that has [autoscaling](https://docs.cloud.google.com/managed-spark/docs/concepts/configuring-clusters/autoscaling.md.txt#enable_autoscaling%20enabled) (the default behavior) because new nodes that start elastic scaling cannot receive new tasks and remain idle. To use XGBoost with a batch workload, you can set the [`spark.dynamicAllocation.enabled = false`](https://docs.cloud.google.com/managed-spark/docs/concepts/autoscaling#spark_dynamic_allocation_properties) property on a Managed Service for Apache Spark job to disable dynamic allocation.

### Python libraries

The `2.3-ml-ubuntu` image contains the following libraries, which support different
stages in the ML lifecycle.
\`2.3-ml-ubuntu\` image Python libraries

| Package | Version |
|---|---|
| accelerate | 1.8.1 |
| conda | 23.11.0 |
| cookiecutter | 2.5.0 |
| curl | 8.12.1 |
| cython | 3.0.12 |
| dask | 2023.12.1 |
| datasets | 3.6.0 |
| deepspeed | 0.17.2 |
| delta-spark | 3.2.0 |
| evaluate | 0.4.5 |
| fastavro | 1.9.7 |
| fastparquet | 2023.10.1 |
| fiona | 1.10.0 |
| gateway-provisioners\[yarn\] | 0.4.0 |
| gcsfs | 2023.12.2.post1 |
| google-auth-oauthlib | 1.2.2 |
| google-cloud-aiplatform | 1.88.0 |
| google-cloud-bigquery\[pandas\] | 3.31.0 |
| google-cloud-bigquery-storage | 2.30.0 |
| google-cloud-bigtable | 2.30.1 |
| google-cloud-container | 2.56.1 |
| google-cloud-datacatalog | 3.26.1 |
| google-cloud-dataproc | 5.18.1 |
| google-cloud-datastore | 2.21.0 |
| google-cloud-language | 2.17.2 |
| google-cloud-logging | 3.11.4 |
| google-cloud-monitoring | 2.27.2 |
| google-cloud-pubsub | 2.29.1 |
| google-cloud-redis | 2.18.1 |
| google-cloud-spanner | 3.53.0 |
| google-cloud-speech | 2.32.0 |
| google-cloud-storage | 2.19.0 |
| google-cloud-texttospeech | 2.25.1 |
| google-cloud-translate | 3.20.3 |
| google-cloud-vision | 3.10.2 |
| huggingface_hub | 0.33.1 |
| httplib2 | 0.22.0 |
| ipyparallel | 8.6.1 |
| ipython-sql | 0.3.9 |
| ipywidgets | 8.1.7 |
| jupyter_contrib_nbextensions | 0.7.0 |
| jupyter_http_over_ws | 0.0.8 |
| jupyter_kernel_gateway | 2.5.2 |
| jupyter_server | 1.24.0 |
| jupyterhub | 4.1.6 |
| jupyterlab | 3.6.8 |
| jupyterlab-git | 0.44.0 |
| jupyterlab_widgets | 3.0.15 |
| koalas | 0.22.0 |
| langchain | 0.3.26 |
| lightgbm | 4.6.0 |
| markdown | 3.5.2 |
| matplotlib | 3.8.4 |
| mlflow | 3.1.1 |
| nbconvert | 7.14.2 |
| nbdime | 3.2.1 |
| nltk | 3.9.1 |
| notebook | 6.5.7 |
| numba | 0.58.1 |
| numpy | 1.26.4 |
| oauth2client | 4.1.3 |
| onnx | 1.17.0 |
| openblas | 0.3.25 |
| opencv | 4.11.0 |
| orc | 2.1.1 |
| pandas | 2.1.4 |
| pandas-profiling | 3.0.0 |
| papermill | 2.4.0 |
| pyarrow | 16.1.0 |
| pydot | 2.0.0 |
| pyhive | 0.7.0 |
| pynvml | 12.0.0 |
| pysal | 23.7 |
| pytables | 3.9.2 |
| python | 3.11 |
| regex | 2023.12.25 |
| requests | 2.32.2 |
| requests-kerberos | 0.12.0 |
| rtree | 1.1.0 |
| scikit-image | 0.22.0 |
| scikit-learn | 1.5.2 |
| scipy | 1.11.4 |
| seaborn | 0.13.2 |
| sentence-transformers | 5.0.0 |
| setuptools | 79.0.1 |
| shap | 0.48.0 |
| shapely | 2.1.1 |
| spacy | 3.8.7 |
| spark-tensorflow-distributor | 1.0.0 |
| spyder | 5.5.6 |
| sqlalchemy | 2.0.41 |
| sympy | 1.13.3 |
| tensorflow | 2.18.0 |
| tokenizers | 0.21.4.dev0 |
| toree | 0.5.0 |
| torch | 2.6.0 |
| torch-model-archiver | 0.11.1 |
| torcheval | 0.0.7 |
| tornado | 6.4.2 |
| torchvision | 0.21.0 |
| traitlets | 5.14.3 |
| transformers | 4.53.1 |
| uritemplate | 4.1.1 |
| virtualenv | 20.26.6 |
| wordcloud | 1.9.4 |
| xgboost | 2.1.4 |

### R libraries

The following R library versions are included in `2.3-ml-ubuntu` image.
\`2.3-ml-ubuntu\` image R libraries

| Package Name | Version |
|---|---|
| r-ggplot2 | 3.4.4 |
| r-irkernel | 1.3.2 |
| r-rcurl | 1.98-1.16 |
| r-recommended | 4.3 |