Managed Service for Apache Spark runtime 3.0

See the Managed Service for Apache Spark release notes for the latest Managed Service for Apache Spark runtime update notices.

Spark runtime version 3.0 components

Component	3.0.13 2026/03/27	3.0.12 2026/03/20	3.0.11 2026/03/13	3.0.10 2026/03/06	3.0.9 2026/02/27
Apache Spark	4.0.1	4.0.1	4.0.0	4.0.0	4.0.0
Cloud Storage Connector	3.1.9	3.1.9	3.1.9	3.1.9	3.1.9
BigQuery Connector	0.44.0	0.44.0	0.43.1-preview	0.43.1-preview	0.43.1-preview
Java	21	21	21	21	21
Python	3.12	3.12	3.12	3.12	3.12
Scala	2.13	2.13	2.13	2.13	2.13

Notes:

• The 3.0 runtime uses the UTF-8 as a default character encoding.

• 3.0 runtime new features and improvements:

  • Regional and multi-zonal workloads are used by default to increase obtainability of compute resources
  • Faster startup than previous runtimes
  • Fast resource cleanup that allows faster release of VPC IPs after workload completion
  • End-user credentials are used for all workloads by default
  • New bigquery Spark catalog, pre-configured for out-of-the-box BigQuery native table interactions
  • New Spark Serverless-specific IAM roles
  • New dataprocrm.googleapis.com API enablement is required

• 3.0 runtime unsupported and deprecated functionality:

  • 3.0 runtimes don't support Lightning Engine and Native Query Execution.

  • 3.0+ runtimes don't use Cloud Storage staging buckets.

  • The Persistent History Server (PHS) is not supported. Instead, use the Spark UI.

  • SparkR batches are not supported. Instead, use sparklyr.

  • Jupyter sessions are not supported. Instead, use Spark Connect sessions, the industry standard for interactive Spark development.

Python libraries

Note: You can't use distributed Spark XGBoost on a Managed Service for Apache Spark batch workload that has autoscaling enabled (the default behavior) because new nodes that start elastic scaling can't receive new tasks and remain idle.

To use XGBoost with a batch workload, you can set the spark.dynamicAllocation.enabled = false property on a batch workload to disable dynamic allocation.

Package Name	Version
accelerate	1.11
bigframes	2.24
cookiecutter	2.6
cuda	12.9
cudnn	9.10
cython	3.1
dask	2025.10
dataproc-spark-connect	1.0.0
dataproc-ml	1.0.0rc1
datasets	4.0
deepspeed	0.17
delta-spark	4.0.0
evaluate	0.4
fastavro	1.12
fastparquet	2024.11
gcsfs	2025.3
git	2.51
google-auth-oauthlib	1.2
google-cloud-aiplatform	1.121
google-cloud-bigquery	3.38
google-cloud-bigquery-storage	2.32
google-cloud-bigtable	2.34
google-cloud-container	2.59
google-cloud-datacatalog	3.27
google-cloud-dataproc	5.22
google-cloud-datastore	2.21
google-cloud-dlp	3.32
google-cloud-language	2.17
google-cloud-logging	3.12
google-cloud-monitoring	2.28
google-cloud-pubsub	2.31
google-cloud-redis	2.18
google-cloud-secret-manager	2.25
google-cloud-spanner	3.59
google-cloud-speech	2.33
google-cloud-storage	2.19
google-cloud-texttospeech	2.31
google-cloud-translate	3.21
google-cloud-vision	3.10
httplib2	0.31
huggingface_hub	0.36
ipyparallel	9.0
keyrings.google-artifactregistry-auth	1.1
langchain	1.0
lightgbm	4.6
mamba	2.3
markdown	3.9
nccl	2.27
nltk	3.9
nodejs	24.9
numba	0.61
numpy	2.1
oauth2client	4.1
onnx	1.17
openblas	0.3
opencv	4.11
orc	2.1
pandas	2.3
pyarrow	19.0
pydot	4.0
pyhive	0.7
pyiceberg	0.10
pymongo	4.15
pynvml	13.0
pytables	3.10
python	3.12
pytorch-gpu	2.6
regex	2025.10
requests	2.32
rtree	1.4
scikit-image	0.25
scikit-learn	1.7
scipy	1.15
seaborn	0.13
sentence-transformers	5.1
shap	0.48
spark-tensorflow-distributor	1.0
spacy	3.8
sqlalchemy	2.0
statsforecast	2.0
sympy	1.14
tensorflow-gpu	2.18
torcheval	0.0.7
torch	2.6
torchvision	0.21
tornado	6.5
transformers	4.57
uritemplate	4.2
virtualenv	20.35
wordcloud	1.9
xgboost	3.0.4
ydata-profiling	4.17