This document explains the key concepts, fundamental building blocks, core
features, and benefits of Managed Service for Apache Spark. Understanding these fundamentals
help you effectively use Managed Service for Apache Spark for your data processing tasks.

## The serverless model

[Managed Service for Apache Spark serverless](https://docs.cloud.google.com/managed-spark/docs/serverless-overview.md.txt) is
the modern, automated-execution Managed Service for Apache Spark model. It lets you run
jobs without provisioning, managing, or scaling the underlying infrastructure:
Managed Service for Apache Spark handles the details for you.

- **Batches** : A batch (also called a *batch workload*) is the serverless
  equivalent of a Managed Service for Apache Spark job. You submit your code, such as a
  Spark job, to the service. Managed Service for Apache Spark provisions the necessary
  resources on-demand, runs the job, and then tears them down. You don't
  create or manage cluster or job resources; the service does the work
  for you.

- **Interactive sessions**: Interactive sessions provide a live, on-demand
  environment for exploratory data analysis, typically within a Jupyter notebook.
  Interactive sessions provide the convenience of a temporary, serverless workspace
  that you can use to run queries and develop code without having to provision and
  manage cluster and notebook resources.

- **Runtime profiles (session templates)** : A runtime profile (called a
  *session template* in the Google Cloud CLI and Dataproc API) is a reusable
  configuration that you can use to define interactive sessions. The profile
  contains session settings, such as compute tier, executor configuration, Spark
  properties, and library dependencies. You use the runtime profile to create
  interactive session environments for development, typically within a Jupyter
  notebook.

## The cluster-based model

[Managed Service for Apache Spark on clusters](https://docs.cloud.google.com/managed-spark/docs/concepts/clusters-overview.md.txt)
is the standard, infrastructure-centric way of using Managed Service for Apache Spark. It
gives you full control over a dedicated set of virtual machines for your data
processing tasks.

- **Clusters**: A cluster is your personal data processing engine, made up of Google Cloud virtual machines. You create a cluster to run open-source frameworks such as Apache Spark and Apache Hadoop. You have full control over cluster size, machine types, and configuration.
- **Jobs**: A job is a specific task, such as a PySpark script or Hadoop query. Instead of running a job directly on a cluster, you submit the job to Managed Service for Apache Spark, which manages job execution for you. You can submit multiple jobs to the cluster.
- **Workflow Templates**: A workflow template is a reusable definition that orchestrates a series of jobs (a workflow). It can define dependencies between jobs, for example to run a machine learning job only after a data cleaning job successfully completes. The templated workflow can run on an existing cluster or on a temporary (ephemeral) cluster that is created to run the workflow, and then deleted after the workflow completes. You can use the template to run the defined workflow whenever needed.
- **Autoscaling policies**: An autoscaling policy contains rules that you define to add or remove worker machines from a cluster based upon cluster workload in order to dynamically optimize cluster cost and performance.

### Environment customization

Managed Service for Apache Spark on clusters offers cluster features and
[components](https://docs.cloud.google.com/managed-spark/docs/concepts/components/overview.md.txt) you can
use to customize your application environment.

### Notebook and development environments

Managed Service for Apache Spark serverless notebooks and IDEs link to integrated
development environments where you can write and execute your code.

- **BigQuery Studio \& Workbench**: These are unified analytics and notebook environments. They allow you to write code (for example in a Jupyter notebook) and use a Managed Service for Apache Spark cluster or serverless session as the powerful backend engine to execute your code on large datasets.
- **Managed Service for Apache Spark JupyterLab Plugin** : This official [`JupyterLabextension`](https://github.com/GoogleCloudDataproc/dataproc-jupyter-plugin) acts as a control panel for Managed Service for Apache Spark serverless inside your notebook environment. It simplifies your workflow by allowing you to browse, create, and manage clusters and submit jobs without having to leave the Jupyter interface.
- **Google Cloud Data Agent Kit** : [Data Agent Kit](https://docs.cloud.google.com/data-agent-kit/overview.md.txt) lets data scientists, engineers, and developers manage their entire data workload lifecycle within their IDE. Data Agent Kit provides support for Managed Service for Apache Spark, enabling you to develop code, create interactive sessions, and [build pipelines](https://docs.cloud.google.com/data-agent-kit/build-pipelines.md.txt) directly from VS Code or a supported agentic CLI.
- **Managed Service for Apache Spark Connect Python Connector** : This Python library streamlines the process of using [Spark Connect](https://spark.apache.org/docs/latest/spark-connect-overview.html) with Managed Service for Apache Spark. It handles authentication and endpoint configuration, making it much simpler to connect your local Python environment, such as a notebook or IDE, to a remote Managed Service for Apache Spark cluster for interactive development.

## The container model

Managed Service for Apache Spark on Google Kubernetes Engine deploys Managed Service for Apache Spark **virtual**
clusters on a GKE cluster. Unlike Managed Service for Apache Spark
clusters, Managed Service for Apache Spark virtual clusters don't provision separate
master and worker VMs. Instead, they provision node pools within a
GKE cluster. Managed Service for Apache Spark on GKE
jobs are run as pods on these node pools. The node pools and
scheduling of pods on the node pools are managed by Managed Service for Apache Spark on GKE.