The Jupyter [optional component](https://docs.cloud.google.com/managed-spark/docs/concepts/components/overview.md.txt#available_optional_components) lets you run interactive data analytics notebooks on a Managed Service for Apache Spark cluster. The component provides a web-based, single-user [Jupyter](http://jupyter.org/) notebook environment that supports the [JupyterLab](https://jupyterlab.readthedocs.io/en/stable/index.html) web UI.

## Multi-user notebook alternatives

To serve notebooks to multiple users, create a Managed Service for Apache Spark-enabled [Vertex AI Workbench instance](https://docs.cloud.google.com/vertex-ai/docs/workbench/instances/create-dataproc-enabled). Alternatively, [install the Managed Service for Apache Spark JupyterLab plugin](https://docs.cloud.google.com/managed-spark/docs/quickstarts/jupyterlab-sessions) on a VM.

## Configuration

**Configure Jupyter.** Jupyter can be configured by providing `dataproc:jupyter`
[cluster properties](https://docs.cloud.google.com/managed-spark/docs/concepts/configuring-clusters/cluster-properties.md.txt#service_properties).
To reduce the risk of remote code execution over unsecured notebook server
APIs, the default `dataproc:jupyter.listen.all.interfaces` cluster property
setting is `false`, which restricts connections to `localhost (127.0.0.1)` when
the [Component Gateway](https://docs.cloud.google.com/managed-spark/docs/concepts/accessing/gateways.md.txt) is
enabled (Component Gateway activation is required when installing the Jupyter component).

The Jupyter notebook provides a Python kernel to run [Spark](https://spark.apache.org/) code, and a
PySpark kernel. By default, notebooks are [saved in Cloud Storage](https://github.com/src-d/jgscm)
in the Managed Service for Apache Spark staging bucket, which is specified by the user or
[auto-created](https://docs.cloud.google.com/managed-spark/docs/guides/create-cluster.md.txt#auto-created_staging_bucket)
when the cluster is created. The location can be changed at cluster creation time using the
[`dataproc:jupyter.notebook.gcs.dir`](https://docs.cloud.google.com/managed-spark/docs/concepts/configuring-clusters/cluster-properties.md.txt#dataproc-properties) cluster property.

**Work with data files.** You can use a Jupyter notebook to work with data files that have been
[uploaded to Cloud Storage](https://docs.cloud.google.com/storage/docs/uploading-objects.md.txt).
Since the [Cloud Storage connector](https://docs.cloud.google.com/managed-spark/docs/concepts/connectors/cloud-storage.md.txt)
is pre-installed on a Managed Service for Apache Spark cluster, you can reference the
files directly in your notebook. Here's an example that accesses CSV files in
Cloud Storage:

```
df = spark.read.csv("gs://bucket/path/file.csv")
df.show()
```

See
[Generic Load and Save Functions](https://spark.apache.org/docs/latest/sql-data-sources-load-save-functions.html)
for PySpark examples.

## Install Jupyter

Install the component when you create a Managed Service for Apache Spark cluster.
The Jupyter component requires activation of the Managed Service for Apache Spark
[Component Gateway](https://docs.cloud.google.com/managed-spark/docs/concepts/accessing/gateways.md.txt).

> [!NOTE]
> **Note:** Only when using [image version 1.5](https://docs.cloud.google.com/managed-spark/docs/concepts/versioning/dataproc-version-clusters#unsupported-dataproc-image-versions), installation of the Jupyter component also requires installation of the [Anaconda](https://docs.cloud.google.com/managed-spark/docs/concepts/components/anaconda) component.

### Google Cloud console

1. In the Google Cloud console, open the [**Create cluster**](https://console.cloud.google.com/dataproc/clusters/create) page.
2. Click **Additional configuration** to expand that section.
3. Edit **Optional components**.
4. In the panel that opens, select the checkbox for **Jupyter Notebook** , then click **Save**.

### gcloud CLI

To create a Managed Service for Apache Spark cluster that includes the Jupyter component,
use the
[gcloud dataproc clusters create](https://docs.cloud.google.com/sdk/gcloud/reference/dataproc/clusters/create.md.txt) <var translate="no">cluster-name</var> command with the `--optional-components` flag.

**Latest default image version example**

The following example installs the Jupyter
component on a cluster that uses the latest default image version.

```
gcloud dataproc clusters create cluster-name \
    --optional-components=JUPYTER \
    --region=region \
    --enable-component-gateway \
    ... other flags
```

### REST API

The Jupyter component
can be installed through the Dataproc API using
[`SoftwareConfig.Component`](https://docs.cloud.google.com/managed-spark/docs/reference/rest/v1/ClusterConfig.md.txt#Component)
as part of a
[`clusters.create`](https://docs.cloud.google.com/managed-spark/docs/reference/rest/v1/projects.regions.clusters/create.md.txt)
request.

- Set the [EndpointConfig.enableHttpPortAccess](https://docs.cloud.google.com/managed-spark/docs/reference/rest/v1/ClusterConfig.md.txt#endpointconfig) property to `true` as part of the `clusters.create` request to enable connecting to the Jupyter notebook Web UI using the [Component Gateway](https://docs.cloud.google.com/managed-spark/docs/concepts/accessing/gateways.md.txt).

## Open the Jupyter and JupyterLab UIs

Click the [Google Cloud console Component Gateway links](https://docs.cloud.google.com/managed-spark/docs/concepts/accessing/gateways.md.txt#viewing_and_accessing_component_gateway_urls)
to open in your local browser the Jupyter notebook or JupyterLab UI running on
the cluster master node.

**Select "GCS" or "Local Disk" to create a new Jupyter Notebook in
either location.**

## Attach GPUs to master and worker nodes

You can [add GPUs](https://docs.cloud.google.com/managed-spark/docs/concepts/compute/gpus.md.txt)
to your cluster's master and worker nodes when using a Jupyter notebook to:

1. Preprocess data in Spark, then collect a [DataFrame](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.html) onto the master and run [TensorFlow](https://www.tensorflow.org/)
2. Use Spark to orchestrate TensorFlow runs in parallel
3. Run [Tensorflow-on-YARN](https://github.com/criteo/tf-yarn)
4. Use with other machine learning scenarios that use GPUs