> [!CAUTION]
>
> Dataproc Hub and
> Vertex AI Workbench user-managed notebooks are
> deprecated. On January 30, 2025, support for user-managed notebooks
> will end and the ability to create user-managed notebooks instances
> will be removed. For alternative notebook solutions
> on Google Cloud, see:
>
> - [Install
>   the Jupyter component on your Managed Service for Apache Spark cluster](https://docs.cloud.google.com/managed-spark/docs/concepts/components/jupyter#install_jupyter).
> - [Create
>   a Managed Service for Apache Spark-enabled
>   Vertex AI Workbench instance](https://docs.cloud.google.com/vertex-ai/docs/workbench/instances/create-dataproc-enabled).

## Objectives

1. Use Dataproc Hub to create a single-user
   JupyterLab notebook environment running on a Managed Service for Apache Spark cluster.

2. Create a notebook and run a Spark job on the Managed Service for Apache Spark cluster.

3. Delete your cluster and preserve your notebook in Cloud Storage.

## Before you begin

1. The administrator must grant you `notebooks.instances.use` permission (see [Set Identity and Access Management (IAM) roles](https://docs.cloud.google.com/managed-spark/docs/tutorials/hub-admins.md.txt#set_identity_and_access_management_iam_roles)).

## Create a Dataproc JupyterLab cluster from Dataproc Hub

1. Select the **User-Managed Notebooks** tab on the
   **[Dataproc→Workbench](https://console.cloud.google.com/dataproc/workbench)**
   page in the Google Cloud console.

2. Click **Open JupyterLab** in the row that
   lists the Dataproc Hub instance created by the administrator.

   1. If you do not have access to the Google Cloud console, enter the Dataproc Hub instance URL that an administrator shared with you in your web browser.
3. On the **Jupyterhub→Dataproc Options** page, select
   a cluster configuration and zone. If enabled, specify any customizations, then
   click **Create**.

   ![](https://docs.cloud.google.com/static/managed-spark/images/hub-user-jupyterhub-form.png)

   After the Managed Service for Apache Spark cluster is created, you are redirected
   to the JupyterLab interface running on the cluster.

## Create a notebook and run a Spark job

1. On the left panel of the JupyterLab interface, click on `GCS` (Cloud Storage).

2. Create a PySpark notebook from the JupyterLab launcher.

   ![](https://docs.cloud.google.com/static/managed-spark/images/hub-user-create-notebook.png)
3. The PySpark kernel initializes a SparkContext (using the `sc` variable).
   You can examine the SparkContext and run a Spark job from the notebook.

   ```
   rdd = (sc.parallelize(['lorem', 'ipsum', 'dolor', 'sit', 'amet', 'lorem'])
          .map(lambda word: (word, 1))
          .reduceByKey(lambda a, b: a + b))
   print(rdd.collect())
   ```
   ![](https://docs.cloud.google.com/static/managed-spark/images/hub-user-run-code.png)
4. Name and save the notebook. The notebook is saved and remains in
   Cloud Storage after the Managed Service for Apache Spark cluster is deleted.

## Shut down the Dataproc cluster

1. From the JupyterLab interface, select **File→Hub Control Panel** to
   open the **Jupyterhub** page.

   When using Managed Service for Apache Spark image versions 1.4 or earlier, navigate to `/hub/home` to access the **Jupyterhub** page.
2. Click **Stop My Cluster** to shut down (delete) the JupyterLab server, which
   deletes the Managed Service for Apache Spark cluster.

   > [!IMPORTANT]
   > Stopping the server and deleting the cluster **does not delete the Dataproc Hub instance** . You can click **Start my server** on the **Jupyterhub** (Hub Control Panel) page or select the **Open JupyterLab** link for your Dataproc Hub instance on the **Dataproc→Workbench→User-Managed Notebooks** page in the Google Cloud console to open configure and create another Managed Service for Apache Spark JupyterLab cluster.

## What's next

- Explore [Spark and Jupyter Notebooks on Dataproc](https://github.com/GoogleCloudDataproc/cloud-managed-spark/tree/master/notebooks) on GitHub.