# Create a cluster by using the Google Cloud console

This page shows you how to use the Google Cloud console to create a
Managed Service for Apache Spark cluster, run a basic [Apache
Spark](http://spark.apache.org/) job in the cluster, and then modify the number
of workers in the cluster.

*** ** * ** ***

To follow step-by-step guidance for this task directly in the
Google Cloud console, click **Guide me**:

[Guide me](https://console.cloud.google.com/?walkthrough_id=dataproc--quickstart-dataproc-console)

*** ** * ** ***

## Before you begin

<br />

### Required roles

Certain Identity and Access Management (IAM) roles are required to
run the examples on this page. Depending on organization policies, these
roles may have already been granted. To check role grants, see
[Do you need to grant roles?](https://docs.cloud.google.com/managed-spark/docs/concepts/iam#do_you_need_to_grant_roles).

For more information about granting roles, see
[Manage access to projects, folders, and organizations](https://docs.cloud.google.com/iam/docs/granting-changing-revoking-access.md.txt).

#### User roles


To get the permissions that
you need to create a Managed Service for Apache Spark cluster,

ask your administrator to grant you the
following IAM roles:

- [Dataproc Editor](https://docs.cloud.google.com/iam/docs/roles-permissions/dataproc.md.txt#dataproc.editor) (`roles/dataproc.editor`) on the project
- [Service Account User](https://docs.cloud.google.com/iam/docs/roles-permissions/iam.md.txt#iam.serviceAccountUser) (`roles/iam.serviceAccountUser`) on the Compute Engine default service account


#### Service account role


To ensure that the Compute Engine default service account has the necessary
permissions to create a Managed Service for Apache Spark cluster,

ask your administrator to grant the
following IAM roles to the Compute Engine default service account on the project:

**Important:** You must grant these roles to the Compute Engine default service account, *not* to your user account. Failure to grant the roles to the correct principal might result in permission errors.

- All: [Dataproc Worker](https://docs.cloud.google.com/iam/docs/roles-permissions/dataproc.md.txt#dataproc.worker) (`roles/dataproc.worker`)


## Create a cluster

1. In the Google Cloud console navigation menu, under **Infrastructure** , go
   to the Managed Service for Apache Spark **Clusters** page.

   [Go to Clusters](https://console.cloud.google.com/dataproc/clusters)
2. Click **Create cluster** , and then choose **Cluster on Compute Engine**.

3. In the **Cluster name** field, enter `example-cluster`.

4. In the **Region** and **Zone** lists, select a region and zone.

   Select a region (for example, `us-east1` or `europe-west1`) to isolate
   resources, such as virtual machine (VM) instances and Cloud Storage and
   metadata storage locations that are used by
   Managed Service for Apache Spark, in the region. For more information, see
   [Available regions and zones](https://docs.cloud.google.com/compute/docs/regions-zones/regions-zones#available)
   and [Cluster region](https://docs.cloud.google.com/managed-spark/docs/concepts/cluster-region.md.txt)
5. For all the other options, use the default settings.

6. To create the cluster, click **Create**.

   Your new cluster appears in a list on the **Clusters** page. The status is
   **Provisioning** until the cluster is ready to use, and then the status
   changes to **Running**. Provisioning the cluster might take a couple of
   minutes.

## Submit a Spark job

Submit a Spark job that estimates a value of Pi:

1. In the Managed Service for Apache Spark navigation menu, click **Jobs (and Batches)** (or **Jobs**).
2. On the **Jobs** page, click
   **Submit job** , and then do
   the following on the **Submit a job** page:

   1. Under **Select compute type for job** , select **Spark on Compute
      Engine**.
   2. In the **Cluster job details** section:
      1. In the **Job ID** field, use the default setting, or provide an ID that is unique to your Google Cloud project.
      2. In the **Cluster** menu, select **`example-cluster`**.
   3. In the **Files and artifacts** section:

      1. For **Job type** , select **Spark (Scala / Java)**.
      2. Under **Driver configuration** , select **Main class** , and in the **Main class** field, enter `org.apache.spark.examples.SparkPi`.
      3. In the **Jar files** field, enter
         `file:///usr/lib/spark/examples/jars/spark-examples.jar`. (This
         sample jar is pre-installed on the cluster. For custom jobs, specify
         a Cloud Storage path; see
         [Submit a job](https://docs.cloud.google.com/managed-spark/docs/guides/submit-job.md.txt).)

         > [!CAUTION]
         > **Caution:** Don't copy custom JAR files directly into cluster system directories such as `/usr/lib/spark/jars/`. Placing custom JARs in system directories pollutes the Managed Service for Apache Spark agent classpath and can cause job cancellation failures and orphaned YARN applications.

      4. In the **Arguments** field, enter `1000` to set the number of tasks.

      > [!NOTE]
      > **Note:** The Spark job estimates Pi by using the [Monte Carlo
      > method](https://wikipedia.org/wiki/Monte_Carlo_method)

      It generates *x* and *y* points on a coordinate plane that models a
      circle enclosed by a unit square. The input argument (`1000`) determines
      the number of x-y pairs to generate; the more pairs generated, the
      greater the accuracy of the estimation. This estimation uses
      Managed Service for Apache Spark worker nodes to parallelize the
      computation. For more information, see [Estimating Pi using the Monte
      Carlo Method](https://academo.org/demos/estimating-pi-monte-carlo/)
      and [JavaSparkPi.java on
      GitHub](https://github.com/apache/spark/blob/master/examples/src/main/java/org/apache/spark/examples/JavaSparkPi.java).
   4. Click **Submit**.

      Your job is displayed on the **Job details** page. The job status is
      **Running** or **Starting** , and then it changes to **Succeeded** after
      it's submitted.

      To avoid scrolling in the output, click **Line wrap: off**. The output
      is similar to the following:

      ```
      Pi is roughly 3.1416759514167594
      ```

      To view job details, click the **Configuration** tab.

## Update a cluster

Update your cluster by changing the number of worker instances:

1. In the Managed Service for Apache Spark navigation menu, click **Clusters**.
2. In the list of clusters, click **`example-cluster`**.
3. On the **Cluster details** page, click the **Configuration** tab.

   Your cluster settings are displayed.
4. Click **Edit**.

5. In the **Worker nodes** field, enter `5`.

6. Click **Save**.

Your cluster is now updated. To decrease the number of worker nodes to the
original value, follow the same procedure.

## Clean up


To avoid incurring charges to your Google Cloud account for
the resources used on this page, follow these steps.

1. To delete the cluster, on the **Cluster details** page for **`example-cluster`** , click **Delete**.
2. To confirm that you want to delete the cluster, click **Delete**.

## What's next

- Try this quickstart by using other tools:

  - [Use the API
    Explorer](https://docs.cloud.google.com/managed-spark/docs/quickstarts/create-cluster-template)

  - [Use the
    Google Cloud CLI](https://docs.cloud.google.com/managed-spark/docs/quickstarts/create-cluster-gcloud)

- Learn how to [create robust firewall rules when you create a
  project](https://docs.cloud.google.com/managed-spark/docs/concepts/configuring-clusters/network)

- Learn how to [write and run a Spark Scala
  job](https://docs.cloud.google.com/managed-spark/docs/tutorials/spark-scala).