This page shows you how to use the Google Cloud CLI [gcloud](https://docs.cloud.google.com/sdk/gcloud/reference/dataproc) command-line tool to create a Managed Service for Apache Spark cluster, run a [Apache Spark](http://spark.apache.org/) job in the cluster, then modify the number of workers in the cluster. You can find out how to do the same or similar tasks with [Quickstarts Using the API Explorer](https://docs.cloud.google.com/managed-spark/docs/quickstarts/create-cluster-template), the Google Cloud console in [Create a cluster by using the Google Cloud console](https://docs.cloud.google.com/managed-spark/docs/quickstarts/create-cluster-console), and using the client libraries in [Create a cluster by using client libraries](https://docs.cloud.google.com/managed-spark/docs/quickstarts/create-cluster-client-libraries). ## Before you begin
### Required roles Certain IAM roles are required to run the examples on this page. Depending on organization policies, these roles may have already been granted. To check role grants, see [Do you need to grant roles?](https://docs.cloud.google.com/managed-spark/docs/concepts/iam#do_you_need_to_grant_roles). For more information about granting roles, see [Manage access to projects,folders, and organizations](https://docs.cloud.google.com/iam/docs/granting-changing-revoking-access.md.txt). #### User roles To get the permissions that you need to create a Managed Service for Apache Spark cluster, ask your administrator to grant you the following IAM roles: - [Dataproc Editor](https://docs.cloud.google.com/iam/docs/roles-permissions/dataproc.md.txt#dataproc.editor) (`roles/dataproc.editor`) on the project - [Service Account User](https://docs.cloud.google.com/iam/docs/roles-permissions/iam.md.txt#iam.serviceAccountUser) (`roles/iam.serviceAccountUser`) on the Compute Engine default service account #### Service account role To ensure that the Compute Engine default service account has the necessary permissions to create a Managed Service for Apache Spark cluster, ask your administrator to grant the following IAM roles to the Compute Engine default service account on the project: **Important:** You must grant these roles to the Compute Engine default service account, *not* to your user account. Failure to grant the roles to the correct principal might result in permission errors. - All: [Dataproc Worker](https://docs.cloud.google.com/iam/docs/roles-permissions/dataproc.md.txt#dataproc.worker) (`roles/dataproc.worker`) ## Create a cluster To create a cluster called `example-cluster`, run the following [`gcloud Managed Service for Apache Spark clusters create`](https://docs.cloud.google.com/sdk/gcloud/reference/dataproc/clusters/create) command. > [!IMPORTANT] > A convenient way to run the `gcloud` command-line tool is from [Cloud Shell](https://console.cloud.google.com/?cloudshell=true), which has the Google Cloud CLI pre-installed. Cloud Shell is free for Google Cloud customers. To use Cloud Shell, you need a Google Cloud project. ``` gcloud dataproc clusters create example-cluster --region=REGION ``` Replace the following: REGION: Specify a [region](https://docs.cloud.google.com/compute/docs/regions-zones/regions-zones#available) where the cluster will be located. ## Submit a job To submit a sample Spark job that calculates a rough value for `pi`, run the following [gcloud Managed Service for Apache Spark jobs submit spark](https://docs.cloud.google.com/sdk/gcloud/reference/dataproc/jobs/submit/spark) command: ``` gcloud dataproc jobs submit spark --cluster example-cluster \     --region=REGION \     --class org.apache.spark.examples.SparkPi \     --jars file:///usr/lib/spark/examples/jars/spark-examples.jar -- 1000 ``` Notes: Replace the following: REGION: Specify the cluster region. - The job runs on the `example-cluster`. - The `class` contains the main method for the SparkPi, which calculates an approximate value of `pi` application. - The jar file contains the job code. Note: `file:///usr/lib/spark/examples/jars/spark-examples.jar` is a sample jar pre-installed on the cluster. For custom jobs, specify your jar file in Cloud Storage (see [Submit a job](https://docs.cloud.google.com/managed-spark/docs/guides/submit-job.md.txt)). > [!CAUTION] > **Caution:** Don't copy custom JAR files directly into cluster system directories such as `/usr/lib/spark/jars/`. Placing custom JARs in system directories pollutes the Managed Service for Apache Spark agent classpath and can cause job cancellation failures and orphaned YARN applications. - `1000` is a job parameter. It specifies the number of tasks (iterations) the job performs to calculate the value of `pi`. > [!NOTE] > Parameters passed to the job must follow a double dash (`--`). For more information, see the [Google Cloud CLI documentation](https://docs.cloud.google.com/sdk/gcloud/reference/dataproc/jobs/submit/spark). The job's running and final output is displayed in the terminal window: ``` Waiting for job output... ... Pi is roughly 3.14118528 ... Job finished successfully. ``` ## Update a cluster To change the number of workers in the cluster to five, run the following command: ``` gcloud dataproc clusters update example-cluster \     --region=REGION \     --num-workers 5 ``` The command output displays cluster details: ``` workerConfig: ... instanceNames: - example-cluster-w-0 - example-cluster-w-1 - example-cluster-w-2 - example-cluster-w-3 - example-cluster-w-4 numInstances: 5 statusHistory: ... - detail: Add 3 workers. ``` To decrease the number of worker nodes to the original value of `2`, run the following command: ``` gcloud dataproc clusters update example-cluster \     --region=REGION \     --num-workers 2 ``` ## Clean up To avoid incurring charges to your Google Cloud account for the resources used on this page, follow these steps. 1. To delete the `example-cluster`, run the [`clusters delete`](https://docs.cloud.google.com/sdk/gcloud/reference/dataproc/clusters/delete) command: ``` gcloud dataproc clusters delete example-cluster \     --region=REGION ``` ## What's next - Learn how to [write and run a Spark Scala job](https://docs.cloud.google.com/managed-spark/docs/tutorials/spark-scala).