This document shows you how to create a Managed Service for Apache Spark on GKE virtual cluster,
and then run a Spark job on the cluster.

## Overview of options

While Managed Service for Apache Spark on GKE offers powerful control for containerized environments,
Google Cloud also provides fully managed and serverless options that can
simplify operations and accelerate development. For a comparison of Spark
Managed Service for Apache Spark deployment options, see
[Decide on the best Spark service](https://docs.cloud.google.com/managed-spark/docs/concepts/compare#spark-decision-guide).

## Before you begin

1. You must have created a standard (not autopilot) Google Kubernetes Engine (GKE)
   [**zonal**](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/creating-a-zonal-cluster.md.txt) or [**regional**](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/creating-a-regional-cluster.md.txt) cluster
   that has [Workload Identity](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/workload-identity.md.txt)
   enabled on the cluster.

   > [!IMPORTANT]
   > **Performance tip:** Enable [image streaming](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/image-streaming.md.txt) for faster workload initialization.

<br />

### Required roles

Certain IAM roles are required to
run the examples on this page. Depending on organization policies, these
roles may have already been granted. To check role grants, see
[Do you need to grant roles?](https://docs.cloud.google.com/managed-spark/docs/concepts/iam#do_you_need_to_grant_roles).

For more information about granting roles, see
[Manage access to projects, folders, and organizations](https://docs.cloud.google.com/iam/docs/granting-changing-revoking-access.md.txt).

#### User roles


To get the permissions that
you need to create a Managed Service for Apache Spark cluster,

ask your administrator to grant you the
following IAM roles:

- [Dataproc Editor](https://docs.cloud.google.com/iam/docs/roles-permissions/dataproc.md.txt#dataproc.editor) (`roles/dataproc.editor`) on the project
- [Service Account User](https://docs.cloud.google.com/iam/docs/roles-permissions/iam.md.txt#iam.serviceAccountUser) (`roles/iam.serviceAccountUser`) on the Compute Engine default service account


#### Service account role


To ensure that the Compute Engine default service account has the necessary
permissions to create a Managed Service for Apache Spark cluster,

ask your administrator to grant the
[Dataproc Worker](https://docs.cloud.google.com/iam/docs/roles-permissions/dataproc.md.txt#dataproc.worker) (`roles/dataproc.worker`) IAM role to the Compute Engine default service account on the project.


**Important:** You must grant this role to the Compute Engine default service account, *not* to your user account. Failure to grant the role to the correct principal might result in permission errors.

<br />

## Create a virtual cluster

A Managed Service for Apache Spark on GKE virtual cluster is created as the deployment platform for
Managed Service for Apache Spark components. It's a **virtual** resource, and unlike a
Managed Service for Apache Spark on Compute Engine cluster, does not include separate Managed Service for Apache Spark
master and worker VMs.

- Managed Service for Apache Spark on GKE creates node pools within a GKE
  cluster when you create a Managed Service for Apache Spark on GKE virtual cluster.

- Managed Service for Apache Spark on GKE jobs are run as pods on these node pools. The node pools
  and scheduling of pods on the node pools are managed by GKE.

- **Create multiple virtual clusters.** You can create and run multiple virtual
  clusters on a GKE cluster to obtain improved
  resource utilization by sharing node pools across the virtual clusters.

  - Each virtual cluster:
    - is created with separate properties, including Spark engine version and workload identity
    - is isolated within a separate GKE namespace on the GKE cluster

> [!NOTE]
> **Note:** Deletion of one or more Managed Service for Apache Spark on GKE clusters does not delete associated node pools. Node pools are deleted when the GKE cluster is deleted (see [Node pool deletion](https://docs.cloud.google.com/managed-spark/docs/guides/dpgke/gke-nodepools.md.txt#node_pool_deletion)).

### Console

Creating a Managed Service for Apache Spark on GKE cluster is not supported in the Google Cloud console.


### gcloud

Set environment variables, then run the
[`gcloud dataproc clusters gke create`](https://docs.cloud.google.com/sdk/gcloud/reference/dataproc/clusters/gke/create)
command locally or in Cloud Shell to create a Managed Service for Apache Spark on GKE cluster.

<br />

1. Set environment variables:

   ```
   DP_CLUSTER=Managed Service for Apache Spark on GKE  cluster-name \
     REGION=region \
     GKE_CLUSTER=GKE cluster-name \
     BUCKET=Cloud Storage bucket-name \
     DP_POOLNAME=node pool-name
     PHS_CLUSTER=Managed Service for Apache Spark PHS server name
   ```
   Notes:

   <br />

   - `DP_CLUSTER`: Set the Managed Service for Apache Spark virtual cluster name, which must start with a lowercase letter, followed by up to 54 lowercase letters, numbers, or hyphens. It and cannot end with a hyphen.
   - `REGION`: The <var translate="no">[region](https://docs.cloud.google.com/compute/docs/regions-zones.md.txt#available)</var> must be the same as the region where the GKE cluster is located.
   - `GKE_CLUSTER`: The name of your existing GKE cluster.
   - `BUCKET`: (Optional) You can specify the name of a [Cloud Storage bucket](https://docs.cloud.google.com/storage/docs/creating-buckets.md.txt), which Managed Service for Apache Spark will use to stage artifacts. If you do not specify a bucket, Managed Service for Apache Spark on GKE will create a staging bucket.
   - `DP_POOLNAME`: The name of a [node pool](https://docs.cloud.google.com/kubernetes-engine/docs/concepts/node-pools.md.txt) to create on the GKE cluster.
   - `PHS_CLUSTER`: (Optional) [Managed Service for Apache Spark PHS Server](https://docs.cloud.google.com/managed-spark/docs/concepts/jobs/history-server.md.txt#set_up_a_persistent_history_server) to use to view Spark job history on active and deleted Managed Service for Apache Spark on GKE clusters. The PHS cluster must be located in the same region as the Managed Service for Apache Spark on GKE virtual cluster.
2. Run the command:

   ```
   gcloud dataproc clusters gke create ${DP_CLUSTER} \
       --region=${REGION} \
       --gke-cluster=${GKE_CLUSTER} \
       --spark-engine-version=latest \
       --staging-bucket=${BUCKET} \
       --pools="name=${DP_POOLNAME},roles=default" \
       --setup-workload-identity \
       --history-server-cluster=${PHS_CLUSTER}
   ```
   Notes:

   <br />

   - `--spark-engine-version`: The [Spark image version](https://docs.cloud.google.com/managed-spark/docs/guides/dpgke/gke-versions.md.txt) used on the Managed Service for Apache Spark cluster. You can use an identifier, such as `3`, `3.1`, or `latest`, or you can specify the full subminor version, such as `3.1-dataproc-5`.
   - `--staging-bucket`: Delete this flag to have Managed Service for Apache Spark on GKE create a staging bucket.
   - `--pools`: This flag is used to specify a new or existing node pool that Managed Service for Apache Spark will create or use to perform the workload. List [Managed Service for Apache Spark on GKE node pool settings](https://docs.cloud.google.com/managed-spark/docs/guides/dpgke/gke-nodepools.md.txt#node_pool_settings), separated by commas, for example:

     ```
     --pools=name=dp-default,roles=default,machineType=e2-standard-4,min=0,max=10
     ```
     You must specify the node pool `name` and `role`. Other node pool settings are optional. You can use multiple `--pools` flags to specify multiple node pools. At least one node pool must have the `default` role. All node pools must have the same location.
   - `--setup-workload-identity`: This flag enables [Workload Identity](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/workload-identity.md.txt) bindings. These bindings allow the Kubernetes service accounts (KSAs) to act as the default [Managed Service for Apache Spark VM Service Account (Data Plane identity)](https://docs.cloud.google.com/managed-spark/docs/concepts/iam/principals.md.txt#vm_service_account_data_plane_identity) of the virtual cluster. You need elevated permissions to set workload identity on a Google service account (GSA) (see [Managed Service for Apache Spark on Google Kubernetes Engine IAM Permissions](https://docs.cloud.google.com/managed-spark/docs/guides/dpgke/gke-iam.md.txt#data_plane_identity)). To use your own GSA with your Managed Service for Apache Spark on GKE virtual cluster, see [Custom IAM configuration](https://docs.cloud.google.com/managed-spark/docs/guides/dpgke/gke-iam.md.txt#custom_iam_configuration).

### REST

Complete a
[virtualClusterConfig](https://docs.cloud.google.com/managed-spark/docs/reference/rest/v1/projects.regions.clusters.md.txt#VirtualClusterConfig)
as part of an
[`cluster.create`](https://docs.cloud.google.com/managed-spark/docs/reference/rest/v1/projects.regions.clusters/create.md.txt)
API request.


Before using any of the request data,
make the following replacements:

- <var translate="no">PROJECT</var>: Google Cloud project ID
- <var translate="no">REGION</var>: Dataproc virtual cluster [region](https://docs.cloud.google.com/compute/docs/regions-zones/regions-zones#available) (same region as the existing GKE cluster region)
- <var translate="no">DP_CLUSTER</var>: Dataproc cluster name
- <var translate="no">GKE_CLUSTER</var>: GKE cluster name
- <var translate="no">NODE_POOL</var>: Node pool name
- <var translate="no">PHS_CLUSTER</var>: [Persistent History Server (PHS) cluster name](https://docs.cloud.google.com/managed-spark/docs/concepts/jobs/history-server.md.txt)
- <var translate="no">BUCKET</var>: (Optional) Staging bucket name. Leave this empty to have Managed Service for Apache Spark on GKE create a staging bucket.


HTTP method and URL:

```
POST https://dataproc.googleapis.com/v1/projects/project-id/regions/region/clusters
```


Request JSON body:

```
{
  "clusterName":"DP_CLUSTER",
  "projectId":"PROJECT",
  "virtualClusterConfig":{
    "auxiliaryServicesConfig":{
      "sparkHistoryServerConfig":{
        "dataprocCluster":"projects/PROJECT/regions/REGION/clusters/PHS_CLUSTER"
      }
    },
    "kubernetesClusterConfig":{
      "gkeClusterConfig":{
        "gkeClusterTarget":"projects/PROJECT/locations/REGION/clusters/GKE_CLUSTER",
        "nodePoolTarget":[
          {
"nodePool":"projects/PROJECT/locations/REGION/clusters/GKE_CLUSTER/nodePools/NODE_POOL",
            "roles":[
              "DEFAULT"
            ]
          }
        ]
      },
      "kubernetesSoftwareConfig":{
        "componentVersion":{
          "SPARK":"latest"
        }
      }
    },
    "stagingBucket":"BUCKET"
  }
}

```

To send your request, expand one of these options:

#### curl (Linux, macOS, or Cloud Shell)

> [!NOTE]
> **Note:** The following command assumes that you have logged in to the `gcloud` CLI with your user account by running [`gcloud init`](https://docs.cloud.google.com/sdk/gcloud/reference/init) or [`gcloud auth login`](https://docs.cloud.google.com/sdk/gcloud/reference/auth/login.md.txt) , or by using [Cloud Shell](https://docs.cloud.google.com/shell/docs.md.txt), which automatically logs you into the `gcloud` CLI . You can check the currently active account by running [`gcloud auth list`](https://docs.cloud.google.com/sdk/gcloud/reference/auth/list.md.txt).


Save the request body in a file named `request.json`,
and execute the following command:

```
curl -X POST \
     -H "Authorization: Bearer $(gcloud auth print-access-token)" \
     -H "Content-Type: application/json; charset=utf-8" \
     -d @request.json \
     "https://dataproc.googleapis.com/v1/projects/project-id/regions/region/clusters"
```

#### PowerShell (Windows)

> [!NOTE]
> **Note:** The following command assumes that you have logged in to the `gcloud` CLI with your user account by running [`gcloud init`](https://docs.cloud.google.com/sdk/gcloud/reference/init) or [`gcloud auth login`](https://docs.cloud.google.com/sdk/gcloud/reference/auth/login.md.txt) . You can check the currently active account by running [`gcloud auth list`](https://docs.cloud.google.com/sdk/gcloud/reference/auth/list.md.txt).


Save the request body in a file named `request.json`,
and execute the following command:

```
$cred = gcloud auth print-access-token
$headers = @{ "Authorization" = "Bearer $cred" }

Invoke-WebRequest `
    -Method POST `
    -Headers $headers `
    -ContentType: "application/json; charset=utf-8" `
    -InFile request.json `
    -Uri "https://dataproc.googleapis.com/v1/projects/project-id/regions/region/clusters" | Select-Object -Expand Content
```

You should receive a JSON response similar to the following:

```
{
  "projectId":"PROJECT",
  "clusterName":"DP_CLUSTER",
  "status":{
    "state":"RUNNING",
    "stateStartTime":"2022-04-01T19:16:39.865716Z"
  },
  "clusterUuid":"98060b77-...",
  "statusHistory":[
    {
      "state":"CREATING",
      "stateStartTime":"2022-04-01T19:14:27.340544Z"
    }
  ],
  "labels":{
    "goog-dataproc-cluster-name":"DP_CLUSTER",
    "goog-dataproc-cluster-uuid":"98060b77-...",
    "goog-dataproc-location":"REGION",
    "goog-dataproc-environment":"prod"
  },
  "virtualClusterConfig":{
    "stagingBucket":"BUCKET",
    "kubernetesClusterConfig":{
      "kubernetesNamespace":"dp-cluster",
      "gkeClusterConfig":{
"gkeClusterTarget":"projects/PROJECT/locations/REGION/clusters/GKE_CLUSTER",
        "nodePoolTarget":[
          {
"nodePool":"projects/PROJECT/locations/REGION/clusters/GKE_CLUSTER/nodePools/NODE_POOL",
            "roles":[
              "DEFAULT"
            ]
          }
        ]
      },
      "kubernetesSoftwareConfig":{
        "componentVersion":{
          "SPARK":"3.1-..."
        },
        "properties":{
          "dpgke:dpgke.unstable.outputOnly.endpoints.sparkHistoryServer":"https://...",
          "spark:spark.eventLog.dir":"gs://BUCKET/.../spark-job-history",
          "spark:spark.eventLog.enabled":"true"
        }
      }
    },
    "auxiliaryServicesConfig":{
      "sparkHistoryServerConfig":{
        "dataprocCluster":"projects/PROJECT/regions/REGION/clusters/PHS_CLUSTER"
      }
    }
  }
```

<br />

## Submit a Spark job

After your Managed Service for Apache Spark on GKE virtual cluster is running,
[submit a Spark job](https://docs.cloud.google.com/managed-spark/docs/guides/submit-job.md.txt#submit_a_spark_job)
using the Google Cloud console,
[gcloud CLI](https://docs.cloud.google.com/sdk/gcloud/reference/dataproc/jobs/submit), or the
Managed Service for Apache Spark
[`jobs.submit`](https://docs.cloud.google.com/managed-spark/docs/reference/rest/v1/projects.regions.jobs/submit.md.txt)
API (by using direct HTTP requests or the
[Cloud Client Libraries](https://docs.cloud.google.com/apis/docs/cloud-client-libraries.md.txt)).

> [!NOTE]
> **Note:** In the following examples, the job jars are pre-installed and run "locally" on the Managed Service for Apache Spark virtual cluster.

\*\*gcloud CLI Spark job example:\*\*

```
gcloud dataproc jobs submit spark \
    --region=${REGION} \
    --cluster=${DP_CLUSTER} \
    --class=org.apache.spark.examples.SparkPi \
    --jars=local:///usr/lib/spark/examples/jars/spark-examples.jar \
    -- 1000
```

\*\*gcloud CLI PySpark job example:\*\*

```
gcloud dataproc jobs submit pyspark \
    --region=${REGION} \
    --cluster=${DP_CLUSTER} \
    local:///usr/lib/spark/examples/src/main/python/pi.py \
    -- 10
```

\*\*gcloud CLI SparkR job example:\*\*

```
gcloud dataproc jobs submit spark-r \
    --region=${REGION} \
    --cluster=${DP_CLUSTER} \
    local:///usr/lib/spark/examples/src/main/r/dataframe.R
```

## Clean up

- Delete any of the following resources used in this quickstart that
  you do not want to continue to use.

- [Delete the Managed Service for Apache Spark on GKE cluster](https://docs.cloud.google.com/managed-spark/docs/guides/dpgke/gke-delete-cluster.md.txt).

- [Delete node pools](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/node-pools.md.txt#deleting_a_node_pool)
  used by the Managed Service for Apache Spark on GKE cluster.

- [Delete the GKE cluster](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/deleting-a-cluster.md.txt).