## Data plane Identity

Managed Service for Apache Spark on GKE uses
[GKE workload identity](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/workload-identity.md.txt)
to allow pods within the Managed Service for Apache Spark on GKE cluster to act with
the authority of the default
[Managed Service for Apache Spark VM service account (data plane identity)](https://docs.cloud.google.com/managed-spark/docs/concepts/iam/principals.md.txt#vm_service_account_data_plane_identity).

> [!NOTE]
> **Note:** Although granting IAM roles directly to a Kubernetes ServiceAccount principal (`principal://`) is the recommended best practice for [Workload Identity Federation for GKE](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/workload-identity.md.txt), Managed Service for Apache Spark on GKE requires IAM service account impersonation because the `--setup-workload-identity` flag and `dataproc:dataproc.gke.*.google-service-account` cluster properties require an IAM service account email address.

Workload identity requires the following permissions
to update IAM policies on the GSA used by your Managed Service for Apache Spark on GKE
virtual cluster:

- `compute.projects.get`
- `iam.serviceAccounts.getIamPolicy`
- `iam.serviceAccounts.setIamPolicy`

> [!NOTE]
> **Note:** You can use workload identity with a different GSA: see [Custom IAM configuration](https://docs.cloud.google.com/managed-spark/docs/guides/dpgke/gke-iam.md.txt#custom-iam-configuration).

GKE workload identity links the following
GKE Service Accounts (KSAs) to the Managed Service for Apache Spark VM Service Account:

1. `agent` KSA (interacts with Managed Service for Apache Spark control plane):  
   `serviceAccount:${PROJECT}.svc.id.goog[${DPGKE_NAMESPACE}/agent]`
2. `spark-driver` KSA (runs Spark drivers):  
   `serviceAccount:${PROJECT}.svc.id.goog[${DPGKE_NAMESPACE}/spark-driver]`
3. `spark-executor` KSA (runs Spark executors):  
   `serviceAccount:${PROJECT}.svc.id.goog[${DPGKE_NAMESPACE}/spark-executor]`

> [!IMPORTANT]
> Use the `gcloud dataproc clusters gke create --setup-workload-identity` flag when you [create a Managed Service for Apache Spark on GKE cluster](https://docs.cloud.google.com/managed-spark/docs/guides/dpgke/quickstarts/dataproc-gke-quickstart-create-cluster) to create the workload identity bindings required for the cluster.

### Assign roles

Grant permissions to the
[Managed Service for Apache Spark VM service account](https://docs.cloud.google.com/managed-spark/docs/concepts/iam/principals.md.txt#vm_service_account_data_plane_identity)
to allow the `spark-driver` and `spark-executor` to access project resources,
data sources, data sinks, and any other services required by your workload.

Example:

The following command assigns roles to the
[default Managed Service for Apache Spark VM service account](https://docs.cloud.google.com/managed-spark/docs/concepts/configuring-clusters/service-accounts.md.txt#dataproc_service_accounts_2)
to allow Spark workloads running on
Managed Service for Apache Spark on GKE cluster VMs to access Cloud Storage buckets and
BigQuery data sets in the project.

```
gcloud projects add-iam-policy-binding \
    --role=roles/storage.objectAdmin \
    --role=roles/bigquery.dataEditor \
    --member="project-number-compute@developer.gserviceaccount.com" \
    "${PROJECT}"
```

## Custom IAM configuration

Managed Service for Apache Spark on GKE uses
[GKE workload identity](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/workload-identity.md.txt)
to link the default
[Managed Service for Apache Spark VM service account (data plane identity)](https://docs.cloud.google.com/managed-spark/docs/concepts/iam/principals.md.txt#vm_service_account_data_plane_identity)
to the three [GKE service accounts (KSAs)](https://docs.cloud.google.com/managed-spark/docs/guides/dpgke/gke-iam.md.txt#ksa-sas).

To create and use a different Google service account (GSA) to link to
the KSAs:

1. Create the GSA (see
   [Creating and managing service accounts](https://docs.cloud.google.com/iam/docs/creating-managing-service-accounts)).

   gcloud CLI example:

   ```
   gcloud iam service-accounts create "dataproc-${USER}" \
       --description "Used by Managed Service for Apache Spark on GKE workloads."
   ```
   Notes:

   <br />

   - The example sets the GSA name as "dataproc-${USER}", but you can use a different name.
2. Set environmental variables:

   ```
   PROJECT=project-id \
     DPGKE_GSA="dataproc-${USER}@${PROJECT}.iam.gserviceaccount.com"
     DPGKE_NAMESPACE=GKE namespace
   ```
   Notes:

   <br />

   - `DPGKE_GSA`: The examples set and use `DPGKE_GSA` as the name of the variable that contains the email address of your GSA. You can set and use a different variable name.
   - `DPGKE_NAMESPACE`: The default GKE namespace is the name of your Managed Service for Apache Spark on GKE cluster.
3. When you create the Managed Service for Apache Spark on GKE cluster, add the following properties
   for Managed Service for Apache Spark to use your GSA instead of the default GSA:

   ```
   --properties "dataproc:dataproc.gke.agent.google-service-account=${DPGKE_GSA}" \
   --properties "dataproc:dataproc.gke.nodepool.google-service-account=${DPGKE_GSA}" \
   --properties "dataproc:dataproc.gke.spark.driver.google-service-account=${DPGKE_GSA}" \
   --properties "dataproc:dataproc.gke.spark.executor.google-service-account=${DPGKE_GSA}" \
   ```

   <br />

4. Run the following commands to assign necessary
   [Workload Identity](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/workload-identity.md.txt)
   permissions to the service accounts:

   1. Assign your GSA the `dataproc.worker` role to allow it to act as agent:

      ```
      gcloud projects add-iam-policy-binding \
          --role=roles/dataproc.worker \
          --member="serviceAccount:${DPGKE_GSA}" \
          "${PROJECT}"
      ```
   2. Assign the `agent` KSA the `iam.workloadIdentityUser` role to
      allow it to act as your GSA:

      ```
      gcloud iam service-accounts add-iam-policy-binding \
          --role=roles/iam.workloadIdentityUser \
          --member="serviceAccount:${PROJECT}.svc.id.goog[${DPGKE_NAMESPACE}/agent]" \
          "${DPGKE_GSA}"
      ```

      <br />

   3. Grant the `spark-driver` KSA the `iam.workloadIdentityUser` role to
      allow it to act as your GSA:

      ```
      gcloud iam service-accounts add-iam-policy-binding \
          --role=roles/iam.workloadIdentityUser \
          --member="serviceAccount:${PROJECT}.svc.id.goog[${DPGKE_NAMESPACE}/spark-driver]" \
          "${DPGKE_GSA}"
      ```

      <br />

   4. Grant the `spark-executor` KSA the `iam.workloadIdentityUser` role to
      allow it to act as your GSA:

      ```
      gcloud iam service-accounts add-iam-policy-binding \
          --role=roles/iam.workloadIdentityUser \
          --member="serviceAccount:${PROJECT}.svc.id.goog[${DPGKE_NAMESPACE}/spark-executor]" \
          "${DPGKE_GSA}"
      ```

      <br />