Identity and Access Management (IAM) lets you control access to your project's resources. This document focuses on the IAM permissions relevant to Serverless for Apache Spark and the IAM roles that grant those permissions.
Dataproc permissions for Serverless for Apache Spark
Dataproc permissions allow users and
service accounts,
to perform actions on Serverless for Apache Spark
resources. For example, the dataproc.batches.create
permission lets you create batch workloads in a project.
You don't directly give users permissions; instead, you grant them IAM roles, which have one or more permissions bundled within them. You can grant predefined roles that contain a list of permissions, or you can create and grant custom roles that contain one or more permissions that you include in the custom role.
The following tables list the basic permissions necessary to call
Dataproc APIs (methods) that create or access Serverless for Apache Spark
resources. The tables are organized according to the APIs associated with each
Serverless for Apache Spark resource, which include batches, sessions,
sessionTemplates, and operations.
Examples:
dataproc.batches.createallows the creation of batches in the containing project.dataproc.sessions.createallows the creation of interactive sessions in the containing project.
Batch permissions
| Method | Required Permission(s) |
|---|---|
| projects.locations.batches.create | dataproc.batches.create 1 |
| projects.locations.batches.delete | dataproc.batches.delete |
| projects.locations.batches.get | dataproc.batches.get |
| projects.locations.batches.list | dataproc.batches.list |
1 dataproc.batches.create also requires dataproc.batches.get and
dataproc.operations.get permissions to allow it to get status updates
from the gcloud command-line tool.
Session permissions
| Method | Required Permission(s) |
|---|---|
| projects.locations.sessions.create | dataproc.sessions.create 1 |
| projects.locations.sessions.delete | dataproc.sessions.delete |
| projects.locations.sessions.get | dataproc.sessions.get |
| projects.locations.sessions.list | dataproc.sessions.list |
| projects.locations.sessions.terminate | dataproc.sessions.terminate |
1 dataproc.sessions.create also requires dataproc.sessions.get and
dataproc.operations.get permissions to allow it to get status updates
from the gcloud command-line tool.
Session template permissions
| Method | Required Permission(s) |
|---|---|
| projects.locations.sessionTemplates.create | dataproc.sessionTemplates.create 1 |
| projects.locations.sessionTemplates.delete | dataproc.sessionTemplates.delete |
| projects.locations.sessionTemplates.get | dataproc.sessionTemplates.get |
| projects.locations.sessionTemplates.list | dataproc.sessionTemplates.list |
| projects.locations.sessionTemplates.update | dataproc.sessionTemplates.update |
1 dataproc.sessionTemplates.create also requires dataproc.sessionTemplates.get and
dataproc.operations.get permissions to allow it to get status updates
from the gcloud command-line tool.
Operations permissions
| Method | Required Permission(s) |
|---|---|
| projects.regions.operations.get | dataproc.operations.get |
| projects.regions.operations.list | dataproc.operations.list |
| projects.regions.operations.cancel 1 | dataproc.operations.cancel |
| projects.regions.operations.delete | dataproc.operations.delete |
| projects.regions.operations.getIamPolicy | dataproc.operations.getIamPolicy |
| projects.regions.operations.setIamPolicy | dataproc.operations.setIamPolicy |
1 To cancel batch operations, dataproc.operations.cancel also requires
dataproc.batches.cancel permission.
Serverless for Apache Spark 3.0+ runtime permissions
The following permissions apply to Serverless for Apache Spark 3.0 and
later runtimes.
Workloads permissions
| Method | Required Permission(s) |
|---|---|
| dataprocrm.v1.dataprocrm.projects.locations.workloads.create | dataprocrm.workloads.create |
| dataprocrm.v1.dataprocrm.projects.locations.workloads.cancel | dataprocrm.workloads.cancel |
| dataprocrm.v1.dataprocrm.projects.locations.workloads.delete | dataprocrm.workloads.delete |
| dataprocrm.v1.dataprocrm.projects.locations.workloads.get | dataprocrm.workloads.get |
| dataprocrm.v1.dataprocrm.projects.locations.workloads.list | dataprocrm.workloads.list |
| dataprocrm.v1.dataprocrm.projects.locations.workloads.use | dataprocrm.workloads.use |
NodePools permissions
| Method | Required Permission(s) |
|---|---|
| dataprocrm.v1.dataprocrm.projects.locations.workloads.nodePools.create | dataprocrm.nodePools.create |
| dataprocrm.v1.dataprocrm.projects.locations.workloads.nodePools.delete | dataprocrm.nodePools.delete |
| dataprocrm.v1.dataprocrm.projects.locations.workloads.nodePools.resize | dataprocrm.nodePools.resize |
| dataprocrm.v1.dataprocrm.projects.locations.workloads.nodePools.deleteNodes | dataprocrm.nodePools.deleteNodes |
| dataprocrm.v1.dataprocrm.projects.locations.workloads.nodePools.update | dataprocrm.nodePools.update |
| dataprocrm.v1.dataprocrm.projects.locations.workloads.nodePools.get | dataprocrm.nodePools.get |
| dataprocrm.v1.dataprocrm.projects.locations.workloads.nodePools.list | dataprocrm.nodePools.list |
Nodes permissions
| Method | Required Permission(s) |
|---|---|
| dataprocrm.v1.dataprocrm.projects.locations.workloads.nodePools.nodes.create | dataprocrm.nodes.create |
| dataprocrm.v1.dataprocrm.projects.locations.workloads.nodePools.nodes.delete | dataprocrm.nodes.delete |
| dataprocrm.v1.dataprocrm.projects.locations.workloads.nodePools.nodes.update | dataprocrm.nodes.update |
| dataprocrm.v1.dataprocrm.projects.locations.workloads.nodePools.nodes.heartbeat | dataprocrm.nodes.heartbeat |
| dataprocrm.v1.dataprocrm.projects.locations.workloads.nodePools.nodes.get | dataprocrm.nodes.get |
| dataprocrm.v1.dataprocrm.projects.locations.workloads.nodePools.nodes.list | dataprocrm.nodes.list |
| dataprocrm.v1.dataprocrm.projects.locations.workloads.nodePools.nodes.mintOAuthToken | dataprocrm.nodes.mintOAuthToken |
Operations permissions
| Method | Required Permission(s) |
|---|---|
| dataprocrm.v1.dataprocrm.projects.locations.operations.get | dataprocrm.operations.get |
| dataprocrm.v1.dataprocrm.projects.locations.operations.list | dataprocrm.operations.list |
Serverless for Apache Spark role requirements
The following table lists roles that contain the permissions required to manage batch workloads and sessions. The requirements can vary depending on the batch or session runtime version and whether the batch or session is running with service account or end-user credentials (EUC).
| Runtime version | IAM roles |
|---|---|
pre-3.0 |
Grant users the following roles:
|
3.0+ |
Grant users the following roles:
|
Notes:
When submitting a batch workload or creating an interactive session with the a
3.0+runtime and end user credentials (the 3.0+ default), dataplane system operations are executed by the Dataproc Resource Manager Node Service Agent. For more information, see 3.0+ runtime service agent service account.For backward compatibility, the legacy Dataproc Editor and Dataproc Viewer roles can be granted with
3.0+runtimes instead of the Dataproc Serverless Editor and Dataproc Serverless Viewer roles. Also, the Dataproc Worker role can be granted instead of the Dataproc Serverless Node role.If a service account has been granted the project Editor role, it contains the permissions included in the Dataproc Worker role.
For more information: see Serverless for Apache Spark service accounts.
Do you need to grant roles?
Depending on your organization policy, a required role may already have been granted.
Check roles granted to users
To see if a user has been granted a role, follow the instructions in Manage access to projects, folders, and organizations > View current access.
Check roles granted to service accounts
To see if the a service account has been granted a role, see View and manage IAM service account roles.
Check roles granted on a service account
To see if a user has been granted a role on a service account, follow the instructions in Manage access to service accounts > View current access.
Lookup Dataproc roles and permissions
You can use the following sections to lookup Dataproc roles and permissions.
| Role | Permissions |
|---|---|
Dataproc Administrator( Full control of Dataproc resources. |
|
Dataproc Editor( Provides the permissions necessary for viewing the resources required to manage Dataproc, including machine types, networks, projects, and zones. Lowest-level resources where you can grant this role:
|
|
Dataproc Hub Agent( Allows management of Dataproc resources. Intended for service accounts running Dataproc Hub instances. |
|
Dataproc Serverless Editor( Permissions needed to run serverless sessions and batches as a user |
|
Dataproc Serverless Node.( Node access to Dataproc Serverless sessions and batches. Intended for service accounts. |
|
Dataproc Serverless Viewer( Permissions needed to view serverless sessions and batches |
|
Dataproc Service Agent( Gives Dataproc Service Account access to service accounts, compute resources, storage resources, and kubernetes resources. Includes access to service accounts. |
|
Dataproc Viewer( Provides read-only access to Dataproc resources. Lowest-level resources where you can grant this role:
|
|
Dataproc Worker( Provides worker access to Dataproc resources. Intended for service accounts. |
|
Project roles
You can also set permissions at the project level by using IAM Project roles. The following table summarizes the permissions associated with IAM project roles:
| Project Role | Permissions |
|---|---|
| Project Viewer | All project permissions for read-only actions that preserve state (get, list) |
| Project Editor | All Project Viewer permissions plus all project permissions for actions that modify state (create, delete, update, use, cancel, stop, start) |
| Project Owner | All Project Editor permissions plus permissions to manage access control for the project (get/set IamPolicy) and to set up project billing |
What's next
- Learn how to Manage access to projects, folders, and organizations.