Use the Managed Service for Apache Spark MCP server

This document shows you how to use the Managed Service for Apache Spark remote Model Context Protocol (MCP) server to connect with AI applications including Gemini CLI, ChatGPT, Claude, and custom applications you are developing. The Managed Service for Apache Spark remote MCP server lets you perform cluster-based operations, such as creating a cluster, submitting a job, or creating an autoscaling policy from your AI application.

The Managed Service for Apache Spark remote MCP server is enabled when you enable the Managed Service for Apache Spark API.

Model Context Protocol (MCP) standardizes how large language models (LLMs) and AI applications or agents connect to external data sources. MCP servers let you use their tools, resources, and prompts to take actions and get updated data from their backend service.

What's the difference between local and remote MCP servers?

Local MCP servers

Typically run on your local machine and use the standard input and output streams (stdio) for communication between services on the same device.

Remote MCP servers

Run on the service's infrastructure and offer an HTTP endpoint to AI applications for communication between the AI MCP client and the MCP server. For more information about MCP architecture, see MCP architecture.

For information on the Managed Service for Apache Spark local MCP server, see An introduction to MCP Toolbox for Databases.

Google and Google Cloud remote MCP servers

Google and Google Cloud remote MCP servers have the following features and benefits:

• Simplified, centralized discovery.
• Managed global or regional HTTP endpoints.
• Fine-grained authorization.
• Optional prompt and response security with Model Armor protection.
• Centralized audit logging.

For information about other MCP servers and information about security and governance controls available for Google Cloud MCP servers, see Google Cloud MCP servers overview.

Before you begin

Sign in to your Google Cloud account. If you're new to Google Cloud, create an account to evaluate how our products perform in real-world scenarios. New customers also get $300 in free credits to run, test, and deploy workloads.

In the Google Cloud console, on the project selector page, select or create a Google Cloud project.

Roles required to select or create a project

• Select a project: Selecting a project doesn't require a specific IAM role—you can select any project that you've been granted a role on.
• Create a project: To create a project, you need the Project Creator role (roles/resourcemanager.projectCreator), which contains the resourcemanager.projects.create permission. Learn how to grant roles.

Note: If you don't plan to keep the resources that you create in this procedure, create a project instead of selecting an existing project. After you finish these steps, you can delete the project, removing all resources associated with the project.

Go to project selector

Verify that you have the permissions required to complete this guide.

Verify that billing is enabled for your Google Cloud project.

Enable the Managed Service for Apache Spark, Cloud Storage, and BigQuery APIs.

Roles required to enable APIs

To enable APIs, you need the Service Usage Admin IAM role (roles/serviceusage.serviceUsageAdmin), which contains the serviceusage.services.enable permission. Learn how to grant roles.

Enable the APIs

In the Google Cloud console, on the project selector page, select or create a Google Cloud project.

Roles required to select or create a project

• Select a project: Selecting a project doesn't require a specific IAM role—you can select any project that you've been granted a role on.
• Create a project: To create a project, you need the Project Creator role (roles/resourcemanager.projectCreator), which contains the resourcemanager.projects.create permission. Learn how to grant roles.

Note: If you don't plan to keep the resources that you create in this procedure, create a project instead of selecting an existing project. After you finish these steps, you can delete the project, removing all resources associated with the project.

Go to project selector

Verify that you have the permissions required to complete this guide.

Verify that billing is enabled for your Google Cloud project.

Enable the Managed Service for Apache Spark, Cloud Storage, and BigQuery APIs.

Roles required to enable APIs

To enable APIs, you need the Service Usage Admin IAM role (roles/serviceusage.serviceUsageAdmin), which contains the serviceusage.services.enable permission. Learn how to grant roles.

Enable the APIs

Required roles

Certain IAM roles are required to run the examples on this page. Depending on organization policies, these roles may have already been granted. To check role grants, see Do you need to grant roles?.

For more information about granting roles, see Manage access to projects, folders, and organizations.

User roles

To get the permissions that you need to use the Managed Service for Apache Spark service, ask your administrator to grant you the following IAM roles:

• Dataproc Editor (roles/dataproc.editor) on the project
• Service Account User (roles/iam.serviceAccountUser) on the Compute Engine default service account

To get the permissions that you need to use the Managed Service for Apache Spark MCP server, ask your administrator to grant you the following IAM roles on the project:

• Make MCP tool calls: MCP Tool User (roles/mcp.toolUser)

These predefined roles contain the permissions required to use the Managed Service for Apache Spark MCP server. To see the exact permissions that are required, expand the Required permissions section:

Required permissions

The following permissions are required to use the Managed Service for Apache Spark MCP server:

• Make MCP tool calls: mcp.tools.call

Service account role

To ensure that the Compute Engine default service account has the necessary permissions to create clusters and submit jobs, ask your administrator to grant the Dataproc Worker (roles/dataproc.worker) IAM role to the Compute Engine default service account on the project.

This predefined role contains the permissions required to create clusters and submit jobs. To see the exact permissions that are required, expand the Required permissions section:

Required permissions

The following permissions are required to create clusters and submit jobs:

• Make MCP tool calls: mcp.tools.call

Authentication and authorization

The Managed Service for Apache Spark remote MCP server uses the OAuth 2.0 protocol with Identity and Access Management (IAM) for authentication and authorization. All Google Cloud identities are supported for authentication to MCP servers.

The Managed Service for Apache Spark remote MCP server doesn't accept API keys.

We recommend that you create a separate identity for agents that are using MCP tools so that access to resources can be controlled and monitored. For more information about authentication, see Authenticate to MCP servers.

Managed Service for Apache Spark MCP OAuth scopes

OAuth 2.0 uses scopes and credentials to determine if an authenticated principal is authorized to take a specific action on a resource. For more information about OAuth 2.0 scopes at Google, read Using OAuth 2.0 to access Google APIs.

Managed Service for Apache Spark has the following MCP tool OAuth scopes:

Scope URI for gcloud CLI	Description
https://www.googleapis.com/auth/dataproc	View and manage your data in Managed Service for Apache Spark and see the email address for your Google Account

Additional scopes might be required on the resources accessed during a tool call. To view a list of scopes required for Managed Service for Apache Spark, see Managed Service for Apache Spark API.

Configure an MCP client to use the Managed Service for Apache Spark MCP server

AI applications and agents, such as Claude or Gemini CLI, can instantiate an MCP client that connects to a single MCP server. An AI application can have multiple clients that connect to different MCP servers. To connect to a remote MCP server, the MCP client must know the remote MCP server's URL.

In your AI application, look for a way to connect to a remote MCP server. You are prompted to enter details about the server, such as its name and URL.

For the Managed Service for Apache Spark MCP server, enter the following as required:

• Server name: Managed Service for Apache Spark MCP server
• Server URL or Endpoint: https://dataproc-us-central1.googleapis.com/mcp
• Transport: HTTP
• Authentication details: Depending on how you want to authenticate, you can enter your Google Cloud credentials, your OAuth Client ID and secret, or an agent identity and credentials. For more information about authentication, see Authenticate to MCP servers.
• OAuth scope: the OAuth 2.0 scope that you want to use when connecting to the Managed Service for Apache Spark MCP server. For more information, see Managed Service for Apache Spark MCP OAuth scopes.

For host-specific guidance about setting up and connecting to MCP server, see the following:

• Claude.ai
• Gemini CLI

For more general guidance, see the following resources:

• Connect to remote MCP servers.
• Configure MCP in an AI application.

Available tools

MCP tools that are read-only have the MCP attribute mcp.tool.isReadOnly set to true. You might want to only allow read-only tools in certain environments through your organization policy.

To view details of available MCP tools and their descriptions for the Managed Service for Apache Spark MCP server, see the Managed Service for Apache Spark MCP reference.

List tools

Use the MCP inspector to list tools, or send a tools/list HTTP request directly to the Managed Service for Apache Spark remote MCP server. The tools/list method doesn't require authentication.

POST /mcp HTTP/1.1
Host: dataproc.googleapis.com
Content-Type: application/json

{
  "jsonrpc": "2.0",
  "method": "tools/list",
}

Example use cases

The following are sample use cases for the Managed Service for Apache Spark MCP server:

• "List the Managed Service for Apache Spark clusters in PROJECT_ID and REGION."
• "Get the latest status for CLUSTER_NAME cluster in PROJECT_ID and REGION."
• "Delete CLUSTER_NAME in PROJECT_ID and REGION and poll till completion. Report any errors if it failed."

Replace the following:

• PROJECT_ID: The Google Cloud project ID. Project IDs are listed in the Project info section IN the Google Cloud console Dashboard.
• REGION: The Compute Engine region where the Managed Service for Apache Spark cluster is located.
• CLUSTER_NAME: The name of the existing Managed Service for Apache Spark cluster.

Optional security and safety configurations

MCP introduces new security risks and considerations due to the wide variety of actions that you can do with the MCP tools. To minimize and manage these risks, Google Cloud offers default settings and customizable policies to control the use of MCP tools in your Google Cloud organization or project.

For more information about MCP security and governance, see AI security and safety.

Use Model Armor

Model Armor is a Google Cloud service designed to enhance the security and safety of your AI applications. It works by proactively screening LLM prompts and responses, protecting against various risks and supporting responsible AI practices. Whether you are deploying AI in your cloud environment, or on external cloud providers, Model Armor can help you prevent malicious input, verify content safety, protect sensitive data, maintain compliance, and enforce your AI safety and security policies consistently across your diverse AI landscape.

When Model Armor is enabled with logging enabled, Model Armor logs the entire payload. This might expose sensitive information in your logs.

Caution: Model Armor is available in certain regions. When Model Armor is enabled and you use an MCP server in a jurisdiction that Model Armor doesn't support, the routing behavior of the call might be different for different MCP servers. For more information about the behavior of individual MCP servers, see Model Armor supported products.

Enable Model Armor

You must enable Model Armor APIs before you can use Model Armor.

Console

1. Enable the Model Armor API.

   Roles required to enable APIs

   To enable APIs, you need the Service Usage Admin IAM role (roles/serviceusage.serviceUsageAdmin), which contains the serviceusage.services.enable permission. Learn how to grant roles.

   Enable the API

2. Select the project where you want to activate Model Armor.

gcloud

Before you begin, follow these steps using the Google Cloud CLI with the Model Armor API:

1. In the Google Cloud console, activate Cloud Shell.

   Activate Cloud Shell

   At the bottom of the Google Cloud console, a Cloud Shell session starts and displays a command-line prompt. Cloud Shell is a shell environment with the Google Cloud CLI already installed and with values already set for your current project. It can take a few seconds for the session to initialize.

2. Run the following command to set the API endpoint for the Model Armor service.

   gcloud config set api_endpoint_overrides/modelarmor "https://modelarmor.LOCATION.rep.googleapis.com/"

   Replace LOCATION with the region where you want to use Model Armor.

Configure protection for Google and Google Cloud remote MCP servers

To help protect your MCP tool calls and responses you can use Model Armor floor settings. A floor setting defines the minimum security filters that apply across the project. This configuration applies a consistent set of filters to all MCP tool calls and responses within the project.

Tip: Don't enable the prompt injection and jailbreak filter unless your MCP traffic carries natural language data.

Set up a Model Armor floor setting with MCP sanitization enabled. For more information, see Configure Model Armor floor settings.

Note: If the agent and the MCP server are in different projects, you can create floor settings in both projects (the client project and the resource project). In this case, Model Armor is invoked twice, once for each project.

See the following example command:

gcloud model-armor floorsettings update \
--full-uri='projects/PROJECT_ID/locations/global/floorSetting' \
--enable-floor-setting-enforcement=TRUE \
--add-integrated-services=GOOGLE_MCP_SERVER \
--google-mcp-server-enforcement-type=INSPECT_AND_BLOCK \
--enable-google-mcp-server-cloud-logging \
--malicious-uri-filter-settings-enforcement=ENABLED \
--add-rai-settings-filters='[{"confidenceLevel": "MEDIUM_AND_ABOVE", "filterType": "DANGEROUS"}]'

Replace PROJECT_ID with your Google Cloud project ID.

Note the following settings:

• INSPECT_AND_BLOCK: The enforcement type that inspects content for the Google MCP server and blocks prompts and responses that match the filters.
• ENABLED: The setting that enables a filter or enforcement.
• MEDIUM_AND_ABOVE: The confidence level for the Responsible AI - Dangerous filter settings. You can modify this setting, though lower values might result in more false positives. For more information, see Model Armor confidence levels.

Disable scanning MCP traffic with Model Armor

If you want to stop scanning Google MCP traffic with Model Armor, run the following command:

gcloud model-armor floorsettings update \
  --full-uri='projects/PROJECT_ID/locations/global/floorSetting' \
  --remove-integrated-services=GOOGLE_MCP_SERVER

Replace PROJECT_ID with the Google Cloud project ID.

Model Armor won't scan MCP traffic in the project.

Control MCP use with IAM deny policies

Identity and Access Management (IAM) deny policies help you secure Google Cloud remote MCP servers. Configure these policies to block unwanted MCP tool access.

For example, you can deny or allow access based on:

• The principal
• Tool properties like read-only
• The application's OAuth client ID

For more information, see Control MCP use with Identity and Access Management.

What's next

• Read the Managed Service for Apache Spark MCP reference documentation.
• Learn more about Google Cloud MCP servers.