# Use the Managed Service for Apache Spark MCP server

<br />

This document shows you how to use the Managed Service for Apache Spark remote Model Context Protocol (MCP) server to connect with AI applications including Antigravity, ChatGPT, Claude, and custom applications you are developing. The Managed Service for Apache Spark remote MCP server lets you perform cluster-based operations, such as creating a cluster, submitting a job, or creating an autoscaling policy from your AI application The Managed Service for Apache Spark remote MCP server is enabled when you enable the Managed Service for Apache Spark API.

[Model Context Protocol](https://modelcontextprotocol.io/docs/getting-started/intro)
(MCP) standardizes how large language models (LLMs) and AI applications or
agents connect to external data sources. MCP servers let you use their tools,
resources, and prompts to take actions and get updated data from their backend
service.

## What's the difference between local and remote MCP servers?

Local MCP servers
:   Typically run on your local machine and use the standard input
    and output streams (stdio) for communication between services on the same
    device.

Remote MCP servers
:   Run on the service's infrastructure and offer an HTTP
    endpoint to AI applications for communication between the AI MCP client and
    the MCP server. For more information about MCP architecture, see
    [MCP architecture](https://modelcontextprotocol.io/docs/learn/architecture).

## Stateless core

With
[MCP version 2026-07-28](https://modelcontextprotocol.io/specification/2026-07-28),
MCP changes from a bidirectional, stateful protocol to a stateless protocol.
Each MCP request is self-describing and can be routed using headers. There isn't
a need for the `initialize`/`initialized` handshake or `Mcp-Session-Id` because
each request includes all the information needed in HTTP headers or the `_meta`
parameter. MCP servers can request additional information required by a tool
through
[multi-round-trip requests (MRTR)](https://modelcontextprotocol.io/specification/latest/basic/patterns/mrtr).

To help route and process requests without parsing the request body, some MCP
headers are required, including the following:

- Headers that are required by the MCP specification such as the [protocol version header](https://modelcontextprotocol.io/specification/latest/basic/transports/streamable-http#protocol-version-header) and [standard request headers](https://modelcontextprotocol.io/specification/latest/basic/transports/streamable-http#standard-request-headers).
- [Custom headers](https://modelcontextprotocol.io/specification/latest/basic/transports/streamable-http#custom-headers-from-tool-parameters) that are defined by the MCP server. These headers are mirrored into HTTP headers from the tool's input schema using the `x-mcp-header` property. For example, an MCP server might define a custom header to specify the Google Cloud region or project ID.

For more information about MCP architecture, see the MCP version 2026-07-28
[specification](https://modelcontextprotocol.io/specification/2026-07-28) and
[key changes](https://modelcontextprotocol.io/specification/2026-07-28/changelog).

For information on the Managed Service for Apache Spark local MCP server, see [An introduction to MCP Toolbox for Databases](https://mcp-toolbox.dev/documentation/introduction/).

## Google and Google Cloud remote MCP servers

Google and Google Cloud remote MCP servers have the following features and benefits:

<br />

- Simplified, centralized discovery
- Managed global or regional HTTP endpoints
- Fine-grained authorization
- Optional prompt and response security with Model Armor protection
- Centralized audit logging

For information about other MCP servers and information about security
and governance controls available for Google Cloud MCP servers,
see [Google Cloud MCP servers overview](https://docs.cloud.google.com/mcp/overview.md.txt).

## Before you begin

<br />

### Required roles

Certain IAM roles are required to
run the examples on this page. Depending on organization policies, these
roles may have already been granted. To check role grants, see
[Do you need to grant roles?](https://docs.cloud.google.com/dataproc/docs/concepts/iam#do_you_need_to_grant_roles).

For more information about granting roles, see
[Manage access to projects, folders, and organizations](https://docs.cloud.google.com/iam/docs/granting-changing-revoking-access.md.txt).

#### User roles


To get the permissions that
you need to use Managed Service for Apache Spark,

ask your administrator to grant you the
following IAM roles:

- [Dataproc Editor](https://docs.cloud.google.com/iam/docs/roles-permissions/dataproc.md.txt#dataproc.editor) (`roles/dataproc.editor`) on the project
- [Service Account User](https://docs.cloud.google.com/iam/docs/roles-permissions/iam.md.txt#iam.serviceAccountUser) (`roles/iam.serviceAccountUser`) on the Compute Engine default service account


To get the permissions that
you need to use the Managed Service for Apache Spark MCP server,

ask your administrator to grant you the
following IAM roles on the project:

- Make MCP tool calls: [MCP Tool User](https://docs.cloud.google.com/iam/docs/roles-permissions/mcp.md.txt#mcp.toolUser) (`roles/mcp.toolUser`)


These predefined roles contain

the permissions required to use the Managed Service for Apache Spark MCP server. To see the exact permissions that are
required, expand the **Required permissions** section:


#### Required permissions

The following permissions are required to use the Managed Service for Apache Spark MCP server:

- Make MCP tool calls: `mcp.tools.call`

#### Service account role


To ensure that the Compute Engine default service account has the necessary
permissions to create clusters and submit jobs,

ask your administrator to grant the
[Dataproc Worker](https://docs.cloud.google.com/iam/docs/roles-permissions/dataproc.md.txt#dataproc.worker) (`roles/dataproc.worker`) IAM role to the Compute Engine default service account on the project.


This predefined role contains

the permissions required to create clusters and submit jobs. To see the exact permissions that are
required, expand the **Required permissions** section:


#### Required permissions

The following permissions are required to create clusters and submit jobs:

- Make MCP tool calls: `mcp.tools.call`

## Authentication and authorization

The Managed Service for Apache Spark remote MCP server uses the [OAuth 2.0](https://developers.google.com/identity/protocols/oauth2) protocol with [Identity and Access Management (IAM)](https://docs.cloud.google.com/iam/docs/overview.md.txt) for authentication and authorization. All [Google Cloud identities](https://docs.cloud.google.com/docs/authentication/identity-products.md.txt) are supported for authentication to MCP servers.

The Managed Service for Apache Spark remote MCP server doesn't accept API keys.

We recommend that you create a separate identity for agents that are using MCP
tools so that access to resources can be controlled and monitored. For more
information about authentication, see
[Authenticate to MCP servers](https://docs.cloud.google.com/mcp/authenticate-mcp.md.txt).

## Managed Service for Apache Spark MCP OAuth scopes

OAuth 2.0 uses scopes and credentials to determine if an authenticated
principal is authorized to take a specific action on a resource. For more
information about OAuth 2.0 scopes at Google, read
[Using OAuth 2.0 to access Google APIs](https://developers.google.com/identity/protocols/oauth2).

Managed Service for Apache Spark has the following MCP tool OAuth scopes:

| Scope URI for gcloud CLI | Description |
|---|---|
| `https://www.googleapis.com/auth/dataproc` | View and manage your data in Managed Service for Apache Spark and see the email address for your Google Account |

Additional scopes might be required on the resources accessed during a tool
call. To view a list of scopes required for
Managed Service for Apache Spark, see
[Managed Service for Apache Spark API](https://developers.google.com/identity/protocols/oauth2/scopes#dataproc).

## Configure an MCP client to use the Managed Service for Apache Spark MCP server

AI applications and agents, such as Claude or Antigravity, can instantiate an MCP client that connects to a single MCP server. An AI application can have multiple clients that connect to different MCP servers. If your application isn't listed in the [client-specific guidance](https://docs.cloud.google.com/mcp/configure-mcp-ai-application.md.txt#client-specific-guidance), then you can use the following information to connect from most applications.

In your AI application, look for a way to add or connect to a remote MCP server.
For the Managed Service for Apache Spark MCP server, enter the following as
required:

- **Server name**: Managed Service for Apache Spark MCP server
- **Server URL** or **Endpoint** : `https://dataproc-us-central1.googleapis.com/mcp`
- **Transport** : [Streamable HTTP](https://modelcontextprotocol.io/specification/latest/basic/transports/streamable-http)
- **Authentication details** : depending on how you want to authenticate, you can enter your Google Cloud credentials, your OAuth Client ID and secret, or an agent identity and credentials. For more information about authentication, see [Authenticate to MCP servers](https://docs.cloud.google.com/mcp/authenticate-mcp.md.txt).
- **OAuth scope** : the [OAuth 2.0 scope](https://developers.google.com/identity/protocols/oauth2/scopes) that you want to use when connecting to the Managed Service for Apache Spark MCP server. For more information, see [Managed Service for Apache Spark MCP OAuth scopes](https://docs.cloud.google.com/managed-spark/docs/guides/use-dataproc-mcp.md.txt#mcp-oauth-scopes). ### Redirect URIs

For web-based applications, and some desktop applications, you must allowlist a
redirect URI when you create a client ID and secret for authentication.
Redirect URIs are used by the authorization server to send tokens
to your application. Your application's documentation should specify the
redirect URI that you must use. [Custom redirect URIs](https://developers.google.com/identity/protocols/oauth2/native-app#redirect-uri_custom-scheme)
aren't supported.
For application-specific guidance about setting up and connecting to MCP server, see [Client-specific guidance](https://docs.cloud.google.com/mcp/configure-mcp-ai-application.md.txt#client-specific-guidance).

For more general guidance, see the following resources:

- [Connect to remote MCP servers](https://modelcontextprotocol.io/docs/develop/connect-remote-servers).
- [Configure MCP in an AI application](https://docs.cloud.google.com/mcp/configure-mcp-ai-application.md.txt).

## Available tools

MCP tools that are read-only have
the MCP attribute `mcp.tool.isReadOnly` set to `true`. You might want to only
allow read-only tools in certain environments through your
[organization policy](https://docs.cloud.google.com/managed-spark/docs/guides/use-dataproc-mcp.md.txt#organization-level-mcp-control).

To view details of available MCP tools and their descriptions for the
Managed Service for Apache Spark MCP server, see the
[Managed Service for Apache Spark MCP reference](https://docs.cloud.google.com/dataproc/docs/reference/mcp).

### List tools

Use the [MCP inspector](https://modelcontextprotocol.io/docs/tools/inspector) to list tools, or send a
`tools/list` HTTP request directly to the Managed Service for Apache Spark
remote MCP server. The `tools/list` method doesn't require authentication.

    curl -X POST https://dataproc.googleapis.com/TOOLSET_ENDPOINT \
        -H 'Content-Type: application/json' \
        -H 'Accept: application/json' \
        -H 'MCP-Protocol-Version: MCP_PROTOCOL_VERSION' \
        -H 'Mcp-Method: tools/list' \
        -d '{
          "jsonrpc": "2.0",
          "id": 1,
          "method": "tools/list",
          "params": {
            "_meta": {
              "io.modelcontextprotocol/protocolVersion": "MCP_PROTOCOL_VERSION",
              "io.modelcontextprotocol/clientCapabilities": {
                "extensions": {
                  "io.modelcontextprotocol/ui": {
                    "mimeTypes": ["text/html;profile=mcp-app"]
                  }
                }
              }
            }
          }
        }'

Replace the following:

- <var translate="no">`TOOLSET_ENDPOINT`</var>: the remainder of the MCP endpoint after the service name. For example, for Managed Service for Apache Spark, this might be `mcp/toolset-name`.
- <var translate="no">`MCP_PROTOCOL_VERSION`</var>: the MCP protocol version. For example, `2026-07-28`.

## Example use cases

The following are sample use cases for the Managed Service for Apache Spark MCP server:

- "List the Managed Service for Apache Spark clusters in <var translate="no">PROJECT_ID</var> and <var translate="no">REGION</var>."
- "Get the latest status for <var translate="no">CLUSTER_NAME</var> cluster in <var translate="no">PROJECT_ID</var> and <var translate="no">REGION</var>."
- "Delete <var translate="no">CLUSTER_NAME</var> in <var translate="no">PROJECT_ID</var> and <var translate="no">REGION</var> and poll till completion. Report any errors if it failed."

Replace the following:

- <var translate="no">PROJECT_ID</var>: the Google Cloud project ID. Project IDs are listed in the **Project info** section IN the Google Cloud console [Dashboard](https://console.cloud.google.com/home/dashboard).
- <var translate="no">REGION</var>: the Compute Engine [region](https://docs.cloud.google.com/compute/docs/regions-zones.md.txt#available) where the Managed Service for Apache Spark cluster is located.
- <var translate="no">CLUSTER_NAME</var>: the name of the existing Managed Service for Apache Spark cluster.

## Optional security and safety configurations

MCP introduces new security risks and considerations due to the wide variety of
actions that you can do with the MCP tools. To minimize and manage these risks,
Google Cloud offers default settings and customizable policies to
control the use of MCP tools in your Google Cloud
organization or project.

For more information about MCP security and governance, see
[AI security and safety](https://docs.cloud.google.com/mcp/ai-security-safety.md.txt).

### Use Model Armor

[Model Armor](https://docs.cloud.google.com/model-armor/overview.md.txt) is a
Google Cloud service designed to enhance the security and
safety of your AI applications. It works by proactively screening LLM prompts
and responses, protecting against various risks and supporting responsible AI
practices. Whether you are deploying AI in your cloud environment, or on
external cloud providers, Model Armor can help
you prevent malicious input, verify content safety, protect sensitive data,
maintain compliance, and enforce your AI safety and security policies
consistently across your diverse AI landscape.

When Model Armor is enabled with
[logging enabled](https://docs.cloud.google.com/model-armor/configure-logging.md.txt), Model Armor logs the entire
payload. This might expose sensitive information in your logs.

#### MCP request routing to Model Armor

Model Armor is available in [certain regions](https://docs.cloud.google.com/model-armor/locations.md.txt). When Model Armor is enabled and you use an MCP server in a jurisdiction that Model Armor doesn't support, the routing behavior of the call might be different for different MCP servers and might break data residency compliance for in-use and in-transit data. For more information about the behavior of individual MCP servers, see [Model Armor supported products](https://docs.cloud.google.com/mcp/model-armor-supported-products.md.txt).

#### Enable Model Armor

You must enable Model Armor APIs before you can use Model Armor.

### Console

1.


   Enable the Model Armor API, if it is not already enabled.


   **Roles required to enable APIs**


   To enable APIs, you need the `serviceusage.services.enable` permission. If you
   created the project, then you likely already have this permission through the
   Owner role (`roles/owner`). Otherwise, you can get this permission through the
   Service Usage Admin role (`roles/serviceusage.serviceUsageAdmin`).
   [Learn how to grant roles](https://docs.cloud.google.com/iam/docs/granting-changing-revoking-access.md.txt).

   [Enable the API](https://console.cloud.google.com/apis/enableflow?apiid=modelarmor.googleapis.com)

   <br />

2. Select the project where you want to activate Model Armor.

### gcloud

Before you begin, follow these steps using the Google Cloud CLI with the
Model Armor API:

1.


   In the Google Cloud console, activate Cloud Shell.

   [Activate Cloud Shell](https://console.cloud.google.com/?cloudshell=true)


   At the bottom of the Google Cloud console, a
   [Cloud Shell](https://docs.cloud.google.com/shell/docs/how-cloud-shell-works.md.txt)
   session starts and displays a command-line prompt. Cloud Shell is a shell environment
   with the Google Cloud CLI
   already installed and with values already set for
   your current project. It can take a few seconds for the session to initialize.

   <br />

2.

   Run the following command to set the API endpoint for the
   Model Armor service.

   ```bash
   gcloud config set api_endpoint_overrides/modelarmor "https://modelarmor.LOCATION.rep.googleapis.com/"
   ```

   Replace `LOCATION` with the region where you want to use
   Model Armor.

   <br />

#### Configure protection for Google and Google Cloud remote MCP servers

To help protect your MCP tool calls and responses you can use
Model Armor floor settings. A floor setting defines the minimum
security filters that apply across the project. This configuration applies a
consistent set of filters to all MCP tool calls and responses within
the project.

> [!TIP]
> **Tip:** Don't enable the prompt injection and jailbreak filter unless your MCP traffic carries natural language data.

Set up a Model Armor floor setting with MCP sanitization
enabled. For more information, see [Configure Model Armor floor
settings](https://docs.cloud.google.com/model-armor/configure-floor-settings).

> [!NOTE]
> **Note:** If the agent and the MCP server are in different projects, you can create floor settings in both projects (the client project and the resource project). In this case, Model Armor is invoked twice, once for each project.

See the following example command:

```bash
gcloud model-armor floorsettings update \
--full-uri='projects/PROJECT_ID/locations/global/floorSetting' \
--enable-floor-setting-enforcement=TRUE \
--add-integrated-services=GOOGLE_MCP_SERVER \
--google-mcp-server-enforcement-type=INSPECT_AND_BLOCK \
--enable-google-mcp-server-cloud-logging \
--malicious-uri-filter-settings-enforcement=ENABLED \
--add-rai-settings-filters='[{"confidenceLevel": "MEDIUM_AND_ABOVE", "filterType": "DANGEROUS"}]'
```

Replace `PROJECT_ID` with your Google Cloud project ID.

Note the following settings:

- <var translate="no">`INSPECT_AND_BLOCK`</var>: The enforcement type that inspects content for the Google MCP server and blocks prompts and responses that match the filters.
- <var translate="no">`ENABLED`</var>: The setting that enables a filter or enforcement.
- <var translate="no">`MEDIUM_AND_ABOVE`</var>: The confidence level for the Responsible AI - Dangerous filter settings. You can modify this setting, though lower values might result in more false positives. For more information, see [Model Armor confidence levels](https://docs.cloud.google.com/model-armor/overview.md.txt#ma-confidence-levels).

#### Disable scanning MCP traffic with Model Armor

To stop Model Armor from automatically scanning traffic to and
from Google MCP servers based on the project's floor settings, run the following
command:

```bash
gcloud model-armor floorsettings update \
  --full-uri='projects/PROJECT_ID/locations/global/floorSetting' \
  --remove-integrated-services=GOOGLE_MCP_SERVER
```

Replace `PROJECT_ID` with the Google Cloud project
ID. Model Armor doesn't automatically apply the rules defined in
this project's floor settings to any Google MCP server traffic.

Model Armor floor settings and general configuration can impact
more than just MCP. Because Model Armor integrates with services
like Vertex AI, any changes you make to floor settings can affect
traffic scanning and safety behaviors across all integrated services, not just
MCP.

### Control MCP use with IAM policies

Identity and Access Management (IAM)
[deny policies](https://docs.cloud.google.com/iam/docs/deny-overview.md.txt) and
[allow policies](https://docs.cloud.google.com/iam/docs/allow-policies.md.txt) help you
secure Google Cloud and Google MCP servers.

You can combine multiple criteria to build customized security and governance
policies by allowing or denying access based on the following:

- The principal.
- Tool properties like the read-only attribute.
- The service name or tool name.
- The application's OAuth client ID.

For more information, see
[Control MCP use with Identity and Access Management](https://docs.cloud.google.com/mcp/control-mcp-use-iam.md.txt).

## What's next

- Read the [Managed Service for Apache Spark MCP reference documentation](https://docs.cloud.google.com/dataproc/docs/reference/mcp).
- Learn more about [Google Cloud MCP servers](https://docs.cloud.google.com/mcp/overview.md.txt).