Configuring a Virtual Private Cloud (VPC) (VPC) network for Managed Service for Apache Spark lets you securely connect workloads to internal project resources and Google APIs. These subnetwork and firewall settings help Spark drivers and executors communicate directly while complying with your organization's security policies.

## VPC network requirements

This document describes VPC network requirements for Managed Service for Apache Spark
batch workloads and interactive sessions.

### Private Google Access

Managed Service for Apache Spark batch workloads and interactive sessions run on
VMs with internal IP addresses only and on a regional subnet with [Private Google Access (PGA)](https://docs.cloud.google.com/vpc/docs/configure-private-google-access.md.txt)
enabled automatically on the subnet.

If you don't specify a subnet, Managed Service for Apache Spark selects the
`default` subnet in the batch workload or session region as the subnet for a
batch workload or session.

If your workload requires external network or internet access, for example, to
download resources such as ML models from [PyTorch
Hub](https://pytorch.org/hub/) or [Hugging Face](https://huggingface.co/), you
can set up [Cloud NAT](https://docs.cloud.google.com/nat/docs/overview.md.txt) to allow outbound traffic using
internal IPs on your VPC network.

### Open subnet connectivity

The VPC subnet for the region selected for the
Managed Service for Apache Spark batch workload or interactive session must allow
internal communication on all ports between VM instances within the subnet.

To prevent malicious scripts in one workload from affecting other workloads,
Managed Service for Apache Spark deploys [default security
measures](https://docs.cloud.google.com/managed-spark/docs/concepts/security).

The following Google Cloud CLI command attaches a network firewall to a subnet
that allows internal ingress communications among VMs using all protocols on all
ports:

```
gcloud compute firewall-rules create allow-internal-ingress \
    --network=NETWORK_NAME \
    --source-ranges=SUBNET_RANGES \
    --destination-ranges=SUBNET_RANGES \
    --direction=ingress \
    --action=allow \
    --rules=all
```

Notes:

- <var translate="no">SUBNET_RANGES:</var> See [Allow internal ingress connections between
  VMs](https://docs.cloud.google.com/firewall/docs/using-firewalls#common-use-cases-allow-internal). The
  `default` VPC network in a project with the
  `default-allow-internal` firewall rule, which allows ingress communication
  on all ports (`tcp:0-65535`, `udp:0-65535`, and `icmp protocols:ports`),
  meets the open-subnet-connectivity requirement. However, this rule also
  allows ingress by any VM instance on the network.

  > [!IMPORTANT]
  > **Use network tags to limit connectivity**. In production, the recommended practice is to limit firewall rules to the IP addresses used by your Spark workloads.

#### Automatically created [regional system firewall policy](https://docs.cloud.google.com/firewall/docs/firewall-policies-overview.md.txt)

To satisfy the [open subnet connectivity](https://docs.cloud.google.com/managed-spark/docs/concepts/network-serverless.md.txt#open-subnet) requirement,
Managed Service for Apache Spark batch workloads and interactive sessions
that use runtime version `3.0` or later automatically create a regional
system firewall policy `dataproc-firewall-policy-[network-id]-region` or
`dataproc-fw-[network-id]-region` on the batch or session VPC subnet.
This policy contains the following ingress and egress rules.

| Name | Purpose | Priority | Direction | Action | Source and Destination | Protocol and ports |
|---|---|---|---|---|---|---|
| `dataproc-allow-internal-ingress-rule-[subnetworkId]` | Allows all necessary internal communication only from other tagged Managed Service for Apache Spark VMs within the same subnet. | 4 | INGRESS | ALLOW | - `srcSecureTag`: secure tag value for this subnet. - `targetSecureTags`:secure tag value for this subnet. | tcp:0-65535, udp:0-65535, icmp protocols:ports |
| `dataproc-allow-internal-egress-rule-[subnetworkId]` | Allows Managed Service for Apache Spark VMs to download packages, for example pip and apt-get, and access Google APIs using Private Google Access. | 5 | EGRESS | ALLOW | - `destIpRanges`: 0.0.0.0/0. - `targetSecureTags`:secure tag value for this subnet. | tcp:0-65535, udp:0-65535, icmp protocols:ports |

Notes:

- Managed Service for Apache Spark provisions a [tenant project](https://docs.cloud.google.com/service-infrastructure/docs/glossary.md.txt#tenant)
  associated with the user project to store secure tags.
  Managed Service for Apache Spark creates a
  [secure tag](https://docs.cloud.google.com/managed-spark/docs/guides/use-secure-tags.md.txt#benefits_of_using_secure_tags)
  for the subnet in the [tenant project](https://docs.cloud.google.com/service-infrastructure/docs/glossary.md.txt#tenant)
  and attaches it to Managed Service for Apache Spark VMs, which ensures that
  the created system firewall policy only applies to Managed Service for Apache Spark VMs.

- Automatically created system firewall policy is not supported for [Shared VPC](https://docs.cloud.google.com/vpc/docs/shared-vpc.md.txt).

## Managed Service for Apache Spark and VPC-SC networks

With [VPC Service Controls](https://docs.cloud.google.com/vpc-service-controls/docs.md.txt), network administrators
can define a security perimeter around resources of Google-managed services to
control communication to and between those services.

Consider the following strategies when you use VPC-SC networks
with Managed Service for Apache Spark:

- [Set up private
  connectivity](https://docs.cloud.google.com/vpc-service-controls/docs/set-up-private-connectivity).

- Create a [custom container image](https://docs.cloud.google.com/managed-spark/docs/guides/custom-containers.md.txt)
  that pre-installs dependencies outside the VPC-SC perimeter,
  and then [submit a Spark batch
  workload](https://docs.cloud.google.com/managed-spark/docs/guides/custom-containers#submit_a_spark_batch_workload_using_a_custom_container_image)
  that uses your custom container image.

For more information, see [VPC Service Controls---
Managed Service for Apache Spark](https://docs.cloud.google.com/vpc-service-controls/docs/supported-products#table_dataproc_serverless).