Configure a VPC network

Google Cloud Managed Lustre runs within a Virtual Private Cloud (VPC) which provides networking functionality to Compute Engine virtual machine (VM) instances, Google Kubernetes Engine (GKE) clusters, and serverless workloads.

You can connect your Managed Lustre instance to client Compute Engine VMs or Google Kubernetes Engine clusters using either of the following network setups:

Same VPC network: Specify the same VPC network when creating the Managed Lustre instance and your client resources.
Peered VPC network: Connect a peered VPC network through Network Connectivity Center (NCC). NCC lets you connect multiple VPC networks and on-premises networks to a central hub, providing connectivity between them. For instructions on how to set up NCC, see the Network Connectivity Center documentation.

Required permissions

You must have the following IAM permissions:

serviceusage.services.enable
compute.networks.create
compute.addresses.create
compute.addresses.get
compute.firewalls.create
servicenetworking.services.addPeering

These permissions can be granted by adding all of the following predefined roles:

Service Usage Admin (roles/serviceusage.serviceUsageAdmin)
Compute Network Admin (roles/compute.networkAdmin)
Compute Security Admin (roles/compute.securityAdmin)

Or, create a custom role containing the specific permissions.

To grant a role to a user:

gcloud projects add-iam-policy-binding PROJECT_ID \
  --member="user:EMAIL_ADDRESS"
  --role=ROLE

Create and configure the VPC

Enable service networking.

gcloud services enable servicenetworking.googleapis.com

Create a VPC Network in custom mode.
```
gcloud compute networks create NETWORK_NAME \
  --subnet-mode=custom \
  --mtu=8896
```
Note: Setting the value of mtu (maximum transmission unit, or the size of the largest IP packet that can be transmitted on this network) to the maximum allowed value of 8896 improves performance up to 10% compared to the default value of 1460 bytes.

Create a primary subnet for your GKE or Compute Engine resources.

gcloud compute networks subnets create SUBNET_NAME \
  --network=NETWORK_NAME \
  --range=10.128.0.0/20 \
  --region=REGION

Allocate an IP range for private services access.

This internal IP range is used for the private services access connection, which peers your VPC network with the Google-managed network where Managed Lustre resources are provisioned. This allocated range is used to provide IPs for Managed Lustre instances, and must not overlap with any subnets in your VPC network.

Each Managed Lustre instance requires a contiguous CIDR block with a prefix length of at least 23.

We recommend creating a larger IP range of /20 to allow for the creation of multiple Managed Lustre instances or the use of other Google Cloud services.
```
gcloud compute addresses create IP_RANGE_NAME \
  --global \
  --purpose=VPC_PEERING \
  --prefix-length=20 \
  --description="Managed Lustre VPC Peering" \
  --network=NETWORK_NAME
```

Get the CIDR block associated with the range you created in the previous step.

CIDR_BLOCK=$(
  gcloud compute addresses describe IP_RANGE_NAME \
    --global  \
    --format="value[separator=/](address, prefixLength)"
)

Create a firewall rule to allow TCP traffic from the IP range you created.
```
gcloud compute firewall-rules create FIREWALL_NAME \
  --allow=tcp:988,tcp:6988 \
  --network=NETWORK_NAME \
  --source-ranges=$CIDR_BLOCK
```
If your client VMs or GKE clusters reside in a peered VPC network connected through Network Connectivity Center (NCC), the firewall rules in both VPC networks must allow traffic between your client subnets and the allocated IP range ($CIDR_BLOCK). Run the following command to create a firewall rule in the NCC-connected VPC network:
```
gcloud compute firewall-rules create PEER_FIREWALL_NAME \
  --allow=tcp:988,tcp:6988 \
  --network=PEER_NETWORK_NAME \
  --source-ranges=$CIDR_BLOCK
```
Establish the private services access connection.
```
gcloud services vpc-peerings connect \
  --network=NETWORK_NAME \
  --ranges=IP_RANGE_NAME \
  --service=servicenetworking.googleapis.com
```
This connection is only required in the VPC network where the Managed Lustre instance is provisioned.

Create additional subnets for multi-NIC

If you plan to use multiple network interface cards (multi-NIC) to aggregate bandwidth, you must create a separate subnet within your VPC network for each NIC.

To benefit from multi-NIC, you must use Compute Engine machine types with multiple physical NICs that are attached to regular VPCs. NICs that attach to VPCs with RDMA network profiles cannot be used to increase general networking bandwidth. See Networking and GPU machines for additional details.

To create a subnet for an additional physical NIC:

gcloud compute networks subnets create SUBNET_NAME_2 \
  --network=NETWORK_NAME \
  --range=10.130.0.0/20 \
  --region=REGION

Repeat this step for each additional NIC. Ensure that the IP ranges for each subnet don't overlap each other.

VPC Service Controls

Managed Lustre supports VPC Service Controls (VPC-SC). See Secure instances with a service perimeter for details.

Troubleshooting VPC setup

Permission denied to add peering for service `servicenetworking.googleapis.com`

ERROR: (gcloud.services.vpc-peerings.connect) User [$(USER)] does not have
permission to access services instance [servicenetworking.googleapis.com]
(or it may not exist): Permission denied to add peering for service
'servicenetworking.googleapis.com'.

This error means that you don't have servicenetworking.services.addPeering IAM permission on your user account.

See Access control with IAM for instructions on adding one of the following roles to your account:

roles/compute.networkAdmin or
roles/servicenetworking.networksAdmin

Cannot modify allocated ranges in CreateConnection

ERROR: (gcloud.services.vpc-peerings.connect) The operation
"operations/[operation_id]" resulted in a failure "Cannot modify allocated
ranges in CreateConnection. Please use UpdateConnection."

This error is returned when you have already created a VPC peering on this network with different IP ranges. There are two possible solutions:

Replace the existing IP ranges:

gcloud services vpc-peerings update \
  --network=NETWORK_NAME \
  --ranges=IP_RANGE_NAME \
  --service=servicenetworking.googleapis.com \
  --force

Or, add the new IP range to the existing connection:

Retrieve the list of existing IP ranges for the peering:

EXISTING_RANGES="$(
  gcloud services vpc-peerings list \
    --network=NETWORK_NAME \
    --service=servicenetworking.googleapis.com \
    --format="value(reservedPeeringRanges.list())" \
    --flatten=reservedPeeringRanges
)

Then, add the new range to the peering:

gcloud services vpc-peerings update \
  --network=NETWORK_NAME \
  --ranges="${EXISTING_RANGES}",IP_RANGE_NAME \
  --service=servicenetworking.googleapis.com

IP address range exhausted

If instance creation fails with an IP address range exhausted error:

ERROR: (gcloud.alpha.Google Cloud Managed Lustre.instances.create) FAILED_PRECONDITION: Invalid
resource state for "NETWORK_RANGES_NOT_AVAILABLE": IP address range exhausted

Follow the VPC guide to modify the existing private connection to add IP address ranges.

We recommend a prefix length of at least /20 (4096 addresses).