You can enable your App Engine service to send traffic to a [Shared VPC network](https://docs.cloud.google.com/vpc/docs/shared-vpc.md.txt#shared_vpc_networks) by using Direct VPC egress with no Serverless VPC Access connector required. Direct VPC egress supports [VPC Flow Logs](https://docs.cloud.google.com/vpc/docs/using-flow-logs.md.txt) and [Cloud NAT](https://docs.cloud.google.com/nat/docs/overview.md.txt) (both [Public NAT](https://docs.cloud.google.com/nat/docs/public-nat.md.txt) and [Private NAT](https://docs.cloud.google.com/nat/docs/private-nat.md.txt)). This page describes how to configure App Engine's IAM permissions to use the subnet of the Shared VPC network, and then place your service on the shared subnet. ## Before you begin Review the [IP address allocation](https://docs.cloud.google.com/appengine/docs/standard/shared-vpc-direct-vpc.md.txt#direct-vpc-ip-addresses) section for guidance with allocating IP addresses from your subnet. ## Limitations - **IP address consumption**: Your service's IP address usage scales directly with the number of running instances, and your scaling ability is limited by the number of available IP addresses in your chosen subnet. - **Maintenance events**: Your service might experience brief connection breaks during networking infrastructure maintenance events. We recommend that you use client libraries for handling occasional connection resets. - **Cold starts**: Initial cold start times are dependent on the region and the specific use case. In rare cases, cold starts might last up to a minute. - **Direct VPC ingress**: App Engine doesn't support Direct VPC ingress. - **Number of instances** : By default, you can only configure up to 100 instances per App Engine version to use Direct VPC egress. You can [request a quota increase](https://docs.cloud.google.com/appengine/docs/standard/quotas.md.txt#Services_Versions_and_Instances) for this limit if your workload requires more instances. > [!NOTE] > **Note:** Starting from March 2025, for new applications, if you haven't set `max_instances` in your `app.yaml` file, App Engine sets this field to 20. For applications created before March 2025, App Engine sets this field to your Direct VPC instance quota, which is 100 by default. - **VPC Flow Logs** : Doesn't display your App Engine service version in the log records. The service name is displayed in the [`AppEngineServiceDetails`](https://docs.cloud.google.com/vpc/docs/about-flow-logs-records.md.txt#app-engine-service-details) field. - **Regional availability** : Direct VPC egress is available only by invitation in [certain regions](https://docs.cloud.google.com/appengine/docs/standard/locations.md.txt#direct-vpc-locations). To request access in these regions, contact support. ## Set up IAM permissions Before App Engine in a Shared VPC service project can access a Shared VPC network, make sure the App Engine service agent has sufficient permissions to use the subnet. - **App Engine Service Agent role** : To access the Shared VPC network, grant sufficient permissions to the [App Engine service agent](https://docs.cloud.google.com/iam/docs/roles-permissions/appengine.md.txt#appengine.serviceAgent) by adding one of the following roles: - [Compute Network User (`roles/compute.networkUser`)](https://docs.cloud.google.com/iam/docs/roles-permissions/compute.md.txt#compute.networkUser) on the Shared VPC host project. Subnets with external IPv6 also require the [Compute Public IP Admin (`roles/compute.publicIpAdmin`)](https://docs.cloud.google.com/iam/docs/roles-permissions/compute.md.txt#compute.publicIpAdmin) role. For example, to grant the Compute Network User role, run the following command: ```bash gcloud projects add-iam-policy-binding HOST_PROJECT_ID \ --member "serviceAccount:service-SERVICE_PROJECT_NUMBER@gcp-gae-service.iam.gserviceaccount.com" \ --role "roles/compute.networkUser" ``` Replace the following: - HOST_PROJECT_ID: the ID of your Shared VPC host project. - SERVICE_PROJECT_NUMBER: the Shared VPC service project number where you deploy your App Engine service. - [Compute Network Viewer (`roles/compute.networkViewer`)](https://docs.cloud.google.com/iam/docs/roles-permissions/compute.md.txt#compute.networkViewer) on the Shared VPC host project and the [Compute Network User (`roles/compute.networkUser`)](https://docs.cloud.google.com/iam/docs/roles-permissions/compute.md.txt#compute.networkUser) role on the Shared VPC subnet. Subnets with external IPv6 also require the [Compute Public IP Admin (`roles/compute.publicIpAdmin`)](https://docs.cloud.google.com/iam/docs/roles-permissions/compute.md.txt#compute.publicIpAdmin) role. For example, to grant the Compute Network Viewer role on the host project and the Compute Network User role on the subnet, run the following commands: ```bash gcloud projects add-iam-policy-binding HOST_PROJECT_ID \ --member "serviceAccount:service-SERVICE_PROJECT_NUMBER@gcp-gae-service.iam.gserviceaccount.com" \ --role "roles/compute.networkViewer" gcloud compute networks subnets add-iam-policy-binding SUBNET_NAME \ --region REGION \ --member "serviceAccount:service-SERVICE_PROJECT_NUMBER@gcp-gae-service.iam.gserviceaccount.com" \ --role "roles/compute.networkUser" \ --project HOST_PROJECT_ID ``` Replace the following: - HOST_PROJECT_ID: the ID of your Shared VPC host project. - SERVICE_PROJECT_NUMBER: the Shared VPC service project number where you deploy your App Engine service. - SUBNET_NAME: the name of the subnet that you want to run your App Engine services. - REGION: the region for your App Engine service, which must match the region of your subnet. - **Custom permissions**: For more granular control, grant the App Engine service agent the following additional permissions on the project: - `compute.networks.get` on the Shared VPC host project - `compute.subnetworks.get` on the host project or the specific subnet - `compute.subnetworks.use` on the host project or the specific subnet - `compute.addresses.get` on the Shared VPC service project - `compute.addresses.list` on the Shared VPC service project - `compute.addresses.create` on the Shared VPC service project - `compute.addresses.delete` on the Shared VPC service project - `compute.addresses.createInternal` on the Shared VPC service project - `compute.addresses.deleteInternal` on the Shared VPC service project - `compute.regionOperations.get` on the Shared VPC service project ## IP address allocation To place your App Engine service on a VPC network, specify either a VPC network or a subnet, or both. If you specify only a network, the subnet uses the same name as the network. App Engine allocates IP addresses from your subnet. IP addresses are ephemeral, so don't create policies based on individual IPs. If you need to create a policy based on IPs, such as in firewall rules, you must use the IP address range of the entire subnet. To change the network or subnet that your service uses, deploy a new version that uses the new network and subnet values. ### Scale up and scale down For faster scale up during a traffic surge, App Engine reserves IP addresses in blocks of 16 (`28` subnet mask) at a time. To ensure that you have enough IPv4 addresses available for use across App Engine, your subnet's IPv4 address range must be `/26` or larger. For IP allocation efficiency and ease of management, place multiple resources on the same subnet. If your IPv4 address space is limited, see [Supported IPv4 ranges](https://docs.cloud.google.com/appengine/docs/standard/shared-vpc-direct-vpc.md.txt#supported-ip-ranges) for more options. To delete the subnet, you must first delete or redeploy your App Engine service to stop using the subnet, and then wait 1-2 hours. ### IP address consumption for services At steady state, App Engine uses 2 times (2X) as many IP addresses as the number of instances. When a version scales down, App Engine retains its IP addresses for up to 20 minutes. In total, reserve at least 2X the number of IP addresses, plus a buffer to account for version updates. For example, if you upgrade versions so that `version 1` scales from 100 instances down to zero while `version 2` scales from zero up to 100, App Engine retains the `version 1` IP addresses for up to 20 minutes after scaling down. During the 20-minute retention window, you must reserve at least 400 IP addresses (`(100 + 100) * 2`). ### Supported IPv4 ranges App Engine supports the following IPv4 ranges for your subnet: - [RFC 1918](https://tools.ietf.org/html/rfc1918#section-3) - `10.0.0.0/8` - `172.16.0.0/12` - `192.168.0.0/16` - [RFC 6598](https://tools.ietf.org/html/rfc6598#section-7) - `100.64.0.0/10` - Class E - `240.0.0.0/4` > [!NOTE] > **Note:** Test non-RFC 1918 setups to identify possible compatibility issues with non-Google Cloud resources or hardware. Learn more about [creating subnets with non-RFC 1918 IP ranges](https://docs.cloud.google.com/vpc/docs/subnets.md.txt#ipv4-ranges). ## Deploy an App Engine service Direct VPC egress allows your App Engine service to send traffic to a Shared VPC network without a Serverless VPC Access connector. You can assign network tags to a specific version of your App Engine service for more granular network security, such as applying [VPC firewall rules](https://docs.cloud.google.com/firewall/docs/firewalls.md.txt). To place your service on the shared subnet, specify the fully qualified resource names for the Shared VPC network and subnet. You can configure Direct VPC egress for an App Engine service by using YAML or Terraform. To enable or disconnect Direct VPC egress, deploy a new version and [migrate traffic](https://docs.cloud.google.com/appengine/docs/standard/migrating-traffic.md.txt) to it. You can also delete the deployed version first, instead of overwriting an existing version ID. ### YAML 1. Update your `app.yaml` file to include the `vpc_access` section: ```yaml vpc_access: network_interface: network: projects/HOST_PROJECT_ID/global/networks/VPC_NETWORK subnet: projects/HOST_PROJECT_ID/regions/REGION/subnetworks/SUBNET_NAME tags: - NETWORK_TAGS vpc_egress: EGRESS_SETTING ``` Replace the following: - HOST_PROJECT_ID: the ID of your Shared VPC project. - VPC_NETWORK: the name of your Shared VPC network. - REGION: the region for your App Engine service, which must match the region of your subnet. - SUBNET_NAME: the name of your subnet. - Optional: NETWORK_TAGS lists [network tags](https://docs.cloud.google.com/vpc/docs/add-remove-network-tags.md.txt#restrictions) to associate with your App Engine service's instances for use in firewall rules and routing policies. - Optional: EGRESS_SETTING controls how outbound traffic is routed. This field supports the following configuration settings: - `all-traffic`: Sends all outbound traffic through the Shared VPC network. - `private-ranges-only` (default): Sends only traffic to internal addresses through the Shared VPC network. 2. Deploy your service by running the following command: ```bash gcloud app deploy ``` ### Terraform To learn how to apply or remove a Terraform configuration, see [Basic Terraform commands](https://docs.cloud.google.com/docs/terraform/basic-commands.md.txt). 1. Add the following `vpc_access` block to a [`google_app_engine_standard_app_version`](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/app_engine_standard_app_version) resource in your Terraform configuration: ```terraform vpc_access { vpc_egress = "EGRESS_SETTING" network_interfaces { network = "projects/HOST_PROJECT_ID/global/networks/VPC_NETWORK" subnetwork = "projects/HOST_PROJECT_ID/regions/REGION/subnetworks/SUBNET_NAME" tags = ["TAG_1", "TAG_2"] } } ``` Replace the following: - HOST_PROJECT_ID: the ID of your Shared VPC host project. - VPC_NETWORK: the name of your Shared VPC network. - REGION: the region for your App Engine service, which must match the region of your subnet. - SUBNET_NAME: the name of your subnet. - Optional: TAG_1, TAG_2 lists [network tags](https://docs.cloud.google.com/vpc/docs/add-remove-network-tags.md.txt#restrictions) to associate with your App Engine service's instances for use in firewall rules and routing policies. - Optional: EGRESS_SETTING controls how outbound traffic is routed. This field supports the following configuration settings: - `ALL_TRAFFIC`: Sends all outbound traffic through the Shared VPC network. - `PRIVATE_IP_RANGES` (default): Sends only traffic to internal addresses through the Shared VPC network. 2. If the version is already deployed without Direct VPC egress, update the `version_id` field to a new value. You can't enable Direct VPC egress on an existing version. 3. Apply the configuration: ```bash terraform apply ``` ## Disconnect a service To remove your service from the Shared VPC network, use YAML or Terraform: ### YAML 1. Remove the `vpc_access` section from your `app.yaml` file. 2. Redeploy your service: ```bash gcloud app deploy ``` ### Terraform 1. Remove the `vpc_access` block from the `google_app_engine_standard_app_version` resource in your Terraform configuration. 2. Update the `version_id` field to a new value. You can't disable Direct VPC egress on an existing version. 3. Apply the configuration: ```bash terraform apply ``` ## Troubleshoot This section describes common errors you might encounter when using Direct VPC egress with a Shared VPC network and provides guidance on how to resolve them. ### Cannot delete subnet To delete a subnet, you must first delete all resources that use it. If App Engine is using a subnet, you must [disconnect](https://docs.cloud.google.com/appengine/docs/standard/shared-vpc-direct-vpc.md.txt#disconnect-service) it or move it to a different subnet before deleting the subnet. After you delete or move your App Engine resources, wait 1-2 hours for App Engine to release the IPs before you delete the subnet. ### Deployment failures If your deployment fails, the Google Cloud CLI displays error messages indicating the root cause. Some common issues include the following: - Incorrect VPC network metadata, such as a misspelled network or subnet name in your `app.yaml` file. To fix potential errors, review the VPC network configuration in your `app.yaml` file. - Insufficient IAM permissions. Make sure you [grant the required permissions](https://docs.cloud.google.com/appengine/docs/standard/shared-vpc-direct-vpc.md.txt#iam) to the App Engine service agent (`service-SERVICE_PROJECT_NUMBER@gcp-gae-service.iam.gserviceaccount.com`). If you encounter permission errors during deployment, ensure that you grant the following additional roles to your deploying service account: - [Cloud Build Service Account](https://docs.cloud.google.com/iam/docs/roles-permissions/cloudbuild.md.txt#cloudbuild.builds.builder) (`roles/cloudbuild.builds.builder`) - [Service Account Token Creator](https://docs.cloud.google.com/iam/docs/roles-permissions/iam.md.txt#iam.serviceAccountTokenCreator) (`roles/iam.serviceAccountTokenCreator`) ### Cannot detach Shared VPC network To detach the Shared VPC network in the host project, follow the steps to [Deprovision Shared VPC](https://docs.cloud.google.com/vpc/docs/deprovisioning-shared-vpc.md.txt) and be sure to disconnect any App Engine [services](https://docs.cloud.google.com/appengine/docs/standard/shared-vpc-direct-vpc.md.txt#disconnect-service) from the Shared VPC network. To see which App Engine resources are using the Shared VPC network, run the following command: ```bash gcloud compute shared-vpc list-associated-resources HOST_PROJECT_ID ``` Replace HOST_PROJECT_ID with the ID of your Shared VPC host project. ### Direct VPC egress subnet runs out of IPv4 addresses If the subnet of the VPC network runs out of IPv4 addresses, App Engine cannot start any more service instances. To resolve this issue, consider moving to a subnetwork with more available IP addresses or expanding the subnet range. ### View allocated IP addresses To see which IP addresses App Engine has allocated, go to the [IP addresses page](https://console.cloud.google.com/networking/addresses) in the Google Cloud console or run the following command from the Google Cloud CLI: ```bash gcloud compute addresses list ```