Direct VPC egress provides a high-performing networking solution for your App Engine service to send traffic to a Virtual Private Cloud (VPC) network. Direct VPC egress lets workloads seamlessly access VPC network resources and removes the need to configure Serverless VPC Access connectors. ## Key benefits - **Simplified management** : Eliminates the operational overhead of managing connector instances, machine types, and scaling settings. App Engine handles configurations directly within your service's `app.yaml` file. - **Cost efficiency**: Using Direct VPC egress has no additional charge, and you don't have to pay fixed monthly charges for connector VMs. - **Improved performance and reliability**: By eliminating the need to use a connector, Direct VPC egress offers a faster, more reliable connection to your VPC network resources. It scales as quickly as your App Engine service and avoids connection drops that might occur with connectors during maintenance. - **Granular security**: You can apply network tags directly to your App Engine service versions, enabling precise, service-specific firewall rules and network policies. - **Supports VPC Flow Logs** : You can enable [VPC Flow Logs on your subnet](https://docs.cloud.google.com/vpc/docs/using-flow-logs.md.txt) to log egress traffic from your App Engine service. - **Supports Cloud NAT** : You can configure a [Cloud NAT gateway](https://docs.cloud.google.com/nat/docs/public-nat.md.txt) on your subnet to provide NAT services, such as Public NAT or Private NAT for outbound traffic from your App Engine service. ## Limitations - **IP address consumption**: Your service's IP address usage scales directly with the number of running instances, and your scaling ability is limited by the number of available IP addresses in your chosen subnet. - **Maintenance events**: Your service might experience brief connection breaks during networking infrastructure maintenance events. We recommend that you use client libraries for handling occasional connection resets. - **Cold starts**: Initial cold start times are dependent on the region and the specific use case. In rare cases, cold starts might last up to a minute. - **Direct VPC ingress**: App Engine doesn't support Direct VPC ingress. - **Number of instances** : By default, you can only configure up to 100 instances per App Engine version to use Direct VPC egress. You can [request a quota increase](https://docs.cloud.google.com/appengine/docs/standard/quotas.md.txt#Services_Versions_and_Instances) for this limit if your workload requires more instances. > [!NOTE] > **Note:** Starting from March 2025, for new applications, if you haven't set `max_instances` in your `app.yaml` file, App Engine sets this field to 20. For applications created before March 2025, App Engine sets this field to your Direct VPC instance quota, which is 100 by default. - **VPC Flow Logs** : Doesn't display your App Engine service version in the log records. The service name is displayed in the [`AppEngineServiceDetails`](https://docs.cloud.google.com/vpc/docs/about-flow-logs-records.md.txt#app-engine-service-details) field. - **Regional availability** : Direct VPC egress is available only by invitation in [certain regions](https://docs.cloud.google.com/appengine/docs/standard/locations.md.txt#direct-vpc-locations). To request access in these regions, contact support. ## IP address allocation To place your App Engine service on a VPC network, specify either a VPC network or a subnet, or both. If you specify only a network, the subnet uses the same name as the network. App Engine allocates IP addresses from your subnet. IP addresses are ephemeral, so don't create policies based on individual IPs. If you need to create a policy based on IPs, such as in firewall rules, you must use the IP address range of the entire subnet. To change the network or subnet that your service uses, deploy a new version that uses the new network and subnet values. ### Scale up and scale down For faster scale up during a traffic surge, App Engine reserves IP addresses in blocks of 16 (`28` subnet mask) at a time. To ensure that you have enough IPv4 addresses available for use across App Engine, your subnet's IPv4 address range must be `/26` or larger. For IP allocation efficiency and ease of management, place multiple resources on the same subnet. If your IPv4 address space is limited, see [Supported IPv4 ranges](https://docs.cloud.google.com/appengine/docs/standard/vpc-direct-vpc.md.txt#supported-ip-ranges) for more options. To delete the subnet, you must first delete or redeploy your App Engine service to stop using the subnet, and then wait 1-2 hours. ### IP address consumption for services At steady state, App Engine uses 2 times (2X) as many IP addresses as the number of instances. When a version scales down, App Engine retains its IP addresses for up to 20 minutes. In total, reserve at least 2X the number of IP addresses, plus a buffer to account for version updates. For example, if you upgrade versions so that `version 1` scales from 100 instances down to zero while `version 2` scales from zero up to 100, App Engine retains the `version 1` IP addresses for up to 20 minutes after scaling down. During the 20-minute retention window, you must reserve at least 400 IP addresses (`(100 + 100) * 2`). ### Supported IPv4 ranges App Engine supports the following IPv4 ranges for your subnet: - [RFC 1918](https://tools.ietf.org/html/rfc1918#section-3) - `10.0.0.0/8` - `172.16.0.0/12` - `192.168.0.0/16` - [RFC 6598](https://tools.ietf.org/html/rfc6598#section-7) - `100.64.0.0/10` - Class E - `240.0.0.0/4` > [!NOTE] > **Note:** Test non-RFC 1918 setups to identify possible compatibility issues with non-Google Cloud resources or hardware. Learn more about [creating subnets with non-RFC 1918 IP ranges](https://docs.cloud.google.com/vpc/docs/subnets.md.txt#ipv4-ranges). ## Before you begin 1. Make sure you have an existing VPC network and a subnet in your project. If you don't already have an existing VPC, follow the instructions to create one in [Create a VPC network](https://docs.cloud.google.com/vpc/docs/create-modify-vpc-networks.md.txt). 2. Enable the Compute Engine API and the Cloud Build API: [Enable APIs](https://console.cloud.google.com/flows/enableapi?apiid=compute.googleapis.com,cloudbuild.googleapis.com) 3. To use Direct VPC egress, make sure you are running the most recent version of the Google Cloud CLI: ```bash gcloud components update ``` ## Required roles Ensure that App Engine has access to the VPC network by granting the following roles to the App Engine service agent: - **App Engine Service Agent role** : By default, the App Engine service agent has the [App Engine Service Agent role (`roles/appengine.serviceAgent`)](https://docs.cloud.google.com/iam/docs/roles-permissions/appengine.md.txt#appengine.serviceAgent) that contains the necessary permissions. - **Custom permissions**: For more granular control, grant the App Engine service agent the following additional permissions on the project: - `compute.networks.get` - `compute.subnetworks.get` - `compute.subnetworks.use` on the project or the specific subnet - `compute.addresses.get` - `compute.addresses.list` - `compute.addresses.create` - `compute.addresses.delete` - `compute.addresses.createInternal` - `compute.addresses.deleteInternal` - `compute.regionOperations.get` - **Compute Network User role** : If you don't use the default App Engine Service Agent role or the custom permissions, grant the [Compute Network User role (`roles/compute.networkUser`)](https://docs.cloud.google.com/iam/docs/roles-permissions/compute.md.txt#compute.networkUser) to the App Engine service agent. Subnets with external IPv6 also require the [Compute Public IP Admin role (`roles/compute.publicIpAdmin`)](https://docs.cloud.google.com/iam/docs/roles-permissions/compute.md.txt#compute.publicIpAdmin). For example, to grant the Compute Network User role, run the following command: ```bash gcloud projects add-iam-policy-binding PROJECT_ID \ --member "serviceAccount:service-PROJECT_NUMBER@gcp-gae-service.iam.gserviceaccount.com" \ --role "roles/compute.networkUser" ``` Replace the following: - PROJECT_ID: the ID of your project. - PROJECT_NUMBER: the project number where you deploy your App Engine service. ## Configure an App Engine service with Direct VPC egress You can configure Direct VPC egress for an App Engine service by using YAML or Terraform. To enable or disconnect Direct VPC egress, deploy a new version and [migrate traffic](https://docs.cloud.google.com/appengine/docs/standard/migrating-traffic.md.txt) to it. You can also delete the deployed version first, instead of overwriting an existing version ID. ### YAML 1. Add the following `vpc_access` setting to your `app.yaml` file: ```yaml vpc_access: network_interface: network: NETWORK subnet: SUBNET tags: - NETWORK_TAGS vpc_egress: EGRESS_SETTING ``` Replace the following: - NETWORK: the name of the existing network your application instances connect to, for example, `default`. Specify either a VPC network or a subnet, or both. If you specify only a network, the subnet uses the same name as the network. - SUBNET: the name of the existing subnetwork your application instances connect to, for example, `default`. Specify either a VPC network or a subnet, or both. If you specify only a network, the subnet uses the same name as the network. - Optional: NETWORK_TAGS lists [network tags](https://docs.cloud.google.com/vpc/docs/add-remove-network-tags.md.txt#restrictions) to associate with your App Engine service's instances for use in firewall rules and routing policies. - Optional: EGRESS_SETTING controls how outbound traffic is routed. This field supports the following configuration settings: - `all-traffic`: All outbound requests are routed through the VPC network. - `private-ranges-only` (default): Only traffic to internal IP addresses is routed through the VPC network. Internet traffic uses the default App Engine path. 2. Deploy to App Engine by running the following command: ```bash gcloud app deploy ``` ### Terraform To learn how to apply or remove a Terraform configuration, see [Basic Terraform commands](https://docs.cloud.google.com/docs/terraform/basic-commands.md.txt). 1. Add the following `vpc_access` block to a [`google_app_engine_standard_app_version`](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/app_engine_standard_app_version) resource in your Terraform configuration: ```terraform vpc_access { vpc_egress = "EGRESS_SETTING" network_interfaces { network = "NETWORK" subnetwork = "SUBNET" tags = ["TAG_1", "TAG_2"] } } ``` Replace the following: - NETWORK: the name of the existing VPC network your application instances connect to, for example, `default`. Specify either a VPC network or a subnet, or both. If you specify only a network, the subnet uses the same name as the network. You can also reference a `google_compute_network` resource name (for example, `google_compute_network.custom.name`). - SUBNET: the name of the existing subnetwork your application instances connect to, for example, `default`. Specify either a VPC network or a subnet, or both. If you specify only a network, the subnet uses the same name as the network. You can also reference a `google_compute_subnetwork` resource name (for example, `google_compute_subnetwork.custom.name`). - Optional: TAG_1, TAG_2 lists [network tags](https://docs.cloud.google.com/vpc/docs/add-remove-network-tags.md.txt#restrictions) to associate with your App Engine service's instances for use in firewall rules and routing policies. - Optional: EGRESS_SETTING controls how outbound traffic is routed. This field supports the following configuration settings: - `ALL_TRAFFIC`: All outbound requests are routed through the VPC network. - `PRIVATE_IP_RANGES` (default): Only traffic to internal IP addresses is routed through the VPC network. Internet traffic uses the default App Engine path. 2. If the version is already deployed without Direct VPC egress, update the `version_id` field to a new value. You can't enable Direct VPC egress on an existing version. 3. Apply the configuration: ```bash terraform apply ``` ### Disconnect a service To disconnect your service from the VPC network, use YAML or Terraform: ### YAML 1. Remove the `vpc_access` section from your `app.yaml` file. 2. Redeploy your service: ```bash gcloud app deploy ``` ### Terraform 1. Remove the `vpc_access` block from the `google_app_engine_standard_app_version` resource in your Terraform configuration. 2. Update the `version_id` field to a new value. You can't disable Direct VPC egress on an existing version. 3. Apply the configuration: ```bash terraform apply ``` ### Isolate outbound internet traffic By default, when `vpc_egress` is set to `private-ranges-only`, App Engine sends outbound internet traffic through a regional pool of dynamic IP addresses shared across serverless workloads. Don't use this default shared pool for workloads that depend on IP reputation or generate high-volume probing traffic. Instead, set `vpc_egress: all-traffic` and route outbound traffic through Cloud NAT with dedicated [static outbound IP addresses](https://docs.cloud.google.com/appengine/docs/standard/outbound-ip-addresses.md.txt#static-ip) in these cases: - Your service connects to reputation-sensitive external endpoints, such as payment gateways, hosted databases, or partner APIs behind third-party web application firewalls (WAFs). Dedicated static IP addresses prevent noisy traffic from other tenants on the shared pool from triggering external WAF blocks or `HTTP 403` errors. - Your service runs authorized network scanning, port enumeration, host fingerprinting, vulnerability probing, web scraping, or load testing. Running these workloads over the default shared IP pool degrades IP reputation, can disrupt service for other customers in violation of the [Google Cloud Acceptable Use Policy](https://cloud.google.com/terms/aup), and might cause Google to throttle or suspend your workload. When you run authorized scanning or high-connection workloads through Cloud NAT, allocate enough static IP addresses and configure [port reservation](https://docs.cloud.google.com/nat/docs/ports-and-addresses.md.txt) to prevent port exhaustion. ## Troubleshoot This section describes the common errors you might encounter while deploying your App Engine service with Direct VPC egress. ### Cannot delete subnet To delete a subnet, you must first delete or redeploy all resources that use it. If App Engine is using a subnet, disconnect the App Engine [service](https://docs.cloud.google.com/appengine/docs/standard/vpc-direct-vpc.md.txt#disconnect-service) from the VPC network or move it to a different subnet before deleting the subnet. After you delete or move your App Engine service, wait 1-2 hours for App Engine to release the IP addresses before you delete the subnet. ### Deployment failures If your deployment fails, the Google Cloud CLI displays error messages indicating the root cause. Some common issues include the following: - Incorrect VPC metadata, such as a misspelled network or subnet name in your `app.yaml` file. To fix potential errors, review the VPC configuration in your `app.yaml` file. - Insufficient IAM permissions. Make sure you [grant the required permissions](https://docs.cloud.google.com/appengine/docs/standard/vpc-direct-vpc.md.txt#required-roles) to the App Engine service agent. If you encounter permission errors during deployment, ensure that you grant the following additional roles to your deploying service account: - [Cloud Build Service Account](https://docs.cloud.google.com/iam/docs/roles-permissions/cloudbuild.md.txt#cloudbuild.builds.builder) (`roles/cloudbuild.builds.builder`) - [Service Account Token Creator](https://docs.cloud.google.com/iam/docs/roles-permissions/iam.md.txt#iam.serviceAccountTokenCreator) (`roles/iam.serviceAccountTokenCreator`) ### IP address exhaustion If your subnet runs out of available IP addresses, App Engine fails to start new instances and [logs an error](https://docs.cloud.google.com/appengine/docs/standard/writing-application-logs.md.txt?tab=python#view_logs). To resolve this issue, expand the IP range of your subnet or move your service to a larger subnet. ### IP address leaks IP address leaks might lead to IP address exhaustion. IP address leaks are unlikely during standard operations, but might occur due to the following causes: - The App Engine service agent only has permissions to reserve addresses (`create`, `createInternal`) and lacks permissions to release addresses (`delete`, `deleteInternal`). - The App Engine service agent was deleted while other Google Cloud address reservations were active. - The Cloud Billing account was removed and later re-enabled in the project while Serverless address reservations were active. - Your Google Cloud project was deleted and later restored while Serverless address reservations still existed. To resolve this issue, follow these steps: 1. Query for the following logs in the [Logs Explorer](https://docs.cloud.google.com/logging/docs/view/building-queries.md.txt#building-queries) to identify the leaked addresses: protoPayload.authorizationInfo.permission=~"compute.addresses.delete.*" protoPayload.authorizationInfo.resourceAttributes.type="compute.addresses" protoPayload.resourceName=~"projects/.*/regions/.*/addresses/serverless-.*" severity>=WARNING If this query doesn't return results, there are no IP address leaks, and no further action is required. 2. If your query returns results, make sure that the App Engine service agent has the [App Engine Service Agent role (`roles/appengine.serviceAgent`)](https://docs.cloud.google.com/iam/docs/roles-permissions/appengine.md.txt#appengine.serviceAgent). If you can't use this role, ensure that you grant the other [required roles and permissions](https://docs.cloud.google.com/appengine/docs/standard/vpc-direct-vpc.md.txt#required-roles) to the App Engine service agent. 3. Manually delete leaked IP addresses by using the Google Cloud console or the Google Cloud CLI: ### Console 1. Go to the **IP addresses** page in the Google Cloud console: [Go to IP addresses](https://console.cloud.google.com/networking/addresses) 2. Select the leaked IP addresses you identified in the previous step by running the query. 3. Click **Release static address** to remove the leaked addresses. ### gcloud Run the [`gcloud compute addresses delete`](https://docs.cloud.google.com/sdk/gcloud/reference/compute/addresses/delete) command: ```bash gcloud compute addresses delete ADDRESS_NAME --region=REGION ``` Replace the following: - ADDRESS_NAME: the name of the leaked IP address. - REGION: the region of the leaked IP address.