This page outlines the best practices for configuring networking options for App Engine resources. We recommend that you review all the sections on this page to understand the networking options that App Engine supports, and their implications. **Best practices**: [IP address management](https://docs.cloud.google.com/appengine/docs/standard/networking-best-practices.md.txt#ip-management). [Monitor IP address usage](https://docs.cloud.google.com/appengine/docs/standard/networking-best-practices.md.txt#monitor-ip-usage). [Use non-RFC 1918 IP addresses](https://docs.cloud.google.com/appengine/docs/standard/networking-best-practices.md.txt#non-rfc-1918). [Use IPv4 and IPv6 (dual-stack) subnets](https://docs.cloud.google.com/appengine/docs/standard/networking-best-practices.md.txt#dual-stack-subnets). [Use connection pooling and reuse connections](https://docs.cloud.google.com/appengine/docs/standard/networking-best-practices.md.txt#connection-pooling). [Use supported runtimes](https://docs.cloud.google.com/appengine/docs/standard/networking-best-practices.md.txt#second-gen-runtimes). [Faster external throughput to the internet](https://docs.cloud.google.com/appengine/docs/standard/networking-best-practices.md.txt#external-throughput). [Faster internal throughput to a Google API](https://docs.cloud.google.com/appengine/docs/standard/networking-best-practices.md.txt#internal-vpc-throughput). ## IP management You must manage your IP addresses because each instance of your App Engine service consumes IP addresses from your subnet. Use the following strategies for managing your IP addresses: - **Alternative IP allocation** : Allocate standard private ranges or alternative IP ranges to mitigate IP address exhaustion. See [IP address exhaustion strategies](https://docs.cloud.google.com/appengine/docs/standard/networking-best-practices.md.txt#alt-ip-address). - **Subnet sizing** : Ensure your subnet's IPv4 address range is `/26` or larger to provide enough IP addresses for scaling. - **Overprovision IP addresses**: We recommend overprovisioning the number of available IP addresses in your subnet to avoid exhaustion. Similar to other App Engine services, you should typically use four times as many IP addresses (2X for steady state and an additional 2X during deployment) as the number of running instances to facilitate smooth scaling and updates. ## Monitor IP address usage If you're using Direct VPC egress, [make sure that you have enough IP addresses allocated for your subnet](https://docs.cloud.google.com/appengine/docs/standard/vpc-direct-vpc.md.txt#direct-vpc-ip-addresses). The number of IP addresses you use depends on the number of instances that your workloads run, so we recommend monitoring your IP address usage. Be sure that your IP usage over time stays within the bounds supported by the subnet. To estimate your IP address usage: 1. In the Google Cloud console, go to the Metrics Explorer page: [Go to Metrics Explorer](https://console.cloud.google.com/monitoring/metrics-explorer) 2. Look up the number of instances in your project by using the [`appengine.googleapis.com/system/instance_count`](https://docs.cloud.google.com/monitoring/api/metrics_gcp_a_b.md.txt#gcp-appengine) metric type. This metric counts all App Engine instances across the project. For versions configured with Direct VPC egress, filter by the `module_id` label, which identifies the service, or by the `version_id` label. Cloud Monitoring lets you view this metric's value over time. 3. Multiply the instance count metric's value by 2 to get an estimate of the number of IP addresses in use. ## IP address exhaustion strategies Having a large number of App Engine workloads can cause IP exhaustion challenges when using the RFC 1918 private IP address space with Direct VPC egress. The following strategies can help you manage IP address exhaustion by using alternative IP address ranges. ### Use non-RFC 1918 IPv4 addresses Aside from the RFC 1918 IPv4 address ranges, App Engine also supports [RFC 6598](https://datatracker.ietf.org/doc/html/rfc6598#section-7) and [Class E/RFC 5735](https://tools.ietf.org/html/rfc5735) ranges. All Google Cloud services and features work with these non-RFC 1918 ranges, including VPC networks, Cloud Load Balancing, and Private Service Connect. For the best compatibility, we recommend starting with the RFC 6598 (100.64.0.0/10) range. If you already use this range elsewhere, consider Class E/RFC 5735 (240.0.0.0/4) instead. Class E is a huge space with over 268 million IP addresses available, so it'll support your growth for a long time. However, Class E has some limitations. For example, it's not supported on Windows and on some on-premises hardware. Read about [leveraging Class E IPv4 Address space to mitigate IPv4 exhaustion issues in GKE](https://cloud.google.com/blog/products/containers-kubernetes/how-class-e-addresses-solve-for-ip-address-exhaustion-in-gke). ### Use Cloud NAT or Private Service Connect If your App Engine workload is in a non-RFC 1918 range and needs to reach an on-premises destination that accepts only RFC 1918, choose one of the following solutions: - Use [Hybrid NAT](https://docs.cloud.google.com/nat/docs/private-nat.md.txt) to perform address translation and egress through a small RFC 1918 range. - Expose the on-premises service as a [Private Service Connect](https://docs.cloud.google.com/vpc/docs/configure-private-service-connect-producer.md.txt) hybrid service. ### Use IPv4 and IPv6 (dual-stack) subnets Although it won't reduce IPv4 exhaustion, consider moving your apps to [IPv6](https://docs.cloud.google.com/vpc/docs/ipv6-support.md.txt). To avoid IPv4 exhaustion problems in the future, [Set up dual-stack resources](https://docs.cloud.google.com/appengine/docs/standard/vpc-dual-stack-subnet.md.txt). ## Port exhaustion reduction strategies The following section describes strategies for reducing port exhaustion with App Engine. ### Use connection pooling and reuse connections When sending a large number of requests from your App Engine app to a single destination IP address, use connection pooling to maintain and reuse connections to the destination. High connection rates to a single IP address can exhaust outbound ports and cause connection refused errors. ## Performance and throughput strategies This section covers scalable options for improving network performance and throughput towards the internet and Google services. ### Use supported runtimes Direct VPC egress is only supported on standard [second-generation runtimes](https://docs.cloud.google.com/appengine/docs/standard/runtimes). Supported runtimes run standard language implementations, support modern library integrations, and offer faster, more robust networking performance. ### Use Direct VPC egress for faster network egress throughput To achieve faster throughput across network egress connections, use Direct VPC egress to route traffic through your VPC network. #### Example 1: External traffic to the internet If you're sending external traffic to the public internet, route all traffic through the VPC network by setting the `vpc_egress` field to `all-traffic` in your `app.yaml` file. With this approach, you must set up Cloud NAT to reach the public internet. To set up Cloud NAT for your App Engine service, see [Set up a static outbound IP address](https://docs.cloud.google.com/appengine/docs/standard/outbound-ip-addresses.md.txt#static-ip). For the Cloud NAT gateway settings to use with Direct VPC egress for Public NAT or Private NAT, see [Cloud NAT Direct VPC egress interactions](https://docs.cloud.google.com/nat/docs/nat-product-interactions.md.txt#interactions-direct-vpc). #### Example 2: Internal traffic to a Google API If you're using Direct VPC egress to send traffic to a Google API, such as Cloud Storage, choose one of the following options: - Specify `private-ranges-only` (default) with [Private Google Access](https://docs.cloud.google.com/vpc/docs/configure-private-google-access.md.txt): 1. Set `vpc_egress: private-ranges-only` in your `app.yaml` file. 2. Enable Private Google Access. 3. [Configure DNS for Private Google Access](https://docs.cloud.google.com/vpc/docs/configure-private-google-access.md.txt#config-domain). Make sure your target domain (such as `storage.googleapis.com`) maps to one of the following internal IP address ranges: - `199.36.153.8/30` - `199.36.153.4/30` - Specify `all-traffic` with [Private Google Access](https://docs.cloud.google.com/vpc/docs/configure-private-google-access.md.txt): 1. Set `vpc_egress: all-traffic` in your `app.yaml` file. 2. Enable Private Google Access. ## Use the default MTU setting for App Engine Don't change the [maximum transmission unit (MTU)](https://docs.cloud.google.com/vpc/docs/mtu.md.txt) setting of a VPC network when using it with App Engine. Use the default MTU of 1,460 bytes instead. ## Cost considerations When configuring networking options for your resources, consider the following: - **Colocate your resources**: Try to deploy your App Engine resources in the same region as your backend databases, such as Cloud SQL, Firestore, and Cloud Storage buckets. Data transfer between Google Cloud resources within the same region is free. - **Switch to Direct VPC egress** : If you are securely routing traffic to internal VPC network resources, consider switching to [Direct VPC egress](https://docs.cloud.google.com/appengine/docs/standard/vpc-direct-vpc.md.txt) from Serverless VPC Access connectors. Direct VPC egress doesn't incur baseline compute charges, eliminating the baseline compute overhead and idle costs associated with connector instances. - **Use Cloud CDN** : Offload static assets and highly cacheable content by placing [Cloud CDN](https://docs.cloud.google.com/cdn/docs.md.txt) in front of your App Engine resources. Serving data from the edge is significantly cheaper than paying for standard internet egress directly from App Engine. You can't enable Cloud CDN directly on the default `appspot.com` URL. Instead, route requests through an external Application Load Balancer that has a serverless network endpoint group (NEG) backend. For more information, see [Set up a global external Application Load Balancer with App Engine](https://docs.cloud.google.com/load-balancing/docs/https/setup-global-ext-https-serverless.md.txt). - **Monitor internet egress**: Outbound traffic that crosses region boundaries or goes to the public internet incurs standard charges. Focus your monitoring efforts on outbound traffic that crosses region boundaries. Review [App Engine pricing](https://cloud.google.com/appengine/pricing) or estimate costs with the [pricing calculator](https://cloud.google.com/products/calculator) for more information. ## What's next - [Compare Direct VPC egress and Serverless VPC Access connectors](https://docs.cloud.google.com/appengine/docs/standard/compare-direct-vpc-egress-connectors.md.txt) - [Configure Direct VPC egress](https://docs.cloud.google.com/appengine/docs/standard/vpc-direct-vpc.md.txt)