# Configuration updates for modernization

This document describes configuration updates you must make to your managed
Cloud Service Mesh clusters if flagged by compatibility checks before
modernizing from the [deprecated](https://docs.cloud.google.com/service-mesh/deprecations.md.txt)
`ISTIOD` control plane to the `TRAFFIC_DIRECTOR`
control plane before the end-of-support deadline.

The following are a list of possible configuration updates necessary to prepare
your cluster for modernization. See each section for update instructions:

- [Multi-cluster](https://docs.cloud.google.com/service-mesh/docs/migrate/modernization-configuration-updates.md.txt#multicluster_secrets)
- [Enable Workload Identity Federation for GKE](https://docs.cloud.google.com/service-mesh/docs/migrate/modernization-configuration-updates.md.txt#enable_workload_identity)
- [Enable managed CNI](https://docs.cloud.google.com/service-mesh/docs/migrate/modernization-configuration-updates.md.txt#enable_managed_cni)
- [Non-Standard Binary Usage in Sidecar](https://docs.cloud.google.com/service-mesh/docs/migrate/modernization-configuration-updates.md.txt#nonstandard_sidecar_binary_usage)
- [Migrate to the Istio Ingress Gateway](https://docs.cloud.google.com/service-mesh/docs/migrate/modernization-configuration-updates.md.txt#migrate_to_the_istio_ingress_gateway)
- [Modernize EnvoyFilter compressor configurations](https://docs.cloud.google.com/service-mesh/docs/migrate/modernization-configuration-updates.md.txt#modernize_envoyfilter_compressor_configurations)
- [Enable Internet Network Endpoint Groups](https://docs.cloud.google.com/service-mesh/docs/migrate/modernization-configuration-updates.md.txt#org_policy_disabled)
- [Shared VPC Permissions](https://docs.cloud.google.com/service-mesh/docs/migrate/modernization-configuration-updates.md.txt#shared_vpc_permissions)
- [Missing Control Plane Configuration](https://docs.cloud.google.com/service-mesh/docs/migrate/modernization-configuration-updates.md.txt#missing_control_plane_config)

For more information on the modernization workflow, see the
[Managed control plane modernization](https://docs.cloud.google.com/service-mesh/docs/modernization.md.txt) page.

## Migrate from Istio secrets to multicluster_mode

Multi-cluster secrets are not supported when a cluster is using the
`TRAFFIC_DIRECTOR` control plane. This document describes how you can
modernize from using Istio multi-cluster secrets to using `multicluster_mode`.

### Istio secrets versus declarative API overview

[Open source istio multi-cluster](https://docs.cloud.google.com/kubernetes-engine/docs/concepts/workload-identity.md.txt#how_works) endpoint discovery works by using
`istioctl` or other tools to create a [Kubernetes Secret](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/managing-clusters.md.txt#before_you_begin) in a cluster. This
secret allows a cluster to load balance traffic to another cluster in the mesh.
The `ISTIOD` control plane then reads this secret and
begins routing traffic to that other cluster.

Cloud Service Mesh has a
[declarative API](https://docs.cloud.google.com/service-mesh/docs/operate-and-maintain/multi-cluster.md.txt#endpoint-discovery-declarative-api)
to control multi-cluster traffic instead of directly creating Istio secrets.
This API treats Istio secrets as an implementation detail and is more reliable
than creating Istio secrets manually. Future Cloud Service Mesh features will
depend on the declarative API, and you won't be able to use those new features
with Istio secrets directly. The declarative API is the only supported path
forward.

If you are using Istio Secrets, migrate to using the declarative API as soon as
possible. Note that the `multicluster_mode` setting directs each cluster to
direct traffic to every other cluster in the mesh. Using secrets allows a more
flexible configuration, letting you configure for each cluster which other
cluster it should direct traffic to in the mesh. For a full list of the
differences between the supported features of the declarative API and Istio
secrets, see
[Supported features using Istio APIs](https://docs.cloud.google.com/service-mesh/docs/supported-features-managed.md.txt#multi-cluster_deployment).

> [!IMPORTANT]
> **Important:** With the declarative API, an entire cluster is opted into endpoint discovery at a time. This means that every cluster with \`multicluster_mode=connected\` will discover endpoints for every other cluster in the fleet that also has \`multicluster_mode=connected\`.

### Migrate from Istio secrets to declarative API

If you provisioned Cloud Service Mesh using automatic management with the
[fleet feature API](https://docs.cloud.google.com/service-mesh/docs/onboarding/provision-control-plane.md.txt), you don't
need to follow these instructions. These steps only apply if you onboarded using
`asmcli --managed`.

Note, this process changes secrets that point to a cluster. During this process,
the endpoints are removed and then re-added. In between the endpoints being
removed and added, the traffic will briefly revert to routing locally instead of
load balancing to other clusters. For more information, see the
[GitHub issue](https://github.com/istio/istio/issues/36365).

To move from using Istio secrets to the declarative API, follow these steps.
Execute these steps at the same time or in close succession:

1. Enable the declarative API for each cluster in the fleet where you want to
   enable multi cluster endpoint discovery by setting
   `multicluster_mode=connected`. Note that you need to explicitly set
   `multicluster_mode=disconnected` if you don't want the cluster to be
   discoverable.

   Use the following command to opt in a cluster for multi cluster endpoint
   discovery:

        kubectl patch configmap/asm-options -n istio-system --type merge -p '{"data":{"multicluster_mode":"connected"}}'

   Use the following command to opt a cluster out of endpoint discovery:

        kubectl patch configmap/asm-options -n istio-system --type merge -p '{"data":{"multicluster_mode":"disconnected"}}'

2. Delete old secrets.

   After setting `multicluster_mode=connected` on your clusters, each cluster
   will have a new secret generated for every other cluster that also has
   `multicluster_mode=connected` set. The secret is placed in the istio-system
   namespace and have the following format:

   ```bash
   istio-remote-secret-projects-PROJECT_NAME-locations-LOCATION-memberships-MEMBERSHIPS
   ```

   Each secret will also have the label `istio.io/owned-by:
   mesh.googleapis.com` applied.

   Once the new secrets are created, you can delete any secrets manually
   created with `istioctl create-remote-secret`:

   ```bash
   kubectl delete secret SECRET_NAME -n istio-system
   ```

Once migrated, check your request metrics to make sure they're routed as
expected.

## Enable Workload Identity Federation for GKE

Workload Identity Federation is the recommended secure method for Google Kubernetes Engine
workloads. This allows access to Google Cloud services such as Compute Engine,
BigQuery, and Machine Learning APIs. Workload Identity Federation doesn't
require manual configuration or less secure methods like service account key
files because it uses IAM policies. For more details on
Workload Identity Federation, see [How Workload Identity Federation for GKE works](https://docs.cloud.google.com/kubernetes-engine/docs/concepts/workload-identity.md.txt#how_works).

The following section describe how to enable Workload Identity Federation.

### Enable Workload Identity Federation on clusters

1. Check Workload Identity Federation is enabled for your cluster. To do that,
   ensure the GKE cluster has a Workload Identity Federation pool
   set configured, which is essential for IAM credential
   validation.

   Use the following command, to check the workload identity pool set for a
   cluster:

   ```bash
   gcloud container clusters describe CLUSTER_NAME \
     --format="value(workloadIdentityConfig.workloadPool)"
   ```

   Replace `CLUSTER_NAME` with the name of your GKE cluster. If
   you haven't already specified [a default zone or region for gcloud](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/managing-clusters.md.txt#before_you_begin), you
   might also need to specify a `--region` or `--zone` flag when running this
   command.
2. If the output is empty, follow the instructions in
   [Update an existing cluster](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/workload-identity.md.txt#enable-existing-cluster) to enable workload identity on existing
   GKE clusters.

### Enable Workload Identity Federation on node pools

After Workload Identity Federation is enabled on a cluster, [node pools](https://docs.cloud.google.com/kubernetes-engine/docs/concepts/node-pools.md.txt) must be
configured to use the [GKE metadata server](https://docs.cloud.google.com/kubernetes-engine/docs/concepts/workload-identity.md.txt#metadata_server).

1. List all the node pools of a Standard cluster. Run the
   [gcloud container node-pools list](https://docs.cloud.google.com/sdk/gcloud/reference/container/clusters/list) command:

   ```bash
   gcloud container node-pools list --cluster CLUSTER_NAME
   ```

   Replace `CLUSTER_NAME` with the name of your GKE cluster. If
   you haven't already specified [a default zone or region for gcloud](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/managing-clusters.md.txt#before_you_begin), you
   might also need to specify a `--region` or `--zone` flag when running this
   command.
2. Verify that each node pool is using the GKE metadata server:

   ```bash
   gcloud container node-pools describe NODEPOOL_NAME \
       --cluster=CLUSTER_NAME \
       --format="value(config.workloadMetadataConfig.mode)"
   ```

   Replace the following:
   - `NODEPOOL_NAME` with the name of your nodepool.
   - `CLUSTER_NAME` with the name of your GKE cluster.
3. If the output doesn't contain `GKE_METADATA`, update the node pool using the
   [Update an existing node pool](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/workload-identity.md.txt#option_2_node_pool_modification) guide.

## Enable managed container network interface (CNI)

This section guides you through enabling managed CNI for Cloud Service Mesh
on Google Kubernetes Engine.

### Managed CNI overview

Managed container network interface (CNI) is a Google-managed implementation of
the Istio CNI. The
[CNI plugin](https://github.com/containernetworking/cni#cni---the-container-network-interface)
streamlines pod networking by configuring iptables rules. This enables traffic
redirection between applications and Envoy proxies, eliminating the need for
privileged permissions for the init-container required to manage `iptables`.

The [Istio CNI plugin](https://istio.io/latest/docs/setup/additional-setup/cni/)
replaces the `istio-init` container. The `istio-init` container was previously
responsible for setting up the pod's network environment to enable traffic
interception for the Istio sidecar. The CNI plugin performs the same network
redirect function, but with the added benefit of reducing the need for elevated
privileges, thereby enhancing security.

Therefore, for enhanced security and reliability, and to simplify management and
troubleshooting, managed CNI is required across all Managed
Cloud Service Mesh deployments.

### Impact on init containers

Init containers are specialized containers that run before application
containers for setup tasks. Setup tasks can include tasks such as downloading
configuration files, communicating with external services, or performing
pre-application initialization. Init containers that rely on network access
might encounter issues when managed CNI is enabled in the cluster.

The pod setup process with managed CNI is as follows:

1. The CNI plugin sets up pod network interfaces, assigns pod IPs and redirects traffic to the Istio sidecar proxy which hasn't started yet.
2. All init containers execute and complete.
3. The Istio sidecar proxy starts alongside the application containers.

Therefore, if an init container attempts to make outbound network connections or
connect to services within the mesh, the network requests from the init
containers may be dropped or misrouted. This is because the Istio sidecar proxy,
which manages network traffic for the pod, is not running when the requests are
made. For more details, refer
[Istio CNI documentation](https://istio.io/latest/docs/setup/additional-setup/cni/#how-sidecar-traffic-redirection-works).

### Enable managed CNI for your cluster

Follow the steps in this section to enable managed CNI on your cluster.

1. Remove network dependencies from your init container. Consider the following
   alternatives:

   - **Modify application logic or containers:** You can modify your services to remove the dependency on init containers that require network requests or perform network operations within your application containers, after the sidecar proxy has started.
   - **Use Kubernetes ConfigMaps or secrets:** Store configuration data fetched by the network request in Kubernetes ConfigMaps or secrets and mount them into your application containers. For alternative solutions, refer to the [Istio documentation](https://istio.io/latest/docs/setup/additional-setup/cni/#compatibility-with-application-init-containers).
2. Enable managed CNI on your cluster:

   1. Make the following configuration changes:

      1. Run the following command to locate the `controlPlaneRevision`.

         ```bash
         kubectl get controlplanerevision -n istio-system
         ```
      2. In your `ControlPlaneRevision` (CPR) custom resource (CR), set the
         label `mesh.cloud.google.com/managed-cni-enabled` to `true`.

         ```bash
         kubectl label controlplanerevision CPR_NAME \
             -n istio-system mesh.cloud.google.com/managed-cni-enabled=true \
             --overwrite
         ```

         Replace `CPR_NAME` with the value under the NAME column from the
         output of the previous step.
      3. In the asm-options ConfigMap, set the `ASM_OPTS` value to `CNI=on`.

         ```bash
         kubectl patch configmap asm-options -n istio-system \
             -p '{"data":{"ASM_OPTS":"CNI=on"}}'
         ```
      4. In your `ControlPlaneRevision` (CPR) custom resource (CR), set the
         annotation `mesh.cloud.google.com/force-reprovision` to `true`. This
         action triggers control plane restart.

         > [!NOTE]
         > **Note:** This method is not the recommended method for restarting the control plane, and should only be used for Cloud Service Mesh modernization efforts.

         ```bash
         kubectl annotate controlplanerevision CPR_NAME \
             -n istio-system mesh.cloud.google.com/force-reprovision=true \
             --overwrite
         ```
   2. Check the feature state. Retrieve the feature state using the following
      command:

      ```bash
      gcloud container fleet mesh describe --project FLEET_PROJECT_ID
      ```

      Replace `FLEET_PROJECT_ID` with the ID of your Fleet Host project.
      Generally, the `FLEET_PROJECT_ID` has the same name as the project.
      - Verify that the `MANAGED_CNI_NOT_ENABLED` condition is removed from `servicemesh.conditions`.
      - Note, it may take up to 15-20 minutes for the state to update. Try waiting a few minutes and re-run the command.
   3. Once the `controlPlaneManagement.state` is `Active` in the cluster's
      feature state, **restart the pods**.

## Move away from Non-Standard Binary Usage in Sidecar

This section suggest ways to make your deployments compatible with the
distroless envoy proxy image.

### Distroless envoy proxy sidecar images

Cloud Service Mesh uses two types of Envoy proxy sidecar images based on your
[control plane configuration](https://docs.cloud.google.com/service-mesh/docs/supported-features-managed.md.txt#identify_control_plane_implementation):
the **`default`** proxy image (containing various debug binaries) and the **`distroless`** proxy image.

For clusters with the managed `TRAFFIC_DIRECTOR` control plane:

- **Directly onboarded clusters** : Use `distroless` images by default.
- **Clusters migrated from CSM-ISTIOD** : Use `default` images by default. You can explicitly opt-in to `distroless` images to improve your security posture.

To opt-in to `distroless` images for a cluster migrated from CSM-ISTIOD, you can configure it globally using `defaultConfig.image.imageType: distroless` in your `MeshConfig`, or per-workload using the `sidecar.istio.io/proxyImageType: distroless` pod annotation.

Distroless base images are minimal container images that prioritize security and resource
optimization by only including essential components. The attack surface is
reduced to help prevent vulnerabilities. For more information, refer to the
documentation on
[Distroless proxy image](https://docs.cloud.google.com/service-mesh/docs/enable-optional-features-managed.md.txt#distroless_proxy_image).

### Binary compatibility

As a best practice, you should restrict the contents of a container runtime to
only the necessary packages. This approach improves security and the
signal-to-noise ratio of Common Vulnerabilities and Exposures (CVE) scanners.
Distroless Sidecar image has a minimal set of dependencies, stripped of all
non-essential executables, libraries, and debugging tools. It is therefore not
possible to execute a shell command or use curl, ping, or other debug utilities
like `kubectl exec` inside the container.

### Make clusters compatible with distroless images

- Remove references to any unsupported binaries (like bash or curl) from your configuration. Particularly inside Readiness, Startup, and Liveness [probes](https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/), and Lifecycle PostStart and PreStop [hooks](https://kubernetes.io/docs/concepts/containers/container-lifecycle-hooks/) within the istio-proxy, istio-init, or istio-validation containers.
- Consider alternatives like [holdApplicationUntilProxyStarts](https://docs.cloud.google.com/service-mesh/docs/troubleshooting/troubleshoot-pod-startup-time.md.txt#global_mesh_configuration_holdapplicationuntilproxystarts) for certain use cases.
- For debugging, you can use [ephemeral containers](https://kubernetes.io/docs/concepts/workloads/pods/ephemeral-containers/) to attach to a running workload Pod. You can then inspect it and run custom commands. For an example, see [Collecting {service_mesh_name} logs](https://docs.cloud.google.com/service-mesh/docs/troubleshooting/troubleshoot-collect-logs.md.txt#configure_envoy_proxy).

If you can't find a solution for your specific use case, contact Google Cloud
Support at [Getting support](https://docs.cloud.google.com/service-mesh/docs/getting-support.md.txt).

## Migrate to the Istio Ingress Gateway

This section shows you how to migrate to the Istio Ingress Gateway. There are
two methods for migrating to the Istio Ingress Gateway:

1. Phased Migration with Traffic Splitting

   This method prioritizes minimizing disruption in which you'll incrementally
   send traffic to the new Istio gateway, allowing you to monitor its
   performance on a small percentage of requests and quickly revert if
   necessary. Keep in mind that configuring Layer 7 traffic splitting can be
   challenging for some applications, so you need to manage both gateway
   systems concurrently during the transition. See
   [Phased Migration with traffic splitting](https://docs.cloud.google.com/service-mesh/docs/migrate/modernization-configuration-updates.md.txt#phased_migration_with_traffic_splitting)
   for the steps.
2. Direct Migration

   This method involves simultaneously rerouting all traffic to the new Istio
   gateway once you have thoroughly conducted testing. The advantage of this
   approach is complete separation from the old gateway's infrastructure,
   allowing adaptable configuration of the new gateway without the constraints
   of the existing setup. However, there is an increased risk of downtime in
   case unexpected problems arise with the new gateway during the transition.
   See [Direct Migration](https://docs.cloud.google.com/service-mesh/docs/migrate/modernization-configuration-updates.md.txt#direct_migration) for the steps.

The following migration examples assume you have an HTTP service
([httpbin](https://raw.githubusercontent.com/istio/istio/release-1.25/samples/httpbin/httpbin.yaml))
running in the application namespace (default) and exposed externally using
Kubernetes Gateway API. The relevant configurations are:

- Gateway: `k8-api-gateway` (in `istio-ingress` namespace) - configured to listen for HTTP traffic on port 80 for any hostname ending with `.example.com`.
- HTTPRoute: `httpbin-route` (in `default` namespace) - directs any HTTP request with the hostname `httpbin.example.com` and a path starting with `/get` to the `httpbin` service within the `default` namespace.
- The httpbin application is accessible using the external IP 34.57.246.68.

![Basic gateway diagram](https://docs.cloud.google.com/static/service-mesh/docs/images/gateway-basic-setup.svg)

### Phased Migration with traffic splitting

#### Provision a new Istio Ingress Gateway

1. Deploy a new Ingress Gateway following the steps in the
   [Deploy sample gateway section](https://docs.cloud.google.com/service-mesh/docs/operate-and-maintain/gateways.md.txt#deploy_sample_gateway)
   and customize the sample configurations to your requirements. The samples in
   the [anthos-service-mesh](https://github.com/GoogleCloudPlatform/anthos-service-mesh-packages/tree/main/samples/gateways/istio-ingressgateway)
   repository are meant for deploying a `istio-ingressgateway` loadBalancer
   service and the corresponding `ingress-gateway` pods.

   Example Gateway Resource (istio-ingressgateway.yaml)

   ```yaml
    apiVersion: networking.istio.io/v1beta1
    kind: Gateway
    metadata:
      name: istio-api-gateway
      namespace: GATEWAY_NAMESPACE
    spec:
      selector:
        istio: ingressgateway  # The selector should match the ingress-gateway pod labels.
      servers:
      - port:
          number: 80
          name: http
          protocol: HTTP
        hosts:   # or specific hostnames if needed
        - "httpbin.example.com"
   ```
2. Apply the [Gateway](https://istio.io/v1.24/docs/reference/config/networking/gateway/#Gateway)
   configuration to manage traffic:

   ```bash
   kubectl apply -f istio-ingressgateway.yaml -n GATEWAY_NAMESPACE
   ```

   Ensure the 'spec.selector' in your Gateway resource matches the labels of
   your `ingress-gateway` pods. For example, if the `ingress-gateway` pods have
   the label `istio=ingressgateway`, your Gateway configuration must also select
   this the `istio=ingressgateway` label.

#### Configure initial routing for the new Gateway

1. Define the initial routing rules for your application using an Istio
   [VirtualService](https://istio.io/latest/docs/reference/config/networking/virtual-service/).

   Example VirtualService (my-app-vs-new.yaml):

   ```yaml
   apiVersion: networking.istio.io/v1beta1
   kind: VirtualService
   metadata:
     name: httpbin-vs
     namespace: APPLICATION_NAMESPACE
   spec:
       gateways:
       - istio-ingress/istio-api-gateway  # Replace with <gateway-namespace/gateway-name>
       hosts:
       - httpbin.example.com
       http:
       - match:
         - uri:
             prefix: /get
         route:
         - destination:
             host: httpbin
             port:
               number: 8000
   ```
2. Apply the VirtualService:

   ```bash
   kubectl apply -f my-app-vs-new.yaml -n MY_APP_NAMESPACE
   ```

#### Access the backend (httpbin) service through the newly deployed Istio Ingress Gateway

1. Set the Ingress Host environment variable to the external IP address
   associated with the recently deployed `istio-ingressgateway` load balancer:

   ```bash
   export INGRESS_HOST=$(kubectl -n GATEWAY_NAMESPACE get service istio-ingressgateway -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
   ```
2. Verify the application (httpbin) is accessible using the new gateway:

   ```bash
   curl -s -I -HHost:httpbin.example.com "http://$INGRESS_HOST/get"
   ```

   The output is similar to:

   ```bash
   HTTP/1.1 200 OK
   ```

![Request flow with the new istio ingress gateway](https://docs.cloud.google.com/static/service-mesh/docs/images/istio-api-gateway.svg)

#### Modify existing Ingress for traffic splitting

After confirming the successful setup of the new gateway (ex.
istio-api-gateway), you can begin routing a portion of your traffic through it.
To do this, update your current
[HTTPRoute](https://gateway-api.sigs.k8s.io/api-types/httproute/) to direct a
small percentage of traffic to the new gateway, while the larger portion
continues to use the existing gateway (k8-api-gateway).

1. Open the httproute for editing:

   ```bash
   kubectl edit httproute httpbin-route -n MY_APP_NAMESPACE
   ```
2. Add a new backend reference pointing to the new Ingress Gateway's
   loadbalancer service with an initial weight of 10% and update the weight for
   the old gateway's backend.

   ```yaml
   apiVersion: gateway.networking.k8s.io/v1
   kind: HTTPRoute
   metadata:
     name: httpbin-route
     namespace: MY_APP_NAMESPACE  # your application's namespace
   spec:
     parentRefs:
     - name: k8-api-gateway
       namespace: istio-ingress
     hostnames: ["httpbin.example.com"]
     rules:
     - matches:
       - path:
           type: PathPrefix
           value: /get
       backendRefs:
       - name: httpbin
         port: 8000
         weight: 90
       - name: istio-ingressgateway # Newly deployed load balancer service
         namespace: GATEWAY_NAMESPACE
         port: 80
         weight: 10
   ```
3. Grant Permission for Cross-Namespace Referencing with reference grant.

   To allow your `HTTPRoute` in the application namespace (default) to access
   `loadbalancer` service in gateway namespace (istio-ingress), you may need to
   create a
   [reference grant](https://gateway-api.sigs.k8s.io/api-types/referencegrant/).
   This resource serves as a security control, explicitly defining which
   cross-namespace references are permitted.

   The following `istio-ingress-grant.yaml` describes an example reference
   grant:

   ```yaml
   apiVersion: gateway.networking.k8s.io/v1beta1
   kind: ReferenceGrant
   metadata:
     name: istio-ingressgateway-grant
     namespace: istio-ingress # Namespace of the referenced resource
   spec:
     from:
     - group: gateway.networking.k8s.io
       kind: HTTPRoute
       namespace: MY_APP_NAMESPACE # Namespace of the referencing resource
     to:
     - group: ""               # Core Kubernetes API group for Services
       kind: Service
       name: istio-ingressgateway # Loadbalancer Service of the new ingress gateway
   ```
4. Apply the reference grant:

   ```bash
   kubectl apply -f istio-ingress-grant.yaml -n GATEWAY_NAMESPACE
   ```
5. Verify requests to existing external IP address (ex. 34.57.246.68)are not
   failing. The following `check-traffic-flow.sh` describes a script to check
   request failures:

   ```bash
   # Update the following values based on your application setup
   external_ip="34.57.246.68" # Replace with existing external IP
   url="http://$external_ip/get"
   host_name="httpbin.example.com"

   # Counter for successful requests
   success_count=0

   # Loop 50 times
   for i in {1..50}; do
     # Perform the curl request and capture the status code
     status_code=$(curl -s -HHost:"$host_name" -o /dev/null -w "%{http_code}" "$url")
     # Check if the request was successful (status code 200)
     if [ "$status_code" -eq 200 ]; then
       ((success_count++))  # Increment the success counter
     else
       echo "Request $i: Failed with status code $status_code"
     fi
   done

   # After the loop, check if all requests were successful
   if [ "$success_count" -eq 50 ]; then
     echo "All 50 requests were successful!"
   else
     echo "Some requests failed.  Successful requests: $success_count"
   fi
   ```
6. Execute the script to confirm that no requests fail, regardless of the
   traffic route:

   ```bash
   chmod +x check-traffic-flow.sh
   ./check-traffic-flow.sh
   ```

![Request flow with traffic split between existing gateway and new istio ingress gateway](https://docs.cloud.google.com/static/service-mesh/docs/images/istio-api-gateway-split.svg)

#### Slowly increase traffic percentage

If no request failures are seen for the existing external IP address (for
example, `34.57.246.68`), gradually shift more traffic to the new Istio Ingress
Gateway by adjusting the backend weights in your `HTTPRoute`. Increase the
weight for the `istio-ingressgateway` and decrease the weight for the old
gateway in small increments such as 10%, 20%, and on.

> [!IMPORTANT]
> **Important:** Continuously observe key metrics such as request success rate, latency, error rates, and the resource utilization of your application pods to ensure stability at each increment.

Use the following command to update your existing `HTTPRoute`:

```bash
kubectl edit httproute httpbin-route -n MY_APP_NAMESPACE
```

#### Full traffic migration and removing the old gateway

1. When the new Istio Ingress Gateway demonstrates stable performance and
   successful request handling, shift all traffic to it. Update your
   `HTTPRoute` to set the old gateway's backend weight to `0` and the new
   gateway's to `100`.

2. Once traffic is fully routed to the new gateway, update your external DNS
   records for your application's hostname (for example, `httpbin.example.com`)
   to point to the external IP address of the load balancer service created in
   [Provision a new Istio Ingress Gateway](https://docs.cloud.google.com/service-mesh/docs/migrate/modernization-configuration-updates.md.txt#provision_a_new_istio_ingress_gateway).

3. Finally, delete the old gateway and its associated resources:

   ```bash
   kubectl delete gateway OLD_GATEWAY -n GATEWAY_NAMESPACE
   kubectl delete service OLD_GATEWAY_SERVICE -n GATEWAY_NAMESPACE
   ```

### Direct Migration

#### Provision a new Istio Ingress Gateway

1. Deploy a new Ingress Gateway following the steps in the
   [Deploy sample gateway section](https://docs.cloud.google.com/service-mesh/docs/operate-and-maintain/gateways.md.txt#deploy_sample_gateway)
   and customize the sample configurations to your requirements. The samples in
   the [anthos-service-mesh](https://github.com/GoogleCloudPlatform/anthos-service-mesh-packages/tree/main/samples/gateways/istio-ingressgateway)
   repository are meant for deploying a `istio-ingressgateway` loadBalancer
   service and the corresponding `ingress-gateway` pods.

   Example Gateway Resource (istio-ingressgateway.yaml)

   ```yaml
    apiVersion: networking.istio.io/v1beta1
    kind: Gateway
    metadata:
      name: istio-api-gateway
      namespace: GATEWAY_NAMESPACE
    spec:
      selector:
        istio: ingressgateway  # The selector should match the ingress-gateway pod labels.
      servers:
      - port:
          number: 80
          name: http
          protocol: HTTP
        hosts:   # or specific hostnames if needed
        - "httpbin.example.com"
   ```
2. Apply the [Gateway](https://istio.io/v1.24/docs/reference/config/networking/gateway/#Gateway)
   configuration to manage traffic:

   ```bash
   kubectl apply -f istio-ingressgateway.yaml -n GATEWAY_NAMESPACE
   ```

   Ensure the 'spec.selector' in your Gateway resource matches the labels of
   your `ingress-gateway` pods. For example, if the `ingress-gateway` pods have
   the label `istio=ingressgateway`, your Gateway configuration must also select
   this the `istio=ingressgateway` label.

#### Configure initial routing for the new Gateway

1. Define the initial routing rules for your application using an Istio
   [VirtualService](https://istio.io/latest/docs/reference/config/networking/virtual-service/).

   Example VirtualService (my-app-vs-new.yaml):

   ```yaml
   apiVersion: networking.istio.io/v1beta1
   kind: VirtualService
   metadata:
     name: httpbin-vs
     namespace: APPLICATION_NAMESPACE
   spec:
       gateways:
       - istio-ingress/istio-api-gateway  # Replace with <gateway-namespace/gateway-name>
       hosts:
       - httpbin.example.com
       http:
       - match:
         - uri:
             prefix: /get
         route:
         - destination:
             host: httpbin
             port:
               number: 8000
   ```
2. Apply the VirtualService:

   ```bash
   kubectl apply -f my-app-vs-new.yaml -n MY_APP_NAMESPACE
   ```

#### Access the backend (httpbin) service through the newly deployed Istio Ingress Gateway

1. Set the Ingress Host environment variable to the external IP address
   associated with the recently deployed `istio-ingressgateway` load balancer:

   ```bash
   export INGRESS_HOST=$(kubectl -n GATEWAY_NAMESPACE get service istio-ingressgateway -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
   ```
2. Verify the application (httpbin) is accessible using the new gateway:

   ```bash
   curl -s -I -HHost:httpbin.example.com "http://$INGRESS_HOST/get"
   ```

   The output is similar to:

   ```bash
   HTTP/1.1 200 OK
   ```

![Request flow with the new istio ingress gateway](https://docs.cloud.google.com/static/service-mesh/docs/images/istio-api-gateway.svg)

#### Test and Monitor new gateway

1. Test all routing rules, validate TLS configuration, security policies, and
   other features. Perform load testing to verify the new gateway can handle
   expected traffic.

2. Once the new gateway is fully tested, update your external DNS records for
   your application's hostname (for example, `httpbin.example.com`) to point to
   the external IP address of the load balancer service created in
   [Provision a new Istio Ingress Gateway](https://docs.cloud.google.com/service-mesh/docs/migrate/modernization-configuration-updates.md.txt#provision_a_new_istio_ingress_gateway_2).

3. Monitor key metrics such as request success rate, latency, error rates, and
   the resource utilization of your application pods to verify stability with
   the new Istio Ingress Gateway. Once stable, you can delete the old gateway
   and its associated resources

   ```bash
   kubectl delete gateway OLD_GATEWAY -n GATEWAY_NAMESPACE
   kubectl delete service OLD_GATEWAY_SERVICE -n GATEWAY_NAMESPACE
   ```

Important Considerations: Ensure TLS certificates and configurations are
correctly set up on the new Istio Ingress Gateway if your application requires
HTTPS. See
[Set up TLS termination in ingress gateway](https://docs.cloud.google.com/service-mesh/docs/operate-and-maintain/gateway-tls-termination.md.txt)
for more details.

## Fix multiple control planes

Cloud Service Mesh previously supported
[onboarding using `asmcli` (deprecated)](https://docs.cloud.google.com/service-mesh/docs/release-notes.md.txt#August_22_2024)
which did not block provisioning multiple control planes. Cloud Service Mesh
now enforces the best practice of deploying only one channel per cluster that
matches the cluster channel and does not support using multiple deployed
channels in the same cluster.

If you want canary deployments on new versions of mesh in rapid before they
become available in stable or regular, you need to use two different clusters
where each has a separate channel. Note that channels are controlled by the
GKE Cluster Channel, and Mesh does not have a separate
channel associated with it.

You can check if you have multiple channels by looking for the
`UNSUPPORTED_MULTIPLE_CONTROL_PLANES` status condition on your membership. If
this warning does not appear, then you are not impacted and may skip reading
this section.

1. Run the following command to check if your cluster has multiple control
   plane channels:

   ```bash
   gcloud container fleet mesh describe
   ```

   The output is similar to:

       ...
       projects/.../locations/global/memberships/my-membership:
           servicemesh:
             conditions:
             - code: UNSUPPORTED_MULTIPLE_CONTROL_PLANES
               details: 'Using multiple control planes is not supported. Please remove a control plane from your cluster.'
               documentationLink: https://cloud.google.com/service-mesh/docs/migrate/modernization-configuration-updates#multiple_control_planes
               severity: WARNING
             controlPlaneManagement:
               details:
               - code: REVISION_READY
                 details: 'Ready: asm-managed-stable'
               implementation: ISTIOD
               state: ACTIVE
       ...

2. If the `UNSUPPORTED_MULTIPLE_CONTROL_PLANES` condition appeared, determine
   which channels exist for your cluster:

   ```bash
   kubectl get controlplanerevisions -n istio-system
   ```

   The output is similar to:

       NAME                 RECONCILED   STALLED   AGE
       asm-managed-stable   True         False     97d
       asm-managed          True         False     97d
       asm-managed-rapid    True         False     97d

   In this example, all three channels were provisioned:
   - asm-managed-stable -\> STABLE
   - asm-managed -\> REGULAR
   - asm-managed-rapid -\> RAPID

   If only 1 result appears, then only one channel is provisioned on your
   cluster, and you can skip the rest of these steps.

   If 2 or more results appear, then follow the rest of these steps to remove
   the surplus channels.

### Consolidate workloads to one channel

Before you can remove additional channels, you must ensure your workloads are
only using a single channel.

1. Find all the labels you are using in your cluster:

   ```bash
   kubectl get namespaces -l istio.io/rev=RELEASE_CHANNEL
   ```

   Depending on the output from the previous command, replace
   <var translate="no">RELEASE_CHANNEL</var> with `asm-managed-stable`, `asm-managed`, or
   `asm-managed-rapid`. Repeat this step for each provisioned channel.

   The output is similar to:

       NAME      STATUS   AGE
       default   Active   110d

   Note that in this example, the default namespace is being injected with the
   regular channel.

   If all of your workloads are already using the same channel, you can skip to
   step [Remove the extra channels](https://docs.cloud.google.com/service-mesh/docs/migrate/modernization-configuration-updates.md.txt#remove_the_extra_channels). Otherwise,
   continue in this section.
2. Change the labels so that only one channel is being used:

   > [!CAUTION]
   > **Caution:** Consider the following before changing labels.

   - Pods can also be injected directly in some cases with the `sidecar.istio.io/inject` label. Make sure to check those for that usage as well.
   - You can ignore `istio-injection=enabled` labels for this step. Namespaces with that label will automatically change to match whichever channel is left in the cluster.
   - When selecting a channel to keep, try to select the channel that is the same as your GKE Cluster Channel. If this channel does not exist, then select any one of the active channels.
   - The actual channel you select does not matter. The GKE Cluster Channel determines which version of mesh you get, not the Mesh channel.
   - Check your meshconfig configuration between any active channels that are in use to make sure there are no differences between them. Each channel uses a separate configmap for configuration, so consolidating two channels down to one should ensure consistent behavior between the two channels. `kubectl get configmap istio-asm-managed{-rapid | -stable} -n
     istio-system -o yaml`

   ```bash
   kubectl label namespace NAMESPACE istio.io/rev- istio-injection=enabled --overwrite
   ```

   Replace <var translate="no">NAMESPACE</var> with the name of your namespace.

   The best practice is to use `istio-injection=enabled`. However, if you don't
   want to use that label, then you can also use
   `istio.io/rev=RELEASE_CHANNEL`.

   Once you have changed the label for a namespace / pod, you must restart all
   workloads so that they are injected by the correct control plane.

### Remove the extra channels

Once you have verified that all of your workloads are running on a single
channel, you can remove the unused, extra channels. If all three release
channels were provisioned, remember to run the following commands for each
channel.

1. Delete the extra `ControlPlaneRevision` resource:

   ```bash
   kubectl delete controlplanerevision RELEASE_CHANNEL -n istio-system
   ```

   Replace <var translate="no">RELEASE_CHANNEL</var> with `asm-managed-stable`, `asm-managed`, or
   `asm-managed-rapid`.
2. Delete the `MutatingWebhookConfiguration`:

   ```bash
   kubectl delete mutatingwebhookconfiguration istiod-RELEASE_CHANNEL
   ```
3. Delete the `meshconfig` configmap:

   ```bash
   kubectl delete configmap istio-RELEASE_CHANNEL
   ```

### Enable automatic management

> [!CAUTION]
> **Caution:** Enabling automatic control plane management automatically enables multi-cluster and managed data plane on that cluster.

1. Run the following command to enable automatic management:

   ```bash
   gcloud container fleet mesh update \
       --management automatic \
       --memberships MEMBERSHIP_NAME \
       --project PROJECT_ID \
       --location MEMBERSHIP_LOCATION
   ```

   Replace the following:
   - <var translate="no">MEMBERSHIP_NAME</var> is the membership name listed when you verified that your cluster was registered to the fleet.
   - <var translate="no">PROJECT_ID</var> is the project ID of your project.
   - <var translate="no">MEMBERSHIP_LOCATION</var> is the location of your membership (either a region, or `global`). You can check your membership's location with `gcloud container fleet memberships list --project PROJECT_ID`.
2. Verify that automatic management is enabled:

   ```bash
   gcloud container fleet mesh describe
   ```

   The output is similar to:

       ...
       membershipSpecs:
         projects/.../locations/us-central1/memberships/my-member:
           mesh:
             management: MANAGEMENT_AUTOMATIC
       membershipStates:
         projects/.../locations/us-central1/memberships/my-member:
           servicemesh:
             conditions:
             - code: VPCSC_GA_SUPPORTED
               details: This control plane supports VPC-SC GA.
               documentationLink: http://cloud.google.com/service-mesh/docs/managed/vpc-sc
               severity: INFO
             controlPlaneManagement:
               details:
               - code: REVISION_READY
                 details: 'Ready: asm-managed'
               implementation: TRAFFIC_DIRECTOR
               state: ACTIVE
             dataPlaneManagement:
               details:
               - code: OK
                 details: Service is running.
               state: ACTIVE
           state:
             code: OK
             description: |-
               Revision ready for use: asm-managed.
               All Canonical Services have been reconciled successfully.
       ...

## Modernize EnvoyFilter compressor configurations

If you are using the `envoy.extensions.filters.http.compressor.v3.Compressor`
filter, several top-level fields have been deprecated in Envoy and moved to
`response_direction_config.common_config` and
`request_direction_config.common_config`.

To learn how to update your `EnvoyFilter` resources to the modern and supported
format, see the
[Modernize EnvoyFilter compressor configurations](https://docs.cloud.google.com/service-mesh/docs/migrate/modernize-envoyfilter-compressor.md.txt)
guide.

## Enabling a Required Organization Policy

Cloud Service Mesh requires specific policies to be enabled in order to
operate successfully.

### Required Policies

#### compute.disableInternetNetworkEndpointGroup

Cloud Service Mesh requires the ability to create Internet
[Network Endpoint Groups](https://docs.cloud.google.com/service-mesh/docs/onboarding/provision-control-plane.md.txt#negs-in-csm)
(NEGs). These NEGs are essential components for routing traffic to services
outside of Google Cloud.

The `constraints/compute.disableInternetNetworkEndpointGroup` constraint can
prevent the creation of these necessary Internet NEGs.

If this organization policy is enforced on your project, folder, or
organization, it will block Cloud Service Mesh from functioning correctly
with the Traffic Director control plane. This will prevent migration to or
successful operation of this Cloud Service Mesh configuration.

If `Enforced` is set to `true`, the creation of Internet NEGs is disabled.

### How do I know if I am impacted?

You can check the effective policy for your project using the Google Cloud console
or Google Cloud CLI. Instructions for viewing organization policies can be found
in the
[Viewing organization policies](https://cloud.google.com/resource-manager/docs/organization-policy/using-constraints#viewing_organization_policies)
section of the official documentation.

For example, you can use this Google Cloud CLI command:

```bash
gcloud resource-manager org-policies describe constraints/compute.disableInternetNetworkEndpointGroup --project YOUR_PROJECT_ID --effective
```

If the policy shows the `enforced: true` value, then you are impacted.

```bash
constraint: constraints/compute.disableInternetNetworkEndpointGroup
booleanPolicy:
  enforced: true
```

If the policy shows a different value or is empty, then you are not impacted.

```bash
constraint: constraints/compute.disableInternetNetworkEndpointGroup
booleanPolicy: {}
```

### Resolving the Issue

If the policy is enforced, you will need to disable this constraint (set to
`false`) at the organization, folder, or project level.

Detailed instructions on how to modify organization policies, including boolean
constraints like this one, can be found in the
[Creating and managing organization policies](https://cloud.google.com/resource-manager/docs/organization-policy/creating-managing-policies)
guide. You will typically need the `roles/orgpolicy.policyAdmin` role to make
these changes.

## Shared VPC Permissions

When using managed Cloud Service Mesh with a Shared VPC where your
clusters reside in a service project and the VPC network is in a host project,
additional permissions are required for the Cloud Service Mesh service agent
in the fleet project.

As part of the migration to a `TRAFFIC_DIRECTOR` control plane,
Cloud Service Mesh requires permissions to manage firewall rules in the host
project to ensure proper health checking and network functionality. This was not
required for the previous `ISTIOD` control plane.

### How do I know if I am impacted?

You are impacted if:

1. Your GKE clusters are in a service project.
2. These clusters use a Shared VPC network hosted in a **different** project (the VPC host project).
3. The Fleet project Cloud Service Mesh service agent does not have permission to manage firewall rules in the VPC host project.

You may see a warning message with the code `SHARED_VPC_MISSING_PERMISSIONS` in
your fleet feature state. For more details on understanding and resolving this
issue, including both system-managed and user-managed options, see
[Understand health checks](https://docs.cloud.google.com/service-mesh/docs/operate-and-maintain/understand-health-checks.md.txt).

## Missing Control Plane Configuration

Cloud Service Mesh's best practices for managing a service mesh uses the
Google Kubernetes Engine Fleet API to set the Management field to `AUTOMATIC`. This provides
fully automatic management of many mesh features such as webhooks and the
control plane. Additionally, it sets up the cluster to automatically receive new
updates and features as they are released. Some legacy installation methods did
not configure Cloud Service Mesh in this way, and you must take some manual
steps to patch them.

### How do I know if I am impacted?

Check whether you have a `ControlPlaneRevision` in your cluster:

```bash
kubectl get controlplanerevisions -n istio-system
```

If you already have a `ControlPlaneRevision` in your cluster, then this guide
doesn't apply to you. If you don't have a `ControlPlaneRevision`, then use this
guide to modernize your cluster(s).

> [!NOTE]
> **Note:** There is a known issue where the `MISSING_CONTROL_PLANE_CONFIG` warning is reported even though your cluster already has a `ControlPlaneRevision`. This happens if your cluster previously had more than one control plane channel and you deleted the `ControlPlaneRevision` for one of those channels. If the preceding command returns at least one `ControlPlaneRevision`, then your mesh is configured correctly, no action is required, and you can ignore the warning.

If you are receiving the `MISSING_CONTROL_PLANE_CONFIG` warning and don't want
to be using Cloud Service Mesh, then
[disable Cloud Service Mesh](https://docs.cloud.google.com/service-mesh/docs/uninstall.md.txt).

### Modernization Steps

Follow these steps to determine what needs to be done to bring your
configuration into alignment with current standards.

```bash
gcloud container fleet mesh describe --project PROJECT_ID
```

Perform these steps depending on the output:

| Output | Step to perform |
|---|---|
| `API [gkehub.googleapis.com] not enabled on project [my-project]. Would you like to enable and retry (this will take a few minutes)? (y/N)?` | Respond `y` and retry. |
| `ERROR: (gcloud.container.fleet.mesh.describe) Service Mesh Feature for project [my-project] is not enabled` | Proceed to [1. Enable the mesh feature](https://docs.cloud.google.com/service-mesh/docs/migrate/modernization-configuration-updates.md.txt#enable_the_mesh_feature). |
| `... resourceState: state: ACTIVE ...` | Proceed to [2. Register your cluster to Fleet](https://docs.cloud.google.com/service-mesh/docs/migrate/modernization-configuration-updates.md.txt#register_your_cluster_to_fleet). |

#### 1. Enable the mesh feature

Enabling the mesh feature can be done using the following command. This allows
[Cloud Service Mesh](https://docs.cloud.google.com/service-mesh/docs/onboarding/provision-control-plane.md.txt) to run
on your project. For more details and potential side effects, see
[Provision Service Mesh](https://docs.cloud.google.com/service-mesh/docs/onboarding/provision-control-plane.md.txt).

```bash
gcloud container fleet mesh enable --project PROJECT_ID
```

After enabling the mesh feature, proceed to
[2. Register your cluster to Fleet](https://docs.cloud.google.com/service-mesh/docs/migrate/modernization-configuration-updates.md.txt#register_your_cluster_to_fleet).

#### 2. Register your cluster to Fleet

Check if your cluster is already registered to the fleet:

```bash
gcloud container fleet memberships list --project PROJECT_ID
```

If your cluster is not listed, follow these instructions to register it:
[Registering a cluster to the fleet](https://docs.cloud.google.com/kubernetes-engine/fleet-management/docs/register/gke.md.txt#register_an_existing_cluster).

Registering a cluster to your fleet will allow that cluster to manage features,
such as Cloud Service Mesh, through the fleet.

#### 3. Determine your mesh channel

Cloud Service Mesh depends on your cluster's Google Kubernetes Engine channel rather than
an independent mesh channel. In the past, Cloud Service Mesh supported
choosing one or more [**mesh channels**](https://docs.cloud.google.com/service-mesh/legacy/anthos-service-mesh/managed-anthos-service-mesh/select-a-release-channel.md.txt)
(different from the Google Kubernetes Engine cluster channel). You may still have
references to these channels in your mesh configuration.

First determine what channels you are using. If you are using multiple channels,
follow the instructions in [Fix multiple control planes](https://docs.cloud.google.com/service-mesh/docs/migrate/modernization-configuration-updates.md.txt#multiple_control_planes)
to reduce your usage down to a single channel.

To determine what mesh channel you are using, check the labels on your
namespaces. Note you may also need to check workloads if you have added
injection labels specifically on workloads. The following command checks all
workloads in the cluster and filters the output to show the control planes and
injection tags used for those workloads.

```bash
kubectl get namespaces,pods,deployments,statefulsets,daemonsets --all-namespaces --show-labels | grep -E "istio.io/rev|istio-injection"
```

If you manually created any custom tags or webhooks, you need to check those too.

The output is similar to:

    namespace/default                                 Active   153d   istio-injection=enabled,istio.io/rev=asm-managed,kubernetes.io/metadata.name=default
    namespace/payments                                Active   42d    istio.io/rev=asm-managed-stable,kubernetes.io/metadata.name=payments
    default    pod/frontend-847f569874-v9kjz          2/2      Running  0   2m     app=frontend,istio.io/rev=asm-managed,pod-template-hash=847f569874
    payments   pod/checkout-0                         2/2      Running  0   5d     app=checkout,istio.io/rev=asm-managed-stable,controller-revision-hash=5c68b9487
    default    deployment.apps/frontend               1/1      1        1   153d   app=frontend,istio.io/rev=asm-managed
    payments   statefulset.apps/checkout              1/1      1        1   42d    app=checkout,istio.io/rev=asm-managed-stable

Note the values present for `istio.io/rev={channel}`. The names match to channels
in this way. If there are multiple channels in use, then follow the instructions in [Fix multiple control planes](https://docs.cloud.google.com/service-mesh/docs/migrate/modernization-configuration-updates.md.txt#multiple_control_planes) to
reduce down to a single channel:

| Value | Channel |
|---|---|
| `asm-managed-stable` | Stable |
| `asm-managed` | Regular |
| `asm-managed-rapid` | Rapid |

If your mesh channel matches your Google Kubernetes Engine Cluster channel, then skip to step
5.

#### 4. Create a ControlPlaneRevision resource for your mesh channel

If your mesh channel matches your Google Kubernetes Engine Cluster channel, you can skip this
step. Enabling automatic control plane management will automatically create this
`ControlPlaneRevision` resource if your channel matches the cluster channel.

Create a `ControlPlaneRevision` resource for the mesh channel you are using:

### Stable

```yaml
kubectl create ns istio-system
cat <<EOF | kubectl apply -f -
apiVersion: mesh.cloud.google.com/v1beta1
kind: ControlPlaneRevision
metadata:
  name: asm-managed-stable
  namespace: istio-system
spec:
  type: managed_service
  channel: stable
EOF
```

### Regular

```yaml
kubectl create ns istio-system
cat <<EOF | kubectl apply -f -
apiVersion: mesh.cloud.google.com/v1beta1
kind: ControlPlaneRevision
metadata:
  name: asm-managed
  namespace: istio-system
spec:
  type: managed_service
  channel: regular
EOF
```

### Rapid

```yaml
kubectl create ns istio-system
cat <<EOF | kubectl apply -f -
apiVersion: mesh.cloud.google.com/v1beta1
kind: ControlPlaneRevision
metadata:
  name: asm-managed-rapid
  namespace: istio-system
spec:
  type: managed_service
  channel: rapid
EOF
```

#### 5. Enable Automatic Management

> [!CAUTION]
> **Caution:** Enabling automatic control plane management has side effects such as automatically enabling [multi-cluster endpoint discovery](https://docs.cloud.google.com/service-mesh/docs/onboarding/provision-control-plane.md.txt#configure_endpoint_discovery_only_for_multi-cluster_installations) and [managed data plane](https://docs.cloud.google.com/service-mesh/docs/onboarding/provision-control-plane.md.txt#managed-data-plane).

To disable multi-cluster endpoint discovery, follow the "Disable" instructions in
[Endpoint discovery declarative API](https://docs.cloud.google.com/service-mesh/docs/operate-and-maintain/multi-cluster.md.txt#endpoint-discovery-declarative-api)
before enabling automatic control plane management.

To disable the managed data plane, follow the instructions in
[Disable the managed data plane](https://docs.cloud.google.com/service-mesh/docs/operate-and-maintain/managed-data-plane.md.txt#disable_the_managed_data_plane_optional).

To enable [automatic management](https://docs.cloud.google.com/service-mesh/docs/onboarding/provision-control-plane.md.txt#enable_automatic_management),
[run the following command](https://docs.cloud.google.com/sdk/gcloud/reference/container/fleet/mesh/update). This will allow
Fleet-managed Cloud Service Mesh to manage your cluster's control plane
configuration.

```bash
gcloud container fleet mesh update --management automatic --memberships MY_CLUSTER_MEMBERSHIP --project PROJECT_ID
```

Replace <var translate="no">MY_CLUSTER_MEMBERSHIP</var> with your cluster's membership name.

#### 6. Verify Installation

Run the following command:

```bash
gcloud container fleet mesh describe --project PROJECT_ID
```

The output is similar to:

```yaml
createTime: '2025-02-24T22:10:10.826263363Z'
etag: XjjRZB3-ZsueoaKihEBUupYhMDUjvoSyUHTVD2YgR6k
membershipSpecs:
  projects/1234567890/locations/us-central1/memberships/my-cluster-membership:
    mesh:
      management: MANAGEMENT_AUTOMATIC # See item 1 below
membershipStates:
  projects/1234567890/locations/us-central1/memberships/my-cluster-membership:
    servicemesh:
      conditions: [] # See item 2 below
      controlPlaneManagement: # See item 3 below
        details:
        - code: REVISION_READY
          details: 'Ready: asm-managed'
        implementation: TRAFFIC_DIRECTOR
        state: ACTIVE
      dataPlaneManagement: # See item 4 below
        details:
        - code: OK
          details: Service is running.
        state: ACTIVE
    state:
      code: OK
      description: |-
        Revision ready for use: asm-managed.
        All Canonical Services have been reconciled successfully.
      updateTime: '2025-06-17T23:39:05.267540058Z'
name: projects/my-project/locations/global/features/servicemesh
resourceState:
  state: ACTIVE
spec: {}
updateTime: '2025-02-25T04:12:16.280672401Z'
```

1. Within `membershipSpecs`, each membership within your fleet has a separate spec. Ensure that each member correctly has management set to `MANAGEMENT_AUTOMATIC`.
2. Each membership has a list of status conditions associated with it. Ensure there are no errors or warnings here, and follow the documentation links provided to resolve any issues.
3. `controlPlaneManagement` contains the status of your control plane. Ensure that the control plane is in the `ACTIVE` state.
4. `dataPlaneManagement` contains the status of the data plane. It is recommended to have this enabled.