# Provision a managed Cloud Service Mesh control plane on GKE

> [!NOTE]
> **Note:** This guide only supports Cloud Service Mesh with Istio APIs and does not support Google Cloud APIs. For more information see, [Cloud Service Mesh overview](https://docs.cloud.google.com/service-mesh/docs/overview.md.txt).

Cloud Service Mesh is a Google-managed service mesh that you simply enable.
Google handles reliability, upgrades, scaling, and security for you.

This page shows you how to use the
[fleet](https://docs.cloud.google.com/anthos/fleet-management/docs/fleet-concepts) API to set up managed
Cloud Service Mesh using Istio APIs.

## Prerequisites

As a starting point, this guide assumes that you have:

- [A Cloud project](https://docs.cloud.google.com/resource-manager/docs/creating-managing-projects.md.txt)
- [A Cloud Billing account](https://docs.cloud.google.com/billing/docs/how-to/manage-billing-account.md.txt)
- Obtained the [required permissions](https://docs.cloud.google.com/service-mesh/v1.22/docs/onboarding/provision-control-plane.md.txt#roles-required) to provision Cloud Service Mesh
- Enabled [Workload Identity](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/workload-identity.md.txt) on your clusters and node pools.

## Requirements

- One or more clusters with a [supported version of GKE](https://docs.cloud.google.com/service-mesh/docs/managed/supported-features-mcp#environments), in one of the supported [regions](https://docs.cloud.google.com/service-mesh/docs/managed/supported-features-mcp#regions).
- Ensure that your cluster has enough capacity for the required components that managed Cloud Service Mesh installs in the cluster.
  - The `mdp-controller` deployment in `kube-system` namespace requests cpu: 50m, memory: 128Mi.
  - The `istio-cni-node` daemonset in `kube-system` namespace requests cpu: 100m, memory: 100Mi on each node.
- Ensure that the organizational policy `constraints/compute.disableInternetNetworkEndpointGroup` is disabled. If the policy is enabled, ServiceEntry may not work.
- Ensure that the client machine that you provision managed Cloud Service Mesh from has network connectivity to the API server.
- Your clusters must be [registered to a fleet](https://docs.cloud.google.com/anthos/multicluster-management/connect/registering-a-cluster). This is included in the instructions, or can be done separately prior to the provision.
- Your project must have the Service Mesh fleet feature enabled. This is included in the instructions or can be done separately.
- GKE Autopilot is only supported with GKE version
  1.21.3+.

- Cloud Service Mesh can use multiple GKE clusters in a
  single-project single-network environment or a multi-project single-network
  environment.

  - If you join clusters that are not in the same project, they must be registered to the same [fleet host project](https://docs.cloud.google.com/anthos/multicluster-management/fleets#fleet-host-project), and the clusters must be in a [shared VPC](https://docs.cloud.google.com/vpc/docs/shared-vpc.md.txt) configuration together on the same network.
  - For a single-project multi-cluster environment, the fleet project can be the same as the cluster project. For more information about fleets, see [Fleets Overview](https://docs.cloud.google.com/kubernetes-engine/docs/fleets-overview.md.txt).
  - For a multi-project environment, we recommend that you host the fleet in a separate project from the cluster projects. If your organizational policies and existing configuration allow it, we recommend that you use the shared VPC project as the fleet host project. For more information, see [Setting up clusters with Shared VPC](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/cluster-shared-vpc.md.txt).

## Roles required to install Cloud Service Mesh

The following table describes the roles that are required to install managed
Cloud Service Mesh.

| **Role name** | **Role ID** | **Grant location** | **Description** |
|---|---|---|---|
| GKE Hub Admin | roles/gkehub.admin | Fleet project | Full access to GKE Hubs and related resources. |
| Service Usage Admin | roles/serviceusage.serviceUsageAdmin | Fleet project | Ability to enable, disable, and inspect service states, inspect operations, and consume quota and billing for a consumer project. ^[(Note 1)](https://docs.cloud.google.com/service-mesh/v1.22/docs/onboarding/provision-control-plane.md.txt#serviceusageadmin)^ |
| CA Service Admin ^[Beta](https://cloud.google.com/products#product-launch-stages)^ | roles/privateca.admin | Fleet project | Full access to all CA Service resources. ^[(Note 2)](https://docs.cloud.google.com/service-mesh/v1.22/docs/onboarding/provision-control-plane.md.txt#caservice)^ |

## Limitations

> [!CAUTION]
> **Caution:** Note the following fleet-specific limitations: - All clusters in a particular Cloud Service Mesh must be registered to the same fleet. The clusters cannot be registered to different fleets. - All clusters in a project that includes a Cloud Service Mesh cluster must be registered to the same fleet.

We recommend that you review the list of
[Cloud Service Mesh supported features and limitations](https://docs.cloud.google.com/service-mesh/docs/managed/supported-features-mcp).
In particular, note the following:

- The `IstioOperator` API isn't supported since its main purpose is to control
  in-cluster components.

  > [!NOTE]
  > **Note:** You can still enable optional features without using the IstioOperator API, see [Enabling optional features on managed Cloud Service Mesh](https://docs.cloud.google.com/service-mesh/v1.22/docs/managed-service-mesh-optional-features). You can also use the migration tool included with `asmcli` to automatically convert other `IstioOperator` optional features to be compatible with managed control plane. For more information, see [Migrate from `IstioOperator`](https://docs.cloud.google.com/service-mesh/v1.22/docs/migrate-istio-operator).

- Using Certificate Authority Service (CA Service) requires [configuring Cloud Service Mesh per cluster](https://docs.cloud.google.com/service-mesh/v1.22/docs/onboarding/provision-control-plane.md.txt#configure_per_cluster),
  and is not supported when using the [fleet-default configuration](https://docs.cloud.google.com/service-mesh/v1.22/docs/onboarding/provision-control-plane.md.txt#configure_your_fleet)
  in GKE Enterprise.

- For GKE Autopilot clusters, cross-project setup is only
  supported with GKE 1.23 or later.

- For GKE Autopilot clusters, in order to adapt to the
  [GKE Autopilot resource limit](https://docs.cloud.google.com/kubernetes-engine/docs/concepts/autopilot-overview.md.txt#default_contaizner_resource_requests),
  the default proxy resource requests and limits are set to 500m CPU and 512 Mb
  memory. You can override the default values using
  [custom injection](https://docs.cloud.google.com/service-mesh/v1.22/docs/onboarding/provision-control-plane.md.txt#custom-injection).

- During the provisioning process for a managed control plane, Istio CRDs are
  provisioned in the specified cluster. If there are existing Istio CRDs in the
  cluster, they will be overwritten

- [Istio CNI](https://istio.io/latest/docs/setup/additional-setup/cni/)
  and Traffic Director are not compatible with GKE Sandbox. Therefore,
  managed Cloud Service Mesh with the `TRAFFIC_DIRECTOR` implementation does
  not support clusters with GKE Sandbox enabled.

## Before you begin

1. Configure `gcloud` (even if you are using Cloud Shell).
   1. Authenticate with the Google Cloud CLI, where <var translate="no">FLEET_PROJECT_ID</var> is the ID of your Fleet Host project. Generally, the <var translate="no">FLEET_PROJECT_ID</var> is created by default and has the same name as the project.

      <br />

                 gcloud auth login --project FLEET_PROJECT_ID

      <br />

   2. Update the components:<br />

                 gcloud components update

      <br />

2. Enable the required APIs on your fleet host project.<br />

         gcloud services enable mesh.googleapis.com \
             --project=FLEET_PROJECT_ID

   <br />

#### Enabling mesh.googleapis.com enables the following APIs:

> [!NOTE]
> **Note:** Some required APIs have transitive dependencies on other APIs.

| API | Purpose | Can Be Disabled |
|---|---|---|
| `meshconfig.googleapis.com` | Cloud Service Mesh uses the Mesh Configuration API to relay configuration data from your mesh to Google Cloud. Additionally, enabling the Mesh Configuration API allows you to access the Cloud Service Mesh pages in the Google Cloud console and to use the Cloud Service Mesh certificate authority. | No |
| `meshca.googleapis.com` | Related to Cloud Service Mesh certificate authority used by managed Cloud Service Mesh. | No |
| `container.googleapis.com` | Required to create Google Kubernetes Engine (GKE) clusters. | No |
| `gkehub.googleapis.com` | Required to manage the mesh as a [fleet](https://docs.cloud.google.com/anthos/fleet-management/docs). | No |
| `monitoring.googleapis.com` | Required to capture telemetry for mesh workloads. | No |
| `stackdriver.googleapis.com` | Required to use the Services UI. | No |
| `opsconfigmonitoring.googleapis.com` | Required to use the Services UI for off-Google Cloud clusters. | No |
| `connectgateway.googleapis.com` | Required so that the managed Cloud Service Mesh control plane can access mesh workloads. | Yes\* |
| `trafficdirector.googleapis.com` | Enables a highly available and scalable managed control plane. | Yes\* |
| `networkservices.googleapis.com` | Enables a highly available and scalable managed control plane. | Yes\* |
| `networksecurity.googleapis.com` | Enables a highly available and scalable managed control plane. | Yes\* |

> [!WARNING]
> **Warning:** Disabling `connectgateway.googleapis.com`, `trafficdirector.googleapis.com`, `networkservices.googleapis.com`, and/or `networksecurity.googleapis.com` causes managed Cloud Service Mesh control plane to stop working. If the fleet does not use managed Cloud Service Mesh on any cluster, then these APIs can be disabled.

## Configure managed Cloud Service Mesh

3. The steps required to provision managed Cloud Service Mesh using the fleet API depend on whether you prefer to enable by [default for new fleet clusters](https://docs.cloud.google.com/service-mesh/v1.22/docs/onboarding/provision-control-plane.md.txt#configure_your_fleet) or [enable it per cluster](https://docs.cloud.google.com/service-mesh/v1.22/docs/onboarding/provision-control-plane.md.txt#configure_per_cluster).

### Configure for your fleet

4. If you have [enabled
   Google Kubernetes Engine (GKE) Enterprise edition](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/enable-gkee), you can enable managed Cloud Service Mesh as a default configuration for your fleet. This means that every new GKE on Google Cloud cluster [registered
   during cluster
   creation](https://docs.cloud.google.com/anthos/fleet-management/docs/register/gke#register_your_cluster) will have managed Cloud Service Mesh enabled on the cluster. You can find out more about fleet default configuration in [Manage fleet-level
   features](https://docs.cloud.google.com/anthos/fleet-management/docs/manage-features).
5. Enabling managed Cloud Service Mesh as a default configuration for your fleet and registering clusters to the fleet during cluster creation only supports [Mesh CA](https://docs.cloud.google.com/service-mesh/v1.22/docs/security/security-overview.md.txt#features). If you want to use Certificate Authority Service, we recommend that you [enable it per cluster](https://docs.cloud.google.com/service-mesh/v1.22/docs/onboarding/provision-control-plane.md.txt#configure_per_cluster).
6. To enable fleet-level defaults for managed Cloud Service Mesh, complete the following steps:

### Console

1. In the Google Cloud console, go to the **Feature Manager** page.

   [Go to Feature Manager](https://console.cloud.google.com/kubernetes/features/overview)
2. In the **Service Mesh** pane, click **Configure**.

3. Review the settings that are inherited by all new clusters that you
   create in the Google Cloud console and register to the fleet.

   > [!NOTE]
   > **Note:** These settings are GKE settings that are prerequisites for using Cloud Service Mesh.

4. To apply these settings, click **Configure**.

5. In the confirmation dialog, click **Confirm**.

6. Optional: Sync existing clusters to the default settings:

   1. In the **Clusters in the fleet** list, select the clusters that you want to sync. You can only select clusters that have Cloud Service Mesh installed.
   2. Click **Sync to fleet settings** and click **Confirm** in the confirmation dialog that appears. This operation can take a few minutes to complete.

### gcloud

To configure fleet-level defaults using the Google Cloud CLI, you must
establish the following settings:

- Fleet-level settings

  - Create a `mesh.yaml` file that only contains the single line
    `management: automatic`:

    ```bash
    echo "management: automatic" > mesh.yaml
    ```
  - Enable Cloud Service Mesh for your fleet:

    ```bash
    gcloud container fleet mesh enable --project FLEET_PROJECT_ID \
        --fleet-default-member-config mesh.yaml
    ```

    If you see the following error, then you need to
    [enable GKE Enterprise](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/enable-gkee).

        ERROR: (gcloud.container.fleet.mesh.enable) FAILED_PRECONDITION: The
        [anthos.googleapis.com] service is required for this operation and is not
        enabled for the project [PROJECT_NUMBER]. Please use the Google Developers
        Console to enable it.: failed precondition

- Network-level settings

  - If your network's project differs from your fleet host project (for example
    you are using a [shared VPC](https://docs.cloud.google.com/vpc/docs/shared-vpc.md.txt)), you must allow
    Cloud Service Mesh service accounts in the fleet project to access the
    network project. You only need to do this once for the network project.

    Grant service accounts in the fleet project permission to access the
    network project:

    ```bash
    gcloud projects add-iam-policy-binding "NETWORK_PROJECT_ID"  \
        --member "serviceAccount:service-FLEET_PROJECT_NUMBER@gcp-sa-servicemesh.iam.gserviceaccount.com" \
        --role roles/anthosservicemesh.serviceAgent
    ```
- Cluster-level settings

  - When you're ready to create clusters to use with Cloud Service Mesh, create and
    register them in a single step with Google Cloud CLI to use the default
    configuration. For example:

    ```bash
    gcloud container clusters create-auto CLUSTER_NAME \
        --fleet-project FLEET_PROJECT_ID \
        --location=LOCATION
    ```

    You can get the project number for your fleet project by running the
    following command:

    ```bash
    gcloud projects list --filter="FLEET_PROJECT_ID" --format="value(PROJECT_ID)"
    ```

    The `--location` flag is the compute zone or region (such as
    `us-central1-a` or `us-central1`) for the cluster.
  - If your cluster's project differs from your fleet host project, you must
    allow Cloud Service Mesh service accounts in the fleet project to access the
    cluster project, and enable required APIs on the cluster project. You only
    need to do this once for each cluster project.

    Grant service accounts in the fleet project permission to access the
    cluster project:

    ```bash
    gcloud projects add-iam-policy-binding "CLUSTER_PROJECT_ID"  \
        --member "serviceAccount:service-FLEET_PROJECT_NUMBER@gcp-sa-servicemesh.iam.gserviceaccount.com" \
        --role roles/anthosservicemesh.serviceAgent
    ```

    Enable the Mesh API on the cluster's project:

    ```bash
    gcloud services enable mesh.googleapis.com \
      --project=CLUSTER_PROJECT_ID
    ```

    Replace <var translate="no">CLUSTER_PROJECT_ID</var> with the unique identifier of
    your cluster project. If you created your cluster in the same project
    as your fleet, then the <var translate="no">CLUSTER_PROJECT_ID</var> is the same as
    the <var translate="no">FLEET_PROJECT_ID</var>.
7. Proceed to [Verify the control plane has been provisioned](https://docs.cloud.google.com/service-mesh/v1.22/docs/onboarding/provision-control-plane.md.txt#verify_the_control_plane_has_been_provisioned).

### Configure per cluster

8. Use the following steps to configure managed Cloud Service Mesh for each cluster in your mesh individually.

#### Enable the Cloud Service Mesh fleet feature

9. Enable Cloud Service Mesh on the fleet project. Note that if you plan to register multiple clusters, enabling Cloud Service Mesh happens at the fleet-level so you only have to run this command once.

    gcloud container fleet mesh enable --project FLEET_PROJECT_ID

#### Register clusters to a fleet

   1. [Register a GKE cluster](https://docs.cloud.google.com/anthos/fleet-management/docs/register/gke)
      using fleet workload identity. The `--location` flag is the compute zone or
      region (such as `us-central1-a` or `us-central1`) for the cluster.

          gcloud container clusters update CLUSTER_NAME \
            --location CLUSTER_LOCATION \
            --fleet-project FLEET_PROJECT_ID

   2. Verify your cluster is registered:

          gcloud container fleet memberships list --project FLEET_PROJECT_ID

      Example output:

          NAME                 EXTERNAL_ID                           LOCATION
          cluster-1            1d8e255d-2b55-4df9-8793-0435461a2cbc  us-central1

      Make note of the <var translate="no">MEMBERSHIP_NAME</var>, as you will need it when you
      [enable automatic management](https://docs.cloud.google.com/service-mesh/v1.22/docs/onboarding/provision-control-plane.md.txt#enable_automatic_management).
   3. If your cluster's network's project differs from your fleet host project
      (for example you are using a [shared VPC](https://docs.cloud.google.com/vpc/docs/shared-vpc.md.txt)), you must allow
      Cloud Service Mesh service accounts in the fleet project to access the
      network project. You only need to do this once for the network project.

      Grant service accounts in the fleet project permission to access the
      network project:

      ```bash
       gcloud projects add-iam-policy-binding "NETWORK_PROJECT_ID"  \
            --member "serviceAccount:service-FLEET_PROJECT_NUMBER@gcp-sa-servicemesh.iam.gserviceaccount.com" \
            --role roles/anthosservicemesh.serviceAgent
      ```
   4. If your cluster's project differs from your fleet host project, you must allow
      Cloud Service Mesh service accounts in the fleet project to access the cluster
      project, and enable required APIs on the cluster project.

      You only need to do this once for each cluster project. If you previously
      configured managed Cloud Service Mesh for this combination of cluster and
      fleet projects, then these changes have already been applied and you don't
      have to run the following commands.

      Grant service accounts in the fleet project permission to access the cluster
      project:

      ```bash
       gcloud projects add-iam-policy-binding "CLUSTER_PROJECT_ID" \
           --member "serviceAccount:service-FLEET_PROJECT_NUMBER@gcp-sa-servicemesh.iam.gserviceaccount.com" \
           --role roles/anthosservicemesh.serviceAgent
      ```

      Enable the Mesh API on the cluster's project:

      ```bash
       gcloud services enable mesh.googleapis.com \
           --project=CLUSTER_PROJECT_ID
      ```

#### Configure Certificate Authority Service (Optional)

10. If your service mesh deployment requires Certificate Authority Service (CA Service), then follow [Configure Certificate Authority Service for managed Cloud Service Mesh](https://docs.cloud.google.com/service-mesh/v1.22/docs/security/certificate-authority-service.md.txt) to enable it for your fleet. Make sure to complete all steps before proceeding to the next section.

#### Enable automatic management

11. Run the following command to enable automatic management:

> [!NOTE]
> **Note:** You can apply this change to multiple clusters at a time by passing a comma-separated list of membership names as the `--memberships` parameter.

      gcloud container fleet mesh update \
         --management automatic \
         --memberships MEMBERSHIP_NAME \
         --project FLEET_PROJECT_ID \
         --location MEMBERSHIP_LOCATION

12. where:
    - <var translate="no">MEMBERSHIP_NAME</var> is the membership name listed when you verified that your cluster was registered to the fleet.
    - <var translate="no">MEMBERSHIP_LOCATION</var> is the location of your membership (either a
      region, or `global`).

      If you recently created the membership using the command in this guide, this
      should be the region of your cluster. If you have a zonal cluster, use the
      region corresponding to the cluster's zone. For example, if you have a zonal
      cluster in `us-central1-c`, then use the value `us-central1`.

      This value may be `global` if you registered prior to May 2023, or if you
      specified the `global` location when registering the membership. You can
      check your membership's location with
      `gcloud container fleet memberships list --project FLEET_PROJECT_ID`.

## Verify the control plane has been provisioned

13. After a few minutes, verify that the control plane status is `ACTIVE`:

    gcloud container fleet mesh describe --project FLEET_PROJECT_ID

14. The output is similar to:

    ...
    membershipSpecs:
      projects/746296320118/locations/us-central1/memberships/demo-cluster-1:
        mesh:
          management: MANAGEMENT_AUTOMATIC
    membershipStates:
      projects/746296320118/locations/us-central1/memberships/demo-cluster-1:
        servicemesh:
          controlPlaneManagement:
            details:
            - code: REVISION_READY
              details: 'Ready: asm-managed'
            state: ACTIVE
            implementation: ISTIOD | TRAFFIC_DIRECTOR
          dataPlaneManagement:
            details:
            - code: OK
              details: Service is running.
            state: ACTIVE
        state:
          code: OK
          description: 'Revision(s) ready for use: asm-managed.'
    ...

15. Take note of the control plane displayed in the `implementation` field, either `ISTIOD` or `TRAFFIC_DIRECTOR`. See [Cloud Service Mesh supported features](https://docs.cloud.google.com/service-mesh/v1.22/docs/supported-features-managed.md.txt) for control plane differences and supported configurations, and for how the control plane implementation is selected.

## Configure `kubectl` to point to the cluster

16. The following sections involve running `kubectl` commands against each one of your clusters. Before proceeding through the following sections, run the following command for each of your clusters to configure `kubectl` to point to the cluster.

```bash
gcloud container clusters get-credentials CLUSTER_NAME \
      --location CLUSTER_LOCATION \
      --project CLUSTER_PROJECT_ID
```
17. Note that an ingress gateway isn't automatically deployed with the control plane. Decoupling the deployment of the ingress gateway and control plane lets you manage your gateways in a production environment. If you want to use an Istio ingress gateway or an egress gateway, see [Deploy gateways](https://docs.cloud.google.com/service-mesh/v1.22/docs/operate-and-maintain/gateways.md.txt#deploy_gateways). If you want to use the Kubernetes Gateway API, see [Prepare Gateway for Mesh](https://docs.cloud.google.com/service-mesh/v1.22/docs/gateway/prepare-gateway.md.txt). To enable other optional features, see [Enabling optional features on Cloud Service Mesh](https://docs.cloud.google.com/service-mesh/v1.22/docs/managed-service-mesh-optional-features).

## Managed data plane

18. If you use managed Cloud Service Mesh, Google fully manages upgrades of your proxies unless you [disable it](https://docs.cloud.google.com/service-mesh/v1.22/docs/onboarding/provision-control-plane.md.txt#disable_the_managed_data_plane_optional).
19. With the managed data plane, the sidecar proxies and injected gateways are automatically updated in conjunction with the managed control plane by restarting workloads to re-inject new versions of the proxy. This normally completes 1-2 weeks after the managed control plane is upgraded.
20. If disabled, proxy management is driven by the natural lifecycle of the pods in the cluster and must be manually triggered by the user to control the update rate.
21. The managed data plane upgrades proxies by evicting pods that are running earlier versions of the proxy. The evictions are done gradually, honoring the pod disruption budget and controlling the rate of change.

> [!NOTE]
> **Note:** If the cluster has a [GKE maintenance window](https://docs.cloud.google.com/kubernetes-engine/docs/concepts/maintenance-windows-and-exclusions.md.txt) defined, pod evictions will commence at the start of the next available maintenance window and continue without pause until all managed pods have been updated (usually 12 hours). Maintenance window is not observed for CVE related rollouts.

22. The managed data plane doesn't manage the following:
    - Uninjected pods
    - Manually injected pods
    - Jobs
    - StatefulSets
    - DaemonSets

### Disable the managed data plane (optional)

23. If you are provisioning managed Cloud Service Mesh on a new cluster, then you can disable the managed data plane completely, or for individual namespaces or pods. The managed data plane will continue to be disabled for existing clusters where it was disabled by default or manually.
24. To disable the managed data plane at the cluster level and revert back to managing the sidecar proxies yourself, change the annotation:

    kubectl annotate --overwrite controlplanerevision -n istio-system \
      mesh.cloud.google.com/proxy='{"managed":"false"}'

25. To disable the managed data plane for a namespace:

    kubectl annotate --overwrite namespace NAMESPACE \
      mesh.cloud.google.com/proxy='{"managed":"false"}'

26. To disable the managed data plane for a pod:

    kubectl annotate --overwrite pod POD_NAME \
      mesh.cloud.google.com/proxy='{"managed":"false"}'

### Enable maintenance notifications

27. You can request to be notified about upcoming managed data plane maintenance up to week before maintenance is scheduled. Maintenance notifications are not sent by default. You must also [Configure a GKE maintenance window](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/maintenance-windows-and-exclusions.md.txt) before you can receive notifications. When enabled, notifications are sent at least two days before the upgrade operation.
28. To opt in to managed data plane maintenance notifications:
    1. Go to the **Communication** page.

       [Go to the Communication page](https://console.cloud.google.com/user-preferences/communication/)
    2. In the **Cloud Service Mesh Upgrade** row, under the **Email** column, select the
       radio button to turn maintenance notifications **ON**.

29. Each user who wants to receive notifications must opt in separately. If you want to set an email filter for these notifications, the subject line is:
30. `Upcoming upgrade for your Cloud Service Mesh cluster "CLUSTER_LOCATION/CLUSTER_NAME"`.
31. The following example shows a typical managed data plane maintenance notification:
> Subject Line: Upcoming upgrade for your Cloud Service Mesh cluster "`<location/cluster-name>`"
>
> Dear Cloud Service Mesh user,
>
> The Cloud Service Mesh components in your cluster ${instance_id} (https://console.cloud.google.com/kubernetes/clusters/details/${instance_id}/details?project=${project_id}) are scheduled to upgrade on ${scheduled_date_human_readable} at ${scheduled_time_human_readable}.
>
> You can check the release notes (https://cloud.google.com/service-mesh/docs/release-notes) to learn about the new update.
>
> In the event that this maintenance gets canceled, you'll receive another email.
>
> Sincerely,
>
> The Cloud Service Mesh Team
>
> (c) 2023 Google LLC 1600 Amphitheater Parkway, Mountain View, CA 94043
> You have received this announcement to update you about important changes to Google Cloud Platform or your account.
> You can opt out of maintenance window notifications by editing your user preferences: https://console.cloud.google.com/user-preferences/communication?project=${project_id}

## Configure endpoint discovery (only for multi-cluster installations)

32. If your mesh has only one cluster, skip these multi-cluster steps and proceed to [Deploy applications](https://docs.cloud.google.com/service-mesh/v1.22/docs/onboarding/provision-control-plane.md.txt#deploy_applications) or [Migrate applications](https://docs.cloud.google.com/service-mesh/v1.22/docs/onboarding/provision-control-plane.md.txt#migrate_applications_to_managed_anthos_service_mesh).
33. Before you continue, ensure that Cloud Service Mesh is configured on each cluster.

> [!NOTE]
> **Note:** For more information on endpoint discovery, refer to [Endpoint discovery with multiple control planes](https://istio.io/v1.22/docs/ops/deployment/deployment-models/#endpoint-discovery-with-multiple-control-planes).

34. Enabling Cloud Service Mesh with the fleet API will enable endpoint discovery for this cluster. However, you must [open firewall ports](https://docs.cloud.google.com/service-mesh/docs/managed/multi-cluster#create_firewall_rule). To disable endpoint discovery for one or more clusters, see the instructions to disable it in [Endpoint discovery between clusters with declarative API](https://docs.cloud.google.com/service-mesh/docs/managed/multi-cluster#endpoint-discovery-declarative-api).
35. For an example application with two clusters, see [HelloWorld service example](https://docs.cloud.google.com/service-mesh/docs/managed/multi-cluster#install_the_helloworld_service).

> [!NOTE]
> **Note:** The **HelloWorld service** example uses [Docker Hub](https://hub.docker.com/). In a private cluster, the container runtime can pull container images from [Artifact Registry](https://docs.cloud.google.com/artifact-registry/docs.md.txt) by default. The container runtime cannot pull images from any other container image registry on the internet. You can download the image and push to Artifact Registry or use [Cloud NAT](https://docs.cloud.google.com/nat/docs/overview.md.txt) to provide outbound internet access for certain private nodes. For more information, see [Migrate external containers](https://docs.cloud.google.com/container-registry/docs/migrate-external-containers) and [Creating a private cluster](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/private-clusters#pulling_images).

## Deploy applications

If you have more than one cluster in the fleet using managed Cloud Service Mesh, then ensure endpoint discovery or firewall ports are configured as intended before proceeding and deploying applications.
36. Enable the namespace for injection. The steps depend on your [control plane implementation](https://docs.cloud.google.com/service-mesh/v1.22/docs/check-control-plane-implementation.md.txt).

### Managed (TD)

1. Apply the default injection label to the namespace:

```bash
kubectl label namespace NAMESPACE \
    istio.io/rev- istio-injection=enabled --overwrite
```

### Managed (Istiod)

**Recommended:** Run the following command to apply the default injection
label to the namespace:

```bash
  kubectl label namespace NAMESPACE \
      istio.io/rev- istio-injection=enabled --overwrite
```

**If you are an existing user with the Managed Istiod control plane:**
We recommend that you use default injection, but revision-based injection is
supported. Use the following instructions:

1. Run the following command to locate the available release channels:

   ```bash
   kubectl -n istio-system get controlplanerevision
   ```

   The output is similar to the following:

       NAME                AGE
       asm-managed-rapid   6d7h

   NOTE: If two control plane revisions appear in the list above, remove one. Having multiple control plane channels in the cluster is not supported.

   In the output, the value under the `NAME` column is the revision label that corresponds to the available [release channel](https://docs.cloud.google.com/service-mesh/docs/managed/select-a-release-channel#anthos_service_mesh_versions_per_channel) for the Cloud Service Mesh version.
2. Apply the revision label to the namespace:

   ```bash
   kubectl label namespace NAMESPACE \
       istio-injection- istio.io/rev=REVISION_LABEL --overwrite
   ```
37. At this point, you have successfully configured managed Cloud Service Mesh. If you have any existing workloads in labeled namespaces, then restart them so they get proxies injected.
38. If you deploy an application in a multi-cluster setup, replicate the Kubernetes and control plane configuration in all clusters, unless you plan to limit that particular config to a subset of clusters. The configuration applied to a particular cluster is the source of truth for that cluster.

### Customize injection (optional)

39. You can override default values and customize injection settings but this can lead to unforeseen configuration errors and resulting issues with sidecar containers. Before you customize injection, read the information after the sample for notes on particular settings and recommendations.
40. Per-pod configuration is available to override these options on individual pods. This is done by adding an `istio-proxy` container to your pod. The sidecar injection will treat any configuration defined here as an override to the default injection template.
41. For example, the following configuration customizes a variety of settings, including lowering the CPU requests, adding a volume mount, and adding a `preStop` hook:

```yaml
apiVersion: v1
kind: Pod
metadata:
  name: example
spec:
  containers:
  - name: hello
    image: alpine
  - name: istio-proxy
    image: auto
    resources:
      requests:
        cpu: "200m"
        memory: "256Mi"
      limits:
        cpu: "200m"
        memory: "256Mi"
    volumeMounts:
    - mountPath: /etc/certs
      name: certs
    lifecycle:
      preStop:
        exec:
          command: ["sleep", "10"]
  volumes:
  - name: certs
    secret:
      secretName: istio-certs
```
42. In general, any field in a pod can be set. However, care must be taken for certain fields:
    - Kubernetes requires the `image` field to be set before the injection has run. While you can set a specific image to override the default one, we recommend that you set the `image` to `auto`, which will cause the sidecar injector to automatically select the image to use.
    - Some fields in `containers` are dependent on related settings. For example, must be less than or equal to the CPU limit. If both fields are not properly configured, the pod may fail to start.
    - Kubernetes lets you set both `requests` and `limits` for resources in your Pod `spec`. **GKE Autopilot** only considers `requests`. For more information, see [Setting resource limits in Autopilot](https://docs.cloud.google.com/kubernetes-engine/docs/concepts/autopilot-resource-requests.md.txt#resource-limits).
43. Additionally, certain fields are configurable by annotations on the Pod, although it is recommended to use the above approach to customizing settings. Take additional care for the following annotations:
    - For GKE Standard, if `sidecar.istio.io/proxyCPU` is set, make sure to explicitly set `sidecar.istio.io/proxyCPULimit`. Otherwise the sidecar's CPU limit will be set as unlimited.
    - For GKE Standard, if `sidecar.istio.io/proxyMemory` is set, make sure to explicitly set `sidecar.istio.io/proxyMemoryLimit`. Otherwise the sidecar's memory limit will be set as unlimited.
    - For GKE Autopilot, configuring resource `requests` and `limits` using annotations might overprovision resources. Use the image template approach to avoid over provisioning resources. See [Resource modification examples in Autopilot](https://docs.cloud.google.com/kubernetes-engine/docs/concepts/autopilot-resource-requests.md.txt#examples).
44. For example, see the below resources annotation:

```yaml
spec:
  template:
    metadata:
      annotations:
        sidecar.istio.io/proxyCPU: "200m"
        sidecar.istio.io/proxyCPULimit: "200m"
        sidecar.istio.io/proxyMemory: "256Mi"
        sidecar.istio.io/proxyMemoryLimit: "256Mi"
```

## Migrate applications to managed Cloud Service Mesh

> [!WARNING]
> **Warning:** During the migration process, Istio CRDs are installed in the specified cluster. If there are existing Istio CRDs in the cluster, they will be overwritten.

45. To migrate applications from in-cluster Cloud Service Mesh to managed Cloud Service Mesh, perform the following steps:

> [!NOTE]
> **Note:** Ensure you have provisioned managed Cloud Service Mesh by following all of the preceding sections of this guide.

    1. Replace the current namespace label. The steps depend on your [control plane implementation](https://docs.cloud.google.com/service-mesh/v1.22/docs/check-control-plane-implementation.md.txt).

### Managed (TD)

1. Apply the default injection label to the namespace:

```bash
kubectl label namespace NAMESPACE \
    istio.io/rev- istio-injection=enabled --overwrite
```

### Managed (Istiod)

**Recommended:** Run the following command to apply the default injection
label to the namespace:

```bash
  kubectl label namespace NAMESPACE \
      istio.io/rev- istio-injection=enabled --overwrite
```

**If you are an existing user with the Managed Istiod control plane:**
We recommend that you use default injection, but revision-based injection is
supported. Use the following instructions:

1. Run the following command to locate the available release channels:

   ```bash
   kubectl -n istio-system get controlplanerevision
   ```

   The output is similar to the following:

       NAME                AGE
       asm-managed-rapid   6d7h

   NOTE: If two control plane revisions appear in the list above, remove one. Having multiple control plane channels in the cluster is not supported.

   In the output, the value under the `NAME` column is the revision label that corresponds to the available [release channel](https://docs.cloud.google.com/service-mesh/docs/managed/select-a-release-channel#anthos_service_mesh_versions_per_channel) for the Cloud Service Mesh version.
2. Apply the revision label to the namespace:

   ```bash
   kubectl label namespace NAMESPACE \
       istio-injection- istio.io/rev=REVISION_LABEL --overwrite
   ```
    1. Perform a rolling upgrade of deployments in the namespace:

           kubectl rollout restart deployment -n NAMESPACE

    2. Test your application to verify that the workloads function correctly.

    3. If you have workloads in other namespaces, repeat the previous steps for
       each namespace.

    4. If you deployed the application in a multi-cluster setup, replicate the
       Kubernetes and Istio configuration in all clusters, unless there is a
       desire to limit that configuration to a subset of clusters only. The
       configuration applied to a particular cluster is the source of truth for
       that cluster.

46. If you are satisfied that your application works as expected, you can remove the in-cluster `istiod` after you switch all namespaces to the managed control plane, or keep them as a backup - `istiod` will automatically scale down to use fewer resources. To remove, skip to [Delete old control plane](https://docs.cloud.google.com/service-mesh/v1.22/docs/onboarding/provision-control-plane.md.txt#delete-old-control-plane).
47. If you encounter problems, you can identify and resolve them by using the information in [Resolving managed control plane issues](https://docs.cloud.google.com/service-mesh/v1.22/docs/troubleshooting/troubleshoot-managed-service-mesh.md.txt) and if necessary, roll back to the previous version.

### Delete old control plane

After you install and confirm that all namespaces use the Google-managed control
plane, you can delete the old control plane.

    kubectl delete Service,Deployment,HorizontalPodAutoscaler,PodDisruptionBudget istiod -n istio-system --ignore-not-found=true

If you used `istioctl kube-inject` instead of automatic injection, or if
you installed additional gateways, check the metrics for the control plane,
and verify that the number of connected endpoints is zero.

### Roll back

Perform the following steps if you need to roll back to the previous control
plane version:

1. Update workloads to be injected with the previous version of the
   control plane. In the following command, the revision value `asm-191-1` is
   used only as an example. Replace the example value with the
   revision label of your previous control plane.

       kubectl label namespace NAMESPACE istio-injection- istio.io/rev=asm-191-1 --overwrite

2. Restart the Pods to trigger re-injection so the proxies have the
   previous version:

       kubectl rollout restart deployment -n NAMESPACE

The managed control plane will automatically scale to zero and not use any
resource when not in use. The mutating webhooks and provisioning will remain
and do not affect cluster behavior.

The gateway is now set to the `asm-managed` revision. To roll back, re-run
the Cloud Service Mesh install command, which will re-deploy gateway pointing back
to your in-cluster control plane:

    kubectl -n istio-system rollout undo deploy istio-ingressgateway

Expect this output on success:

    deployment.apps/istio-ingressgateway rolled back

## Uninstall Cloud Service Mesh

48. Managed control plane auto-scales to zero when no namespaces are using it. For detailed steps, see [Uninstall Cloud Service Mesh](https://docs.cloud.google.com/service-mesh/v1.22/docs/uninstall.md.txt).