> [!CAUTION]
> You are currently viewing version 1.8 of the Apigee hybrid documentation. **This version is end of life.** You should upgrade to a newer version. For more information, see [Supported versions](https://docs.cloud.google.com/apigee/docs/hybrid/supported-platforms.md.txt#supported-versions).

<button class="button button-with-icon cloud-select-dropdown__button button-primary" aria-haspopup="true">Version 1.8 (unsupported)</button>

- Supported versions:
- [v1.17 (latest)](https://docs.cloud.google.com/apigee/docs/hybrid/v1.17/upgrade.md.txt)
- [v1.16](https://docs.cloud.google.com/apigee/docs/hybrid/v1.16/upgrade.md.txt)
- [v1.15](https://docs.cloud.google.com/apigee/docs/hybrid/v1.15/upgrade.md.txt)
- [v1.14](https://docs.cloud.google.com/apigee/docs/hybrid/v1.14/upgrade.md.txt)
- [List of supported versions](https://docs.cloud.google.com/apigee/docs/hybrid/supported-platforms.md.txt#supported-versions)
- Unsupported versions:
- [v1.13](https://docs.cloud.google.com/apigee/docs/hybrid/v1.13/upgrade.md.txt)
- [v1.12](https://docs.cloud.google.com/apigee/docs/hybrid/v1.12/upgrade.md.txt)
- [v1.11](https://docs.cloud.google.com/apigee/docs/hybrid/v1.11/upgrade.md.txt)
- [v1.10](https://docs.cloud.google.com/apigee/docs/hybrid/v1.10/upgrade.md.txt)
- [v1.9](https://docs.cloud.google.com/apigee/docs/hybrid/v1.9/upgrade.md.txt)
- [v1.8](https://docs.cloud.google.com/apigee/docs/hybrid/v1.8/upgrade.md.txt)
- [v1.7](https://docs.cloud.google.com/apigee/docs/hybrid/v1.7/upgrade.md.txt)
- [v1.6](https://docs.cloud.google.com/apigee/docs/hybrid/v1.6/upgrade.md.txt)
- [v1.5](https://docs.cloud.google.com/apigee/docs/hybrid/v1.5/upgrade.md.txt)
- [v1.4](https://docs.cloud.google.com/apigee/docs/hybrid/v1.4/upgrade.md.txt)
- [v1.3](https://docs.cloud.google.com/apigee/docs/hybrid/v1.3/upgrade.md.txt)
- [v1.2](https://docs.cloud.google.com/apigee/docs/hybrid/v1.2/upgrade.md.txt)
- [v1.1](https://docs.cloud.google.com/apigee/docs/hybrid/v1.1/upgrade.md.txt)

> [!NOTE]
> **Note:** This procedure covers upgrading from Apigee hybrid version 1.7.x or from previous releases of Apigee hybrid version 1.8.x to Apigee hybrid version 1.8.8.
>
>
> Use the same procedures for minor version upgrades (for example version 1.7
> to 1.8) and for patch release upgrades (for example
> 1.8.0 to 1.8.8).
>
>
> If you are upgrading from Apigee hybrid version 1.6 or older, you must first upgrade to
> hybrid version 1.7 before upgrading to version 1.8.8.
> See the instructions for [Upgrading
> Apigee hybrid to version 1.7](https://docs.cloud.google.com/apigee/docs/hybrid/v1.7/upgrade).
>
> If you are already on hybrid v1.8.0 and are looking to migrate to
> Apigee ingress gateway from Anthos Service Mesh, see
> [Migrating to Apigee ingress gateway](https://docs.cloud.google.com/apigee/docs/hybrid/v1.8/migrating-to-apigee-ingress.md.txt).

## Introducing Apigee ingress gateway


Starting in version 1.8, Apigee hybrid offers a new feature to manage the ingress gateway for
your hybrid installation, *Apigee ingress gateway*. Anthos Service Mesh is no longer a
prerequisite for hybrid installation. With Apigee ingress gateway, Apigee will stop supplying routing
configuration to Anthos Service Mesh. After the upgrade, you have to migrate the traffic to the new
Apigee ingress gateway before you can start using the feature.


Apigee uses a small subset of Anthos Service Mesh features for the ingress gateway.
Starting with hybrid version 1.8 Apigee hybrid includes an ingress gateway which is
installed and upgraded as part of Apigee hybrid upgrades. Therefore you do not need
to build expertise around Anthos Service Mesh to install, upgrade, and manage Apigee
hybrid. Issues around ingress gateway versions and compatibility with Apigee hybrid
releases are handled automatically.


Two scenarios for migrating are:

- **Multi-cluster or multi-region migration** (recommended):

  Before switching to a new Ingress for Apigee, drain all the traffic to another cluster or
  region from the cluster you are migrating. This will give you time to test if the new
  Apigee ingress gateway is working as expected. Then shift the traffic back to the upgraded
  cluster.
- **In-place upgrade** (not recommended in production environments):

  During the upgrade Apigee will bring up the new ingress gateway with an IP address you specify.
  You can then test if the new Apigee ingress gateway is working as expected, and then shift
  traffic to the new ingress. There might be downtime during this upgrade.


When upgrading Apigee hybrid to version 1.8, you must configure
Apigee ingress gateway in your overrides file. After upgrading, you control which type of ingress
gateway your clusters will use by directing the A or CNAME records at your registrar to the IP
address for Apigee ingress gateway or for Anthos Service Mesh.

> [!WARNING]
> **Caution:** Because Apigee ingress gateway is a new feature which introduces significant change, Apigee recommends you upgrade and test a non-production installation first, before you upgrade a production installation.

## Upgrading to version 1.8.8 overview

> [!WARNING]
> Upgrading to Apigee hybrid version 1.8 may require downtime.
>
> When upgrading the Apigee controller to version 1.8.8, all Apigee
> deployments undergo a rolling restart. To minimize downtime in production hybrid environments
> during a rolling restart, make sure you are running at least two clusters (in the same or
> different region/data center). Divert all production traffic to a single cluster and take the
> cluster you are about to upgrade offline, and then proceed with the upgrade process. Repeat
> the process for each cluster.
>
> Apigee recommends that you upgrade all clusters as soon as possible to reduce the chances
> of production impact. There is no time limit on when all remaining clusters must be upgraded
> after the first one is upgraded. However, until all remaining clusters are upgraded the
> following operations
> will be impacted:
>
> - Cassandra backup and restore cannot work with mixed versions. For example, a backup from Hybrid 1.7 cannot be used to restore a Hybrid 1.8 instance.
> - Cassandra data streaming will not work between mixed Hybrid versions. Therefore, your Cassandra clusters cannot scale horizontally.
> - Region expansion and decommissioning will be impacted, because these operations depend on Cassandra data streaming.


The procedures for upgrading Apigee hybrid are organized in the following sections:

1. [Prepare to upgrade](https://docs.cloud.google.com/apigee/docs/hybrid/v1.8/upgrade.md.txt#prepare).
2. [Install hybrid runtime version 1.8.8](https://docs.cloud.google.com/apigee/docs/hybrid/v1.8/upgrade.md.txt#install-runtime).
3. For the ingress gateway, pick one of the following options:
   - (Recommended) Use the [new Apigee ingress gateway](https://docs.cloud.google.com/apigee/docs/hybrid/v1.8/upgrade.md.txt#introducing), follow the steps in [Switch traffic from Anthos Service Mesh to Apigee ingress gateway](https://docs.cloud.google.com/apigee/docs/hybrid/v1.8/upgrade.md.txt#switch-to-ingress).
   - Continue using Anthos Service Mesh for your ingress gateway.

## Prerequisite

These upgrade instructions assume you have Apigee hybrid version 1.7.x
or an earlier patch release of version 1.8.x installed and wish to
upgrade it to version 1.8.8. If you are updating from an earlier
version see the instructions for [Upgrading
Apigee hybrid to version 1.7](https://docs.cloud.google.com/apigee/docs/hybrid/v1.7/upgrade).


If you prefer to continue using Anthos Service Mesh, you must ensure that Anthos Service Mesh is upgraded to a supported
version. See the [Supported platforms](https://docs.cloud.google.com/apigee/docs/hybrid/supported-platforms.md.txt#components) table for supported
versions of Anthos Service Mesh.

## Prepare to upgrade to version 1.8

### Back up your hybrid installation (recommended)

1. These instructions use the environment variable <var translate="no">APIGEECTL_HOME</var> for the directory in your file system where you have installed `apigeectl`. If needed, change directory into your `apigeectl` directory and define the variable with the following command:

   ### Linux

       export APIGEECTL_HOME=$PWD
       echo $APIGEECTL_HOME

   ### Mac OS

       export APIGEECTL_HOME=$PWD
       echo $APIGEECTL_HOME

   ### Windows

       set APIGEECTL_HOME=%CD%
       echo %APIGEECTL_HOME%

2. Make a backup copy of your version 1.7 `$APIGEECTL_HOME/` directory. For example:

       tar -czvf $APIGEECTL_HOME/../apigeectl-v1.7-backup.tar.gz $APIGEECTL_HOME

3. Back up your Cassandra database following the instructions in [Cassandra backup and recovery](https://docs.cloud.google.com/apigee/docs/hybrid/v1.8/cassandra-backup-overview.md.txt)

### Add the **Cloud Trace Agent** role to the service account for the Apigee runtime. (Optional)

Optional: If you plan to use [Cloud trace](https://docs.cloud.google.com/trace/docs/overview.md.txt) and you have not
already performed this step on your hybrid v1.7 installation, ensure your service
account for your Apigee runtime services has the **Cloud Trace Agent** Google role.
(`roles/cloudtrace.agent`).


For production environments, this is usually the `apigee-runtime` service account. For
non-production environments, this is usually the `apigee-non-prod` service account.


You can add the role in the
[Cloud console \> IAM
\& Admin \> Service accounts](https://console.cloud.google.com/iam-admin/serviceaccounts) UI or with the following commands:

1. Get the email address for your service account with the following command:

   ### Production

   ```
   gcloud iam service-accounts list --filter "apigee-runtime"
   ```


   If it matches the pattern `apigee-runtime@$ORG_NAME.iam.gserviceaccount.com`,
   you can use that pattern in the next step.

   ### Non-Prod

   ```
   gcloud iam service-accounts list --filter "apigee-non-prod"
   ```


   If it matches the pattern `apigee-non-prod@$ORG_NAME.iam.gserviceaccount.com`,
   you can use that pattern in the next step.
2. Assign the **Cloud Trace Agent** role to the service account:

   ### Production

   ```
   gcloud projects add-iam-policy-binding $PROJECT_ID \
       --member="serviceAccount:apigee-runtime@$PROJECT_ID.iam.gserviceaccount.com" \
       --role="roles/cloudtrace.agent"
   ```

   ### Non-Prod

   ```
   gcloud projects add-iam-policy-binding $PROJECT_ID \
       --member="serviceAccount:apigee-non-prod@$PROJECT_ID.iam.gserviceaccount.com" \
       --role="roles/cloudtrace.agent"
   ```

   ### Example

   ```
   gcloud projects add-iam-policy-binding hybrid-example-project \
       --member="serviceAccount:apigee-runtime@hybrid-example-project.iam.gserviceaccount.com" \
       --role="roles/cloudtrace.agent"
   ```


   Where: <var translate="no">$PROJECT_ID</var> is the name of the Google Cloud project where Apigee hybrid is installed.

   > [!NOTE]
   > **Note:** If your installation uses a custom name for the service account, substitute that name for `apigee-runtime` or `apigee-non-prod` in the command.

### Prepare to install Apigee ingress gateway


To install Apigee ingress gateway as part of the upgrade. You need to add the
following [`ingressGateways`](https://docs.cloud.google.com/apigee/docs/hybrid/v1.8/config-prop-ref.md.txt#ingressgateways) property to your overrides file.

### Syntax

```
ingressGateways:
- name: INGRESS_NAME
  replicaCountMin: REPLICAS_MIN
  replicaCountMax: REPLICAS_MAX
  resources:
    requests:
      cpu: CPU_COUNT_REQ
      memory: MEMORY_REQ
    limits:
      cpu: CPU_COUNT_LIMIT
      memory: MEMORY_LIMIT
  svcAnnotations:  # optional. See Known issue 243599452.
    SVC_ANNOTATIONS_KEY: SVC_ANNOTATIONS_VALUE
  svcLoadBalancerIP: SVC_LOAD_BALANCER_IP # optional
```

### Example

```
ingressGateways:
- name: prod1
  replicaCountMin: 2
  replicaCountMax: 100
  resources:
    requests:
      cpu: 1
      memory: 1Gi
    limits:
      cpu: 2
      memory: 2Gi 
```

- <var translate="no">INGRESS_NAME</var> is the name of the ingress deployment. This can be any name that meets the following requirements:
  - Have a maximum length of 17 characters
  - Contain only lowercase alphanumeric characters, '-' or '.'
  - Start with an alphanumeric character
  - End with an alphanumeric character


  See [`ingressGateways[].name`](https://docs.cloud.google.com/apigee/docs/hybrid/v1.8/config-prop-ref.md.txt#ingressgateways-name)
  in the Configuration property reference
- <var translate="no">REPLICAS_MIN</var> and <var translate="no">REPLICAS_MAX</var> are the minimum and maximum replica counts for Apigee ingress gateway in your installation. For more information and default settings, see [`ingressGateways[].replicaCountMin`](https://docs.cloud.google.com/apigee/docs/hybrid/v1.8/config-prop-ref.md.txt#ingressgateways-replicacountmin) and [`ingressGateways[].replicaCountMax`](https://docs.cloud.google.com/apigee/docs/hybrid/v1.8/config-prop-ref.md.txt#ingressgateways-replicacountmax) in the Configuration property reference.
- <var translate="no">CPU_COUNT_REQ</var> and <var translate="no">MEMORY_REQ</var> are the CPU and memory request for each replica of Apigee ingress gateway in your installation.

  > [!NOTE]
  > Only set these properties if you have previously set them for your Anthos Service Mesh ingress gateway, for example in your `overlay.yaml` file.


  For more information and default settings, see
  [`ingressGateways[].resources.requests.cpu`](https://docs.cloud.google.com/apigee/docs/hybrid/v1.8/config-prop-ref.md.txt#ingressgateways-resources-requests-cpu)
  and
  [`ingressGateways[].resources.requests.memory`](https://docs.cloud.google.com/apigee/docs/hybrid/v1.8/config-prop-ref.md.txt#ingressgateways-resources-requests-memory)
  in the Configuration property reference.
- <var translate="no">CPU_COUNT_LIMIT</var> and <var translate="no">MEMORY_LIMIT</var> The maximum CPU and memory limits for each replica of Apigee ingress gateway in your installation.

  > [!NOTE]
  > Only set these properties if you have previously set them for your Anthos Service Mesh ingress gateway, for example in your `overlay.yaml` file.


  For more information and default settings, see
  [`ingressGateways[].resources.limits.cpu`](https://docs.cloud.google.com/apigee/docs/hybrid/v1.8/config-prop-ref.md.txt#ingressgateways-resources-limits-cpu)
  and
  [`ingressGateways[].resources.limits.memory`](https://docs.cloud.google.com/apigee/docs/hybrid/v1.8/config-prop-ref.md.txt#ingressgateways-resources-limits-memory)
  in the Configuration property reference.
- <var translate="no">SVC_ANNOTATIONS_KEY</var> and <var translate="no">SVC_ANNOTATIONS_VALUE</var> (optional):

  > [!NOTE]
  > **Note:** The `ingressGateways[].svcAnnotations` field in `overrides.yaml` is not working as expected. See [Known issue 243599452](https://docs.cloud.google.com/apigee/docs/release/known-issues.md.txt#243599452)


  This is a key-value pair that provides
  annotations for your default ingress service. Annotations are used by your cloud platform to
  help configure your hybrid installation, for example setting the loadbalancer type to
  either internal or external. For example:

  ```
  ingressGateways:
    svcAnnotations:
      networking.gke.io/load-balancer-type: "Internal"
  ```

  Annotations vary from platform to platform. Refer to your platform
  documentation for required and suggested annotations.

  > [!NOTE]
  > **Note:** You do not need to set `Annotations` if you are creating your own Kubernetes service for ingress deployment as documented in [Expose Apigee ingress gateway](https://docs.cloud.google.com/apigee/docs/hybrid/v1.8/install-expose-apigee-ingress.md.txt).

  See [`ingressGateways[].svcAnnotations`](https://docs.cloud.google.com/apigee/docs/hybrid/v1.8/config-prop-ref.md.txt#ingressgateways-svcannotations) in the Configuration property reference.
- <var translate="no">SVC_LOAD_BALANCER_IP</var> (optional) Allows you to assign a static IP address for your load balancer. On platforms that support specifying the load balancer IP address, the load balancer will be created with this IP address. On platforms that do not allow you to specify the load balancer IP address, this property is ignored.

  If you do not have a static IP address allocated for your load balancer, leave this property
  out of your overrides file.

  > [!CAUTION]
  > **Caution:** Make sure not to specify the same IP address as the current istio-ingressgateway. This could cause problems while that service is still on the cluster.

  > [!NOTE]
  > **Note:** You do not need to set `LoadBalancerIP` if you are creating your own Kubernetes service for ingress deployment as documented in [Expose Apigee ingress gateway](https://docs.cloud.google.com/apigee/docs/hybrid/v1.8/install-expose-apigee-ingress.md.txt).

  See [`ingressGateways[].svcLoadBalancerIP`](https://docs.cloud.google.com/apigee/docs/hybrid/v1.8/config-prop-ref.md.txt#ingressgateways-svcloadbalancerip) in the Configuration property reference.

### Make additional changes to your overrides file to enable or disable optional v1.8 features


Add the following properties to your `overrides.yaml` file to enable new features in
hybrid v1.8. These features are optional.

- Org-scoped UDCA is now on by default. Using a single UDCA deployment to handle traffic for all environments prevents under-utilization of UDCA pods and increases availability of node resources for other Apigee components. Org-scoped UDCA uses a single service account for all environments, `apigee-udca`.


  If you are using different service accounts for UDCA in different environments,
  be aware that it will now use the service account specified at the org level in your overrides
  file with `udca:serviceAccountPath`, instead of the ones specified at the env level
  with `envs:udca:serviceAccountPath`.


  Apigee hybrid v 1.8 supports environment-scoped UDCA. To keep per-environment UDCA, set
  `orgScopedUDCA: false`.

  > [!WARNING]
  > **Note**: Support for environment-scoped UDCA will not be available in future releases of Apigee hybrid.


  See [`orgScopedUDCA`](https://docs.cloud.google.com/apigee/docs/hybrid/v1.8/config-prop-ref.md.txt#orgscopedudca) in the Configuration
  properties reference.
- Enable `validateOrg` to require strict validation that the Apigee organization and environment are active and work with the Google Cloud Platform project specified in your `overrides` file.

  ```
  validateOrg: true
  ```

  > [!NOTE]
  > **Note:** Apigee hybrid supports not using strict org validation. To turn the feature off, set `validateOrg: false`.


  See [`validateOrg` in the Configuration
  properties reference](https://docs.cloud.google.com/apigee/docs/hybrid/v1.8/config-prop-ref#validateorg).

## Install the hybrid 1.8.8 runtime

> [!CAUTION]
> **Caution:**Do not create new environments during the upgrade process.

1. Be sure you are in the hybrid base directory (the parent of the directory where the `apigeectl` executable file is located):

   ```
   cd $APIGEECTL_HOME/..
   ```
2. Download the release package for your operating system using the following command.
   Be sure to select your platform in the following table:

   ### Linux

   **Linux 64 bit:**

   ```
   curl -LO \
     https://storage.googleapis.com/apigee-release/hybrid/apigee-hybrid-setup/1.8.8/apigeectl_linux_64.tar.gz
   ```

   ### Mac OS

   **Mac 64 bit:**

   ```
   curl -LO \
     https://storage.googleapis.com/apigee-release/hybrid/apigee-hybrid-setup/1.8.8/apigeectl_mac_64.tar.gz
   ```

   ### Windows

   **Windows 64 bit:**

   ```
   curl -LO ^
     https://storage.googleapis.com/apigee-release/hybrid/apigee-hybrid-setup/1.8.8/apigeectl_windows_64.zip
   ```
3. Rename your current `apigeectl/` directory to a backup directory name. For example:

   ### Linux

   ```
   mv $APIGEECTL_HOME/ $APIGEECTL_HOME-v1.7/
   ```

   ### Mac OS

   ```
   mv $APIGEECTL_HOME/ $APIGEECTL_HOME-v1.7/ 
   ```

   ### Windows

   ```
   rename %APIGEECTL_HOME% %APIGEECTL_HOME%-v1.7 
   ```
4. Extract the downloaded gzip file contents into your hybrid base directory.
   The hybrid base directory is the directory where the renamed `apigeectl-v1.7` directory is located:

   ### Linux

   ```
   tar xvzf filename.tar.gz -C ./
   ```

   ### Mac OS

   ```
   tar xvzf filename.tar.gz -C ./
   ```

   ### Windows

   ```
   tar xvzf filename.zip -C ./
   ```
5. The tar contents are, by default, expanded into a directory with the version and platform in
   its name. For example: `./apigeectl_1.8.8-xxxxxxx_linux_64`. Rename that directory
   to `apigeectl` using the following command:

   ### Linux

   ```
   mv apigeectl_1.8.8-xxxxxxx_linux_64 apigeectl
   ```

   ### Mac OS

   ```
   mv apigeectl_1.8.8-xxxxxxx_mac_64 apigeectl
   ```

   ### Windows

   ```
   rename apigeectl_1.8.8-xxxxxxx_windows_64 apigeectl
   ```
6. Change to the `apigeectl` directory:

   ```
   cd ./apigeectl
   ```


   This directory is the `apigeectl` home directory. It is where
   the `apigeectl` executable command is located.
7. These instructions use the environment variable `$APIGEECTL_HOME` for the directory in your file system where the `apigeectl` utility is installed. If needed, change directory into your `apigeectl` directory and define the variable with the following command:

   ### Linux

   ```
   export APIGEECTL_HOME=$PWD
   ```

   ```
   echo $APIGEECTL_HOME
   ```

   ### Mac OS

   ```
   export APIGEECTL_HOME=$PWD
   ```

   ```
   echo $APIGEECTL_HOME
   ```

   ### Windows

   ```
   set APIGEECTL_HOME=%CD%
   ```

   ```
   echo %APIGEECTL_HOME%
   ```
8. Verify the version of `apigeectl` with the `version` command:

   ```
   ./apigeectl version
   ```

   ```
   Version: 1.8.8
   ```
9. Move to the `hybrid-base-directory/hybrid-files` directory. The `hybrid-files` directory is where configuration files such as the overrides file, certs, and service accounts are located. For example:

   ```
   cd $APIGEECTL_HOME/../hybrid-files
   ```
10. Verify that `kubectl` is set to the correct context using the following command. The current context should be set to the cluster in which you are upgrading Apigee hybrid.

    ```
    kubectl config get-contexts | grep \*
    ```
11. In the `hybrid-files` directory:
    1. Update the following symbolic links to `$APIGEECTL_HOME`. These links allow you to run the newly installed `apigeectl` command from inside the `hybrid-files` directory:

           ln -nfs $APIGEECTL_HOME/tools tools
           ln -nfs `$APIGEECTL_HOME`/config config
           ln -nfs `$APIGEECTL_HOME`/templates templates
           ln -nfs `$APIGEECTL_HOME`/plugins plugins

    2. To check that the symlinks were created correctly, execute the following command and make sure the link paths point to the correct locations:

       ```
       ls -l | grep ^l
       ```
12. Do a dry run initialization to check for errors:

    ```
    ${APIGEECTL_HOME}/apigeectl init -f OVERRIDES_FILE --dry-run=client
    ```


    Where <var translate="no">OVERRIDES_FILE</var> is the name of your overrides file, for example `./overrides/overrides.yaml`.

    > [!TIP]
    > **Tip:** You can replace `OVERRIDES_FILE` in the code sample above with the name and path to your overrides file, and every instance on this page will be replaced.

13. If there are no errors, initialize hybrid 1.8.8. This command also installs and configures Apigee ingress gateway:

    ```
    $APIGEECTL_HOME/apigeectl init -f OVERRIDES_FILE
    ```
14. Check the initialization status:

    ```
    $APIGEECTL_HOME/apigeectl check-ready -f OVERRIDES_FILE
    ```


    On success, the output says: `All containers ready.`

    > [!WARNING]
    > **Known issue:** After you run `init` and confirm that `check-ready` succeeds, you may notice, if you view the pods, that the Cassandra schema validation is in an error state.**This state is considered a [known issue (#245664917)](https://docs.cloud.google.com/apigee/docs/release/known-issues.md.txt)
    > and is harmless** . You can safely ignore the error and go on to the next step: `apigeectl apply`.

    As a further check, you can also run this command to check ApigeeDataStore status:

    ```
    kubectl describe apigeeds -n apigee
    ```


    In the output, look for `State: running`.
15. Check for errors with a dry run of the `apply` command using the `--dry-run` flag:

    ```
    $APIGEECTL_HOME/apigeectl apply -f OVERRIDES_FILE --dry-run=client
    ```
16. If there are no errors, apply your overrides. Select and follow the instructions for production environments or non-prod environments, depending on your installation.

    > [!TIP]
    > During the upgrade process each component will perform a rolling restart. Therefore, for production environments, it is best to apply the upgrade to one component at a time.

    ### Production


    For production environments you should upgrade each hybrid component individually, and
    check the status of the upgraded component before proceeding to the next component.
    1. Be sure you are in the `hybrid-files` directory.
    2. Apply your overrides to upgrade Cassandra:

       ```
       $APIGEECTL_HOME/apigeectl apply -f OVERRIDES_FILE --datastore
       ```
    3. Check completion:

       ```
       $APIGEECTL_HOME/apigeectl check-ready -f OVERRIDES_FILE
       ```

       > [!TIP]
       > **Tip:** If `check-ready` fails, you can get more information about your pods with:
       >
       > ```
       > kubectl -n NAMESPACE get pods
       > ```
       >
       >
       > Where <var translate="no">NAMESPACE</var> is your Apigee hybrid namespace.


       Proceed to the next step only when the pods are ready.
    4. Apply your overrides to upgrade Telemetry components and check completion:

       ```
       $APIGEECTL_HOME/apigeectl apply -f OVERRIDES_FILE --telemetry
       ```

       ```
       $APIGEECTL_HOME/apigeectl check-ready -f OVERRIDES_FILE
       ```
    5. Bring up Redis components:

       ```
       $APIGEECTL_HOME/apigeectl apply -f OVERRIDES_FILE --redis
       ```
    6. Apply your overrides to upgrade the org-level components (MART, Watcher and Apigee Connect) and check completion:

       ```
       $APIGEECTL_HOME/apigeectl apply -f OVERRIDES_FILE --org
       ```

       ```
       $APIGEECTL_HOME/apigeectl check-ready -f OVERRIDES_FILE
       ```
    7. Apply your overrides to upgrade your environments. You have two choices:
       - **Environment by environment** : Apply your overrides to one environment at a time and check completion. Repeat this step for each environment:

         ```
         $APIGEECTL_HOME/apigeectl apply -f OVERRIDES_FILE --env ENV_NAME
         ```

         ```
         $APIGEECTL_HOME/apigeectl check-ready -f OVERRIDES_FILE
         ```


         Where <var translate="no">ENV_NAME</var> is the name of the environment you are upgrading.
       - **All environments at one time** : Apply your overrides to all environments at once and check completion:

         ```
         $APIGEECTL_HOME/apigeectl apply -f OVERRIDES_FILE --all-envs
         ```

         ```
         $APIGEECTL_HOME/apigeectl check-ready -f OVERRIDES_FILE
         ```
    8. Apply your overrides to upgrade the `virtualhosts` components and check completion:

       ```
       $APIGEECTL_HOME/apigeectl apply -f OVERRIDES_FILE --settings virtualhosts
       ```

       ```
       $APIGEECTL_HOME/apigeectl check-ready -f OVERRIDES_FILE
       ```

    ### Non-prod


    In most non-production, demo, or experimental environments, you can apply the overrides
    to all components at once. If your non-production environment is large and complex or
    closely mimics a production environment, you may want to use the instructions for
    upgrading production environments.
    1. Be sure you are in the `hybrid-files` directory.
    2.

       ```
       $APIGEECTL_HOME/apigeectl apply -f OVERRIDES_FILE
       ```
    3. Check the status:

       ```
       $APIGEECTL_HOME/apigeectl check-ready -f OVERRIDES_FILE
       ```

## Upgrade your Kubernetes version


Upgrade your Kubernetes platform to the versions supported by hybrid 1.8.
Follow your platform's documentation if you need help.

### *Click to expand a list of supported platforms*

|   | Apigee hybrid versions |||||
| ### Platforms | 1.6[![not supported](https://docs.cloud.google.com/static/apigee/docs/images/not-supported.svg "EOL - not supported")](https://docs.cloud.google.com/apigee/docs/hybrid/v1.8/upgrade.md.txt#note4)[](https://docs.cloud.google.com/apigee/docs/hybrid/v1.8/upgrade.md.txt#note3) | 1.7 ![not supported](https://docs.cloud.google.com/static/apigee/docs/images/not-supported.svg "EOL - not supported") [](https://docs.cloud.google.com/apigee/docs/hybrid/v1.8/upgrade.md.txt#note3) | 1.8 ![not supported](https://docs.cloud.google.com/static/apigee/docs/images/not-supported.svg "EOL - not supported") [](https://docs.cloud.google.com/apigee/docs/hybrid/v1.8/upgrade.md.txt#note3) | 1.9 | 1.10 |
|---|---|---|---|---|---|
| Anthos (Google Cloud - GKE) | 1.19.x 1.20.x 1.21.x | 1.20.x 1.21.x 1.22.x [](https://docs.cloud.google.com/apigee/docs/hybrid/v1.8/upgrade.md.txt#note2) 1.23.x [](https://docs.cloud.google.com/apigee/docs/hybrid/v1.8/upgrade.md.txt#note2) | 1.21.x [](https://docs.cloud.google.com/apigee/docs/hybrid/v1.8/upgrade.md.txt#note8) 1.22.x [](https://docs.cloud.google.com/apigee/docs/hybrid/v1.8/upgrade.md.txt#note8) 1.23.x [](https://docs.cloud.google.com/apigee/docs/hybrid/v1.8/upgrade.md.txt#note10) 1.24.x [](https://docs.cloud.google.com/apigee/docs/hybrid/v1.8/upgrade.md.txt#note6) 1.25.x [](https://docs.cloud.google.com/apigee/docs/hybrid/v1.8/upgrade.md.txt#note6) | 1.23.x 1.24.x 1.25.x 1.26.x [](https://docs.cloud.google.com/apigee/docs/hybrid/v1.8/upgrade.md.txt#note12) | 1.24.x 1.25.x 1.26.x |
| Anthos (AWS) | 1.7.x 1.8.x 1.9.3+ 1.10.x | 1.9.x 1.10.x 1.12.x [](https://docs.cloud.google.com/apigee/docs/hybrid/v1.8/upgrade.md.txt#note2) | 1.10.x 1.11.x 1.12.x 1.13.x 1.25[](https://docs.cloud.google.com/apigee/docs/hybrid/v1.8/upgrade.md.txt#note13) | 1.12.x 1.13.x 1.25[](https://docs.cloud.google.com/apigee/docs/hybrid/v1.8/upgrade.md.txt#note13) | 1.13.x 1.25[](https://docs.cloud.google.com/apigee/docs/hybrid/v1.8/upgrade.md.txt#note13) 1.26[](https://docs.cloud.google.com/apigee/docs/hybrid/v1.8/upgrade.md.txt#note13) |
| Anthos (Azure) | 1.8.x | 1.9.x 1.10.x 1.12.x [](https://docs.cloud.google.com/apigee/docs/hybrid/v1.8/upgrade.md.txt#note2) | 1.10.x 1.11.x 1.12.x 1.13.x 1.14.x | 1.12.x 1.13.x 1.14.x | 1.13.x 1.14.x 1.15.x |
| Anthos (on-premises - VMware)[^(1)^](https://docs.cloud.google.com/apigee/docs/hybrid/v1.8/upgrade.md.txt#note1) | 1.7.x 1.8.x 1.9.3+ 1.10.x | 1.9.x ![not supported](https://docs.cloud.google.com/static/apigee/docs/images/not-supported.svg "EOL - not supported")[^(4)^](https://docs.cloud.google.com/apigee/docs/hybrid/v1.8/upgrade.md.txt#note4) 1.10.x ![not supported](https://docs.cloud.google.com/static/apigee/docs/images/not-supported.svg "EOL - not supported")[^(4)^](https://docs.cloud.google.com/apigee/docs/hybrid/v1.8/upgrade.md.txt#note4) 1.11.x ![not supported](https://docs.cloud.google.com/static/apigee/docs/images/not-supported.svg "EOL - not supported")[^(4)^](https://docs.cloud.google.com/apigee/docs/hybrid/v1.8/upgrade.md.txt#note4)[](https://docs.cloud.google.com/apigee/docs/hybrid/v1.8/upgrade.md.txt#note2) 1.12.x ![not supported](https://docs.cloud.google.com/static/apigee/docs/images/not-supported.svg "EOL - not supported")[^(4)^](https://docs.cloud.google.com/apigee/docs/hybrid/v1.8/upgrade.md.txt#note4) [](https://docs.cloud.google.com/apigee/docs/hybrid/v1.8/upgrade.md.txt#note2) | 1.10.x ![not supported](https://docs.cloud.google.com/static/apigee/docs/images/not-supported.svg "EOL - not supported")[^(4)^](https://docs.cloud.google.com/apigee/docs/hybrid/v1.8/upgrade.md.txt#note4) 1.11.x ![not supported](https://docs.cloud.google.com/static/apigee/docs/images/not-supported.svg "EOL - not supported")[^(4)^](https://docs.cloud.google.com/apigee/docs/hybrid/v1.8/upgrade.md.txt#note4) 1.12.x ![not supported](https://docs.cloud.google.com/static/apigee/docs/images/not-supported.svg "EOL - not supported")[^(4)^](https://docs.cloud.google.com/apigee/docs/hybrid/v1.8/upgrade.md.txt#note4)[^(5)^](https://docs.cloud.google.com/apigee/docs/hybrid/v1.8/upgrade.md.txt#note5) 1.13.x [^(5)^](https://docs.cloud.google.com/apigee/docs/hybrid/v1.8/upgrade.md.txt#note5) 1.14.x [^(5)^](https://docs.cloud.google.com/apigee/docs/hybrid/v1.8/upgrade.md.txt#note5) 1.15.x | 1.12.x ![not supported](https://docs.cloud.google.com/static/apigee/docs/images/not-supported.svg "EOL - not supported")[^(4)^](https://docs.cloud.google.com/apigee/docs/hybrid/v1.8/upgrade.md.txt#note4) 1.13.x 1.14.x 1.15.x | 1.13.x 1.14.x 1.15.x 1.16.x |
| Anthos (Bare Metal)[^(1)^](https://docs.cloud.google.com/apigee/docs/hybrid/v1.8/upgrade.md.txt#note1) | 1.7.x 1.8.2+ 1.9.3+ 1.10.x | 1.9.x ![not supported](https://docs.cloud.google.com/static/apigee/docs/images/not-supported.svg "EOL - not supported")[^(4)^](https://docs.cloud.google.com/apigee/docs/hybrid/v1.8/upgrade.md.txt#note4) 1.10.x ![not supported](https://docs.cloud.google.com/static/apigee/docs/images/not-supported.svg "EOL - not supported")[^(4)^](https://docs.cloud.google.com/apigee/docs/hybrid/v1.8/upgrade.md.txt#note4) 1.11.x ![not supported](https://docs.cloud.google.com/static/apigee/docs/images/not-supported.svg "EOL - not supported")[^(4)^](https://docs.cloud.google.com/apigee/docs/hybrid/v1.8/upgrade.md.txt#note4) [](https://docs.cloud.google.com/apigee/docs/hybrid/v1.8/upgrade.md.txt#note2) 1.12.x ![not supported](https://docs.cloud.google.com/static/apigee/docs/images/not-supported.svg "EOL - not supported")[^(4)^](https://docs.cloud.google.com/apigee/docs/hybrid/v1.8/upgrade.md.txt#note4) [](https://docs.cloud.google.com/apigee/docs/hybrid/v1.8/upgrade.md.txt#note2) | 1.10.x ![not supported](https://docs.cloud.google.com/static/apigee/docs/images/not-supported.svg "EOL - not supported")[^(4)^](https://docs.cloud.google.com/apigee/docs/hybrid/v1.8/upgrade.md.txt#note4) 1.11.x ![not supported](https://docs.cloud.google.com/static/apigee/docs/images/not-supported.svg "EOL - not supported")[^(4)^](https://docs.cloud.google.com/apigee/docs/hybrid/v1.8/upgrade.md.txt#note4) 1.12.x ![not supported](https://docs.cloud.google.com/static/apigee/docs/images/not-supported.svg "EOL - not supported")[^(4)^](https://docs.cloud.google.com/apigee/docs/hybrid/v1.8/upgrade.md.txt#note4) [^(5)^](https://docs.cloud.google.com/apigee/docs/hybrid/v1.8/upgrade.md.txt#note5) 1.13.x [^(5)^](https://docs.cloud.google.com/apigee/docs/hybrid/v1.8/upgrade.md.txt#note5) 1.14.x [^(5)^](https://docs.cloud.google.com/apigee/docs/hybrid/v1.8/upgrade.md.txt#note5) 1.15.x | 1.12.x ![not supported](https://docs.cloud.google.com/static/apigee/docs/images/not-supported.svg "EOL - not supported")[^(4)^](https://docs.cloud.google.com/apigee/docs/hybrid/v1.8/upgrade.md.txt#note4) 1.13.x 1.14.x 1.15.x | 1.13.x 1.14.x 1.15.x 1.16.x |
| EKS[^(7)^](https://docs.cloud.google.com/apigee/docs/hybrid/v1.8/upgrade.md.txt#note7) | 1.19.x 1.20.x 1.21.x | 1.21.x 1.22.x [](https://docs.cloud.google.com/apigee/docs/hybrid/v1.8/upgrade.md.txt#note2) 1.23.x [](https://docs.cloud.google.com/apigee/docs/hybrid/v1.8/upgrade.md.txt#note2) | 1.22.x [](https://docs.cloud.google.com/apigee/docs/hybrid/v1.8/upgrade.md.txt#note8) 1.23.x [](https://docs.cloud.google.com/apigee/docs/hybrid/v1.8/upgrade.md.txt#note10) 1.24.x [](https://docs.cloud.google.com/apigee/docs/hybrid/v1.8/upgrade.md.txt#note6) 1.25.x [](https://docs.cloud.google.com/apigee/docs/hybrid/v1.8/upgrade.md.txt#note6) | 1.23.x 1.24.x 1.25.x 1.26.x [](https://docs.cloud.google.com/apigee/docs/hybrid/v1.8/upgrade.md.txt#note12) | 1.24.x 1.25.x 1.26.x |
| AKS[^(7)^](https://docs.cloud.google.com/apigee/docs/hybrid/v1.8/upgrade.md.txt#note7) | 1.19.x 1.20.x 1.21.x | 1.21.x 1.22.x [](https://docs.cloud.google.com/apigee/docs/hybrid/v1.8/upgrade.md.txt#note2) 1.23.x [](https://docs.cloud.google.com/apigee/docs/hybrid/v1.8/upgrade.md.txt#note2) | 1.22.x [](https://docs.cloud.google.com/apigee/docs/hybrid/v1.8/upgrade.md.txt#note8) 1.23.x [](https://docs.cloud.google.com/apigee/docs/hybrid/v1.8/upgrade.md.txt#note10) 1.24.x [](https://docs.cloud.google.com/apigee/docs/hybrid/v1.8/upgrade.md.txt#note6) 1.25.x [](https://docs.cloud.google.com/apigee/docs/hybrid/v1.8/upgrade.md.txt#note6) | 1.23.x 1.24.x 1.25.x 1.26.x [](https://docs.cloud.google.com/apigee/docs/hybrid/v1.8/upgrade.md.txt#note12) | 1.24.x 1.25.x 1.26.x |
| OpenShift[^(7)^](https://docs.cloud.google.com/apigee/docs/hybrid/v1.8/upgrade.md.txt#note7) | 4.6 4.7 4.8 | 4.7 4.8 | 4.8 4.9 4.10 | 4.10 4.11 | 4.11 4.12 |
| Rancher Kubernetes Engine (RKE)[^(7)^](https://docs.cloud.google.com/apigee/docs/hybrid/v1.8/upgrade.md.txt#note7) | N/A | N/A | N/A | 1.3.8 | v1.26.2+rke2r1 |
| Anthos Service Mesh (ASM) | 1.9.x ![not supported](https://docs.cloud.google.com/static/apigee/docs/images/not-supported.svg "EOL - not supported") 1.10.x ![not supported](https://docs.cloud.google.com/static/apigee/docs/images/not-supported.svg "EOL - not supported") 1.12.x ![not supported](https://docs.cloud.google.com/static/apigee/docs/images/not-supported.svg "EOL - not supported") 1.13.x ![not supported](https://docs.cloud.google.com/static/apigee/docs/images/not-supported.svg "EOL - not supported") | 1.10.x ![not supported](https://docs.cloud.google.com/static/apigee/docs/images/not-supported.svg "EOL - not supported") 1.11.x ![not supported](https://docs.cloud.google.com/static/apigee/docs/images/not-supported.svg "EOL - not supported") 1.12.x ![not supported](https://docs.cloud.google.com/static/apigee/docs/images/not-supported.svg "EOL - not supported") 1.13.x ![not supported](https://docs.cloud.google.com/static/apigee/docs/images/not-supported.svg "EOL - not supported") [](https://docs.cloud.google.com/apigee/docs/hybrid/v1.8/upgrade.md.txt#note2) 1.14.x ![not supported](https://docs.cloud.google.com/static/apigee/docs/images/not-supported.svg "Skipped - never supported") 1.15.x [](https://docs.cloud.google.com/apigee/docs/hybrid/v1.8/upgrade.md.txt#note9) | 1.11.x ![not supported](https://docs.cloud.google.com/static/apigee/docs/images/not-supported.svg "EOL - not supported") 1.12.x ![not supported](https://docs.cloud.google.com/static/apigee/docs/images/not-supported.svg "EOL - not supported") 1.13.x ![not supported](https://docs.cloud.google.com/static/apigee/docs/images/not-supported.svg "EOL - not supported") 1.14.x ![not supported](https://docs.cloud.google.com/static/apigee/docs/images/not-supported.svg "Skipped - never supported") 1.15.x | 1.17.x[](https://docs.cloud.google.com/apigee/docs/hybrid/v1.8/upgrade.md.txt#note11) | 1.17.x[](https://docs.cloud.google.com/apigee/docs/hybrid/v1.8/upgrade.md.txt#note11) |
| JDK | JDK 11 | JDK 11 | JDK 11 | JDK 11 | JDK 11 |
| cert-manager | 1.5.4 | 1.7.x | 1.7.x 1.8.x 1.9.x 1.10.x 1.11.x | 1.10.x 1.11.x | 1.10.x 1.11.x 1.12.x |
| Cassandra | 3.11.10 | 3.11.10 | 3.11.10 | 3.11.10 | 3.11.10 |
| **^(1)^ On Anthos versions 1.8, 1.9, and 1.10, follow the instructions in these documents to avoid conflict with `cert-manager`:** - **Anthos 1.8.2+** : [Conflict with `cert-manager` when upgrading to version 1.8.2 or above](https://docs.cloud.google.com/anthos/clusters/docs/on-prem/1.8/known-issues.md.txt#conflict_with_cert-manager_when_upgrading_to_version_182_or_above). - **Anthos 1.9** : [Conflict with `cert-manager` when upgrading to version 1.9.0 or 1.9.1](https://docs.cloud.google.com/anthos/clusters/docs/on-prem/1.9/known-issues.md.txt#conflict_with_cert-manager_when_upgrading_to_version_190_or_191). - **Anthos 1.10+** : [Conflicting `cert-manager` installation](https://docs.cloud.google.com/anthos/clusters/docs/on-prem/1.10/known-issues.md.txt#conflicting_cert-manager_installation). **^(2)^ Support available with Apigee hybrid version 1.7.2 and newer**. **^(3)^ The official EOL dates for Apigee hybrid versions 1.8 and older have been reached. Regular monthly patches are no longer available. These releases are no longer officially supported except for customers with explicit and official exceptions for continued support. Other customers must upgrade**. **^(4)^Anthos on Bare Metal and VMWare versions 1.12 and earlier are out of support. See the [Anthos on Bare Metal Version Support Policy](https://docs.cloud.google.com/anthos/clusters/docs/bare-metal/latest/getting-support#version-support) and the [Anthos clusters on VMware versions](https://docs.cloud.google.com/anthos/clusters/docs/on-prem/latest/version-history#supported-versions).** **^(5)^Anthos on Bare Metal and VMWare [requires ASM 1.14](https://docs.cloud.google.com/anthos/docs/version-and-upgrade-support#112) or later. We recommend that you upgrade to hybrid v1.8 and switch to Apigee ingress gateway which no longer requires you to install ASM on your hybrid cluster.** **^(6)^ Support available with Apigee hybrid version 1.8.4 and newer**. **^(7)^ [Attached clusters](https://docs.cloud.google.com/apigee/docs/hybrid/v1.8/upgrade.md.txt#aboutattachedclusters) are not required for Apigee hybrid v1.8 and newer using [Apigee ingressgateway](https://cloud.google.com/apigee/docs/hybrid/latest/managing-ingress).**. **^(8)^ Not supported with Apigee hybrid version 1.8.4 and newer**. **^(9)^ Support available with Apigee hybrid version 1.7.6 and newer**. **^(10)^ Not supported with Apigee hybrid version 1.8.5 and newer**. **^(11)^ ASM is automatically installed with Apigee hybrid 1.9 and newer**. **^(12)^ Support available with Apigee hybrid version 1.9.2 and newer**. **^(13)^ Anthos clusters on AWS (Multi-Cloud) version numbers now reflect the Kubernetes versions. See [Anthos version and upgrade support](https://docs.cloud.google.com/anthos/docs/version-and-upgrade-support#anthos-on-other-public-clouds) for version details and recommended patches**. |||||

### About attached clusters


For Apigee hybrid versions 1.7.x and older, you must use Anthos
[attached
clusters](https://cloud.google.com/anthos/clusters/docs/attached) if you want to run Apigee hybrid in a multi-cloud context on Elastic Kubernetes
Service (EKS), Azure Kubernetes Service (AKS), or another supported third-party Kubernetes service
provider. Cluster attachment allows Google to measure the usage of Anthos Service Mesh (ASM).
[*Registering*](https://docs.cloud.google.com/anthos/clusters/docs/attached/how-to/attach-kubernetes-clusters#register_your_cluster)
the third-party cluster is optional. Register only if you wish to view the attached cluster in the
Google Cloud console. For more information, see [Attach
third-party Kubernetes clusters to Google Cloud](https://docs.cloud.google.com/anthos/clusters/docs/attached/how-to/attach-kubernetes-clusters).


For Apigee hybrid version 1.8.x, Anthos attached clusters are required if you are using
Anthos Service Mesh for your ingress gateway. If you are using Apigee ingress gateway, Anthos attached
clusters are optional.

> [!NOTE]
> **Note:** Hybrid installations are not currently supported on GKE Autopilot.

## Switch traffic from Anthos Service Mesh to Apigee ingress gateway


To switch your traffic to Apigee ingress gateway:

1. Expose the Apigee ingress gateway. Follow the procedures in [Expose Apigee ingress gateway](https://docs.cloud.google.com/apigee/docs/hybrid/v1.8/install-expose-apigee-ingress.md.txt).
2. Test your new ingress gateway by calling a proxy. Ideally, test all crucial proxies you currently have deployed.
3. To switch the traffic, update your DNS records to point to the IP address for your new Apigee ingress gateway. Depending on your DNS provider, you might be able to gradually shift traffic to the new endpoint. **Tip:** You can find the external IP address of Apigee ingress gateway with the following command:

   ```
   kubectl get svc -n apigee -l app=apigee-ingressgateway
   ```


   Your output should look something like:

   ```
   NAME                                        TYPE           CLUSTER-IP    EXTERNAL-IP     PORT(S)                                      AGE
   apigee-ingressgateway-prod-hybrid-37a39bd   LoadBalancer   192.0.2.123   233.252.0.123   15021:32049/TCP,80:31624/TCP,443:30723/TCP   16h
   ```
4. Ensure all the runtime traffic is working by monitoring your dashboards. Only proceed to the next step if everything is working as expected. Make sure no traffic is going through your old ingress gateway (Anthos Service Mesh), as the DNS update may be slow to propagate because of DNS caching.
5. To stop Apigee from supplying configuration to Anthos Service Mesh, follow the steps in [Stop supplying configuration to ASM](https://docs.cloud.google.com/apigee/docs/hybrid/v1.8/managing-ingress.md.txt#stop-supplying-configuration) in the Managing Apigee ingress gateway guide.
6. Retest and monitor API proxy traffic.
7. Follow the instructions in the Anthos Service Mesh documentation to [Uninstall Anthos Service Mesh](https://cloud.google.com/service-mesh/docs/uninstall) from the cluster.

## Upgrade Anthos Service Mesh to version 1.15

> [!WARNING]
> Perform this step only if you intend to continue using Anthos Service Mesh for your ingress gateway instead of using [Apigee ingress gateway](https://docs.cloud.google.com/apigee/docs/hybrid/v1.8/upgrade.md.txt#introducing).

> [!NOTE]
> You may experience a brief interruption of incoming traffic (typically less than two seconds) when you upgrade Anthos Service Mesh. If you are running hybrid in two or more clusters, you can avoid this interruption by upgrading one cluster at a time. Divert all production traffic away from the cluster you are about to upgrade, take the cluster offline, and then proceed with the upgrade process. Repeat the process for each cluster.

Perform the procedures using the Anthos Service Mesh documentation appropriate for your platform:


The instructions to install and configure Anthos Service Mesh are different depending on your platform. The
platforms are divided into the following categories:

- **GKE**: Google Kubernetes Engine clusters running on Google Cloud.
- **Outside Google Cloud** : Anthos clusters running on:
  - **Anthos clusters on VMware (GKE on-prem)**
  - **Anthos on bare metal**
  - **Anthos clusters on AWS**
  - **Amazon EKS**
- **Other Kubernetes Platforms** : Conformant clusters created and running on:
  - **AKS**
  - **EKS**
  - **OpenShift**

### GKE


The sequence for upgrading to Anthos Service Mesh version 1.17.8 for your hybrid
installation is as follows:

1. Prepare for the upgrade.
2. Install the new version of Anthos Service Mesh.
3. Delete the previous Anthos Service Mesh version's deployments, services, and webhooks from your current installation.
4. Upgrade your gateways and configure the new webhooks.

### Prepare to upgrade Anthos Service Mesh to version 1.17.8

> [!NOTE]
> **Note:** You can upgrade directly from Anthos Service Mesh version 1.11 or higher to Anthos Service Mesh version 1.17.8. If you are running a version of Anthos Service Mesh older than 1.11, you must first upgrade to version 1.11. Follow the instructions in [Upgrading from earlier versions](https://docs.cloud.google.com/service-mesh/v1.15/docs/upgrade-path-old-versions-gke)

1. Review the requirements in [Upgrade Anthos
   Service Mesh](https://docs.cloud.google.com/service-mesh/v1.15/docs/unified-install/upgrade), but do not perform the upgrade yet.
2. Before installing the new version, determine the current revision. You will need this information to delete the previous Anthos Service Mesh version's deployments, services, and webhooks from your current installation. Use the following command to store the current `istiod` revision to an environment variable:

       export DELETE_REV=$(kubectl get deploy -n istio-system -l app=istiod -o jsonpath={.items[*].metadata.labels.'istio\.io\/rev'}'{"\n"}')
       echo $DELETE_REV


   Your output should look something like `1.16`

   > [!NOTE]
   > **Note:** If the command returns multiple values, set the value of `DELETE_REV` to the oldest version and follow the procedure to delete the oldest version, then repeat the process until you have deleted all versions.

3. Create a new `overlay.yaml` file or verify that your existing `overlay.yaml` contains the following contents:

   ```
   apiVersion: install.istio.io/v1alpha1
   kind: IstioOperator
   spec:
     components:
       ingressGateways:
       - name: istio-ingressgateway
         enabled: true
         k8s:
           nodeSelector:
             # default node selector, if different or not using node selectors, change accordingly.
             cloud.google.com/gke-nodepool: apigee-runtime
           resources:
             requests:
               cpu: 1000m
           service:
             type: LoadBalancer
             loadBalancerIP: STATIC_IP # If you do not have a reserved static IP, leave this out.
             ports:
               - name: http-status-port
                 port: 15021
               - name: http2
                 port: 80
                 targetPort: 8080
               - name: https
                 port: 443
                 targetPort: 8443
     meshConfig:
       accessLogFormat:
         '{"start_time":"%START_TIME%","remote_address":"%DOWNSTREAM_DIRECT_REMOTE_ADDRESS%","user_agent":"%REQ(USER-AGENT)%","host":"%REQ(:AUTHORITY)%","request":"%REQ(:METHOD)% %REQ(X-ENVOY-ORIGINAL-PATH?:PATH)% %PROTOCOL%","request_time":"%DURATION%","status":"%RESPONSE_CODE%","status_details":"%RESPONSE_CODE_DETAILS%","bytes_received":"%BYTES_RECEIVED%","bytes_sent":"%BYTES_SENT%","upstream_address":"%UPSTREAM_HOST%","upstream_response_flags":"%RESPONSE_FLAGS%","upstream_response_time":"%RESPONSE_DURATION%","upstream_service_time":"%RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)%","upstream_cluster":"%UPSTREAM_CLUSTER%","x_forwarded_for":"%REQ(X-FORWARDED-FOR)%","request_method":"%REQ(:METHOD)%","request_path":"%REQ(X-ENVOY-ORIGINAL-PATH?:PATH)%","request_protocol":"%PROTOCOL%","tls_protocol":"%DOWNSTREAM_TLS_VERSION%","request_id":"%REQ(X-REQUEST-ID)%","sni_host":"%REQUESTED_SERVER_NAME%","apigee_dynamic_data":"%DYNAMIC_METADATA(envoy.lua)%"}'
   ```
4. Follow the instructions in the following sections in the Anthos Service Mesh documentation:

   > [!NOTE]
   > **Important:** Make sure to follow the instructions to upgrade Anthos Service Mesh with optional features, and to include your `overlay.yaml`.

   1. [Download
      asmcli](https://docs.cloud.google.com/service-mesh/v1.15/docs/unified-install/install-dependent-tools#download_asmcli)
   2. [Grant
      cluster admin permissions](https://docs.cloud.google.com/service-mesh/v1.15/docs/unified-install/install-dependent-tools#grant_cluster_admin_permissions)
   3. [Validate
      project and cluster](https://docs.cloud.google.com/service-mesh/v1.15/docs/unified-install/install-dependent-tools#validate_project_and_cluster)
   4. [Upgrade
      with optional features](https://docs.cloud.google.com/service-mesh/v1.15/docs/unified-install/upgrade#upgrade_with_optional_features). Stop before starting the "Upgrade Gateways section".
5. Switch to the new control plane:
   1. Get the revision label that is on `istiod`:

      ```
      kubectl get pod -n istio-system -L istio.io/rev
      ```


      The output from the command is similar to the following.

      ```
          NAME                                  READY  STATUS  RESTARTS   AGE  REV
          istiod-asm-Cloud Service Mesh 1.17.8-asm.4-67998f4b55-lrzpz    1/1    Running  0         68m  1.16.7-asm
          istiod-asm-Cloud Service Mesh 1.17.8-asm.4-67998f4b55-r76kr    1/1    Running  0         68m  1.16.7-asm
          istiod-Cloud Service Mesh 1.16.7-asm.1-1-5cd96f88f6-n7tj9      1/1    Running  0         27s  asm-1178-1
          istiod-Cloud Service Mesh 1.16.7-asm.1-1-5cd96f88f6-wm68b      1/1    Running  0         27s  asm-1178-1
      ```
   2. Assign the newer revision label to an environment variable.
      In the output, under the `REV` column, note the value of the revision
      label for the new version. In this example, the value is
      `asm-1178-1`

      ```
      export UPGRADE_REV="REVISION_LABEL"
      ```
   3. Add the revision label to `istio-system` namespace and remove the `istio-injection` label (if it exists) with the following command.

      ```
      kubectl label namespace istio-system istio.io/rev=$UPGRADE_REV istio-injection- --overwrite
      ```


      If you see `"istio-injection not found"` in the output, you can
      ignore it. That means that the namespace didn't previously have the
      `istio-injection` label. Because auto-injection fails if a
      namespace has both the `istio-injection` and the revision label,
      all `kubectl label` commands in the Anthos Service Mesh
      documentation include removing the `istio-injection` label.
   4. Restart the Pods to trigger re-injection.

      ```
      kubectl rollout restart deployment -n istio-system
      ```
   5. Test your application to verify that the workloads are working correctly.
   6. If you have workloads in other namespaces, repeat the steps to label the namespace and restart Pods.
6. Delete the previous versions:
   1. Navigate into the directory where you installed `asmcli`.
   2. Store the output directory for your Anthos Service Mesh installation in the **`DIR_PATH`** environment variable. This is the same directory you specified in the [Upgrade
      with optional features](https://docs.cloud.google.com/service-mesh/v1.15/docs/unified-install/upgrade#upgrade_with_optional_features) procedure.

      ```
      export DIR_PATH=OUTPUT_DIR
      ```
   3. Create a shell script containing the following commands:

      ```
      #!/bin/bash

      set -ex

      if [[ "${DELETE_REV}" != "${UPGRADE_REV}" ]]; then
        kubectl apply -f ${DIR_PATH}/asm/istio/istiod-service.yaml
        kubectl delete deploy -l app=istio-ingressgateway,istio.io/rev=${DELETE_REV} -n istio-system --ignore-not-found=true
        kubectl delete deploy -l app=istio-ingressgateway-connectors,istio.io/rev=${DELETE_REV} -n istio-system --ignore-not-found=true
        kubectl delete ValidatingWebhookConfiguration -l app=istiod,istio.io/rev=${DELETE_REV} -n istio-system --ignore-not-found=true
        kubectl delete MutatingWebhookConfiguration -l app=sidecar-injector,istio.io/rev=${DELETE_REV} -n istio-system --ignore-not-found=true
        kubectl delete Service,Deployment,HorizontalPodAutoscaler,PodDisruptionBudget istiod-${DELETE_REV} -n istio-system --ignore-not-found=true
        kubectl delete IstioOperator installed-state-${DELETE_REV} -n istio-system --ignore-not-found=true
      fi
      ```
   4. Execute the script to delete the previous versions.

### Outside Google Cloud


These instructions cover upgrading Anthos Service Mesh on:

- **Anthos clusters on VMware (GKE on-prem)**
- **Anthos on bare metal**
- **Anthos clusters on AWS**
- **Amazon EKS**


The sequence for upgrading to Anthos Service Mesh version 1.17.8 for your hybrid
installation is as follows:

1. Prepare for the upgrade.
2. Install the new version of Anthos Service Mesh.
3. Delete the previous Anthos Service Mesh version's deployments, services, and webhooks from your current installation.
4. Upgrade your gateways and configure the new webhooks.

### Prepare to upgrade Anthos Service Mesh to version 1.17.8

> [!NOTE]
> **Note:** You can upgrade directly from Anthos Service Mesh version 1.11 or higher to Anthos Service Mesh version 1.17.8. If you are running a version of Anthos Service Mesh older than 1.11, you must first upgrade to version 1.11. Follow the instructions in [Upgrading from earlier versions](https://docs.cloud.google.com/service-mesh/v1.15/docs/upgrade-path-old-versions-gke)

1. Review the requirements in [Upgrade Anthos
   Service Mesh](https://docs.cloud.google.com/service-mesh/v1.15/docs/unified-install/upgrade), but do not perform the upgrade yet.
2. Before installing the new version, determine the current revision. You will need this information to delete the previous Anthos Service Mesh version's deployments, services, and webhooks from your current installation. Use the following command to store the current `istiod` revision to an environment variable:

       export DELETE_REV=$(kubectl get deploy -n istio-system -l app=istiod -o jsonpath={.items[*].metadata.labels.'istio\.io\/rev'}'{"\n"}')
       echo $DELETE_REV


   Your output should look something like `1.16`

   > [!NOTE]
   > **Note:** If the command returns multiple values, set the value of `DELETE_REV` to the oldest version and follow the procedure to delete the oldest version, then repeat the process until you have deleted all versions.

3. Create a new `overlay.yaml` file or verify that your existing `overlay.yaml` contains the following contents:

   ```
   apiVersion: install.istio.io/v1alpha1
   kind: IstioOperator
   spec:  
     components:
       ingressGateways:
       - name: istio-ingressgateway
         enabled: true
         k8s:
           nodeSelector:
             # default node selector, if different or not using node selectors, change accordingly.
             cloud.google.com/gke-nodepool: apigee-runtime
           resources:
             requests:
               cpu: 1000m
           service:
             type: LoadBalancer
             loadBalancerIP: STATIC_IP # If you do not have a reserved static IP, leave this out.
             ports:
               - name: http-status-port
                 port: 15021
               - name: http2
                 port: 80
                 targetPort: 8080
               - name: https
                 port: 443
                 targetPort: 8443
     values:
       gateways:
         istio-ingressgateway:
           runAsRoot: true

     meshConfig:
       accessLogFormat:
         '{"start_time":"%START_TIME%","remote_address":"%DOWNSTREAM_DIRECT_REMOTE_ADDRESS%","user_agent":"%REQ(USER-AGENT)%","host":"%REQ(:AUTHORITY)%","request":"%REQ(:METHOD)% %REQ(X-ENVOY-ORIGINAL-PATH?:PATH)% %PROTOCOL%","request_time":"%DURATION%","status":"%RESPONSE_CODE%","status_details":"%RESPONSE_CODE_DETAILS%","bytes_received":"%BYTES_RECEIVED%","bytes_sent":"%BYTES_SENT%","upstream_address":"%UPSTREAM_HOST%","upstream_response_flags":"%RESPONSE_FLAGS%","upstream_response_time":"%RESPONSE_DURATION%","upstream_service_time":"%RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)%","upstream_cluster":"%UPSTREAM_CLUSTER%","x_forwarded_for":"%REQ(X-FORWARDED-FOR)%","request_method":"%REQ(:METHOD)%","request_path":"%REQ(X-ENVOY-ORIGINAL-PATH?:PATH)%","request_protocol":"%PROTOCOL%","tls_protocol":"%DOWNSTREAM_TLS_VERSION%","request_id":"%REQ(X-REQUEST-ID)%","sni_host":"%REQUESTED_SERVER_NAME%","apigee_dynamic_data":"%DYNAMIC_METADATA(envoy.lua)%"}'
   ```
4. Follow the instructions in the following sections in the Anthos Service Mesh documentation:

   > [!NOTE]
   > **Important:** Make sure to follow the instructions to upgrade Anthos Service Mesh with optional features, and to include your `overlay.yaml`.

   1. [Download
      asmcli](https://docs.cloud.google.com/service-mesh/v1.15/docs/unified-install/install-dependent-tools#download_asmcli)
   2. [Grant
      cluster admin permissions](https://docs.cloud.google.com/service-mesh/v1.15/docs/unified-install/install-dependent-tools#grant_cluster_admin_permissions)
   3. [Validate
      project and cluster](https://docs.cloud.google.com/service-mesh/v1.15/docs/unified-install/install-dependent-tools#validate_project_and_cluster)
   4. [Upgrade
      with optional features](https://docs.cloud.google.com/service-mesh/v1.15/docs/unified-install/upgrade#upgrade_with_optional_features). Stop before starting the "Upgrade Gateways section".
5. Switch to the new control plane:
   1. Get the revision label that is on `istiod`:

      ```
      kubectl get pod -n istio-system -L istio.io/rev
      ```


      The output from the command is similar to the following.

      ```
          NAME                                  READY  STATUS  RESTARTS   AGE  REV
          istiod-asm-Cloud Service Mesh 1.17.8-asm.4-67998f4b55-lrzpz    1/1    Running  0         68m  1.16.7-asm
          istiod-asm-Cloud Service Mesh 1.17.8-asm.4-67998f4b55-r76kr    1/1    Running  0         68m  1.16.7-asm
          istiod-Cloud Service Mesh 1.16.7-asm.1-1-5cd96f88f6-n7tj9      1/1    Running  0         27s  asm-1178-1
          istiod-Cloud Service Mesh 1.16.7-asm.1-1-5cd96f88f6-wm68b      1/1    Running  0         27s  asm-1178-1
      ```
   2. Assign the newer revision label to an environment variable.
      In the output, under the `REV` column, note the value of the revision
      label for the new version. In this example, the value is
      `asm-1178-1`

      ```
      export UPGRADE_REV="REVISION_LABEL"
      ```
   3. Add the revision label to `istio-system` namespace and remove the `istio-injection` label (if it exists) with the following command.

      ```
      kubectl label namespace istio-system istio.io/rev=$UPGRADE_REV istio-injection- --overwrite
      ```


      If you see `"istio-injection not found"` in the output, you can
      ignore it. That means that the namespace didn't previously have the
      `istio-injection` label. Because auto-injection fails if a
      namespace has both the `istio-injection` and the revision label,
      all `kubectl label` commands in the Anthos Service Mesh
      documentation include removing the `istio-injection` label.
   4. Restart the Pods to trigger re-injection.

      ```
      kubectl rollout restart deployment -n istio-system
      ```
   5. Test your application to verify that the workloads are working correctly.
   6. If you have workloads in other namespaces, repeat the steps to label the namespace and restart Pods.
6. Delete the previous versions:
   1. Navigate into the directory where you installed `asmcli`.
   2. Store the output directory for your Anthos Service Mesh installation in the **`DIR_PATH`** environment variable. This is the same directory you specified in the [Upgrade
      with optional features](https://docs.cloud.google.com/service-mesh/v1.15/docs/unified-install/upgrade#upgrade_with_optional_features) procedure.

      ```
      export DIR_PATH=OUTPUT_DIR
      ```
   3. Create a shell script containing the following commands:

      ```
      #!/bin/bash

      set -ex

      if [[ "${DELETE_REV}" != "${UPGRADE_REV}" ]]; then
        kubectl apply -f ${DIR_PATH}/asm/istio/istiod-service.yaml
        kubectl delete deploy -l app=istio-ingressgateway,istio.io/rev=${DELETE_REV} -n istio-system --ignore-not-found=true
        kubectl delete deploy -l app=istio-ingressgateway-connectors,istio.io/rev=${DELETE_REV} -n istio-system --ignore-not-found=true
        kubectl delete ValidatingWebhookConfiguration -l app=istiod,istio.io/rev=${DELETE_REV} -n istio-system --ignore-not-found=true
        kubectl delete MutatingWebhookConfiguration -l app=sidecar-injector,istio.io/rev=${DELETE_REV} -n istio-system --ignore-not-found=true
        kubectl delete Service,Deployment,HorizontalPodAutoscaler,PodDisruptionBudget istiod-${DELETE_REV} -n istio-system --ignore-not-found=true
        kubectl delete IstioOperator installed-state-${DELETE_REV} -n istio-system --ignore-not-found=true
      fi
      ```
   4. Execute the script to delete the previous versions.

### AKS / EKS


In these instructions the process of upgrading Anthos Service Mesh (Anthos Service Mesh) version
1.17.8-asm.4-distroless on Anthos attached clusters is the same as performing a fresh install.

### Preparing to install Anthos Service Mesh

1. Before installing the new version, determine the current revision. You will need this information to delete the *validating webhook* and *mutating webhook* from your current Anthos Service Mesh installation. Use the following command to store the current `istiod` revision to an environment variable:

       export DELETE_REV=$(kubectl get deploy -n istio-system -l app=istiod -o jsonpath={.items[*].metadata.labels.'istio\.io\/rev'}'{"\n"}')
       echo $DELETE_REV


   Your output should look something like `1.16`

   > [!NOTE]
   > **Note:** If the command returns multiple values, set the value of `DELETE_REV` to the oldest version and follow the procedure to delete the oldest version, then repeat the process until you have deleted all versions.

### Linux

- Download the Anthos Service Mesh installation file to your current working directory:

  ```
  curl -LO https://storage.googleapis.com/gke-release/asm/1.17.8-asm.4-distroless-linux-amd64.tar.gz
  ```
- Download the signature file and use OpenSSL to verify the signature:

  ```
  curl -LO https://storage.googleapis.com/gke-release/asm/1.17.8-asm.4-distroless-linux-amd64.tar.gz.1.sig
  ```

      openssl dgst -verify /dev/stdin -signature 1.17.8-asm.4-distroless-linux-amd64.tar.gz.1.sig 1.17.8-asm.4-distroless.tar.gz <<'EOF'
      ---BEGIN PUBLIC KEY---
      MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEWZrGCUaJJr1H8a36sG4UUoXvlXvZ
      wQfk16sxprI2gOJ2vFFggdq3ixF2h4qNBt0kI7ciDhgpwS8t+/960IsIgw==
      ---END PUBLIC KEY---
      EOF

- Extract the contents of the file to any location on your file system. For example, to extract the contents to the current working directory:

  ```
  tar xzf 1.17.8-asm.4-distroless-linux-amd64.tar.gz
  ```


  The command creates an installation directory in your current working directory named
  `1.17.8-asm.4-distroless` that contains:
  - Sample applications in the `samples` directory.
  - The `istioctl` command-line tool that you use to install Anthos Service Mesh is in the `bin` directory.
  - The Anthos Service Mesh configuration profiles are in the `manifests/profiles` directory.
- Ensure that you're in the Anthos Service Mesh installation's root directory:

  ```
  cd 1.17.8-asm.4-distroless
  ```
- For convenience, add the tools in the `/bin` directory to your `PATH`:

  ```
  export PATH=$PWD/bin:$PATH
  ```

### Mac OS

- Download the Anthos Service Mesh installation file to your current working directory:

  ```
  curl -LO https://storage.googleapis.com/gke-release/asm/1.17.8-asm.4-distroless-osx.tar.gz
  ```
- Download the signature file and use OpenSSL to verify the signature:

  ```
  curl -LO https://storage.googleapis.com/gke-release/asm/1.17.8-asm.4-distroless-osx.tar.gz.1.sig
  ```

      openssl dgst -sha256 -verify /dev/stdin -signature 1.17.8-asm.4-distroless-osx.tar.gz.1.sig 1.17.8-asm.4-distroless.tar.gz <<'EOF'
      ---BEGIN PUBLIC KEY---
      MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEWZrGCUaJJr1H8a36sG4UUoXvlXvZ
      wQfk16sxprI2gOJ2vFFggdq3ixF2h4qNBt0kI7ciDhgpwS8t+/960IsIgw==
      ---END PUBLIC KEY---
      EOF

- Extract the contents of the file to any location on your file system. For example, to extract the contents to the current working directory:

  ```
  tar xzf 1.17.8-asm.4-distroless-osx.tar.gz
  ```


  The command creates an installation directory in your current working directory named
  `1.17.8-asm.4-distroless` that contains:
  - Sample applications in the `samples` directory.
  - The `istioctl` command-line tool that you use to install Anthos Service Mesh is in the `bin` directory.
  - The Anthos Service Mesh configuration profiles are in the `manifests/profiles` directory.
- Ensure that you're in the Anthos Service Mesh installation's root directory:

  ```
  cd 1.17.8-asm.4-distroless
  ```
- For convenience, add the tools in the `/bin` directory to your `PATH`:

  ```
  export PATH=$PWD/bin:$PATH
  ```

### Windows

- Download the Anthos Service Mesh installation file to your current working directory:

  ```
  curl -LO https://storage.googleapis.com/gke-release/asm/1.17.8-asm.4-distroless-win.zip
  ```
- Download the signature file and use OpenSSL to verify the signature:

  ```
  curl -LO https://storage.googleapis.com/gke-release/asm/1.17.8-asm.4-distroless-win.zip.1.sig
  ```

      openssl dgst -verify - -signature 1.17.8-asm.4-distroless-win.zip.1.sig 1.17.8-asm.4-distroless.win.zip <<'EOF'
      ---BEGIN PUBLIC KEY---
      MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEWZrGCUaJJr1H8a36sG4UUoXvlXvZ
      wQfk16sxprI2gOJ2vFFggdq3ixF2h4qNBt0kI7ciDhgpwS8t+/960IsIgw==
      ---END PUBLIC KEY---
      EOF

- Extract the contents of the file to any location on your file system. For example, to extract the contents to the current working directory:

  ```
  tar xzf 1.17.8-asm.4-distroless-win.zip
  ```


  The command creates an installation directory in your current working directory named
  `1.17.8-asm.4-distroless` that contains:
  - Sample applications in the `samples` directory.
  - The `istioctl` command-line tool that you use to install Anthos Service Mesh is in the `bin` directory.
  - The Anthos Service Mesh configuration profiles are in the `manifests\profiles` directory.
- Ensure that you're in the Anthos Service Mesh installation's root directory:

  ```
  cd 1.17.8-asm.4-distroless
  ```
- For convenience, add the tools in the \\bin directory to your PATH:

  ```
  set PATH=%CD%\bin:%PATH%
  ```

2. Now that Anthos Service Mesh Istio is installed, check the version of `istioctl`:

   ```
   istioctl version
   ```
3. Create a namespace called istio-system for the control plane components:

   ```
   kubectl create namespace istio-system
   ```

### Installing Anthos Service Mesh

1. Edit your `overlay.yaml` file or create a new one with the following contents:

   ```
   apiVersion: install.istio.io/v1alpha1
   kind: IstioOperator
   spec:
     meshConfig:
       accessLogFile: /dev/stdout
       enableTracing: true
       accessLogFormat:
         '{"start_time":"%START_TIME%","remote_address":"%DOWNSTREAM_DIRECT_REMOTE_ADDRESS%","user_agent":"%REQ(USER-AGENT)%","host":"%REQ(:AUTHORITY)%","request":"%REQ(:METHOD)% %REQ(X-ENVOY-ORIGINAL-PATH?:PATH)% %PROTOCOL%","request_time":"%DURATION%","status":"%RESPONSE_CODE%","status_details":"%RESPONSE_CODE_DETAILS%","bytes_received":"%BYTES_RECEIVED%","bytes_sent":"%BYTES_SENT%","upstream_address":"%UPSTREAM_HOST%","upstream_response_flags":"%RESPONSE_FLAGS%","upstream_response_time":"%RESPONSE_DURATION%","upstream_service_time":"%RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)%","upstream_cluster":"%UPSTREAM_CLUSTER%","x_forwarded_for":"%REQ(X-FORWARDED-FOR)%","request_method":"%REQ(:METHOD)%","request_path":"%REQ(X-ENVOY-ORIGINAL-PATH?:PATH)%","request_protocol":"%PROTOCOL%","tls_protocol":"%DOWNSTREAM_TLS_VERSION%","request_id":"%REQ(X-REQUEST-ID)%","sni_host":"%REQUESTED_SERVER_NAME%","apigee_dynamic_data":"%DYNAMIC_METADATA(envoy.lua)%"}'
     components:
       ingressGateways:
       - name: istio-ingressgateway
         enabled: true
         k8s:
           service:
             type: LoadBalancer
             ports:
             - name: status-port
               port: 15021
               targetPort: 15021
             - name: http2
               port: 80
               targetPort: 8080
             - name: https
               port: 443
               targetPort: 8443
   ```
2. Install Anthos Service Mesh with `istioctl` using the `asm-multicloud` profile:

   ```
   istioctl install \
       --set profile=asm-multicloud \
       --set revision="asm-1178-1" \
       --filename overlay.yaml
   ```


   Your output should look something like:

   ```
   kubectl get pods -n istio-system
   NAME                                   READY   STATUS    RESTARTS   AGE
   istio-ingressgateway-88b6fd976-flgp2   1/1     Running   0          3m13s
   istio-ingressgateway-88b6fd976-p5dl9   1/1     Running   0          2m57s
   istiod-asm-1178-1-798ffb964-2ls88       1/1     Running   0          3m21s
   istiod-asm-1178-1-798ffb964-fnj8c       1/1     Running   1          3m21s
   ```


   The `--set revision` argument adds a revision label in the format
   `istio.io/rev=asm-1178-1` to `istiod`. The revision label is used by the
   automatic sidecar injector webhook to associate injected sidecars with a particular `istiod`
   revision. To enable sidecar auto-injection for a namespace, you must label it with a revision
   that matches the label on `istiod`.
3. Verify that your install completed:

   ```
   kubectl get svc -n istio-system
   ```


   Your output should look something like:

   ```
   NAME                   TYPE           CLUSTER-IP       EXTERNAL-IP     PORT(S)                                                                      AGE
   istio-ingressgateway   LoadBalancer   172.200.48.52    34.74.177.168   15021:30479/TCP,80:30030/TCP,443:32200/TCP,15012:32297/TCP,15443:30244/TCP   3m35s
   istiod                 ClusterIP      172.200.18.133   <none>          15010/TCP,15012/TCP,443/TCP,15014/TCP                                        4m46s
   istiod-asm-1178-1       ClusterIP      172.200.63.220   <none>          15010/TCP,15012/TCP,443/TCP,15014/TCP                                        3m43s
   ```
4. Switch to the new control plane:
   1. Get the revision label that is on `istiod`:

      ```
      kubectl get pod -n istio-system -L istio.io/rev
      ```


      The output from the command is similar to the following.

      ```
          NAME                                  READY  STATUS  RESTARTS   AGE  REV
          istiod-asm-Cloud Service Mesh 1.17.8-asm.4-67998f4b55-lrzpz    1/1    Running  0         68m  1.16.7-asm
          istiod-asm-Cloud Service Mesh 1.17.8-asm.4-67998f4b55-r76kr    1/1    Running  0         68m  1.16.7-asm
          istiod-Cloud Service Mesh 1.16.7-asm.1-1-5cd96f88f6-n7tj9      1/1    Running  0         27s  asm-1178-1
          istiod-Cloud Service Mesh 1.16.7-asm.1-1-5cd96f88f6-wm68b      1/1    Running  0         27s  asm-1178-1
      ```
   2. Assign the newer revision label to an environment variable.
      In the output, under the `REV` column, note the value of the revision
      label for the new version. In this example, the value is
      `asm-1178-1`

      ```
      export UPGRADE_REV="REVISION_LABEL"
      ```
   3. Add the revision label to `istio-system` namespace and remove the `istio-injection` label (if it exists) with the following command.

      ```
      kubectl label namespace istio-system istio.io/rev=$UPGRADE_REV istio-injection- --overwrite
      ```


      If you see `"istio-injection not found"` in the output, you can
      ignore it. That means that the namespace didn't previously have the
      `istio-injection` label. Because auto-injection fails if a
      namespace has both the `istio-injection` and the revision label,
      all `kubectl label` commands in the Anthos Service Mesh
      documentation include removing the `istio-injection` label.
   4. Restart the Pods to trigger re-injection.

      ```
      kubectl rollout restart deployment -n istio-system
      ```
   5. Test your application to verify that the workloads are working correctly.
   6. If you have workloads in other namespaces, repeat the steps to label the namespace and restart Pods.
5. Delete the previous versions:
   1. Navigate into the directory where you installed `asmcli`.
   2. Create a shell script containing the following commands:

      ```
      #!/bin/bash

      set -ex

      if [[ "${DELETE_REV}" != "${UPGRADE_REV}" ]]; then
        kubectl delete deploy -l app=istio-ingressgateway,istio.io/rev=${DELETE_REV} -n istio-system --ignore-not-found=true
        kubectl delete deploy -l app=istio-ingressgateway-connectors,istio.io/rev=${DELETE_REV} -n istio-system --ignore-not-found=true
        kubectl delete ValidatingWebhookConfiguration -l app=istiod,istio.io/rev=${DELETE_REV} -n istio-system --ignore-not-found=true
        kubectl delete MutatingWebhookConfiguration -l app=sidecar-injector,istio.io/rev=${DELETE_REV} -n istio-system --ignore-not-found=true
        kubectl delete Service,Deployment,HorizontalPodAutoscaler,PodDisruptionBudget istiod-${DELETE_REV} -n istio-system --ignore-not-found=true
        kubectl delete IstioOperator installed-state-${DELETE_REV} -n istio-system --ignore-not-found=true
      fi
      ```
   3. Execute the script to delete the previous versions.

### OpenShift


In these instructions the process of upgrading Anthos Service Mesh (Anthos Service Mesh) version
1.17.8-asm.4-distroless on Anthos attached clusters is the same as performing a fresh install.

### Preparing to install Anthos Service Mesh

1. Before installing the new version, determine the current revision. You will need this information to delete the *validating webhook* and *mutating webhook* from your current Anthos Service Mesh installation. Use the following command to store the current `istiod` revision to an environment variable:

       export DELETE_REV=$(kubectl get deploy -n istio-system -l app=istiod -o jsonpath={.items[*].metadata.labels.'istio\.io\/rev'}'{"\n"}')
       echo $DELETE_REV


   Your output should look something like `1.16`

   > [!NOTE]
   > **Note:** If the command returns multiple values, set the value of `DELETE_REV` to the oldest version and follow the procedure to delete the oldest version, then repeat the process until you have deleted all versions.

### Linux

- Grant the `anyuid` security context constraint (SCC) to the istio-system with the following OpenShift CLI (`oc`) command:

  ```
  oc adm policy add-scc-to-group anyuid system:serviceaccounts:istio-system
  ```
- Download the Anthos Service Mesh installation file to your current working directory:

  ```
  curl -LO https://storage.googleapis.com/gke-release/asm/1.17.8-asm.4-distroless-linux-amd64.tar.gz
  ```
- Download the signature file and use OpenSSL to verify the signature:

  ```
  curl -LO https://storage.googleapis.com/gke-release/asm/1.17.8-asm.4-distroless-linux-amd64.tar.gz.1.sig
  ```

      openssl dgst -verify /dev/stdin -signature 1.17.8-asm.4-distroless-linux-amd64.tar.gz.1.sig 1.17.8-asm.4-distroless.tar.gz <<'EOF'
      ---BEGIN PUBLIC KEY---
      MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEWZrGCUaJJr1H8a36sG4UUoXvlXvZ
      wQfk16sxprI2gOJ2vFFggdq3ixF2h4qNBt0kI7ciDhgpwS8t+/960IsIgw==
      ---END PUBLIC KEY---
      EOF

- Extract the contents of the file to any location on your file system. For example, to extract the contents to the current working directory:

  ```
  tar xzf 1.17.8-asm.4-distroless-linux-amd64.tar.gz
  ```


  The command creates an installation directory in your current working directory named
  `1.17.8-asm.4-distroless` that contains:
  - Sample applications in the `samples` directory.
  - The `istioctl` command-line tool that you use to install Anthos Service Mesh is in the `bin` directory.
  - The Anthos Service Mesh configuration profiles are in the `manifests/profiles` directory.
- Ensure that you're in the Anthos Service Mesh installation's root directory:

  ```
  cd 1.17.8-asm.4-distroless
  ```
- For convenience, add the tools in the `/bin` directory to your `PATH`:

  ```
  export PATH=$PWD/bin:$PATH
  ```

### Mac OS

- Grant the `anyuid` security context constraint (SCC) to the istio-system with the following OpenShift CLI (`oc`) command:

  ```
  oc adm policy add-scc-to-group anyuid system:serviceaccounts:istio-system
  ```
- Download the Anthos Service Mesh installation file to your current working directory:

  ```
  curl -LO https://storage.googleapis.com/gke-release/asm/1.17.8-asm.4-distroless-osx.tar.gz
  ```
- Download the signature file and use OpenSSL to verify the signature:

  ```
  curl -LO https://storage.googleapis.com/gke-release/asm/1.17.8-asm.4-distroless-osx.tar.gz.1.sig
  ```

      openssl dgst -sha256 -verify /dev/stdin -signature 1.17.8-asm.4-distroless-osx.tar.gz.1.sig 1.17.8-asm.4-distroless.tar.gz <<'EOF'
      ---BEGIN PUBLIC KEY---
      MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEWZrGCUaJJr1H8a36sG4UUoXvlXvZ
      wQfk16sxprI2gOJ2vFFggdq3ixF2h4qNBt0kI7ciDhgpwS8t+/960IsIgw==
      ---END PUBLIC KEY---
      EOF

- Extract the contents of the file to any location on your file system. For example, to extract the contents to the current working directory:

  ```
  tar xzf 1.17.8-asm.4-distroless-osx.tar.gz
  ```


  The command creates an installation directory in your current working directory named
  `1.17.8-asm.4-distroless` that contains:
  - Sample applications in the `samples` directory.
  - The `istioctl` command-line tool that you use to install Anthos Service Mesh is in the `bin` directory.
  - The Anthos Service Mesh configuration profiles are in the `manifests/profiles` directory.
- Ensure that you're in the Anthos Service Mesh installation's root directory:

  ```
  cd 1.17.8-asm.4-distroless
  ```
- For convenience, add the tools in the `/bin` directory to your `PATH`:

  ```
  export PATH=$PWD/bin:$PATH
  ```

### Windows

- Grant the `anyuid` security context constraint (SCC) to the istio-system with the following OpenShift CLI (`oc`) command:

  ```
  oc adm policy add-scc-to-group anyuid system:serviceaccounts:istio-system
  ```
- Download the Anthos Service Mesh installation file to your current working directory:

  ```
  curl -LO https://storage.googleapis.com/gke-release/asm/1.17.8-asm.4-distroless-win.zip
  ```
- Download the signature file and use OpenSSL to verify the signature:

  ```
  curl -LO https://storage.googleapis.com/gke-release/asm/1.17.8-asm.4-distroless-win.zip.1.sig
  ```

      openssl dgst -verify - -signature 1.17.8-asm.4-distroless-win.zip.1.sig 1.17.8-asm.4-distroless.win.zip <<'EOF'
      ---BEGIN PUBLIC KEY---
      MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEWZrGCUaJJr1H8a36sG4UUoXvlXvZ
      wQfk16sxprI2gOJ2vFFggdq3ixF2h4qNBt0kI7ciDhgpwS8t+/960IsIgw==
      ---END PUBLIC KEY---
      EOF

- Extract the contents of the file to any location on your file system. For example, to extract the contents to the current working directory:

  ```
  tar xzf 1.17.8-asm.4-distroless-win.zip
  ```


  The command creates an installation directory in your current working directory named
  `1.17.8-asm.4-distroless` that contains:
  - Sample applications in the `samples` directory.
  - The `istioctl` command-line tool that you use to install Anthos Service Mesh is in the `bin` directory.
  - The Anthos Service Mesh configuration profiles are in the `manifests\profiles` directory.
- Ensure that you're in the Anthos Service Mesh installation's root directory:

  ```
  cd 1.17.8-asm.4-distroless
  ```
- For convenience, add the tools in the \\bin directory to your PATH:

  ```
  set PATH=%CD%\bin:%PATH%
  ```

2. Now that Anthos Service Mesh Istio is installed, check the version of `istioctl`:

   ```
   istioctl version
   ```
3. Create a namespace called istio-system for the control plane components:

   ```
   kubectl create namespace istio-system
   ```

### Configure the validating webhook


When you install Anthos Service Mesh, you set a revision label on `istiod`. You need to set the same
revision on the validating webhook.

1. Create a file called `istiod-service.yaml` with the following contents:

   ```
   apiVersion: v1
   kind: Service
   metadata:
     name: istiod
     namespace: istio-system
     labels:
       istio.io/rev: asm-1178-1
       app: istiod
       istio: pilot
       release: istio
   spec:
     ports:
       - port: 15010
         name: grpc-xds # plaintext
         protocol: TCP
       - port: 15012
         name: https-dns # mTLS with k8s-signed cert
         protocol: TCP
       - port: 443
         name: https-webhook # validation and injection
         targetPort: 15017
         protocol: TCP
       - port: 15014
         name: http-monitoring # prometheus stats
         protocol: TCP
     selector:
       app: istiod
       istio.io/rev: asm-1178-1
     meshConfig:
       accessLogFormat:
         '{"start_time":"%START_TIME%","remote_address":"%DOWNSTREAM_DIRECT_REMOTE_ADDRESS%","user_agent":"%REQ(USER-AGENT)%","host":"%REQ(:AUTHORITY)%","request":"%REQ(:METHOD)% %REQ(X-ENVOY-ORIGINAL-PATH?:PATH)% %PROTOCOL%","request_time":"%DURATION%","status":"%RESPONSE_CODE%","status_details":"%RESPONSE_CODE_DETAILS%","bytes_received":"%BYTES_RECEIVED%","bytes_sent":"%BYTES_SENT%","upstream_address":"%UPSTREAM_HOST%","upstream_response_flags":"%RESPONSE_FLAGS%","upstream_response_time":"%RESPONSE_DURATION%","upstream_service_time":"%RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)%","upstream_cluster":"%UPSTREAM_CLUSTER%","x_forwarded_for":"%REQ(X-FORWARDED-FOR)%","request_method":"%REQ(:METHOD)%","request_path":"%REQ(X-ENVOY-ORIGINAL-PATH?:PATH)%","request_protocol":"%PROTOCOL%","tls_protocol":"%DOWNSTREAM_TLS_VERSION%","request_id":"%REQ(X-REQUEST-ID)%","sni_host":"%REQUESTED_SERVER_NAME%","apigee_dynamic_data":"%DYNAMIC_METADATA(envoy.lua)%"}'
   ```
2. Use `kubectl` to apply the validating webhook configuration:

   ```
   kubectl apply -f istiod-service.yaml
   ```
3. Verify that the configuration was applied:

   ```
   kubectl get svc -n istio-system
   ```


   The response should look similar to:

   ```
   NAME     TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                                 AGE
   istiod   ClusterIP   172.200.18.133   <none>        15010/TCP,15012/TCP,443/TCP,15014/TCP   22s
   ```

### Installing Anthos Service Mesh

1. Edit your `overlay.yaml` file or create a new one with the following contents:

   ```
   apiVersion: install.istio.io/v1alpha1
   kind: IstioOperator
   spec:
     meshConfig:
       accessLogFile: /dev/stdout
       enableTracing: true
       accessLogFormat:
         '{"start_time":"%START_TIME%","remote_address":"%DOWNSTREAM_DIRECT_REMOTE_ADDRESS%","user_agent":"%REQ(USER-AGENT)%","host":"%REQ(:AUTHORITY)%","request":"%REQ(:METHOD)% %REQ(X-ENVOY-ORIGINAL-PATH?:PATH)% %PROTOCOL%","request_time":"%DURATION%","status":"%RESPONSE_CODE%","status_details":"%RESPONSE_CODE_DETAILS%","bytes_received":"%BYTES_RECEIVED%","bytes_sent":"%BYTES_SENT%","upstream_address":"%UPSTREAM_HOST%","upstream_response_flags":"%RESPONSE_FLAGS%","upstream_response_time":"%RESPONSE_DURATION%","upstream_service_time":"%RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)%","upstream_cluster":"%UPSTREAM_CLUSTER%","x_forwarded_for":"%REQ(X-FORWARDED-FOR)%","request_method":"%REQ(:METHOD)%","request_path":"%REQ(X-ENVOY-ORIGINAL-PATH?:PATH)%","request_protocol":"%PROTOCOL%","tls_protocol":"%DOWNSTREAM_TLS_VERSION%","request_id":"%REQ(X-REQUEST-ID)%","sni_host":"%REQUESTED_SERVER_NAME%","apigee_dynamic_data":"%DYNAMIC_METADATA(envoy.lua)%"}'
     components:
       ingressGateways:
         - name: istio-ingressgateway
           enabled: true
           k8s:
             service:
               type: LoadBalancer
               ports:
               - name: status-port
                 port: 15021
                 targetPort: 15021
               - name: http2
                 port: 80
                 targetPort: 8080
               - name: https
                 port: 443
                 targetPort: 8443
   ```
2. Install Anthos Service Mesh with `istioctl` using the `asm-multicloud` profile:

   ```
   istioctl install \
       --set profile=asm-multicloud \
       --set revision="asm-1178-1" \
       --filename overlayfile.yaml
   ```


   Your output should look something like:

   ```
   kubectl get pods -n istio-system
   NAME                                   READY   STATUS    RESTARTS   AGE
   istio-ingressgateway-88b6fd976-flgp2   1/1     Running   0          3m13s
   istio-ingressgateway-88b6fd976-p5dl9   1/1     Running   0          2m57s
   istiod-asm-1178-1-798ffb964-2ls88       1/1     Running   0          3m21s
   istiod-asm-1178-1-798ffb964-fnj8c       1/1     Running   1          3m21s
   ```


   The `--set revision` argument adds a revision label in the format
   `istio.io/rev=1.6.11-asm.1` to `istiod`. The revision label is used by the
   automatic sidecar injector webhook to associate injected sidecars with a particular `istiod`
   revision. To enable sidecar auto-injection for a namespace, you must label it with a revision
   that matches the label on `istiod`.
3. Verify that your install completed:

   ```
   kubectl get svc -n istio-system
   ```


   Your output should look something like:

   ```
   NAME                   TYPE           CLUSTER-IP       EXTERNAL-IP     PORT(S)                                                                      AGE
   istio-ingressgateway   LoadBalancer   172.200.48.52    34.74.177.168   15021:30479/TCP,80:30030/TCP,443:32200/TCP,15012:32297/TCP,15443:30244/TCP   3m35s
   istiod                 ClusterIP      172.200.18.133   <none>          15010/TCP,15012/TCP,443/TCP,15014/TCP                                        4m46s
   istiod-asm-1178-1       ClusterIP      172.200.63.220   <none>          15010/TCP,15012/TCP,443/TCP,15014/TCP                                        3m43s
   ```
4. Switch to the new control plane:
   1. Get the revision label that is on `istiod`:

      ```
      kubectl get pod -n istio-system -L istio.io/rev
      ```


      The output from the command is similar to the following.

      ```
          NAME                                  READY  STATUS  RESTARTS   AGE  REV
          istiod-asm-Cloud Service Mesh 1.17.8-asm.4-67998f4b55-lrzpz    1/1    Running  0         68m  1.16.7-asm
          istiod-asm-Cloud Service Mesh 1.17.8-asm.4-67998f4b55-r76kr    1/1    Running  0         68m  1.16.7-asm
          istiod-Cloud Service Mesh 1.16.7-asm.1-1-5cd96f88f6-n7tj9      1/1    Running  0         27s  asm-1178-1
          istiod-Cloud Service Mesh 1.16.7-asm.1-1-5cd96f88f6-wm68b      1/1    Running  0         27s  asm-1178-1
      ```
   2. Assign the newer revision label to an environment variable.
      In the output, under the `REV` column, note the value of the revision
      label for the new version. In this example, the value is
      `asm-1178-1`

      ```
      export UPGRADE_REV="REVISION_LABEL"
      ```
   3. Add the revision label to `istio-system` namespace and remove the `istio-injection` label (if it exists) with the following command.

      ```
      kubectl label namespace istio-system istio.io/rev=$UPGRADE_REV istio-injection- --overwrite
      ```


      If you see `"istio-injection not found"` in the output, you can
      ignore it. That means that the namespace didn't previously have the
      `istio-injection` label. Because auto-injection fails if a
      namespace has both the `istio-injection` and the revision label,
      all `kubectl label` commands in the Anthos Service Mesh
      documentation include removing the `istio-injection` label.
   4. Restart the Pods to trigger re-injection.

      ```
      kubectl rollout restart deployment -n istio-system
      ```
   5. Test your application to verify that the workloads are working correctly.
   6. If you have workloads in other namespaces, repeat the steps to label the namespace and restart Pods.
5. Delete the previous versions:
   1. Navigate into the directory where you installed `asmcli`.
   2. Create a shell script containing the following commands:

      ```
      #!/bin/bash

      set -ex

      if [[ "${DELETE_REV}" != "${UPGRADE_REV}" ]]; then
        kubectl delete deploy -l app=istio-ingressgateway,istio.io/rev=${DELETE_REV} -n istio-system --ignore-not-found=true
        kubectl delete deploy -l app=istio-ingressgateway-connectors,istio.io/rev=${DELETE_REV} -n istio-system --ignore-not-found=true
        kubectl delete ValidatingWebhookConfiguration -l app=istiod,istio.io/rev=${DELETE_REV} -n istio-system --ignore-not-found=true
        kubectl delete MutatingWebhookConfiguration -l app=sidecar-injector,istio.io/rev=${DELETE_REV} -n istio-system --ignore-not-found=true
        kubectl delete Service,Deployment,HorizontalPodAutoscaler,PodDisruptionBudget istiod-${DELETE_REV} -n istio-system --ignore-not-found=true
        kubectl delete IstioOperator installed-state-${DELETE_REV} -n istio-system --ignore-not-found=true
      fi
      ```
   3. Execute the script to delete the previous versions.

> [!TIP]
> **Congratulations!** You have successfully upgraded to Apigee hybrid version 1.8.8.

## Rolling back an upgrade


Follow these steps to roll back a previous upgrade:

1. Clean up completed jobs for the hybrid runtime namespace, where <var translate="no">NAMESPACE</var> is the namespace specified in your overrides file, if you specified a namespace. If not, the default namespace is `apigee`:

   ```
   kubectl delete job -n NAMESPACE \
     $(kubectl get job -n NAMESPACE \
     -o=jsonpath='{.items[?(@.status.succeeded==1)].metadata.name}')
   ```
2. Clean up completed jobs for the `apigee-system` namespace:

   ```
   kubectl delete job -n apigee-system \
     $(kubectl get job -n apigee-system \
     -o=jsonpath='{.items[?(@.status.succeeded==1)].metadata.name}')
   ```
3. Change the `APIGEECTL_HOME` variable to point to the directory that contains the previous version of `apigeectl`. For example:

   ```
   export APIGEECTL_HOME=PATH_TO_PREVIOUS_APIGEECTL_DIRECTORY
   ```
4. Undo the changes to your `overrides` file:
   1. Remove or comment out `ingressGateways` and all its properties.
   2. Set the value of `virtualhosts.selector.app` to the previous value, for example:

      ```
      virtualhosts:
        - name: my-env-group
          selector:
            app: istio-ingressgateway
      ```
   3. Remove or comment out `ao.args.disableIstioConfigInAPIServer`.
5. In the root directory of the installation you want to roll back to, run `apigeectl apply`, check the status of your pods, and then run `apigeectl init`. Be sure to use the original overrides file for the version you wish to roll back to:
   1. In the hybrid-files directory, run `apigeectl apply`:

          `$APIGEECTL_HOME`/apigeectl apply -f ORIGINAL_OVERRIDES_FILE


      Where <var translate="no">ORIGINAL_OVERRIDES_FILE</var> is relative path and filename of the overrides
      file for your previous version hybrid installation, for example,
      `./overrides/overrides1.7.yaml`.
   2. Check the status of your pods:

      ```
      kubectl -n NAMESPACE get pods
      ```


      Where <var translate="no">NAMESPACE</var> is your Apigee hybrid namespace.
   3. Check the status of `apigeeds`:

      ```
      kubectl describe apigeeds -n apigee
      ```


      Your output should look something like:

      ```
      Status:
        Cassandra Data Replication:
        Cassandra Pod Ips:
          10.8.2.204
        Cassandra Ready Replicas:  1
        Components:
          Cassandra:
            Last Successfully Released Version:
              Revision:  v1-f8aa9a82b9f69613
              Version:   v1
            Replicas:
              Available:  1
              Ready:      1
              Total:      1
              Updated:    1
            State:        running
        Scaling:
          In Progress:         false
          Operation:
          Requested Replicas:  0
        State:                 running
      ```


      Proceed to the next step only when the `apigeeds` pod is running.
   4. Run the following command to make note of what your new replica count values will be for message processor after the upgrade.If these values do not match what you have set previously, change the values in your overrides file to match your previous configuration.

      ```
      apigeectl apply -f ORIGINAL_OVERRIDES_FILE --dry-run=client --print-yaml --env ENV_NAME 2>/dev/null |grep "runtime:" -A 25 -B 1| grep "autoScaler" -A 2
      ```


      Your output should look something like:

      ```
            autoScaler:
              minReplicas: 2
              maxReplicas: 10
      ```
   5. If you are rolling back to hybrid v1.8.4 or earlier, delete the controller deployment used by hybrid v1.8.5 and newer:

          kubectl -n apigee-system delete deploy apigee-controller-manager

   6. Run `apigeectl init`:

          `$APIGEECTL_HOME`/apigeectl init -f ORIGINAL_OVERRIDES_FILE

6. Delete the Apigee ingress gateway manager deployment. This component is relevant only to Apigee hybrid versions 1.8 and newer.

   ```
   kubectl delete deployment -n NAMESPACE apigee-ingress-gateway-manager
   ```


   Where <var translate="no">NAMESPACE</var> is your Apigee hybrid namespace.