# Integrating IAP with Cloud Service Mesh

This guide describes how to integrate
[Identity-Aware Proxy (IAP)](https://docs.cloud.google.com/iap) with Cloud Service Mesh. The
IAP integration with Cloud Service Mesh
enables you to safely access services based on Google's
[BeyondCorp principles](https://cloud.google.com/beyondcorp). IAP verifies user identity
and context of the request to determine if a user should be allowed to access an
application or resource. The IAP integration with Cloud Service Mesh
provides you with the following benefits:

- Complete [context-aware access](https://docs.cloud.google.com/context-aware-access) control to the
  workloads running on Cloud Service Mesh. You can set fine-grained
  [access policies](https://docs.cloud.google.com/access-context-manager/docs/overview.md.txt#access-levels)
  based on attributes of the originating request, such as user identity, the IP
  address, and device type. You can combine your access policies with
  restrictions based on the
  [hostname and path of a request URL](https://docs.cloud.google.com/iap/docs/cloud-iap-context-aware-access-howto.md.txt#using_hostname_and_path_conditions).

- Enable support for context-aware claims in Cloud Service Mesh authorization.

- Scalable, secure, and highly available access to your application through a
  Google Cloud load balancer. High performance load balancing provides built-in
  protection of distributed denial-of-service (DDoS) attacks and support for
  global [anycast IP addressing](https://wikipedia.org/wiki/Anycast).

## Prerequisites

> [!WARNING]
> **Warning:** HTTP Load Balancing with Cloud Service Mesh requires GKE 1.17.6-gke.11 or later.

Follow the steps in [Install dependent tools and validate cluster](https://docs.cloud.google.com/service-mesh/v1.28/docs/onboarding/kubernetes-off-gcp/install/install-dependent-tools.md.txt) to:

- [Install required tools](https://docs.cloud.google.com/service-mesh/v1.28/docs/onboarding/kubernetes-off-gcp/install/install-dependent-tools.md.txt#install_required_tools)
- [Download `asmcli`](https://docs.cloud.google.com/service-mesh/v1.28/docs/onboarding/kubernetes-off-gcp/install/install-dependent-tools.md.txt#download_asmcli)
- [Grant cluster admin permissions](https://docs.cloud.google.com/service-mesh/v1.28/docs/onboarding/kubernetes-off-gcp/install/install-dependent-tools.md.txt#grant_cluster_admin_permissions)
- [Validate your project and cluster](https://docs.cloud.google.com/service-mesh/v1.28/docs/onboarding/kubernetes-off-gcp/install/install-dependent-tools.md.txt#validate_project_and_cluster)

Additionally, this guide assumes that you have:

- [A Google Cloud project](https://docs.cloud.google.com/resource-manager/docs/creating-managing-projects.md.txt).
- [A Cloud Billing account](https://docs.cloud.google.com/billing/docs/how-to/manage-billing-account.md.txt).

## Setting up a cluster with Cloud Service Mesh

This section explains how to get set up for the IAP integration
for both new installations of Cloud Service Mesh and upgrades.

### New installations

1. Enable `iap.googleapis.com`. In the following command, replace
   `PROJECT_ID`with the project that you will install
   Cloud Service Mesh in:

       gcloud services enable \
         --project=PROJECT_ID \
         iap.googleapis.com

2. The cluster that you are updating must have the `--addons=HttpLoadBalancing`
   option set. The `HttpLoadBalancing` add-on enables an HTTP (L7) load
   balancing controller for the cluster. Run the following command to
   update the cluster with the options required by Cloud Service Mesh. Unless you've
   set a default zone or region, you need to supply the region
   (--region=REGION) or zone (--zone=ZONE) in the command.

       gcloud container clusters update CLUSTER_NAME \
         --project=PROJECT_ID \
         --update-addons=HttpLoadBalancing=ENABLED

3. By default, the `iap-operator.yaml` file has port 31223 set as the status
   port, and port 31224 set as the http port. If port 31223 is already in use
   in your cluster, run the following to set another status port:

   > [!IMPORTANT]
   > **Important:** You must use a `kpt` version earlier than v1.0.

       kpt cfg set asm gcloud.container.cluster.ingress.statusPort STATUS_PORT

   If port 31224 is already in use in your cluster, run the following to set
   another http port:

       kpt cfg set asm gcloud.container.cluster.ingress.httpPort HTTP_PORT

4. Follow the steps in
   [Install default features and Mesh CA](https://docs.cloud.google.com/service-mesh/v1.28/docs/unified-install/install-anthos-service-mesh#install_mesh_ca)
   to use a Google-provided script to install Cloud Service Mesh. When you run the
   script, include the following option:

       --option iap-operator

   For example:

       ./asmcli install \
         --project_id "PROJECT_ID" \
         --cluster_name "CLUSTER_NAME" \
         --cluster_location "CLUSTER_LOCATION" \
         --fleet_id FLEET_PROJECT_ID \
         --output_dir DIR_PATH \
         --enable_all \
         --option iap-operator

   When you install Cloud Service Mesh, the
   [`iap-operator.yaml`](https://github.com/GoogleCloudPlatform/anthos-service-mesh-packages/blob/main/asm/istio/options/iap-operator.yaml)
   file sets the `type` field on the `istio-ingressgateway` service to
   `NodePort`, which configures the gateway to open a specific port on the
   service mesh. This allows you to set up a load balancer, which routes
   traffic sent to your domain name to this port.
5. If you are installing managed Cloud Service Mesh, then also complete the
   following steps:

   1. [Add the revision label](https://docs.cloud.google.com/service-mesh/v1.28/docs/managed/select-a-release-channel#injection_labels)
      to the `istio-system` namespace.

   2. Download the
      [Istio ingress gateway service spec for IAP](https://github.com/GoogleCloudPlatform/anthos-service-mesh-packages/blob/main/asm/istio/options/iap-operator.yaml)
      and name it `iap_operator.yaml`.

   3. Install the ingress as a NodePort service.

          asmcli experimental mcp-migrate-check -f iap_operator.yaml

          istioctl install -f /asm-generated-configs/gateways-istiooperator/"GATEWAY_NAME".yaml

After installing Cloud Service Mesh, return to this guide and continue with the
next section to set up the integration with IAP.

### Upgrades

This section covers the following upgrade use cases:

- You have already set up the IAP integration and you are
  upgrading Cloud Service Mesh. In this case, you have already enabled
  `iap.googleapis.com` on your project and the `HttpLoadBalancing` add-on
  on your cluster. Skip to step 3, to download the `asm` package and
  upgrade Cloud Service Mesh.

- You are upgrading Cloud Service Mesh and you want to set up the integration with
  IAP for the first time. In this case, you need to complete
  all of the following steps, upgrade Cloud Service Mesh, and return to this guide
  after the upgrade to complete the integration.

1. Enable `iap.googleapis.com`. In the following command, replace
   `PROJECT_ID`with the project that you will install
   Cloud Service Mesh in.

       gcloud services enable \
         --project=PROJECT_ID \
         iap.googleapis.com

2. The cluster that you are updating must have the `--addons=HttpLoadBalancing`
   option set. The `HttpLoadBalancing` add-on enables an HTTP (L7) load
   balancing controller for the cluster. Run the following command to
   update the cluster with the options required by Cloud Service Mesh. Unless you've
   set a default zone or region, you need to supply the region
   (--region=REGION) or zone (--zone=ZONE) in the command.

       gcloud container clusters update CLUSTER_NAME \
         --project=PROJECT_ID
         --update-addons=HttpLoadBalancing=ENABLED

3. If you are updating an existing, functioning HTTP Cloud Load Balancer, run
   the following to preserve your existing http and status ports:

   > [!IMPORTANT]
   > **Important:** You must use a `kpt` version earlier than v1.0.

       kpt cfg set asm gcloud.container.cluster.ingress.httpPort $(kubectl -n istio-system get service istio-ingressgateway -o jsonpath='{.spec.ports[?(@.name=="http2")].nodePort}')

       kpt cfg set asm gcloud.container.cluster.ingress.statusPort $(kubectl -n istio-system get service istio-ingressgateway -o jsonpath='{.spec.ports[?(@.name=="status-port")].nodePort}')

4. Follow the steps in
   [Upgrading Cloud Service Mesh](https://docs.cloud.google.com/service-mesh/v1.28/docs/unified-install/upgrade)
   to use a Google-provided script to upgrade Cloud Service Mesh.

   When you upgrade Cloud Service Mesh, the `iap-operator.yaml` file sets the
   `type` field on the `istio-ingressgateway` service to `NodePort`, which
   configures the gateway to open a specific port on the service mesh. This
   allows you to set up a load balancer, which routes traffic sent to your
   domain name to this port.

   By default, the `iap-operator.yaml` file has port 31223 set as the status
   port, and port 31224 set as the http port.

   When you run the script, include the following option:

       --option iap-operator

   For example:

       ./asmcli install \
         --project_id "PROJECT_ID" \
         --cluster_name "CLUSTER_NAME" \
         --cluster_location "CLUSTER_LOCATION" \
         --fleet_id FLEET_PROJECT_ID \
         --output_dir DIR_PATH \
         --enable_all \
         --option iap-operator

5. Complete the upgrade by triggering automatic sidecar proxy injection on
   your workloads. For details, see
   [Deploying and redeploying workloads](https://docs.cloud.google.com/service-mesh/v1.28/docs/unified-install/upgrade#deploying_and_redeploying_workloads).

   After completing the upgrade, return to this guide and continue with the
   next section to set up the integration with IAP.

## Reserving a static IP address and configuring DNS

To integrate Identity-Aware Proxy with Cloud Service Mesh, you have to set up a
Google Cloud HTTP(S) load balancer, which requires a domain name that points
to a static IP address. You can
[reserve a static external IP address](https://docs.cloud.google.com/compute/docs/ip-addresses.md.txt#reservedaddress),
which assigns the address to your project indefinitely until you explicitly
release it.

1. Reserve a static external IP address:

       gcloud compute addresses create example-static-ip --global

2. Get the static IP address:

       gcloud compute addresses describe example-static-ip --global

3. In your domain name registrar, configure a fully qualified domain name
   (FQDN) with the static IP address. Typically, you add an `A` record to your
   DNS settings. The configuration steps and terminology for adding an `A`
   record for a FQDN vary depending on your domain name registrar.

   It can take 24 to 48 hours for the DNS setting to propagate. You can
   continue setting up everything in this guide, but you won't be able to
   test the setup until the DNS settings propagate.

## Deploying a sample application

Before you enable IAP, you need an application running on
your GKE cluster so you can verify that all requests have an
identity. This guide uses the Bookinfo sample to demonstrate how to setup the
HTTP(S) load balancer and enable IAP.

Follow the steps to [deploy Bookinfo](https://docs.cloud.google.com/service-mesh/v1.28/docs/deploy-bookinfo.md.txt). Until
you deploy the load balancer, the Bookinfo application isn't accessible outside
of your GKE cluster (such as from a browser).

### External requests

Bookinfo's Gateway resource (defined in
`samples/bookinfo/networking/bookinfo-gateway.yaml`) uses the preconfigured
[`istio-ingressgateway`](https://archive.istio.io/v1.5/docs/concepts/traffic-management/#gateways).
Recall that when you deployed Cloud Service Mesh, you specified `NodePort` for the
`istio-ingressgateway`, which opens a specific port on the service mesh.
Although the nodes in your cluster have external IP addresses, requests coming
from outside your cluster are blocked by Google Cloud firewall rules. With
IAP, the correct way to expose applications to the public
internet is by using a load balancer. Don't expose the node addresses using
firewall rules, which would bypass IAP.

To route requests to Bookinfo, you set up an HTTP(S) load balancer in your
Google Cloud project. Because the load balancer is in your project, it is
inside of the firewall and can access the nodes in your cluster. After you
configure the load balancer with the static IP address and your domain name,
you can send requests to the domain name, and the load balancer forwards the
requests to the nodes in the cluster.

## Enabling IAP

The following steps describe how to enable IAP.

> [!NOTE]
> **Note:** OAuth clients for IAP can be created using gcloud CLI, but there are a few limitations to how these clients can be used. Learn more [here](https://docs.cloud.google.com/iap/docs/programmatic-oauth-clients.md.txt). Alternatively, you can follow [these](https://docs.cloud.google.com/iap/docs/enabling-kubernetes-howto.md.txt#oauth-configure) steps to create a client with more relaxed restrictions.

### Configuring consent screen

1. Check if you already have an existing brand by using the
   [list](https://docs.cloud.google.com/sdk/gcloud/reference/iap/oauth-brands/list) command. You may
   only have one brand per project.

   ```bash
   gcloud iap oauth-brands list
   ```

   The following is an example gcloud response, if the brand exists:

       name: projects/[PROJECT_NUMBER]/brands/[BRAND_ID]
       applicationTitle: [APPLICATION_TITLE]
       supportEmail: [SUPPORT_EMAIL]
       orgInternalOnly: true

   > [!NOTE]
   > **Note:** If a brand already exists for a project and has been configured for external users (`orgInternalOnly: false`), but you want to restrict it to internal users, you must make that change manually from the [OAuth consent screen](https://console.cloud.google.com/auth/audience) in order to create OAuth clients with this API.

2. If no brand exists, use the
   [create](https://docs.cloud.google.com/sdk/gcloud/reference/iap/oauth-brands/create) command:

   ```bash
   gcloud iap oauth-brands create --application_title=APPLICATION_TITLE --support_email=SUPPORT_EMAIL
   ```


   The above fields are required when calling this API:
   - `supportEmail`: The support email displayed on the OAuth consent screen.
     This email address can either be a user's address or a Google Groups alias.
     While service accounts also have an email address, they are not actual
     valid email addresses, and cannot be used when creating a brand. However,
     a service account can be the owner of a Google Group. Either create a
     new Google Group or configure an existing group and set the desired service
     account as an owner of the group.

     > [!NOTE]
     > **Note:** The user issuing the request must be an owner of the specified support email address.

   - `applicationTitle`: The application name displayed on OAuth consent
     screen.

   The response contains the following fields:

       name: projects/[PROJECT_NUMBER]/brands/[BRAND_ID]
       applicationTitle: [APPLICATION_TITLE]
       supportEmail: [SUPPORT_EMAIL]
       orgInternalOnly: true

### Creating an IAP OAuth Client

1. Use the create command to
   [create](https://docs.cloud.google.com/sdk/gcloud/reference/iap/oauth-clients/create) a client. Use
   the brand `name` from previous step.

   ```bash
   gcloud iap oauth-clients create projects/PROJECT_NUMBER/brands/BRAND-ID --display_name=NAME
   ```

   The response contains the following fields:

       name: projects/[PROJECT_NUMBER]/brands/[BRAND_NAME]/identityAwareProxyClients/[CLIENT_ID]
       secret: [CLIENT_SECRET]
       displayName: [NAME]

2. Use the client ID (`CLIENT_ID` in the above step) and `CLIENT_SECRET` to
   enable IAP. Create a kubernetes secret with the materials
   from your OAuth Client:

   ```
   kubectl create secret generic -n istio-system my-secret --from-literal=client_id=CLIENT_ID \
   --from-literal=client_secret=CLIENT_SECRET
   ```

## Deploying the load balancer

You can use an
[Ingress resource](https://docs.cloud.google.com/kubernetes-engine/docs/concepts/ingress.md.txt) to create an
[HTTP(S) load balancer](https://docs.cloud.google.com/load-balancing/docs/https.md.txt) with automatically configured
SSL certificates.
[Managed SSL certificates](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/managed-certs) are
provisioned, renewed, and managed for your domain.

1. Create a ManagedCertificate resource. This resource specifies the domain for
   the SSL certificate. The `spec.domains` list must contain only one domain.
   Wildcard domains aren't supported. In the following YAML, replace
   `DOMAIN_NAME` with the domain name that you configured
   for the external static IP address.

   ```
   cat <<EOF | kubectl apply -f -
   apiVersion: networking.gke.io/v1
   kind: ManagedCertificate
   metadata:
     name: example-certificate
     namespace: istio-system
   spec:
     domains:
       - DOMAIN_NAME
   EOF
   ```
2. Create a BackendConfig resource. This resource instructs GCLB how to perform
   Health Checks on the Ingress Gateway, as well as configure Identity-Aware
   Proxy. First, collect a few values from the Ingress Gateway about health
   checks:

   - Health check ingress port: This is the health check port of istio-ingress.

     ```
     export HC_INGRESS_PORT=$(kubectl -n istio-system get service istio-ingressgateway -o jsonpath='{.spec.ports[?(@.name=="status-port")].nodePort}')
     ```

     <br />

   - Health check ingress path: This is the health check path of istio-ingress.

     ```
     export HC_INGRESS_PATH=$(kubectl get pods -n istio-system -l app=istio-ingressgateway -o jsonpath='{.items[0].spec.containers[?(@.name=="istio-proxy")].readinessProbe.httpGet.path}')
     ```

     <br />

     <br />

   ```
   cat <<EOF | kubectl apply -n istio-system -f -
   apiVersion: cloud.google.com/v1
   kind: BackendConfig
   metadata:
     name: http-hc-config
   spec:
     healthCheck:
       checkIntervalSec: 2
       timeoutSec: 1
       healthyThreshold: 1
       unhealthyThreshold: 10
       port: ${HC_INGRESS_PORT}
       type: HTTP
       requestPath: ${HC_INGRESS_PATH}
     iap:
       enabled: true
       oauthclientCredentials:
         secretName: my-secret
   EOF
   ```
3. Annotate the ingress service with your BackendConfig.

           kubectl annotate -n istio-system service/istio-ingressgateway --overwrite \
             cloud.google.com/backend-config='{"default": "http-hc-config"}' \
             cloud.google.com/neg='{"ingress":false}'

4. Create the load balancer by defining the Ingress resource.

   - Set the `networking.gke.io/managed-certificates` annotation to the name
     of the certificate you created in the previous step,
     `example-certificate`.

   - Set the `kubernetes.io/ingress.global-static-ip-name` annotation to the
     name of the static IP address you reserved, `example-static-ip`.

   - Set the `serviceName` to `istio-ingressgateway`, which is used in
     the Gateway resource for the Bookinfo sample.

   ```
   cat <<EOF | kubectl apply -f -
   apiVersion: networking.k8s.io/v1
   kind: Ingress
   metadata:
     name: example-ingress
     namespace: istio-system
     annotations:
       kubernetes.io/ingress.global-static-ip-name: example-static-ip
       networking.gke.io/managed-certificates: example-certificate
   spec:
     defaultBackend:
       service:
         name: istio-ingressgateway
         port:
           number: 80
   EOF
   ```

   > [!NOTE]
   > **Note:** It can take up to 10-20 minutes for the load balancer to be functional.

5. In the Google Cloud console, go to the **Kubernetes Engine** \> **Services
   \& Ingress** page.

   [Go to the Services \& Ingress page](https://console.cloud.google.com/kubernetes/discovery)

   You should see the "Creating ingress" message in the **Status** column.
   Wait for GKE to fully provision the Ingress before
   continuing. Refresh the page every few minutes to get the most up-to-date
   status on the Ingress. After the Ingress is provisioned, you might see the
   "Ok" status, or the error "All backend services are in UNHEALTHY state."
   One of the resources that GKE provisions is a default health
   check. If you see the error message, that indicates that the Ingress is
   provisioned and that the default health check ran. When you see either the
   "Ok" status or the error, continue with the next section to configure the
   health checks for the load balancer.

### Configure the IAP access list

Add a user to the access policy for IAP:

> [!CAUTION]
> **Caution:** The below commands configure IAP settings and IAP policies at the project level. Therefore, the configurations will be applicable to all IAP resources in the project, including non-Cloud Service Mesh resources. This is required because the resources are deleted and recreated with every upgrade, and configuring IAP at project level allows us to persist IAP configuration across upgrades.

```
gcloud beta iap web add-iam-policy-binding \
    --member=user:EMAIL_ADDRESS \
    --role=roles/iap.httpsResourceAccessor
```

where `EMAIL_ADDRESS` is the user's full email address
such as `alice@example.com`.

1. Test the load balancer. Point your browser to:

   ```
   http://DOMAIN_NAME/productpage
   ```

   where `DOMAIN_NAME` is the domain name that you
   configured with the external static IP address.

   You should see the Bookinfo application's `productpage`. If you refresh the
   page several times, you should see different versions of reviews, presented
   in a round robin style: red stars, black stars, no stars.

   You should also test `https` access to Bookinfo.

## Enable RCToken support on the service mesh

By default, IAP generates a
[JSON Web Token](https://jwt.io/) (JWT) that is scoped to the OAuth client.
For Cloud Service Mesh, you can configure IAP to generate a
RequestContextToken (RCToken), which is a JWT but with a configurable audience.
RCToken lets you configure the audience of the JWT to an arbitrary string, which
can be used in the Cloud Service Mesh policies for fine-grained authorization.

To configure the RCToken:

1. Create an environment variable for the RCToken audience. This can be
   any string that you want.

       export RCTOKEN_AUD="your-rctoken-aud"

2. Optional: The following step requires the
   <var translate="no">`BACKEND_SERVICE_ID`</var>. If you need to find out the
   <var translate="no">`BACKEND_SERVICE_ID`</var>, run the following command:

       kubectl -n istio-system get Ingress example-ingress -o json | jq \
        '.metadata.annotations."ingress.kubernetes.io/backends"'

   The expected output is similar to `"{\"BACKEND_SERVICE_ID\":\"HEALTHY\"}"`.
   For example,
   `"ingress.kubernetes.io/backends": "{\"k8s-be-31224--51f3b55cd1457fb6\":\"HEALTHY\"}"`.
   The `BACKEND_SERVICE_ID` in this example is `k8s-be-31224--51f3b55cd1457fb6`.
3. Fetch the existing [IAP settings](https://docs.cloud.google.com/iap/docs/reference/rest/v1/IapSettings.md.txt).

   ```
   gcloud iap settings get --format json \
   --project=${PROJECT_ID} --resource-type=compute --service=BACKEND_SERVICE_ID > iapSettings.json
   ```
4. Update `IapSettings` with the RCToken audience.

   <br />

       cat iapSettings.json | jq --arg RCTOKEN_AUD_STR $RCTOKEN_AUD \
       '. + {applicationSettings: {csmSettings: {rctokenAud: $RCTOKEN_AUD_STR}}}' \
       > updatedIapSettings.json

   ```
   gcloud iap settings set updatedIapSettings.json --format json \
   --project=${PROJECT_ID} --resource-type=compute --service=BACKEND_SERVICE_ID
   ```
5. Enable RCToken authentication on the Istio ingress gateway.

   ```yaml
   cat <<EOF | kubectl apply -f -
   apiVersion: "security.istio.io/v1beta1"
   kind: "RequestAuthentication"
   metadata:
     name: "ingressgateway-jwt-policy"
     namespace: "istio-system"
   spec:
     selector:
       matchLabels:
         app: istio-ingressgateway
     jwtRules:
     - issuer: "https://cloud.google.com/iap"
       jwksUri: "https://www.gstatic.com/iap/verify/public_key-jwk"
       audiences:
       - $RCTOKEN_AUD
       fromHeaders:
       - name: ingress-authorization
         prefix: "Istio "
       outputPayloadToHeader: "verified-jwt"
       forwardOriginalToken: true
   EOF
   ```
6. Optional: Ensure requests that don't have valid JWTs are rejected:

   > [!NOTE]
   > **Note:** For more information, see [Require a valid token](https://istio.io/latest/docs/tasks/security/authentication/authn-policy/#require-a-valid-token).

   <br />

   ```
     cat <<EOF | kubectl apply -f -
     apiVersion: security.istio.io/v1beta1
     kind: AuthorizationPolicy
     metadata:
       name: iap-gateway-require-jwt
       namespace: istio-system
     spec:
       selector:
         matchLabels:
           app: istio-iap-ingressgateway
       action: DENY
       rules:
         - from:
             - source:
                 notRequestPrincipals: ["*"]
     EOF
     
   ```

   <br />

7. Make sure requests to the Bookinfo `productpage` are still successful:

   ```
   http://DOMAIN_NAME/productpage
   ```

To test the policy:

1. Create an [`IapSettings`](https://docs.cloud.google.com/iap/docs/reference/rest/v1/IapSettings.md.txt) request
   object, but set the `rctokenAud` to a different string:

       cat iapSettings.json | jq --arg RCTOKEN_AUD_STR wrong-rctoken-aud \
       '. + {applicationSettings: {csmSettings: {rctokenAud: $RCTOKEN_AUD_STR}}}' \
       > wrongIapSettings.json

2. Call the `IapSettings` API to set the RCtoken audience.

   ```
   gcloud beta iap settings set wrongIapSettings.json --project=PROJECT_ID --resource-type=compute --service=BACKEND_SERVICE
   ```
3. Make a request to the Bookinfo `productpage` and it should fail:

   ```
   http://DOMAIN_NAME/productpage
   ```

## Cleaning up

After completing this tutorial, remove the following resources to prevent
unwanted charges incurring on your account:

1. Delete the managed certificate:

   ```bash
   kubectl delete managedcertificates example-certificate
   ```
2. Delete the Ingress, which deallocates the load balancing resources:

   ```
   kubectl -n istio-system delete ingress example-ingress
   ```

   <br />

3. Delete the static IP address:

   ```
   gcloud compute addresses delete example-static-ip --global
   ```

   <br />

   If you do this, be sure to delete the IP address from your domain registrar.
4. Delete the cluster, which deletes the resources that make up the
   cluster, such as the compute instances, disks and network resources:

   ```
   gcloud container clusters delete CLUSTER_NAME
   ```

   <br />