# Resolving multi-cluster issues

This section explains common Cloud Service Mesh problems and how to resolve them.
If you need additional assistance, see [Getting support](https://docs.cloud.google.com/service-mesh/v1.20/docs/getting-support.md.txt).

## Missing secrets

Cloud Service Mesh relies on a kubeconfig file embedded in the Kubernetes secret for
proper remote endpoint discovery. Without the secrets, users will always see
requests hit pods in the local cluster during cross-cluster load balancing.

Verify the secret has been created by running the following command in every
cluster:

    kubectl get secret istio-remote-secret-CLUSTER_NAME_i -n istio-system

Verify the expected output:

    NAME                                   TYPE     DATA   AGE
    istio-remote-secret-CLUSTER_NAME_i   Opaque   1      44s

To recover from this, delete all the remote secrets and re-run the `create-mesh`
command.

## Unreachable API server

The control plane of Cloud Service Mesh needs to reach the API server of the remote
cluster. The following situations can cause the remote cluster to become
unreachable:

- The remote cluster is deleted.
- The remote cluster is a [private cluster](https://docs.cloud.google.com/kubernetes-engine/docs/concepts/private-cluster-concept) that does not have [global access](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/private-clusters#enabling_control_plane_private_endpoint_global_access) enabled.
- The remote cluster is a [private cluster](https://docs.cloud.google.com/kubernetes-engine/docs/concepts/private-cluster-concept) with [Master Authorized Network](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/authorized-networks) enabled but the Cloud Service Mesh control plane IP has not been properly allowed via the allow list.

Given an unreachable API server, Istiod will output error messages in the log.
Users will always see requests hit the local pod during cross-cluster load
balancing.

In the [Logs Explorer](https://docs.cloud.google.com/logging/docs/view/logs-explorer-interface.md.txt) interface,
set the query `resource.type` to `istio_control_plane`.

Check to see if there are any invalid secret errors.

To recover from the above situation, first fix the underlying API server
reachability issue. Then, delete all the remote secrets in every cluster and
re-run the `create-mesh` command.

## Missing firewall rule

Without the proper firewall rule, users will experience a 10-second delay
followed by a timeout when doing cross-cluster load balancing.

To recover from this, follow the steps outlined in
[Create firewall rule](https://docs.cloud.google.com/service-mesh/v1.20/docs/unified-install/gke-install-multi-cluster.md.txt#create_firewall_rule).