This document provides troubleshooting guidance for user access issues
when you use third-party identity providers to authenticate to fleet member
clusters.

## `gcloud anthos create-login-config` fails to get clientconfig

This issue occurs in one of the following cases:

- The kubeconfig file passed to `gcloud anthos create-login-config` is incorrect.
- The ClientConfig custom resource is not present in the cluster (third-party authentication is not configured on the cluster).

### Error message

<br />

```
  failed to get clientconfig default in namespace kube-public
  
```

<br />

### Solution

To resolve this issue, do the following:

1. Make sure you have the correct kubeconfig file for your cluster.
2. To verify if the ClientConfig custom resource is in the cluster, run the following command:

   ```
   kubectl --kubeconfig KUBECONFIG  get clientconfig default -n kube-public
   ```

   If the ClientConfig is not present in the cluster, then install and configure
   the ClientConfig on the cluster. For more information on cluster
   setup options, see [Setup options for clusters](https://docs.cloud.google.com/kubernetes-engine/enterprise/identity.md.txt#setup_options).

## `gcloud anthos create-login-config` fails because of duplicate cluster name

This issue occurs if you attempt to create login configuration for a cluster
into a file that already contains a login configuration for this cluster.

### Error message

<br />

```
  error merging with file FILENAME because FILENAME contains a
    cluster with the same name as the one read from KUBECONFIG.
  
```

<br />

### Solution

To resolve this issue, use the `--output` flag to specify a new destination file.

If you do not provide `--output`, this login configuration data is written to a
file named `kubectl-anthos-config.yaml` in the current directory.

## `gcloud anthos auth login` fails with `proxyconnect tcp`

This issue occurs when there is an error in the `https_proxy` or `HTTPS_PROXY`
environment variable configurations. If there's an `https://` specified in the
environment variables, then the GoLang HTTP client libraries might fail if the
proxy is configured to handle HTTPS connections using other protocols such as
SOCK5.

### Error message

<br />

```
  proxyconnect tcp: tls: first record does not look like a TLS handshake
  
```

<br />

### Solution

To resolve this issue, modify the `https_proxy` and `HTTPS_PROXY` environment
variables to omit the `https:// prefix`. On Windows, modify the system
environment variables.
For example, change the value of the `https_proxy`
environment variable from `https://webproxy.example.com:8000` to
`webproxy.example.com:8000`.

## Cluster access fails when using kubeconfig generated by `gcloud anthos auth login`

This issue occurs when the Kubernetes API server is unable to authorize the
user for one of the following reasons:

- There is an error in the configuration used to login with the `gcloud anthos auth login` command.
- The necessary RBAC policies are incorrect or missing for the user.

### Error message

<br />

```
  Unauthorized
  
```

<br />

### Solution

To resolve this issue, do the following:

1. Verify configuration used to login.

   **OIDC configuration**

   The `authentication.oidc` section in the user cluster configuration file has
   `group` and `username` fields that are used to set the `--oidc-group-claim`
   and `--oidc-username-claim` flags in the Kubernetes API server. When the API
   server is presented with a user's identity token, it forwards the token to
   the cluster, which returns the extracted `group-claim` and
   `username-claim` back to the API server. The API server uses the response to
   verify that the corresponding group or user has the correct permissions.

   Verify that the claims set for `group` and `user` in the `authentication.oidc`
   section of the cluster configuration file are present in the ID token.
2. Verify applied RBAC policies.

   To learn how to set up the correct RBAC policies, see
   [Set up role-based access control (RBAC)](https://docs.cloud.google.com/kubernetes-engine/enterprise/identity/setup/user-access.md.txt#optional_set_up_role-based_access_control_rbac).

## RBACs for groups not working for Google Groups

1. Ensure that the fleet or the cluster is configured for third-party authentication.

   Follow the [setup steps](https://docs.cloud.google.com/kubernetes-engine/enterprise/identity.md.txt#setup_process) to ensure that third-party authentication is configured and enabled.
2. Ensure that you have configured Google Workspace groups correctly:

   - The `gke-security-groups` group [has been created in your Google Workspace Organization](https://docs.cloud.google.com/kubernetes-engine/enterprise/multicluster-management/gateway/setup-groups.md.txt#set_up_users_and_groups).
   - The user that you want to give RBAC access to is a member of a child group with `gke-security-groups` as its parent.

## RBACs for groups not working for OIDC providers

> [!NOTE]
> **Note:** The following troubleshooting instructions are applicable only if your provider is configured under the `oidc` section in the ClientConfig custom resource and the `enableAccessToken` field is set to `false`.

1. Verify if the ID token has the group information

   After you run `gcloud anthos auth login` command to initiate the [OIDC authentication flow](https://cloud.google.com/kubernetes-engine/enterprise/identity/accessing#plugin_to_authenticate), the ID Token is stored in the `kubeconfig` file in the `id-token` field. Use [jwt.io](https://jwt.io) to decode the ID token and verify if it contains the group information of the user as expected.
2. If ID token does not have group information of the user, then correctly configure the OIDC provider to return the group information as per the documentation of your OIDC provider. For example, if you're using OIDC configuration of the Okta Identity provider, then follow the documentation of the Okta Identity provider to configure groups in the ID token.

3. If the ID token has group information, then verify if the group information key in the ID token matches the `groupsClaim` field configured under the `oidc` section.

   For example, if the ID token contains group information in the `groups` key:

       "groups" : ["group1", "group2" ...]

   then the value of the `groupsClaim` field should be `groups` in the `oidc` section.

   After modifying the configuration in the `oidc` section, make sure you run the instructions listed in [Set up user access](https://cloud.google.com/kubernetes-engine/enterprise/identity/setup/user-access) and [Accessing clusters](https://cloud.google.com/kubernetes-engine/enterprise/identity/accessing) again.

## Troubleshoot identity providers

If you have problems using OIDC or LDAP with your cluster, follow the
steps in this section to troubleshoot these issues and help
determine if there's an issue with your identity provider configuration.

### Enable the debug log

To help troubleshoot identity-related issues in your cluster, enable the
debug logs for Pods in the `ais` Deployment.

1. Patch your existing cluster with `kubectl patch`:

   ```bash
   kubectl patch deployment ais \
     -n anthos-identity-service --type=json \
     -p='[{"op": "add", "path": "/spec/template/spec/containers/0/args/-", "value":"--vmodule=cloud/identity/hybrid/charon/*=LOG_LEVEL"}]' \
     --kubeconfig KUBECONFIG
   ```

   Replace the following:
   - `LOG_LEVEL`: For the most verbose logs, set this value
     to level `3` when troubleshooting.

     > [!NOTE]
     > **Note:** This value can be set higher to `9`, however at this level it might reveal some credential information in your console.

   - `KUBECONFIG`: The path to your user cluster kubeconfig
     file.

### Check the container log

Review the content of the container logs for any errors or warnings.

1. To review the logs, use `kubectl logs`:

   ```bash
   kubectl logs -f -l k8s-app=ais \
     -n anthos-identity-service \
     --kubeconfig KUBECONFIG
   ```

   Replace `KUBECONFIG` with the path to your user cluster
   kubeconfig file.

### Restart the Pod

If the container logs show problems, restart the Pod.

1. To restart the Pod, delete the existing Pod. A new
   Pod is automatically created as a replacement.

   ```bash
   kubectl delete pod -l k8s-app=ais \
     -n anthos-identity-service \
     --kubeconfig KUBECONFIG
   ```

   Replace `KUBECONFIG` with the path to your user cluster
   kubeconfig file.

### Troubleshoot connectivity to identity provider

If the Pod looks to be running correctly, test the
connectivity to the remote identity provider.

1. Start a busybox Pod in the same namespace:

   ```bash
   kubectl run curl --image=radial/busyboxplus:curl \
     -n anthos-identity-service -- sleep 3000 \
     --kubeconfig KUBECONFIG
   ```

   Replace `KUBECONFIG` with the path to your user cluster
   kubeconfig file.
2. To check if you can fetch the discovery URL, execute into the busybox Pod and
   run the `curl` command:

   ```bash
   kubectl exec pod/curl -n anthos-identity-service -- \
     curl ISSUER_URL \
     --kubeconfig KUBECONFIG
   ```

   Replace the following:
   - `ISSUER_URL`: The issuer URL of your identity provider.
   - `KUBECONFIG`: The path to your user cluster kubeconfig file.

   A successful response is a JSON result with the detailed identity provider
   endpoints.
3. If the previous command doesn't return the expected result, contact your
   identity provider administrator for additional assistance.

## LDAP login not working for Google Distributed Cloud admin cluster

LDAP is currently only supported for Google Distributed Cloud user cluster.