The GKE On-Prem API lets you manage on-premises cluster lifecycles by using the
Google Cloud console, the Google Cloud CLI, or Terraform. When you create a cluster
with these tools, the API stores cluster metadata in your chosen Google Cloud
region. If you create clusters with `bmctl`, you must enroll them in
the GKE On-Prem API to view details and manage their lifecycles through these
interfaces.

Enrolling a cluster is separate from registering it to a
[fleet](https://docs.cloud.google.com/kubernetes-engine/fleet-management/docs.md.txt). All Google Distributed Cloud
clusters automatically register to a
[fleet host project](https://docs.cloud.google.com/kubernetes-engine/fleet-management/docs/fleet-concepts.md.txt#fleet-host-project)
at creation. In the Google Cloud console, registered clusters managed by the
GKE On-Prem API show as

**Bare metal**

in the **Type** column, whereas unenrolled clusters appear as **External**.

## Requirements

- Only user and admin clusters can be enrolled with the GKE On-Prem API. Enrolling hybrid and standalone clusters isn't supported.
- Version 1.13 or higher.
- Enrollment of user clusters created under custom namespaces isn't supported.
  The generated user cluster configuration file uses the default cluster
  namespace of the form `cluster-CLUSTER_NAME`. If you change
  the namespace when you create the cluster, attempts to enroll the cluster with
  the GKE On-Prem API are blocked.

- If you aren't a project owner, minimally, you must be granted the Identity and Access Management
  role `roles/gkeonprem.admin` on the project. For details on the permissions
  included in this role, see
  [GKE on-prem roles](https://docs.cloud.google.com/iam/docs/roles-permissions/gkeonprem.md.txt)
  in the IAM documentation.

## Before you begin

1. Set up the gcloud CLI, if needed.

   1. If you need to install the gcloud CLI, see the
      [gcloud CLI documentation](https://docs.cloud.google.com/sdk/docs/install). Update the
      gcloud CLI components, if needed:

      ```bash
      gcloud components update
      ```
2. Do the following steps to activate and use the GKE On-Prem API:

   1. Enable the API in your project:

      ```bash
      gcloud services enable \
          --project PROJECT_ID \
          gkeonprem.googleapis.com
      ```

      Replace `PROJECT_ID` with the project ID of
      your fleet host project. This is the project ID that was configured in the
      `gkeconnect` section of your
      [cluster configuration file](https://docs.cloud.google.com/kubernetes-engine/distributed-cloud/bare-metal/docs/reference/cluster-config-ref.md.txt#gkeconnect-projectid).

      If you get a `PERMISSION_DENIED` error, double check the project ID
      that you entered. If the project ID is correct, run `gcloud auth login`
      to sign in to the Google Cloud CLI with the account that has access to the
      project.
   2. If this is the first time that you have enabled the GKE On-Prem API
      in your project, you need to
      [initialize the API](https://docs.cloud.google.com/iam/docs/service-agents.md.txt). You can do this
      by calling a gcloud CLI command that displays available
      versions that you can use to create a cluster:

      ```bash
      gcloud container bare-metal clusters query-version-config \
          --project=PROJECT_ID \
          --location=REGION
      ```

      Replace `REGION` with `us-west1` or another
      [supported region](https://docs.cloud.google.com/kubernetes-engine/distributed-cloud/bare-metal/docs/reference/supported-regions-on-prem-api.md.txt).
3. If your organization has set up an allowlist that lets traffic from
   Google APIs and other addresses pass through your
   [proxy server](https://docs.cloud.google.com/kubernetes-engine/distributed-cloud/bare-metal/docs/installing/proxy.md.txt), add the following to the
   allowlist:

   - `gkeonprem.googleapis.com`
   - `gkeonprem.mtls.googleapis.com`

   These are the service names for the GKE On-Prem API.

### Enroll a user cluster

### gcloud CLI

1. List the fleet memberships of registered clusters:

   ```bash
   gcloud container fleet memberships list \
       --project=PROJECT_ID
   ```

   Replace `PROJECT_ID` with the project ID of
   your fleet host project.

   Note the region in which the admin cluster that manages the
   user cluster is a member because you will need this in the next step.
2. Enroll the cluster in the GKE On-Prem API:

   ```bash
   gcloud container bare-metal clusters enroll USER_CLUSTER_NAME  \
       --project=PROJECT_ID \
       --admin-cluster-membership=projects/PROJECT_ID/locations/MEMBERSHIP_REGION/memberships/ADMIN_CLUSTER_NAME \
       --location=REGION
   ```

   Replace the following:
   - `USER_CLUSTER_NAME`: the name of the user cluster
     that you want to enroll.

   - `PROJECT_ID`: the project ID of your fleet host project.

   - `MEMBERSHIP_REGION`: the region in which the
     admin cluster that manages the user cluster is a member. This could be
     `global` or a Google Cloud region.

   - `ADMIN_CLUSTER_NAME`: the name of the admin cluster
     that manages the user cluster.

     The `ADMIN_CLUSTER_NAME`,
     `MEMBERSHIP_REGION`, and
     `PROJECT_ID` form the fully-specified cluster name
     for the `--admin-cluster-membership` flag.
   - `REGION`: the Google Cloud region in which
     the GKE On-Prem API runs and stores cluster metadata. Specify `us-west1`
     or another
     [supported region](https://docs.cloud.google.com/kubernetes-engine/distributed-cloud/bare-metal/docs/reference/supported-regions-on-prem-api.md.txt).
     The region can't be changed after the cluster is enrolled.

### `bmctl`

Do the following steps on your admin workstation.

1. Add the following section to the user cluster configuration file:

   ```yaml
   gkeOnPremAPI:
     enabled: true
     location: REGION
   ```

   Replace `REGION` with the Google Cloud region in
   which the GKE On-Prem API runs and stores cluster metadata. Specify
   `us-west1` or another
   [supported region](https://docs.cloud.google.com/kubernetes-engine/distributed-cloud/bare-metal/docs/reference/supported-regions-on-prem-api.md.txt).
   The region can't be changed after the cluster is enrolled.
2. Update the cluster:

   ```
   bmctl update cluster -c USER_CLUSTER_NAME \
     --kubeconfig ADMIN_CLUSTER_KUBECONFIG
   ```

   Replace the following:
   - `USER_CLUSTER_NAME`: the name of the user cluster
     to update.

   - `ADMIN_CLUSTER_KUBECONFIG`: the path of the admin
     cluster kubeconfig file.

### Enroll an admin cluster

### gcloud CLI

1. List the fleet memberships of registered clusters:

   ```bash
   gcloud container fleet memberships list \
       --project=PROJECT_ID
   ```

   Replace `PROJECT_ID` with the project ID of
   your fleet host project.

   Note the region in which the admin cluster is a member because
   you will need this in the next step.
2. Enroll the admin cluster in the GKE On-Prem API:

   ```bash
   gcloud container bare-metal admin-clusters enroll ADMIN_CLUSTER_NAME \
       --project=PROJECT_ID  \
       --admin-cluster-membership=projects/PROJECT_ID /locations/MEMBERSHIP_REGION /memberships/ADMIN_CLUSTER_NAME  \
       --location=REGION
   ```

Replace the following:

- `ADMIN_CLUSTER_NAME`: the name of the admin cluster
  that you want to enroll.

- `PROJECT_ID` the project ID of your fleet host project.

- `MEMBERSHIP_REGION`: the region in which the admin
  cluster is a member. This could be `global` or a Google Cloud region.

  The `ADMIN_CLUSTER_NAME`,
  `MEMBERSHIP_REGION`, and
  `PROJECT_ID` form the fully-specified cluster name for
  the `--admin-cluster-membership` flag.
- `REGION`: the Google Cloud region in which
  the GKE On-Prem API runs and stores cluster metadata. Specify `us-west1`
  or another
  [supported region](https://docs.cloud.google.com/kubernetes-engine/distributed-cloud/bare-metal/docs/reference/supported-regions-on-prem-api.md.txt).
  The region can't be changed after the cluster is enrolled.

### `bmctl`

Do the following steps on your admin workstation.

1. Add the following section to the admin cluster configuration file:

   ```yaml
   gkeOnPremAPI:
     enabled: true
     location: REGION
   ```

   Replace `REGION` with the Google Cloud region in
   which the GKE On-Prem API runs and stores cluster metadata. Specify
   `us-west1` or another
   [supported region](https://docs.cloud.google.com/kubernetes-engine/distributed-cloud/bare-metal/docs/reference/supported-regions-on-prem-api.md.txt).
   The region can't be changed after the cluster is enrolled.
2. Update the cluster:

   ```
   bmctl update cluster -c ADMIN_CLUSTER_NAME \
   --kubeconfig ADMIN_CLUSTER_KUBECONFIG
   ```

   Replace the following:
   - `ADMIN_CLUSTER_NAME`: the name of the admin cluster
     to update.

   - `ADMIN_CLUSTER_KUBECONFIG`: the path of the admin
     cluster kubeconfig file.

## View registered clusters

All your fleet clusters are displayed on the
[GKE Clusters](https://console.cloud.google.com/kubernetes/list/overview)
pages in the console. This both gives you an overview of your
entire fleet and, for Google Distributed Cloud, lets you see which clusters are
managed by the GKE On-Prem API.

To view your fleet clusters:

1. In the console, go to the GKE clusters page.  
   [Go to GKE clusters](https://console.cloud.google.com/kubernetes/list/overview)
2. Select the Google Cloud project.
   - If **Bare metal** is displayed in the **Type** column, the cluster is managed by the GKE On-Prem API.
   - If **External** is displayed in the **Type** column, the cluster isn't managed by the GKE On-Prem API.

## Get information about your cluster

After the cluster is enrolled, you can use the following commands to get
information about your clusters:

### User cluster

- To describe a user cluster:

```
gcloud container bare-metal clusters describe USER_CLUSTER_NAME \
    --project=PROJECT_ID \
    --location=REGION
```

- To list your user clusters:

```
gcloud container bare-metal clusters list \
    --project=PROJECT_ID \
    --location=-
```

When you set `--location=-`, that means to list all clusters in all
regions. If you need to scope down the list, set `--location` to the region
you specified when you enrolled the cluster.

### Admin cluster

- To describe an admin cluster:

```
gcloud container bare-metal admin-clusters describe ADMIN_CLUSTER_NAME \
    --project=PROJECT_ID \
    --location=REGION
```

- To list your admin clusters:

```
gcloud container bare-metal admin-clusters list \
    --project=PROJECT_ID \
    --location=-
```

When you set `--location=-`, that means to list all clusters in all
regions. If you need to scope down the list, set `--location` to the region
you specified when you enrolled the cluster.

## Connect to the cluster

After the cluster is enrolled with the GKE On-Prem API, you need to choose
and configure an authentication method so that you can
manage the cluster from the Google Cloud console. The authentication method that you
select also controls access to the cluster from the command line. For more
information, see the following:

- [Manage the cluster from the Google Cloud console](https://docs.cloud.google.com/kubernetes-engine/distributed-cloud/bare-metal/docs/how-to/anthos-ui.md.txt)
- [GKE Identity Service](https://docs.cloud.google.com/kubernetes-engine/enterprise/identity.md.txt)