Google Distributed Cloud hybrid clusters let you run user workloads while managing other
clusters and the hybrid cluster itself. Combining admin and user roles
eliminates the overhead of a dedicated admin cluster in resource-constrained
environments. Unlike standalone clusters, which only manage themselves, hybrid
clusters can also create and manage other clusters.

Hosting workloads alongside control components trades security isolation for
flexibility, increasing exposure risks for sensitive assets like SSH keys. You
deploy hybrid clusters with a high-availability (HA) control plane by using the
`bmctl` tool from a separate workstation or directly on a target node.

## Prerequisites

- Latest `bmctl` is downloaded (`gs://anthos-baremetal-release/bmctl/1.36.300-gke.150/linux-amd64/bmctl`) from Cloud Storage.
- Workstation running `bmctl` has network connectivity to all nodes in the target hybrid cluster.
- Workstation running `bmctl` has network connectivity to the control plane VIP of the target hybrid cluster.
- SSH key used to create the hybrid cluster is available to root, or there is SUDO user access on all nodes in the target hybrid cluster.
- Connect-register service account is configured for use with Connect.

See the [Google Distributed Cloud quickstart](https://docs.cloud.google.com/kubernetes-engine/distributed-cloud/bare-metal/docs/quickstart.md.txt) for
expanded step-by-step instructions for creating a hybrid cluster.

## Enable SELinux

If you want to enable SELinux to secure your containers, you must make sure that
SELinux is enabled in `Enforced` mode on all your host machines. Starting with
Google Distributed Cloud release 1.9.0 or later, you can enable or disable SELinux
before or after cluster creation or cluster upgrades. SELinux is enabled by
default on Red Hat Enterprise Linux (RHEL). If SELinux is disabled on
your host machines or you aren't sure, see
[Securing your containers using SELinux](https://docs.cloud.google.com/kubernetes-engine/distributed-cloud/bare-metal/docs/installing/configure-selinux.md.txt)
for instructions on how to enable it.

Google Distributed Cloud supports SELinux in only RHEL systems.

## Log into gcloud and create a cluster configuration file

1. Log into the gcloud CLI as a user using `gcloud auth application-default login`:

   ```bash
   gcloud auth application-default login
   ```

   You need to have a Project Owner or Editor role to use the automatic API
   enablement and Service Account creation features, described in the following
   sections.

   You can also add the following IAM roles to the user:
   - Service Account Admin
   - Service Account Key Admin
   - Project IAM Admin
   - Compute Viewer
   - Service Usage Admin

   Alternatively, if you already have a service account with those roles, run:

   ```bash
   export GOOGLE_APPLICATION_CREDENTIALS=JSON_KEY_FILE
   ```

   Replace `JSON_KEY_FILE` with the path of your service
   account JSON key file.
2. Get your Google Cloud project ID to use with cluster creation:

   ```bash
   export CLOUD_PROJECT_ID=$(gcloud config get-value project)
   ```

### Create the cluster configuration file with `bmctl`

After you've logged into the gcloud CLI and have your project set up,
you can create the cluster configuration file with the `bmctl` command.

To create a cluster configuration file:

1. Run the following command creates the cluster configuration file, creates
   all needed service accounts, and enables all required APIs:

   ```bash
   bmctl create config -c CLUSTER_NAME</var> --enable-apis \
       --create-service-accounts --project-id=CLOUD_PROJECT_ID
   ```

   Replace the following:
   - `CLUSTER_NAME`: the name of the cluster you're creating.
   - `CLOUD_PROJECT_ID`: the ID of your Google Cloud project.

   By default, the configuration file is written to
   `bmctl-workspace/CLUSTER_NAME/CLUSTER_NAME.yaml`.

   If you've already enabled APIs and created service accounts, you can create
   configuration files without enabling APIs and creating service accounts. You
   also aren't required to use the `--project-id` flag. However, you'll need to
   specify credentials in your configuration file manually.

   For more information about command options, see
   [create config](https://docs.cloud.google.com/kubernetes-engine/distributed-cloud/bare-metal/docs/reference/bmctl.md.txt#create_config).

## Edit the cluster configuration file

Now that you have a cluster configuration file, edit it to make the following changes:

1. Provide the SSH private key to access the hybrid cluster nodes:

   ```yaml
   # bmctl configuration variables. Because this section is valid YAML but not a valid Kubernetes
   # resource, this section can only be included when using bmctl to
   # create the initial admin/hybrid cluster. Afterwards, when creating user clusters by directly
   # applying the cluster and node pool resources to the existing cluster, you must remove this
   # section.
   gcrKeyPath: bmctl-workspace/.sa-keys/my-gcp-project-anthos-baremetal-gcr.json
   sshPrivateKeyPath: /path/to/your/ssh_private_key
   gkeConnectAgentServiceAccountKeyPath: bmctl-workspace/.sa-keys/my-gcp-project-anthos-baremetal-connect.json
   gkeConnectRegisterServiceAccountKeyPath: bmctl-workspace/.sa-keys/my-gcp-project-anthos-baremetal-register.json
   cloudOperationsServiceAccountKeyPath: bmctl-workspace/.sa-keys/my-gcp-project-anthos-baremetal-cloud-ops.json
   ```
2. Register your clusters to a
   [fleet](https://docs.cloud.google.com/kubernetes-engine/fleet-management/docs.md.txt). The project ID that you specified
   in the `bmctl create config` command is automatically added to the
   `gkeConnect.projectID` field in the cluster configuration file. This project is
   referred to as the
   [fleet host project](https://docs.cloud.google.com/kubernetes-engine/fleet-management/docs/fleet-concepts.md.txt#fleet-host-project).

   - If you created your configuration file, using the automatic API enablement and Service Account creation features, you can skip this step.
   - If you created the configuration file without using the automatic API enablement and Service Account creation features, reference the downloaded service account JSON keys in the corresponding `gkeConnectAgentServiceAccountKeyPath` and `gkeConnectRegisterServiceAccountKeyPath` fields of the cluster configuration file.
3. Change the config to specify a cluster type of `hybrid` instead of `admin`:

   ```yaml
   spec:
     # Cluster type. This can be:
     #   1) admin:  to create an admin cluster. This can later be used to create user clusters.
     #   2) user:   to create a user cluster. Requires an existing admin cluster.
     #   3) hybrid: to create a hybrid cluster that runs admin cluster components and user workloads.
     #   4) standalone: to create a cluster that manages itself, runs user workloads, but does not manage other clusters.
     type: hybrid
   ```
4. Change the config to specify a multi-node, high availability control plane.
   You want to specify an odd number of nodes to be able to have a majority
   quorum for HA:

   ```yaml
     # Control plane configuration
     controlPlane:
       nodePoolSpec:
         nodes:
         # Control plane node pools. Typically, this is either a single machine
         # or 3 machines if using a high availability deployment.
         - address: 10.200.0.4
         - address: 10.200.0.5
         - address: 10.200.0.6
   ```

   > [!NOTE]
   > **Note:** If you have an even number of nodes temporarily while adding or removing nodes for maintenance or replacement, your deployment maintains HA as long as you have enough quorum.

5. Specify the pod density of cluster nodes:

   ```yaml
   ....
   # NodeConfig specifies the configuration that applies to all nodes in the cluster.
   nodeConfig:
     # podDensity specifies the pod density configuration.
     podDensity:
       # maxPodsPerNode specifies at most how many pods can be run on a single node.
       maxPodsPerNode: 250
   ....
   ```

   For hybrid clusters, allowable values for `maxPodsPerNode` are `32-250` for
   HA clusters and `64-250` for non-HA clusters. The default value for
   `maxPodsPerNode` if unspecified is `110`. Once the cluster is created, this
   value cannot be updated.

   Pod density is also limited by your cluster's available IP resources. For details,
   see
   [Pods and Services networking](https://docs.cloud.google.com/kubernetes-engine/distributed-cloud/bare-metal/docs/concepts/network-reqs.md.txt#pods-and-services-networking).

## Create the hybrid cluster with the cluster configuration

Use the `bmctl` command to create the cluster:

```bash
bmctl create cluster -c CLUSTER_NAME
```

Replace `CLUSTER_NAME` with the name you used when you
created the cluster configuration file the previous section.

The following shows an example of the command to create a cluster named
`hybrid1`:

```bash
bmctl create cluster -c hybrid1
```

## Sample hybrid cluster configurations

For example hybrid cluster configurations, see
[Hybrid clusters](https://docs.cloud.google.com/kubernetes-engine/distributed-cloud/bare-metal/docs/reference/config-samples.md.txt#hybrid_clusters) in the
Cluster configuration samples.