<button class="button button-with-icon cloud-select-dropdown__button button-primary" aria-haspopup="true">Version 1.16 </button>

- [Latest version](https://docs.cloud.google.com/anthos/clusters/docs/on-prem/latest/how-to/minimal-create-clusters)
- [Version 1.16](https://docs.cloud.google.com/anthos/clusters/docs/on-prem/1.16/how-to/minimal-create-clusters.md.txt)
- [Earlier versions (unsupported)](https://docs.cloud.google.com/anthos/clusters/docs/on-prem/latest/version-history#expandable-1)

<br />

This is the second part of a guide that walks you through a small
proof-of-concept installation of Google Distributed Cloud. The first part
is
[Set up minimal infrastructure](https://docs.cloud.google.com/anthos/clusters/docs/on-prem/1.16/how-to/minimal-infrastructure.md.txt),
which shows you how to plan your IP addresses and set up the necessary vSphere
and Google Cloud infrastructure for your deployment. This document builds on the
setup and planning you did in the previous section and shows you how to create
an admin workstation, admin cluster, and user cluster in your vSphere
environment, using simple templates that you can fill in here in this document. You
can then go on to [deploy an application](https://docs.cloud.google.com/anthos/clusters/docs/on-prem/1.16/how-to/deploy-first-app.md.txt).

As with the infrastructure setup of this simple installation, the clusters you
set up using this document might not be suitable for your actual production
needs and use cases. For much more information, best practices, and instructions
for production installations, see the
[installation guides](https://docs.cloud.google.com/anthos/clusters/docs/on-prem/1.16/how-to/install-overview.md.txt).

## Before you begin

1. Ensure that you have set up your vSphere and Google Cloud environments as described in
   [Set up minimal infrastructure](https://docs.cloud.google.com/anthos/clusters/docs/on-prem/1.16/how-to/minimal-infrastructure.md.txt).

2. If you want to use Terraform to create the user cluster, you need
   [Terraform](https://developer.hashicorp.com/terraform/tutorials/gcp-get-started/install-cli)
   either on your admin workstation or another computer.

   <br />

## Procedure overview

These are the primary steps involved in this setup:

1. **Log in to the Google Cloud CLI** with an account that has the necessary
   permissions to create service accounts.

2. **Gather information** that you need to
   configure Google Distributed Cloud, including your vCenter username and
   password, and the IP addresses that you prepared in the previous section.

3. **Create an admin workstation** with the resources and tools you need to
   create admin and user clusters, including the additional service accounts
   you need to finish the setup.

4. **Create an admin cluster** to manage and update your user cluster.

5. **Create a user cluster** to run your workloads.

## 1. Log in to the Google Cloud CLI

Setting up Google Distributed Cloud requires multiple
[service accounts](https://docs.cloud.google.com/anthos/clusters/docs/on-prem/1.16/how-to/service-accounts.md.txt)
with different permissions. While you must
[create your component access service account](https://docs.cloud.google.com/anthos/clusters/docs/on-prem/1.16/how-to/minimal-infrastructure.md.txt#set_up_your_component_access_service_account)
manually, the `gkeadm` command line tool can create and configure the remaining
accounts for you as part of creating the admin workstation. To do this,
however, you must be logged in to the Google Cloud CLI with an account that has
the necessary permissions to create and configure service accounts, as `gkeadm`
uses your current gcloud CLI `account`
[property](https://docs.cloud.google.com/sdk/docs/properties.md.txt) when doing this setup.

1. Log in to the gcloud CLI. You can use any Google account but it
   must have the
   [required permissions](https://docs.cloud.google.com/anthos/clusters/docs/on-prem/1.16/how-to/minimal-infrastructure.md.txt#required_permissions).
   If you have followed the previous part of this guide you have probably
   already logged in with an appropriate account to create your component
   access service account.

   ```
   gcloud auth login
   ```
2. Verify that your gcloud CLI `account` property is set correctly:

   ```
   gcloud config list
   ```

   The output shows the value of your SDK `account` property.
   For example:

   ```
   [core]
   account = my-name@google.com
   disable_usage_reporting = False
   Your active configuration is: [default]
   ```
3. Make sure you have the latest gcloud CLI components installed:

       gcloud components update

   Depending on how you installed the gcloud CLI, you might see
   the following message: "You cannot perform this action because the Google
   Cloud CLI component manager is disabled for this installation. You can run
   the following command to achieve the same result for this installation:"
   Follow the instructions to copy and paste the command to update
   the components.

## 2. Gather information

Use the information that you prepared in
[Set up minimal infrastructure](https://docs.cloud.google.com/anthos/clusters/docs/on-prem/1.16/how-to/minimal-infrastructure.md.txt)
to edit the placeholders in the following table:

> [!CAUTION]
> **Caution:** The editable placeholders in this table aren't persistent. Any values you enter in the table are reset if you close or refresh this browser tab, or if you navigate away from this page. If the placeholders in the table are *not* editable, you may need to refresh this page.

| vSphere details ||
|---|---|
| The username of your vCenter account | <var class="edit" scope="USERNAME" translate="no">USERNAME</var> |
| The password of your vCenter account | <var class="edit" scope="PASSWORD" translate="no">PASSWORD</var> |
| Your [vCenter Server address](https://docs.cloud.google.com/anthos/clusters/docs/on-prem/1.16/how-to/vcenter-server-address.md.txt) | <var class="edit" scope="ADDRESS" translate="no">ADDRESS</var> |
| The path to the [root CA certificate](https://docs.cloud.google.com/anthos/clusters/docs/on-prem/1.16/how-to/vcenter-ca-cert-path.md.txt) for your vCenter Server, on the machine you're going to use to create your admin workstation | <var class="edit" scope="CA_CERT_PATH" translate="no">CA_CERT_PATH</var> |
| The name of your [vSphere data center](https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.vcenterhost.doc/GUID-7FDFBDBE-F8AC-4D00-AE5E-3F14D7472FAF.html) | <var class="edit" scope="DATA_CENTER" translate="no">DATA_CENTER</var> |
| The name of your [vSphere cluster](https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.resmgmt.doc/GUID-487C09CE-8BE2-4B89-BA30-0E4F7E3C66F7.html) | <var class="edit" scope="VSPHERE_CLUSTER" translate="no">VSPHERE_CLUSTER</var> |
| The name or path of your [vSphere resource pool](https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.resmgmt.doc/GUID-60077B40-66FF-4625-934A-641703ED7601.html). For more information, see [vcenter.resourcePool](https://docs.cloud.google.com/anthos/clusters/docs/on-prem/1.16/how-to/admin-cluster-configuration-file.md.txt#vcenter-resourcepool-field). | <var class="edit" scope="RESOURCE_POOL" translate="no">RESOURCE_POOL</var> |
| The name of your [vSphere datastore](https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.storage.doc/GUID-3CC7078E-9C30-402C-B2E1-2542BEE67E8F.html) | <var class="edit" scope="DATASTORE" translate="no">DATASTORE</var> |
| The name of your [vSphere network](https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.networking.doc/GUID-35B40B0B-0C13-43B2-BC85-18C9C91BE2D4.html) | <var class="edit" scope="NETWORK" translate="no">NETWORK</var> |
| One IP address for your admin workstation | <var class="edit" scope="ADMIN_WS_IP" translate="no">ADMIN_WS_IP</var> |
| Four IP addresses for your admin cluster nodes. This includes an address for an extra node that can be used during upgrade and update. | <var class="edit" scope="ADMIN_NODE_IP_1" translate="no">ADMIN_NODE_IP_1</var> <var class="edit" scope="ADMIN_NODE_IP_2" translate="no">ADMIN_NODE_IP_2</var> <var class="edit" scope="ADMIN_NODE_IP_3" translate="no">ADMIN_NODE_IP_3</var> <var class="edit" scope="ADMIN_NODE_IP_4" translate="no">ADMIN_NODE_IP_4</var> |
| An IP address for the control-plane node in the user cluster. | <var class="edit" scope="USER_CONTROL_PLANE_NODE_IP" translate="no">USER_CONTROL_PLANE_NODE_IP</var> |
| Four IP addresses for your user cluster nodes. This includes an address for an extra node that can be used during upgrade and update. | <var class="edit" scope="USER_NODE_IP_1" translate="no">USER_NODE_IP_1</var> <var class="edit" scope="USER_NODE_IP_2" translate="no">USER_NODE_IP_2</var> <var class="edit" scope="USER_NODE_IP_3" translate="no">USER_NODE_IP_3</var> <var class="edit" scope="USER_NODE_IP_4" translate="no">USER_NODE_IP_4</var> |
| A virtual IP address (VIP) for the admin cluster Kubernetes API server | <var class="edit" scope="ADMIN_CONTROL_PLANE_VIP" translate="no">ADMIN_CONTROL_PLANE_VIP</var> |
| A VIP for the user cluster Kubernetes API server | <var class="edit" scope="USER_CONTROL_PLANE_VIP" translate="no">USER_CONTROL_PLANE_VIP</var> |
| An Ingress VIP for the user cluster | <var class="edit" scope="USER_INGRESS_VIP" translate="no">USER_INGRESS_VIP</var> |
| Two VIPs for Services of type LoadBalancer in your user cluster. | <var class="edit" scope="SERVICE_VIP_1" translate="no">SERVICE_VIP_1</var> <var class="edit" scope="SERVICE_VIP_2" translate="no">SERVICE_VIP_2</var> |
| The IP address of a DNS server that is reachable from your admin workstation and cluster nodes | <var class="edit" scope="DNS_SERVER_IP" translate="no">DNS_SERVER_IP</var> |
| The IP address of an NTP server that is reachable from your admin workstation and cluster nodes | <var class="edit" scope="NTP_SERVER_IP" translate="no">NTP_SERVER_IP</var> |
| The IP address of the default gateway for the subnet that has your admin workstation and cluster nodes | <var class="edit" scope="DEFAULT_GATEWAY_IP" translate="no">DEFAULT_GATEWAY_IP</var> |
| The netmask for the subnet that has your admin workstation and cluster nodes Example: `255.255.255.0` | <var class="edit" scope="NETMASK" translate="no">NETMASK</var> |
| If your network is behind a proxy server, the URL of the proxy server. For more information, see [proxy](https://docs.cloud.google.com/anthos/clusters/docs/on-prem/1.16/how-to/admin-cluster-configuration-file.md.txt#proxy-section). **Fill this in manually in your admin workstation configuration file if needed**. | <var class="edit" scope="PROXY_URL" translate="no">PROXY_URL</var> |
| The admin cluster and user cluster each need a CIDR range for Services and a CIDR range for Pods. Use the following prepopulated values unless you need to change them to [avoid overlap](https://docs.cloud.google.com/anthos/clusters/docs/on-prem/1.16/how-to/minimal-infrastructure.md.txt#avoid_overlap)) with other elements in your network: ||
| A CIDR range for Services in the admin cluster | <var class="edit" scope="ADMIN_SERVICE_CIDR" translate="no">10.96.232.0/24</var> |
| A CIDR range for Pods in the admin cluster | <var class="edit" scope="ADMIN_POD_CIDR" translate="no">192.168.0.0/16</var> |
| A CIDR range for Services in the user cluster | <var class="edit" scope="USER_SERVICE_CIDR" translate="no">10.96.0.0/20</var> |
| A CIDR range for Pods in the user cluster | <var class="edit" scope="USER_POD_CIDR" translate="no">192.168.0.0/16</var> |
| The ID of your chosen [Cloud project](https://docs.cloud.google.com/anthos/clusters/docs/on-prem/1.16/how-to/minimal-infrastructure.md.txt#cloud-resources) | <var class="edit" scope="PROJECT_ID" translate="no">PROJECT_ID</var> |
| The path to the JSON key file for the [component access service account](https://docs.cloud.google.com/anthos/clusters/docs/on-prem/1.16/how-to/minimal-infrastructure.md.txt#cloud-resources) that you set up in the previous section, on the machine that you're going to use to create your admin cluster. | <var class="edit" scope="COMPONENT_ACCESS_SA_KEY_PATH" translate="no">COMPONENT_ACCESS_SA_KEY_PATH</var> |
| The email address that is associated with your Google Cloud account. For example: `alex@example.com`. | `GOOGLE_ACCOUNT_EMAIL` |

## 3. Create an admin workstation

Before you can create any clusters, you need to create an admin workstation
and then connect to it using SSH. The admin workstation is a standalone VM with
the tools and resources you need to create GKE Enterprise clusters in your
vSphere environment. You use the `gkeadm` command-line tool to create the
admin workstation.

### Download gkeadm

Download `gkeadm` to your current directory:

```
gcloud storage cp gs://gke-on-prem-release/gkeadm/1.16.7-gke.46/linux/gkeadm ./
chmod +x gkeadm
```

You need the `gkeadm` version (which is also the version of Google Distributed Cloud)
to create your admin and user cluster config files. To check the version of
`gkeadm`, run the following:

```
./gkeadm version
```

The following example output shows the version.

```
gkeadm 1.16.8 (1.16.7-gke.46)
```

Although you can
[download another version of `gkeadm`](https://docs.cloud.google.com/anthos/clusters/docs/on-prem/1.16/how-to/download-gkeadm.md.txt),
this guide assumes that you are installing 1.16.7-gke.46,
and uses that version in all configuration files and commands.

### Create your credentials file

Create and save a file named `credential.yaml` in your current directory with the following content:

```
apiVersion: v1
kind: CredentialFile
items:
- name: vCenter
  username: "USERNAME"
  password: "PASSWORD"
```

### Create your admin workstation configuration file

Create and save a file named `admin-ws-config.yaml`, again in your current directory, with the following content:

```
gcp:
  componentAccessServiceAccountKeyPath: "COMPONENT_ACCESS_SA_KEY_PATH"
vCenter:
  credentials:
    address: "ADDRESS"
    fileRef:
      path: "credential.yaml"
      entry: "vCenter"
  datacenter: "DATA_CENTER"
  datastore: "DATASTORE"
  cluster: "VSPHERE_CLUSTER"
  network: "NETWORK"
  resourcePool: "RESOURCE_POOL"
  caCertPath: "CA_CERT_PATH"
proxyUrl: ""
adminWorkstation:
  name: "minimal-installation-admin-workstation"
  cpus: 4
  memoryMB: 8192
  diskGB: 50
  dataDiskName: gke-on-prem-admin-workstation-data-disk/minimal-installation-data-disk.vmdk
  dataDiskMB: 512
  network:
    ipAllocationMode: "static"
    hostConfig:
      ip: "ADMIN_WS_IP"
      gateway: "DEFAULT_GATEWAY_IP"
      netmask: "NETMASK"
      dns:
      - "DNS_SERVER_IP"
  proxyUrl: ""
  ntpServer: ntp.ubuntu.com
```

### Create your admin workstation

Create your admin workstation using the following command:

```
./gkeadm create admin-workstation --auto-create-service-accounts
```

Running this command:

- Creates your admin workstation
- Automatically creates any additional service accounts you need for your installation
- Creates template configuration files for your admin and user clusters

The output gives detailed information about the creation of your admin
workstation and provides a command that you can use to get an SSH connection
to your admin workstation:

```
...
Admin workstation is ready to use.
Admin workstation information saved to /usr/local/google/home/me/my-admin-workstation
This file is required for future upgrades
SSH into the admin workstation with the following command:
ssh -i /usr/local/google/home/me/.ssh/gke-admin-workstation ubuntu@172.16.20.49
********************************************************************
```

For more detailed information about creating an admin workstation, see
[Create an admin workstation](https://docs.cloud.google.com/anthos/clusters/docs/on-prem/1.16/how-to/create-admin-workstation.md.txt).

### Connect to your admin workstation

Use the command displayed in the preceding output to get an SSH connection to
your admin workstation. For example:

```
ssh -i /usr/local/google/home/me/.ssh/gke-admin-workstation ubuntu@172.16.20.49
```

If you need to find this command again, `gkeadm` generates a file called `gke-admin-ws-...` in the directory on your local machine where you ran `gkeadm create admin-workstation`.
This contains details about your admin workstation, including the SSH command.

### View generated files

On your admin workstation, list the files in the home directory:

```
ls -1
```

The output should include:

- `admin-cluster.yaml`, a template config file for creating your admin cluster.
- `user-cluster.yaml`, a template config file for creating your user cluster.
- The vCenter certificate file that you specified in your admin workstation configuration
- The `credential.yaml` file that you specified in your admin workstation configuration.
- JSON key files for two service accounts that `gkeadm` created for you: a connect-register service account and a logging-monitoring service account, as well as the key file for the component access service account you created earlier.

For example:

```
admin-cluster.yaml
admin-ws-config.yaml
sa-key.json
connect-register-sa-2203040617.json
credential.yaml
log-mon-sa-2203040617.json
logs
vc01-cert.pem
user-cluster.yaml
```

You'll need to specify some of these filenames in your configuration files to create clusters. Use the filenames as values for the placeholders in the following table:

|---|---|
| Connect-register service account key file name Example: `connect-register-sa-2203040617.json` | <var class="edit" scope="CONNECT_REGISTER_SA_KEY_PATH" translate="no">CONNECT_REGISTER_SA_KEY_PATH</var> |
| Logging-monitoring service account key file name Example: `log-mon-sa-2203040617.json` | <var class="edit" scope="LOG_MON_SA_KEY_PATH" translate="no">LOG_MON_SA_KEY_PATH</var> |
| Component access service account key file name Example: `sa-key.json` | <var class="edit" scope="COMPONENT_ACCESS_SA_KEY_FILE" translate="no">COMPONENT_ACCESS_SA_KEY_FILE</var> |
| vCenter certificate file name Example: `vc01-cert.pem` | <var class="edit" scope="CA_CERT_FILE" translate="no">CA_CERT_FILE</var> |

## 4. Create an admin cluster

Now that you have an admin workstation configured with your vCenter and other
details, you can use it to create an admin cluster in your vSphere environment.
Ensure you have an SSH connection to your admin workstation, as described
previously, before starting this step. All of the following commands are run on
the admin workstation.

> [!NOTE]
> **Note:** If you're using internal DNS names, or your workstation cannot access Google public DNS addresses (8.8.8.8 or 8.8.4.4), you must specify your internal nameserver in [`/etc/resolv.conf`](https://ubuntu.com/server/docs#name-resolution).

### Create your admin cluster IP block file

Create and save a file named `admin-ipblock.yaml` in the same directory as `admin-cluster.yaml` on your admin workstation, with the following content:

```yaml
blocks:
  - netmask: "NETMASK"
    gateway: "DEFAULT_GATEWAY_IP"
    ips:
    - ip: "ADMIN_NODE_IP_1"
      hostname: "admin-vm-1"
    - ip: "ADMIN_NODE_IP_2"
      hostname: "admin-vm-2"
    - ip: "ADMIN_NODE_IP_3"
      hostname: "admin-vm-3"
    - ip: "ADMIN_NODE_IP_4"
      hostname: "admin-vm-4"
```

The `ips` field is an array of IP addresses and hostnames. These are the IP
addresses and hostnames that Google Distributed Cloud will assign to
your admin cluster nodes.

The IP block file also specifies a subnet mask and a default gateway
for the admin cluster nodes.

### Create your admin cluster configuration file

Open `admin-cluster.yaml` and replace the content with the following:

```
apiVersion: v1
kind: AdminCluster
name: "minimal-installation-admin-cluster"
bundlePath: "/var/lib/gke/bundles/gke-onprem-vsphere-1.16.7-gke.46-full.tgz"
vCenter:
  address: "ADDRESS"
  datacenter: "DATA_CENTER"
  cluster: "VSPHERE_CLUSTER"
  resourcePool: "RESOURCE_POOL"
  datastore: "DATASTORE"
  caCertPath: "CA_CERT_FILE"
  credentials:
    fileRef:
      path: "credential.yaml"
      entry: "vCenter"
  dataDisk: "data-disks/minimal-installation-admin-disk.vmdk"
network:
  hostConfig:
    dnsServers:
    - "DNS_SERVER_IP"
    ntpServers:
    - "NTP_SERVER_IP"
  ipMode:
    type: "static"
    ipBlockFilePath: admin-ipblock.yaml
  serviceCIDR: "10.96.232.0/24"
  podCIDR: "192.168.0.0/16"
  vCenter:
    networkName: "NETWORK"
loadBalancer:
  vips:
    controlPlaneVIP: "ADMIN_CONTROL_PLANE_VIP"
  kind: "MetalLB"
antiAffinityGroups:
  enabled: false
componentAccessServiceAccountKeyPath: "COMPONENT_ACCESS_SA_KEY_FILE"
gkeConnect:
  projectID: "PROJECT_ID"
  registerServiceAccountKeyPath: "CONNECT_REGISTER_SA_KEY_PATH"
stackdriver:
  projectID: "PROJECT_ID"
  clusterLocation: "us-central1"
  enableVPC: false
  serviceAccountKeyPath: "LOG_MON_SA_KEY_PATH"
  disableVsphereResourceMetrics: false
```

#### Validate the admin cluster configuration file

Verify that the your admin cluster configuration file is valid and can be used
for cluster creation:

```
gkectl check-config --config admin-cluster.yaml
```

### Import OS images to vSphere

Run `gkectl prepare` with your completed config file to import node OS images to vSphere:

```
gkectl prepare --config admin-cluster.yaml --skip-validation-all
```

Running this command imports the images to vSphere and marks them as VM templates, including the image for your admin cluster.

This command can take a few minutes to return.

### Create the admin cluster

Create the admin cluster:

```
gkectl create admin --config admin-cluster.yaml
```

#### Resume admin cluster creation after a failure

If the admin cluster creation fails or is canceled, you can run the `create` command again:

```
gkectl create admin --config admin-cluster.yaml
```

### Locate the admin cluster kubeconfig file

The `gkectl create admin` command creates a kubeconfig file named
`kubeconfig` in the current directory. You will need this kubeconfig file
later to interact with your admin cluster.

### Verify that your admin cluster is running

Verify that your admin cluster is running:

```
kubectl get nodes --kubeconfig kubeconfig
```

The output shows the admin cluster nodes. For example:

```
gke-admin-master-hdn4z            Ready    control-plane,master ...
gke-admin-node-7f46cc8c47-g7w2c   Ready ...
gke-admin-node-7f46cc8c47-kwlrs   Ready ...
```

### Enable RBAC authorization

To grant your user account the Kubernetes `clusterrole/cluster-admin` role
on the cluster, run the following command:

```bash
gcloud container fleet memberships generate-gateway-rbac \
    --membership=minimal-installation-admin-cluster \
    --role=clusterrole/cluster-admin \
    --users=GOOGLE_ACCOUNT_EMAIL \
    --project=PROJECT_ID \
    --kubeconfig=kubeconfig \
    --context=minimal-installation-admin-cluster \
    --apply
```

The output of this command is similar to the following, which is truncated for
readability:

```
Validating input arguments.
Specified Cluster Role is: clusterrole/cluster-admin
Generated RBAC policy is:
---
...
Applying the generate RBAC policy to cluster with kubeconfig: kubeconfig, context: minimal-installation-admin-cluster
Writing RBAC policy for user: GOOGLE_ACCOUNT_EMAIL to cluster.
Successfully applied the RBAC policy to cluster.
```

Among other things, the RBAC policy lets you log in to your cluster in the
Google Cloud console using your Google Identity to see more cluster details.

### Automatic enrollment in the GKE On-Prem API

Because the GKE On-Prem API is enabled in your project, the cluster is
automatically enrolled in the GKE On-Prem API. Enrolling your admin
cluster in the GKE On-Prem API lets you use standard tools---the
Google Cloud console, the Google Cloud CLI, or
[Terraform](https://www.terraform.io/)---to create, upgrade,
update, and delete user clusters that the admin cluster manages. Enrolling
your cluster also lets you run `gcloud` commands to
[get information about your cluster](https://docs.cloud.google.com/anthos/clusters/docs/on-prem/1.16/how-to/enroll-cluster.md.txt#get_cluster_information).

## 5. Create a user cluster

This section provides steps for creating the user cluster using the
console, `gkectl`, Terraform, or the gcloud CLI.

### `gkectl`

Ensure you have an SSH connection to your admin workstation, as described
previously, before starting this procedure. All of the following commands
are run on the admin workstation.

### Create your user cluster IP block file

1. Create a file named `user-ipblock.yaml`.

2. Copy and paste the following content into `user-ipblock.yaml` and save
   the file:

       blocks:
         - netmask: "NETMASK"
           gateway: "DEFAULT_GATEWAY_IP"
           ips:
           - ip: "USER_NODE_IP_1"
             hostname: "user-vm-1"
           - ip: "USER_NODE_IP_2"
             hostname: "user-vm-2"
           - ip: "USER_NODE_IP_3"
             hostname: "user-vm-3"
           - ip: "USER_NODE_IP_4"
             hostname: "user-vm-4"

### Create your user cluster configuration file

1. Create a file named `user-cluster.yaml` in the same directory as
   `user-ipblock.yaml`.

2. Copy and paste the following content into `user-cluster.yaml` and save
   the file:

```
apiVersion: v1
kind: UserCluster
name: "minimal-installation-user-cluster"
gkeOnPremVersion: "1.16.7-gke.46"
enableControlplaneV2: true
network:
  hostConfig:
    dnsServers:
    - "DNS_SERVER_IP"
    ntpServers:
    - "NTP_SERVER_IP"
  ipMode:
    type: "static"
    ipBlockFilePath: "user-ipblock.yaml"
  serviceCIDR: "10.96.0.0/20"
  podCIDR: "192.168.0.0/16"
  controlPlaneIPBlock:
    netmask: "NETMASK"
    gateway: "DEFAULT_GATEWAY_IP"
    ips:
    - ip: "USER_CONTROL_PLANE_NODE_IP"
      hostname: "cp-vm-1"
loadBalancer:
  vips:
    controlPlaneVIP: "USER_CONTROL_PLANE_VIP"
    ingressVIP: "USER_INGRESS_VIP"
  kind: "MetalLB"
  metalLB:
    addressPools:
    - name: "uc-address-pool"
      addresses:
      - "USER_INGRESS_VIP/32"
      - "SERVICE_VIP_1/32"
      - "SERVICE_VIP_2/32"
enableDataplaneV2: true
nodePools:
- name: "uc-node-pool"
  cpus: 4
  memoryMB: 8192
  replicas: 3
  enableLoadBalancer: true
antiAffinityGroups:
  enabled: false
gkeConnect:
  projectID: "PROJECT_ID"
  registerServiceAccountKeyPath: "CONNECT_REGISTER_SA_KEY_PATH"
stackdriver:
  projectID: "PROJECT_ID"
  clusterLocation: "us-central1"
  enableVPC: false
  serviceAccountKeyPath: "LOG_MON_SA_KEY_PATH"
  disableVsphereResourceMetrics: false
autoRepair:
  enabled: true
```

### Validate the configuration and create the cluster

1. Verify that the your user cluster configuration file is valid and can be used
   for cluster creation:

   ```bash
   gkectl check-config --kubeconfig kubeconfig --config user-cluster.yaml
   ```
2. Create the user cluster:

   ```bash
   gkectl create cluster --kubeconfig kubeconfig --config user-cluster.yaml
   ```

   Cluster creation takes approximately 30 minutes.

### Locate the user cluster kubeconfig file

The `gkectl create cluster` command creates a kubeconfig file named
`USER_CLUSTER_NAME-kubeconfig` in the current directory. You will need this
kubeconfig file later to interact with your user cluster.

### Verify that your user cluster is running

Verify that your user cluster is running:

```bash
kubectl get nodes --kubeconfig USER_CLUSTER_KUBECONFIG
```

Replace <var translate="no">USER_CLUSTER_KUBECONFIG</var> with the path of your user cluster
kubeconfig file.

The output shows the user cluster nodes. For example:

```
cp-vm-1     Ready    control-plane,master
user-vm-1   Ready
user-vm-2   Ready
user-vm-3   Ready
```

### Enable RBAC authorization

To grant your user account the Kubernetes `clusterrole/cluster-admin` role
on the cluster, run the following command:

```bash
gcloud container fleet memberships generate-gateway-rbac \
  --membership=minimal-installation-user-cluster \
  --role=clusterrole/cluster-admin \
  --users=GOOGLE_ACCOUNT_EMAIL \
  --project=PROJECT_ID \
  --kubeconfig=USER_CLUSTER_KUBECONFIG \
  --context=minimal-installation-user-cluster \
  --apply
```

The output of this command is similar to the following, which is truncated for
readability:

```
Validating input arguments.
Specified Cluster Role is: clusterrole/cluster-admin
Generated RBAC policy is:
---
...
Applying the generate RBAC policy to cluster with kubeconfig: kubeconfig, context: minimal-installation-admin-cluster
Writing RBAC policy for user: GOOGLE_ACCOUNT_EMAIL to cluster.
Successfully applied the RBAC policy to cluster.
```

Among other things, the RBAC policy lets you log in to your cluster in the
Google Cloud console using your Google Identity to see more cluster details.

### Automatic enrollment in the GKE On-Prem API

Because the GKE On-Prem API is enabled in your project, the cluster is
automatically enrolled in the GKE On-Prem API. This lets you use the
console or the gcloud CLI to view
cluster details and manage the cluster lifecycle. For example, you run can
`gcloud` commands to
[get information about your user cluster](https://docs.cloud.google.com/anthos/clusters/docs/on-prem/1.16/how-to/enroll-cluster.md.txt#get_cluster_information).

### Console

1. In the Google Cloud console, go to the **Create a
   Google Distributed Cloud cluster** page.

   [Go to Create a Google Distributed Cloud cluster](https://console.cloud.google.com/kubernetes/vmware/add)
2. Select the Google Cloud project that you want to create the cluster in.
   The selected project is also used as the fleet host project. This must be
   the same project that the admin cluster is registered to. After the user
   cluster is created, it is automatically registered to the selected
   project's fleet.

The following sections guide you through configuring the user cluster.

### Prerequisites

1. Familiarize yourself with the information on the **Prerequisites** page.

2. At the bottom of the page, click **Next**.

### Cluster basics

1. For **Name** , enter a name for the user cluster: for example,
   `minimal-installation-user-cluster`.

2. For **Admin cluster** , select **minimal-installation-admin-cluster**.

3. For **GCP API Location** field, select the **us-central1**.

4. For **Version**, select 1.16.7-gke.46.

5. You don't need to open the **Authorization** section or the **vCenter
   configuration** section.

6. Click **Next**.

### Control Plane

1. Under **Control plane node IPs** , for **Gateway** , enter
   <var scope="DEFAULT_GATEWAY_IP" translate="no">DEFAULT_GATEWAY_IP</var>.

2. For **Subnet mask** , enter
   <var scope="NETMASK" translate="no">NETMASK</var>.

3. Under **IP addresses** , for **IP address 1** , enter
   <var scope="USER_CONTROL_PLANE_NODE_IP" translate="no">USER_CONTROL_PLANE_NODE_IP</var>.
   Leave **Hostname 1** blank.

4. Click **Next**.

### Networking

In this section, you specify the IP addresses for your cluster's nodes, Pods,
and Services. A user cluster needs to have one IP address for each node and an
additional IP address for a temporary node that is needed during cluster
upgrades, updates, and auto repair. For more information, see
[How many IP addresses does a user cluster need?](https://docs.cloud.google.com/anthos/clusters/docs/on-prem/1.16/how-to/manage-node-ip-addresses.md.txt#how_many_ip_addresses_does_a_user_cluster_need).

1. Under **Worker node IPs** , for **IP mode** , make sure that **Static**
   is selected.

2. For **Gateway** , enter
   <var scope="DEFAULT_GATEWAY_IP" translate="no">DEFAULT_GATEWAY_IP</var>.

3. For **Subnet mask** , enter
   <var scope="NETMASK" translate="no">NETMASK</var>.

4. Under **IP addresses**, enter these addresses:

- <var scope="USER_NODE_IP_1" translate="no">USER_NODE_IP_1</var>
- <var scope="USER_NODE_IP_2" translate="no">USER_NODE_IP_2</var>
- <var scope="USER_NODE_IP_3" translate="no">USER_NODE_IP_3</var>
- <var scope="USER_NODE_IP_4" translate="no">USER_NODE_IP_4</var>

Leave the **Hostname** fields blank.

1. For **Service CIDR** , enter
   <var scope="USER_SERVICE_CIDR" translate="no">10.96.0.0/20</var>.
   For **Pod CIDR** , enter
   <var scope="USER_POD_CIDR" translate="no">192.168.0.0/16</var>.

2. For **DNS Server 1** , enter
   <var scope="DNS_SERVER_IP" translate="no">DNS_SERVER_IP</var>.

3. For **NTP Server 1** , enter
   <var scope="NTP_SERVER_IP" translate="no">NTP_SERVER_IP</var>.

4. Leave **DNS search domain** blank.

5. Click **Next**.

### Load balancer

1. For **Load balancer type** , select **Bundled with MetalLB**.

2. Under **Address pools**, use the default name.

3. Under **IP addresses** , for **IP address range 1** , enter
   <var scope="USER_INGRESS_VIP" translate="no">USER_INGRESS_VIP</var><var translate="no">/32</var>.

   Click **Add IP address range** . For **IP address range 1** , enter
   <var scope="SERVICE_VIP_1" translate="no">SERVICE_VIP_1</var><var translate="no">/32</var>

   Click **Add IP address range** . For **IP address range 2** , enter
   <var scope="SERVICE_VIP_2" translate="no">SERVICE_VIP_2<var translate="no"><var translate="no">/32</var></var></var>
4. For **Assignment of IP addresses** , select **Automatic**.

5. Leave **Avoid buggy IP addresses** unchecked.

6. Under **Virtual IPs** , for **Control plane VIP** enter
   <var scope="USER_CONTROL_PLANE_VIP" translate="no">USER_CONTROL_PLANE_VIP</var>.
   **Ingress VIP** is already filled in.

7. Click **Continue**.

### Features

1. Leave all the defaults in place.

2. Click **Next**.

### Node pools

1. Leave all the defaults in place.

2. Click **Verify and Complete** to create the user cluster. It takes 15
   minutes or more to create the user cluster. The console displays status
   messages as it verifies the settings and creates the cluster in your data
   center.

   If an error is encountered verifying the settings, the console displays an error
   message that should be clear enough for you to fix the configuration issue and
   try again to create the cluster.

   For more information about possible errors and how to fix them, see
   [Troubleshoot user cluster creation in the Google Cloud console](https://docs.cloud.google.com/anthos/clusters/docs/on-prem/1.16/troubleshoot-user-cluster-create-api.md.txt).

### Terraform

This section shows you how to create a user cluster and a node pool using
Terraform. For more information and other examples, see the following:

- [`google_gkeonprem_vmware_cluster`](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/gkeonprem_vmware_cluster)

- [google_gkeonprem_vmware_node_pool](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/gkeonprem_vmware_node_pool)

- [terraform.tfvars.sample](https://github.com/GoogleCloudPlatform/anthos-samples/blob/HEAD/anthos-onprem-terraform/avmw_user_cluster_metallb/terraform.tfvars.sample)

1. Create a directory and a new file within that directory. The filename must
   have the `.tf` extension. In this guide, the file is called `main.tf`.

   ```terraform
   mkdir DIRECTORY && cd DIRECTORY && touch main.tf
   ```
2. Verify the user cluster Terraform resource:

   The following Terraform resource example is filled in with the values that
   you entered in the planning table in the preceding section.


   ```terraform
   resource "google_gkeonprem_vmware_cluster" "cluster-basic" {
     name = "minimal-installation-user-cluster"
     project = "PROJECT_ID"
     location = "us-central1"
     admin_cluster_membership = "projects/PROJECT_ID/locations/global/memberships/minimal-installation-admin-cluster"
     description = "User cluster config with MetalLB, static IPs, and Controlplane V2"
     enable_control_plane_v2 = "true"
     on_prem_version = "1.16.7-gke.46"
     control_plane_node {
       cpus = 4
       memory = 8192
       replicas = 1
     }
     network_config {
       service_address_cidr_blocks = ["10.96.0.0/20"]
       pod_address_cidr_blocks = ["192.168.0.0/16"]
       host_config {
         dns_servers = ["DNS_SERVER_IP"]
         ntp_servers = ["NTP_SERVER_IP"]
       }
       static_ip_config {
         ip_blocks {
           netmask = "NETMASK"
           gateway = "DEFAULT_GATEWAY_IP"
           ips {
             ip = "USER_NODE_IP_1"
             hostname = "user-vm-1"
           }
           ips {
             ip = "USER_NODE_IP_2"
             hostname = "user-vm-2"
           }
           ips {
             ip = "USER_NODE_IP_3"
             hostname = "user-vm-3"
           }
           ips {
             ip = "USER_NODE_IP_4"
             hostname = "user-vm-4"
           }
         }
       }
       control_plane_v2_config {
         control_plane_ip_block {
           netmask = "NETMASK"
           gateway = "DEFAULT_GATEWAY_IP"
           ips {
             ip = "USER_CONTROL_PLANE_NODE_IP"
             hostname = "cp-vm-1"
           }
         }
       }
     }
     load_balancer {
       vip_config {
         control_plane_vip = "USER_CONTROL_PLANE_VIP"
         ingress_vip = "USER_INGRESS_VIP"
       }
       metal_lb_config {
         address_pools {
           pool = "uc-address-pool"
           manual_assign = "true"
           addresses = ["USER_INGRESS_VIP/32", "SERVICE_VIP_1/32", "SERVICE_VIP_2/32"]
         }
       }
     }
     authorization {
       admin_users {
       username = "GOOGLE_ACCOUNT_EMAIL"
     }
   }
     provider = google-beta
   }

   resource "google_gkeonprem_vmware_node_pool" "my-node-pool-1" {
     name = "uc-node-pool"
     project = "PROJECT_ID"
     vmware_cluster = "minimal-installation-user-cluster"
     location = "us-central1"
     config {
       replicas = 3
       image_type = "ubuntu_containerd"
       enable_load_balancer = "true"
     }
     depends_on = [
       google_gkeonprem_vmware_cluster.cluster-basic
     ]
     provider = google-beta
   }
   ```
3. Copy the Terraform resource to `main.tf` and save the file.

4. Initialize and create the Terraform plan:

   ```terraform
   terraform init
   ```

   Terraform installs any needed libraries, such as the Google Cloud provider.
5. Review the configuration and make changes if needed:

   ```terraform
   terraform plan
   ```
6. Apply the Terraform plan to create the user cluster:

   ```terraform
   terraform apply
   ```

   When prompted, enter `yes`.

   It takes about 15 minutes (or more depending on your network) to create
   the basic user cluster and node pool.

### gcloud

Create the cluster:

```
gcloud container vmware clusters create minimal-installation-user-cluster \
  --project=PROJECT_ID \
  --admin-cluster-membership=projects/PROJECT_ID/locations/global/memberships/minimal-installation-admin-cluster \
  --location=us-central1 \
  --version=1.16.7-gke.46 \
  --admin-users=GOOGLE_ACCOUNT_EMAIL \
  --service-address-cidr-blocks=10.96.0.0/20 \
  --pod-address-cidr-blocks=192.168.0.0/16 \
  --metal-lb-config-address-pools='pool=uc-address-pool,avoid-buggy-ips=False,manual-assign=False,addresses=USER_INGRESS_VIP/32;SERVICE_VIP_1/32;SERVICE_VIP_2/32' \
  --control-plane-vip=USER_CONTROL_PLANE_VIP \
  --ingress-vip=USER_INGRESS_VIP \
  --static-ip-config-ip-blocks='gateway=DEFAULT_GATEWAY_IP,netmask=NETMASK,ips=USER_NODE_IP_1;USER_NODE_IP_2;USER_NODE_IP_3;USER_NODE_IP_4' \
  --dns-servers=DNS_SERVER_IP \
  --ntp-servers=NTP_SERVER_IP \
  --enable-control-plane-v2 \
  --enable-dataplane-v2 \
  --control-plane-ip-block='gateway=DEFAULT_GATEWAY_IP,netmask=NETMASK,ips=USER_CONTROL_PLANE_NODE_IP'
```

The output from the command is similar to the following:

```
Waiting for operation [projects/example-project-12345/locations/us-central1/operations/operation-1679543737105-5f7893fd5bae9-942b3f97-75e59179] to complete.
```

In the example output, the string `operation-1679543737105-5f7893fd5bae9-942b3f97-75e59179`
is the `OPERATION_ID` of the long-running operation. You
can find out the status of the operation with the following command:

```
gcloud container vmware operations describe OPERATION_ID \
  --project=PROJECT_ID \
  --location=us-central1
```

For more information, see
[gcloud container vmware operations](https://docs.cloud.google.com/sdk/gcloud/reference/container/vmware/operations).

It takes 15 minutes or more to create the user cluster. You can view
the cluster in the Google Cloud console on the
[GKE clusters](https://console.cloud.google.com/kubernetes/list/overview)
page.

Create a node pool:

```
gcloud container vmware node-pools create  uc-node-pool \
  --cluster=minimal-installation-user-cluster \
  --project=PROJECT_ID \
  --location=us-central1 \
  --image-type=ubuntu_containerd  \
  --boot-disk-size=40 \
  --cpus=4 \
  --memory=8192 \
  --replicas=3 \
  --enable-load-balancer
```

## What's next

You have now completed this minimal installation of Google Distributed Cloud. As
an optional follow-up, you can see your installation in action by
[deploying an application](https://docs.cloud.google.com/anthos/clusters/docs/on-prem/1.16/how-to/deploy-first-app.md.txt).