<button class="button button-with-icon cloud-select-dropdown__button button-primary" aria-haspopup="true">Version 1.16 </button>

- [Latest version](https://docs.cloud.google.com/anthos/clusters/docs/on-prem/latest/how-to/create-user-cluster)
- [Version 1.16](https://docs.cloud.google.com/anthos/clusters/docs/on-prem/1.16/how-to/create-user-cluster.md.txt)
- [Earlier versions (unsupported)](https://docs.cloud.google.com/anthos/clusters/docs/on-prem/latest/version-history#expandable-1)

<br />

This document shows how to create a user cluster that uses kubeception.

## What is kubeception?

The term
[*kubeception*](https://kubernetes.io/blog/2017/01/how-we-run-kubernetes-in-kubernetes-kubeception/)
is used to convey the idea that a Kubernetes cluster is used to create and
manage other Kubernetes clusters. In the context of Google Distributed Cloud,
kubeception refers to the case where the control plane for a user cluster runs
on one or more nodes in an admin cluster.

We don't recommend using kubeception. Instead, we recommend using
[Controlplane V2](https://docs.cloud.google.com/anthos/clusters/docs/on-prem/1.16/how-to/create-user-cluster-controlplane-v2.md.txt). With
Controlplane V2, the control-plane nodes for the user cluster are in the user
cluster itself.

## Plan your IP addresses

Follow the instructions in
[Plan your IP addresses (kubeception)](https://docs.cloud.google.com/anthos/clusters/docs/on-prem/1.16/how-to/plan-ip-addresses-kubeception.md.txt).

## Fill in a cluster configuration file

Follow the instructions in
[Create a user cluster (Controlplane V2)](https://docs.cloud.google.com/anthos/clusters/docs/on-prem/1.16/how-to/create-user-cluster-controlplane-v2.md.txt).

As you fill in your user cluster configuration file:

- Set `enableControlplaneV2` to `false`.

- Decide what type of load balancing you want to use. The options are:

  - MetalLB bundled load balancing. Set
    [`loadBalancer.kind`](https://docs.cloud.google.com/anthos/clusters/docs/on-prem/1.16/how-to/user-cluster-configuration-file.md.txt#loadbalancer-kind-field)
    to `"MetalLB"`. Also fill in the
    [`loadBalancer.metalLB.addressPools`](https://docs.cloud.google.com/anthos/clusters/docs/on-prem/1.16/how-to/user-cluster-configuration-file.md.txt#loadbalancer-metallb-section)
    section, and set `enableLoadBalancer` to `true` for at least one of your
    node pools. For more information, see
    [Bundled load balancing with MetalLB](https://docs.cloud.google.com/anthos/clusters/docs/on-prem/1.16/how-to/bundled-load-balance-metallb.md.txt).

  - Seesaw bundled load balancing. Set `loadBalancer.kind` to `"Seesaw"`, and
    fill in the
    [`loadBalancer.seesaw`](https://docs.cloud.google.com/anthos/clusters/docs/on-prem/1.16/how-to/user-cluster-configuration-file.md.txt#loadbalancer-seesaw-section)
    section. For more information, see
    [Bundled load balancing with Seesaw](https://docs.cloud.google.com/anthos/clusters/docs/on-prem/1.16/how-to/bundled-load-balance.md.txt).

  - Integrated load balancing with F5 BIG-IP. Set `loadBalancer.kind` to
    `"F5BigIP"`, and fill in the
    [`f5BigIP`](https://docs.cloud.google.com/anthos/clusters/docs/on-prem/1.16/how-to/user-cluster-configuration-file.md.txt#loadbalancer-f5bigip-section)
    section. For more information, see
    [Load balancing with F5 BIG-IP](https://docs.cloud.google.com/anthos/clusters/docs/on-prem/1.16/how-to/f5-big-ip-load-balance.md.txt).

  - Manual load balancing. Set `loadBalancer.kind` to `"ManualLB"`, and fill in
    the
    [`manualLB`](https://docs.cloud.google.com/anthos/clusters/docs/on-prem/1.16/how-to/user-cluster-configuration-file.md.txt#loadbalancer-manuallb-section)
    section. For more information, see
    [Manual load balancing](https://docs.cloud.google.com/anthos/clusters/docs/on-prem/1.16/how-to/manual-load-balance.md.txt).

  For more information about load balancing options, see
  [Overview of load balancing](https://docs.cloud.google.com/anthos/clusters/docs/on-prem/1.16/how-to/setup-load-balance.md.txt).
- Decide whether you want to enable
  [Dataplane V2](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/dataplane-v2.md.txt)
  for your user cluster, and set
  [enableDataplaneV2](https://docs.cloud.google.com/anthos/clusters/docs/on-prem/1.16/how-to/user-cluster-configuration-file.md.txt#enabledataplanev2-field)
  accordingly.

- If you include the `stackdriver` and `cloudAuditLogging` sections in the
  configuration file, the ID in `gkeConnect.projectID` must be the same as
  the ID set in `stackdriver.projectID` and `cloudAuditLogging.projectID`.
  If the project IDs aren't the same, cluster creation fails.

- If you include the `gkeOnPremAPI`, `cloudAuditLogging`, and the
  `stackdriver` sections in the configuration file, you must set the same
  [Google Cloud region](https://docs.cloud.google.com/anthos/clusters/docs/on-prem/1.16/reference/supported-regions-on-prem-api.md.txt) in
  `gkeOnPremAPI.location`, `cloudAuditLogging.clusterLocation`, and
  `stackdriver.location`. If the regions aren't the same,
  cluster creation fails.

### Example of filled-in configuration files

Here is an example of a filled-in IP block file and a filled-in user cluster
configuration file. The configuration enables some, but not all, of the
available features.

#### user-ipblock.yaml

```
blocks:
  - netmask: 255.255.252.0
    gateway: 172.16.23.254
    ips:
    - ip: 172.16.20.21
      hostname: user-host1
    - ip: 172.16.20.22
      hostname: user-host2
    - ip: 172.16.20.23
      hostname: user-host3
    - ip: 172.16.20.24
      hostname: user-host4
```

#### user-cluster.yaml

```
apiVersion: v1
kind: UserCluster
name: "my-user-cluster"
gkeOnPremVersion: 1.15.0-gke.581
enableControlplaneV2: false
network:
  hostConfig:
    dnsServers:
    - "203.0.113.1"
    - "198.51.100.1"
    ntpServers:
    - "216.239.35.4"
  ipMode:
    type: static
    ipBlockFilePath: "user-ipblock.yaml"
  serviceCIDR: 10.96.0.0/20
  podCIDR: 192.168.0.0/16
loadBalancer:
  vips:
    controlPlaneVIP: "172.16.20.32"
    ingressVIP: "172.16.21.30"
  kind: "MetalLB"
  metalLB:
    addressPools:
    - name: "my-address-pool"
      addresses:
      - "172.16.21.30 - 172.16.21.39"
enableDataplaneV2: true
masterNode:
  cpus: 4
  memoryMB: 8192
  replicas: 1
nodePools:
- name: "my-node-pool"
  cpus: 4
  memoryMB: 8192
  replicas: 3
  osImageType: "ubuntu_containerd"
  enableLoadBalancer: true
antiAffinityGroups:
  enabled: true
gkeConnect:
  projectID: "my-project-123"
  registerServiceAccountKeyPath: "connect-register-sa-2203040617.json"
stackdriver:
  projectID: "my-project-123"
  clusterLocation: "us-central1"
  enableVPC: false
  serviceAccountKeyPath: "log-mon-sa-2203040617.json"
autoRepair:
  enabled: true
```

## Validate your configuration file

After you've filled in your user cluster configuration file, run
`gkectl check-config` to verify that the file is valid:

```
gkectl check-config --kubeconfig ADMIN_CLUSTER_KUBECONFIG --config USER_CLUSTER_CONFIG
```

Replace the following:

- <var translate="no">ADMIN_CLUSTER_KUBECONFIG</var>: the path of the kubeconfig file for
  your admin cluster

- <var translate="no">USER_CLUSTER_CONFIG</var>: the path of your user cluster configuration
  file

If the command returns any failure messages, fix the issues and validate the
file again.

If you want to skip the more time-consuming validations, pass the `--fast` flag.
To skip individual validations, use the `--skip-validation-xxx` flags. To
learn more about the `check-config` command, see
[Running preflight checks](https://docs.cloud.google.com/anthos/clusters/docs/on-prem/1.16/how-to/preflight-checks.md.txt).

## (Optional) Create a Seesaw load balancer for your user cluster

If you have chosen to use the bundled Seesaw load balancer, do the step in this
section. Otherwise, skip this section.

Create and configure the VMs for your
[Seesaw load balancer](https://docs.cloud.google.com/anthos/clusters/docs/on-prem/1.16/how-to/bundled-load-balance.md.txt):

```
gkectl create loadbalancer --kubeconfig ADMIN_CLUSTER_KUBECONFIG --config USER_CLUSTER_CONFIG
```

## (Optional) Import OS images to vSphere, and push container images to a private registry

Run `gkectl prepare` if any of the following are true:

- Your user cluster is in a different vSphere data center from your admin
  cluster.

- Your user cluster has a different vCenter Server from your admin cluster.

- Your user cluster uses a private container registry that is different from the
  private registry used by your admin cluster.

```
gkectl prepare --kubeconfig ADMIN_CLUSTER_KUBECONFIG \ 
    --bundle-path BUNDLE \
    --user-cluster-config USER_CLUSTER_CONFIG
```

Replace the following:

- <var translate="no">ADMIN_CLUSTER_KUBECONFIG</var>: the path of your admin cluster kubeconfig
  file

- <var translate="no">BUNDLE</var>: the path of the bundle file. This file is on your admin
  workstation in `/var/lib/gke/bundles/`. For example:

  ```
  /var/lib/gke/bundles/gke-onprem-vsphere-1.14.0-gke.421-full.tgz
  ```
- <var translate="no">USER_CLUSTER_CONFIG</var>: the path of your user cluster configuration
  file

## Create the user cluster

Run the following command to create the user cluster:

```
gkectl create cluster --kubeconfig ADMIN_CLUSTER_KUBECONFIG --config USER_CLUSTER_CONFIG 
```

Locate the user cluster kubeconfig file

The `gkectl create cluster` command creates a kubeconfig file named
`USER_CLUSTER_NAME-kubeconfig` in the current directory. You will need this
kubeconfig file later to interact with your user cluster.

The kubeconfig file contains the name of your user cluster. To view the cluster name, you can run:

```
kubectl config get-clusters --kubeconfig USER_CLUSTER_KUBECONFIG
```

The output shows the name of the cluster. For example:

```
NAME
my-user-cluster
```

If you like, you can change the name and location of your kubeconfig file.

## Verify that your user cluster is running

Run the following command to verify that your user cluster is running:

```
kubectl get nodes --kubeconfig USER_CLUSTER_KUBECONFIG
```

Replace <var translate="no">USER_CLUSTER_KUBECONFIG</var> with the path of your user cluster
kubeconfig file.

The output shows the user cluster nodes. For example:

```
my-user-cluster-node-pool-69-d46d77885-7b7tx   Ready ...
my-user-cluster-node-pool-69-d46d77885-lsvzk   Ready ...
my-user-cluster-node-pool-69-d46d77885-sswjk   Ready ...
```

## Troubleshooting

See [Troubleshooting cluster creation and upgrade](https://docs.cloud.google.com/anthos/clusters/docs/on-prem/1.16/troubleshoot-cluster-creation-upgrade.md.txt).

## What's next

- [Connect to your cluster from the Google Cloud console](https://docs.cloud.google.com/anthos/clusters/docs/on-prem/1.16/how-to/connecting-to-a-cluster).
- [Deploy an application](https://docs.cloud.google.com/anthos/clusters/docs/on-prem/1.16/how-to/deploy-first-app.md.txt).