This document explains how to create a shared Kubernetes cluster in a
Google Distributed Cloud (GDC) air-gapped zone. A shared cluster spans multiple projects,
and includes comprehensive GDC-managed services that
offer a highly opinionated Kubernetes cluster configuration that is less
configurable than the standard cluster. For more information about standard
clusters, see
[Kubernetes cluster configurations](https://docs.cloud.google.com/distributed-cloud/hosted/docs/latest/gdcag/platform/pa-user/cluster-configs.md.txt).

Shared clusters are a
[zonal resource](https://docs.cloud.google.com/distributed-cloud/hosted/docs/latest/gdcag/resources/multi-zone/api-servers.md.txt#resource-types)
and cannot span multiple zones. To operate clusters in a multi-zone universe,
you must manually create clusters in each zone.

This document is for audiences such as application developers within the application
operator group, who are responsible for managing container workloads within
their organization. For more information, see
[Audiences for GDC air-gapped documentation](https://docs.cloud.google.com/distributed-cloud/hosted/docs/latest/gdcag/resources/audiences.md.txt).

## Before you begin

- To get the permissions that you need to create a shared cluster, ask your
  IAM Org Admin to grant you the User Cluster Admin role
  (`user-cluster-admin`). This role is not bound to a namespace.

- To use the API or Terraform to create a shared cluster, generate the
  kubeconfig file of the zonal API server to host your cluster. Set the
  environment variable `MANAGEMENT_API_SERVER` to the
  kubeconfig path. For more information, see
  [Zonal management API server resources](https://docs.cloud.google.com/distributed-cloud/hosted/docs/latest/gdcag/platform/pa-user/iam/sign-in.md.txt#zonal-resources-kubeconfig).

- Review the cluster [limits](https://docs.cloud.google.com/distributed-cloud/hosted/docs/latest/gdcag/resources/system-limits.md.txt#cluster-management)
  for resource considerations.

## Plan the pod CIDR block

To allocate the appropriate sized pod CIDR block for your workloads, you must
calculate the amount of IP addresses that are required for your Kubernetes
cluster before you create it. Most networking parameters cannot be changed after
the cluster is created.

A Kubernetes cluster follows the proceeding logic when allocating IP addresses:

- Kubernetes assigns a `/24` CIDR block consisting of 256 addresses to each of the nodes. This amount adheres to the default maximum of 110 pods per node for Kubernetes clusters.
- The size of the CIDR block assigned to a node depends on the maximum pods per node value.
- The block always contains at least twice as many addresses as the maximum number of pods per node.

See the following example to understand how the default value of *Per node mask
size= /24* was calculated to accommodate 110 pods:

    Maximum pods per node = 110
    Total number of IP addresses required = 2 * 110 = 220

    Per node mask size = /24
    Number of IP addresses in a /24 = 2(32 - 24) = 256

Determine the required pod CIDR mask to configure for the Kubernetes
cluster based on the required number of nodes. Plan for future node additions
to the cluster while configuring the CIDR range:

      Total number of nodes supported = 2(Per node mask size - pod CIDR mask)

Given there's a default *Per node mask size= /24* , refer to the following
table that maps the pod CIDR mask to the number of nodes supported.

<br />

| **Pod CIDR Mask** | **Calculation: 2^(Per node mask size - CIDR mask)^** | **Maximum number of nodes supported including control plane nodes** |
|---|---|---|
| /21 | 2^(24 - 21)^ | 8 |
| /20 | 2^(24-20)^ | 16 |
| /19 | 2^(24 - 19)^ | 32 |
| /18 | 2^(24 - 18)^ | 64 |

<br />

> [!WARNING]
> **Warning:** If an insufficient CIDR mask is provided, cluster creation might become stuck in a reconciling phase with pods showing the state `CrashLoopBackOff`.

After calculating your pod CIDR block for your Kubernetes cluster, configure it
as part of the cluster creation workflow in the next section.

## Create a shared cluster

Complete the following steps to create a shared Kubernetes cluster:

### Console

1. In the [project picker](https://docs.cloud.google.com/distributed-cloud/hosted/docs/latest/gdcag/platform/pa-user/manage-projects.md.txt#select-project),
   select your organization.

   Shared clusters are organization-scoped resources. If you select a
   project instead, you'll create a standard cluster, which is
   project-scoped. For more information, see the
   [resource hierarchy](https://docs.cloud.google.com/distributed-cloud/hosted/docs/latest/gdcag/resources/resource-hierarchy.md.txt#cluster).
2. In the navigation menu, select **Kubernetes Engine \> Clusters**.

3. Click **Create Cluster**.

4. In the **Name** field, specify a name for the cluster.

5. Select the Kubernetes version for the cluster.

6. Select the zone in which to create the cluster.

7. Click **Attach Project** and select an existing project to attach to
   your cluster. Then click **Save**. You can attach or detach projects
   after creating the cluster from the project details page. You must have
   a project attached to your cluster before deploying container workloads it.

   ![Create a cluster using the console.](https://docs.cloud.google.com/static/distributed-cloud/hosted/docs/latest/gdcag/images/create-cluster.png)
8. Click **Next**.

9. Configure the network settings for your cluster. You can't change these
   network settings after you create the cluster. The default and only
   supported Internet Protocol for Kubernetes clusters is Internet Protocol version 4
   (IPv4).

   1. If you want to create dedicated *load balancer nodes*, enter the number
      of nodes to create. By default, you receive zero nodes, and load
      balancer traffic runs through the control nodes.

   2. Select the **Service CIDR** (Classless Inter-Domain Routing) to use. Your
      deployed services, such as load balancers, are allocated IP addresses
      from this range.

      > [!IMPORTANT]
      > **Important:** The range can be any RFC 1918 range that doesn't conflict with other IP address ranges in the cluster and node pool resources. For more information, see [RFC 1918](https://tools.ietf.org/html/rfc1918).

   3. Select the **Pod CIDR** to use. The cluster allocates IP addresses from
      this range to your pods and VMs.

   4. Click **Next**.

10. Review the details of the auto-generated default node pool for the
    cluster. Click **Edit** to modify the
    default node pool.

    > [!IMPORTANT]
    > **Important:** If you intend the cluster to run graphics processing unit (GPU) workloads, you must provision a GPU machine type in your cluster. For more information, see [GPU workloads in a Kubernetes cluster](https://docs.cloud.google.com/distributed-cloud/hosted/docs/latest/gdcag/platform/pa-user/clusters.md.txt#gpu).

11. To create additional node pools, select **Add Node Pool**. When editing the
    default node pool or adding a new node pool, you customize it with the
    following options:

    1. Assign a name for the node pool. You cannot modify the name after you create the node pool.
    2. Specify the number of worker nodes to create in the node pool.
    3. Select your machine class that best suits your workload requirements.
       View the list of the following settings:

       - Machine type
       - CPU
       - Memory
    4. Click **Save**.

12. Click **Create** to create the cluster.

Shared cluster creation can take up to 90 minutes to complete.

### API

1. Create the `Cluster` custom resource:

   ```yaml
   kubectl --kubeconfig MANAGEMENT_API_SERVER apply -f - <<EOF
   apiVersion: cluster.gdc.goog/v1
   kind: Cluster
   metadata:
     name: CLUSTER_NAME
     namespace: platform
   spec:
     clusterNetwork:
       podCIDRSize: POD_CIDR
       serviceCIDRSize: SERVICE_CIDR
     initialVersion:
       kubernetesVersion: KUBERNETES_VERSION
     nodePools:
     - machineTypeName: MACHINE_TYPE
       name: NODE_POOL_NAME
       nodeCount: NUMBER_OF_WORKER_NODES
       taints: TAINTS
       labels: LABELS
       acceleratorOptions:
         gpuPartitionScheme: GPU_PARTITION_SCHEME
     releaseChannel:
       channel: UNSPECIFIED
   EOF
   ```

   Replace the following:
   - `MANAGEMENT_API_SERVER`: the zonal API server's kubeconfig path.
   - `CLUSTER_NAME`: the name of the cluster. The cluster name must not end with `-system`. The `-system` suffix is reserved for clusters created by GDC.
   - `POD_CIDR`: the size of network ranges from which pod virtual IP addresses are allocated. If unset, a default value `21` is used.
   - `SERVICE_CIDR`: the size of network ranges from which service virtual IP addresses are allocated. If unset, a default value `23` is used.
   - `KUBERNETES_VERSION`: the Kubernetes version of the cluster, such as `1.26.5-gke.2100`. To list the available Kubernetes versions to configure, see [List available Kubernetes versions for a cluster](https://docs.cloud.google.com/distributed-cloud/hosted/docs/latest/gdcag/platform-application/pa-ao-operations/cluster.md.txt#available-kubernetes-versions).
   - `MACHINE_TYPE`: the machine type for the worker nodes of the node pool. View the [available machine types](https://docs.cloud.google.com/distributed-cloud/hosted/docs/latest/gdcag/platform/pa-user/cluster-node-machines.md.txt#available-machine-types) for what is available to configure.
   - `NODE_POOL_NAME`: the name of the node pool.
   - `NUMBER_OF_WORKER_NODES`: the number of worker nodes to provision in the node pool.
   - `TAINTS`: the taints to apply to the nodes of this node pool. This is an optional field.
   - `LABELS`: the labels to apply to the nodes of this node pool. It contains a list of key-value pairs. This is an optional field.
   - `GPU_PARTITION_SCHEME`: the GPU partitioning scheme, if you're running GPU workloads. This is an optional field. For example, `mixed-2`. The GPU is not partitioned if this field is not set. For more information about available Multi-Instance GPU (MIG) profiles, see [Supported MIG profiles](https://docs.cloud.google.com/distributed-cloud/hosted/docs/latest/gdcag/platform/pa-user/cluster-node-machines.md.txt#mig-profiles).

   > [!IMPORTANT]
   > **Important:** You cannot edit node configurations, such as GPU partitioning, after a cluster node pool is created.

   Shared cluster creation can take up to 90 minutes to complete.
2. Create the `ProjectBinding` custom resource:

   ```bash
   kubectl --kubeconfig MANAGEMENT_API_SERVER apply -f - <<EOF
   apiVersion: resourcemanager.gdc.goog/v1
   kind: ProjectBinding
   metadata:
     name: CLUSTER_NAME-PROJECT_NAME
     namespace: platform
     labels:
       resourcemanager.gdc.goog/projectbinding-for-user-project: "true"
   spec:
     clusterRef:
       name: CLUSTER_NAME
     selector:
       nameSelector:
         matchNames:
         - PROJECT_NAME
   EOF
   ```

   Replace the following:
   - `MANAGEMENT_API_SERVER`: the zonal API server's kubeconfig path.
   - `CLUSTER_NAME`: the name of the cluster.
   - `PROJECT_NAME`: the name of the project to bind to. Each `ProjectBinding` resource can only map to one cluster. If a project requires access to multiple clusters, a unique `ProjectBinding` must be created for each cluster.

   You must attach a project to your cluster before a developer can deploy container
   workloads to the cluster.

### Terraform

1. In a Terraform configuration file, insert the following code snippet to
   create the `Cluster` custom resource:

   ```terraform
   provider "kubernetes" {
     config_path = "MANAGEMENT_API_SERVER"
   }

   resource "kubernetes_manifest" "CLUSTER_RESOURCE_NAME" {
     manifest = {
       "apiVersion" = "cluster.gdc.goog/v1"
       "kind" = "Cluster"
       "metadata" = {
         "name" = "CLUSTER_NAME"
         "namespace" = "platform"
       }
       "spec" = {
         "clusterNetwork" = {
           "podCIDRSize" = "POD_CIDR"
           "serviceCIDRSize" = "SERVICE_CIDR"
         }
         "initialVersion" = {
           "kubernetesVersion" = "KUBERNETES_VERSION"
         }
         "nodePools" = [{
           "machineTypeName" = "MACHINE_TYPE"
           "name" = "NODE_POOL_NAME"
           "nodeCount" = "NUMBER_OF_WORKER_NODES"
           "taints" = "TAINTS"
           "labels" = "LABELS"
           "acceleratorOptions" = {
             "gpuPartitionScheme" = "GPU_PARTITION_SCHEME"
           }
         }]
         "releaseChannel" = {
           "channel" = "UNSPECIFIED"
         }
       }
     }
   }
   ```

   Replace the following:
   - `MANAGEMENT_API_SERVER`: the zonal API server's kubeconfig path.
   - `CLUSTER_RESOURCE_NAME`: the unique Terraform resource name of the cluster, such as `cluster-1`. This name is used by Terraform to identify your cluster, and is not used by GDC.
   - `CLUSTER_NAME`: the name of the cluster. The cluster name must not end with `-system`. The `-system` suffix is reserved for clusters created by GDC.
   - `POD_CIDR`: the size of network ranges from which pod virtual IP addresses are allocated. If unset, a default value `21` is used.
   - `SERVICE_CIDR`: the size of network ranges from which service virtual IP addresses are allocated. If unset, a default value `23` is used.
   - `KUBERNETES_VERSION`: the Kubernetes version of the cluster, such as `1.26.5-gke.2100`. To list the available Kubernetes versions to configure, see [List available Kubernetes versions for a cluster](https://docs.cloud.google.com/distributed-cloud/hosted/docs/latest/gdcag/platform-application/pa-ao-operations/cluster.md.txt#available-kubernetes-versions).
   - `MACHINE_TYPE`: the machine type for the worker nodes of the node pool. View the [available machine types](https://docs.cloud.google.com/distributed-cloud/hosted/docs/latest/gdcag/platform/pa-user/cluster-node-machines.md.txt#available-machine-types) for what is available to configure.
   - `NODE_POOL_NAME`: the name of the node pool.
   - `NUMBER_OF_WORKER_NODES`: the number of worker nodes to provision in the node pool.
   - `TAINTS`: the taints to apply to the nodes of this node pool. This is an optional field.
   - `LABELS`: the labels to apply to the nodes of this node pool. It contains a list of key-value pairs. This is an optional field.
   - `GPU_PARTITION_SCHEME`: the GPU partitioning scheme, if you're running GPU workloads. This is an optional field. For example, `mixed-2`. The GPU is not partitioned if this field is not set. For more information about available Multi-Instance GPU (MIG) profiles, see [Supported MIG profiles](https://docs.cloud.google.com/distributed-cloud/hosted/docs/latest/gdcag/platform/pa-user/cluster-node-machines.md.txt#mig-profiles).
2. In a Terraform configuration file, insert the following code snippet to
   create the `ProjectBinding` custom resource:

   ```terraform
   provider "kubernetes" {
     config_path = "MANAGEMENT_API_SERVER"
   }

   resource "kubernetes_manifest" "PROJECT_BINDING_RESOURCE_NAME" {
     manifest = {
       "apiVersion" = "resourcemanager.gdc.goog/v1"
       "kind" = "ProjectBinding"
       "metadata" = {
         "name" = "CLUSTER_NAME-PROJECT_NAME"
         "namespace" = "platform"
         "labels" = {
           "resourcemanager.gdc.goog/projectbinding-for-user-project" = "true"
         }
       }
       "spec" = {
         "clusterRef" = {
           "name" = "CLUSTER_NAME"
         }
         "selector" = {
           "nameSelector" = {
             "matchNames" = [
               "PROJECT_NAME",
             ]
           }
         }
       }
     }
   }
   ```

   Replace the following:
   - `MANAGEMENT_API_SERVER`: the zonal API server's kubeconfig path.
   - `PROJECT_BINDING_RESOURCE_NAME`: the Terraform resource name of the project binding, such as `project-binding-1`. This name is used by Terraform to identify your project binding, and is not used by GDC.
   - `CLUSTER_NAME`: the name of the cluster.
   - `PROJECT_NAME`: the name of the project to bind to. Each `ProjectBinding` resource can only map to one cluster. If a project requires access to multiple clusters, a unique `ProjectBinding` must be created for each cluster.

   You must attach a project to your cluster before a developer can deploy container
   workloads to the cluster.
3. Apply the new custom resources using Terraform:

   ```bash
   terraform apply
   ```

Shared cluster creation can take up to 90 minutes to complete.

## What's next

- [Deploy an HA container app](https://docs.cloud.google.com/distributed-cloud/hosted/docs/latest/gdcag/platform-application/pa-ao-operations/ha-apps/deploy-ha-container-app.md.txt)
- [Container workloads in GDC](https://docs.cloud.google.com/distributed-cloud/hosted/docs/latest/gdcag/application/ao-user/containers/containers-intro.md.txt)
- [Resource hierarchy](https://docs.cloud.google.com/distributed-cloud/hosted/docs/latest/gdcag/resources/resource-hierarchy.md.txt)