This topic describes how to customize the configuration of a
GKE on AWS
[user cluster](https://docs.cloud.google.com/kubernetes-engine/multi-cloud/docs/aws/previous-generation/concepts/architecture.md.txt#user_clusters).

You might want to create a custom user cluster for the following reasons:

- Creating another cluster for a staging or test environment.
- Adding node pools with different machine types.
- Creating a cluster in specific AWS availability zones (AZ).

## Before you begin


Before you start using GKE on AWS, make sure you have performed the following tasks:

- Complete the [Prerequisites](https://docs.cloud.google.com/kubernetes-engine/multi-cloud/docs/aws/previous-generation/how-to/prerequisites.md.txt).

<!-- -->

- Install a [management service](https://docs.cloud.google.com/kubernetes-engine/multi-cloud/docs/aws/previous-generation/how-to/installing-management.md.txt).

- If you want to create a cluster without using
  [`terraform output example_cluster`](https://docs.cloud.google.com/kubernetes-engine/multi-cloud/docs/aws/previous-generation/how-to/creating-user-cluster.md.txt),
  have private AWS subnets for your control plane. Each subnet should
  belong to a different AZ in the same AWS region. Route tables must be
  configured to allow traffic between private subnets, and each subnet must have
  access to a NAT gateway.

- Have your AWS Virtual Private Cloud (VPC) ID. A VPC ID looks like
  `vpc-012345678abcde`. You can find your VPC ID on the
  [AWS Console](https://console.aws.amazon.com/vpc/home).

To connect to your GKE on AWS resources, perform the following
steps. Select if you have an existing AWS VPC (or direct connection to
your VPC) or created a dedicated VPC when creating your management service.

### Existing VPC

If you have a direct or VPN connection to an existing VPC, omit the line
`env HTTP_PROXY=http://localhost:8118` from commands in this topic.

### Dedicated VPC

When you create a management service in a dedicated VPC,
GKE on AWS includes a
[bastion](https://en.wikipedia.org/wiki/Bastion_host) host in a
public subnet.

> [!IMPORTANT]
> **Important:** If you restart your terminal session or the SSH connection is lost, you need to re-launch the `bastion-tunnel.sh` script.

To connect to your management service, perform the following steps:

1. Change to the directory with your GKE on AWS configuration.
   You created this directory when
   [Installing the management service](https://docs.cloud.google.com/kubernetes-engine/multi-cloud/docs/aws/previous-generation/how-to/installing-management.md.txt).

   ```sh
   cd anthos-aws
   ```

   <br />

2. To open the tunnel, run the `bastion-tunnel.sh` script. The tunnel forwards
   to `localhost:8118`.

   To open a tunnel to the bastion host, run the following command:

   ```bash
   ./bastion-tunnel.sh -N
   ```

   Messages from the SSH tunnel appear in this window. When you are ready to
   close the connection, stop the process by using <kbd>Control+C</kbd> or
   closing the window.
3. Open a new terminal and change into your `anthos-aws` directory.

   ```sh
   cd anthos-aws
   ```
4. Check that you're able to connect to the cluster with `kubectl`.

   ```bash
   env HTTPS_PROXY=http://localhost:8118 \
   kubectl cluster-info
   ```

   The output includes the URL for the management service API server.

### Selecting a control plane instance size

GKE on AWS doesn't support resizing control plane instances. Before
creating your user cluster, select the instance size of your control
planes. Control plane sizes depend on the number of nodes in your cluster. The
following table contains recommended control plane
[instance sizes](https://docs.cloud.google.com/kubernetes-engine/multi-cloud/docs/aws/previous-generation/reference/instance-types.md.txt) for various cluster
sizes.

| Cluster size (nodes) | Control plane instance type |
|---|---|
| 1 -- 10 | m5.large |
| 11 -- 100 | m5.xlarge |
| 101 -- 200 | m5.2xlarge |

## Creating a new cluster with a custom configuration

You can use
[`terraform output example_cluster`](https://docs.cloud.google.com/kubernetes-engine/multi-cloud/docs/aws/previous-generation/how-to/creating-user-cluster.md.txt)
to create configuration for one user cluster per management cluster. If you want
to create additional clusters, you need to apply a custom configuration.

In this example, you create a cluster manually from `AWSCluster` and
`AWSNodePool` CRDs.

1. Change to the directory with your GKE on AWS configuration.
   You created this directory when
   [Installing the management service](https://docs.cloud.google.com/kubernetes-engine/multi-cloud/docs/aws/previous-generation/how-to/installing-management.md.txt).

   ```sh
   cd anthos-aws
   ```

   <br />

2. From your `anthos-aws` directory, use
   `anthos-gke` to switch context to your management service.

   ```sh
   cd anthos-aws
   anthos-gke aws management get-credentials
   ```

   > [!NOTE]
   > **Note:** `anthos-gke` uses the credentials you specified when you ran `aws configure`.

   <br />

3. Open a text editor and copy the following `AWSCluster` definition into a file
   named `custom-cluster.yaml`.

   ```yaml
   apiVersion: multicloud.cluster.gke.io/v1
   kind: AWSCluster
   metadata:
     name: CLUSTER_NAME
   spec:
     region: AWS_REGION
     networking:
       vpcID: VPC_ID
       podAddressCIDRBlocks: POD_ADDRESS_CIDR_BLOCKS
       serviceAddressCIDRBlocks: SERVICE_ADDRESS_CIDR_BLOCKS
       ServiceLoadBalancerSubnetIDs: SERVICE_LOAD_BALANCER_SUBNETS
     controlPlane:
       version:  CLUSTER_VERSION # Latest version is 1.25.5-gke.2100
       instanceType: AWS_INSTANCE_TYPE
       keyName: SSH_KEY_NAME
       subnetIDs:
       - CONTROL_PLANE_SUBNET_IDS
       securityGroupIDs:
       - CONTROL_PLANE_SECURITY_GROUPS
       iamInstanceProfile: CONTROL_PLANE_IAM_ROLE
       rootVolume:
         sizeGiB: ROOT_VOLUME_SIZE
         volumeType: ROOT_VOLUME_TYPE # Optional
         iops: ROOT_VOLUME_IOPS # Optional
         kmsKeyARN: ROOT_VOLUME_KEY # Optional
       etcd:
         mainVolume:
           sizeGiB: ETCD_VOLUME_SIZE
           volumeType: ETCD_VOLUME_TYPE # Optional
           iops: ETCD_VOLUME_IOPS # Optional
           kmsKeyARN: ETCD_VOLUME_KEY # Optional
       databaseEncryption:
         kmsKeyARN: ARN_OF_KMS_KEY
       hub: # Optional
         membershipName: ANTHOS_CONNECT_NAME
       cloudOperations: # Optional
         projectID: YOUR_PROJECT
         location: GCP_REGION
         enableLogging: ENABLE_LOGGING
         enableMonitoring: ENABLE_MONITORING
       workloadIdentity: # Optional
         oidcDiscoveryGCSBucket: WORKLOAD_IDENTITY_BUCKET
   ```

   Replace the following:
   - <var translate="no">CLUSTER_NAME</var>: the name of your cluster.
   - <var translate="no">AWS_REGION</var>: the AWS region where your cluster runs.

     > [!NOTE]
     > **Note:** You must create your user cluster in the same region as your management service.

   - <var translate="no">VPC_ID</var>: the ID of the VPC where your cluster runs.

   - <var translate="no">POD_ADDRESS_CIDR_BLOCKS</var>: the range of IPv4 addresses used by the
     cluster's pods. Currently only a single range is supported. The range
     must not overlap with any subnets reachable from your network. It is
     safe to use the same range across multiple different AWSCluster objects.
     For example, `10.2.0.0/16`.

   - <var translate="no">SERVICE_ADDRESS_CIDR_BLOCKS</var>: the range of IPv4 addresses used
     by the cluster's services. Currently only a single range is supported.
     The range must not overlap with any subnets reachable from your network.
     It is safe to use the same range across multiple different AWSCluster
     objects. For example, `10.1.0.0/16`.

   - <var translate="no">SERVICE_LOAD_BALANCER_SUBNETS</var>: the subnet IDs where
     GKE on AWS can create public or private load balancers.

   - <var translate="no">CLUSTER_VERSION</var>: a
     [Kubernetes version](https://docs.cloud.google.com/kubernetes-engine/multi-cloud/docs/aws/previous-generation/concepts/versioning-and-upgrades.md.txt#versions)
     supported by GKE on AWS. The most recent version is
     1.25.5-gke.2100.

   - <var translate="no">AWS_INSTANCE_TYPE</var>: a
     [supported EC2 instance type](https://docs.cloud.google.com/kubernetes-engine/multi-cloud/docs/aws/previous-generation/reference/instance-types.md.txt).

   - <var translate="no">SSH_KEY_NAME</var>: an
     [AWS EC2 key pair](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-key-pairs.html#prepare-key-pair).

   - <var translate="no">CONTROL_PLANE_SUBNET_IDS</var>: the subnet IDs in the AZs
     where your control plane instances run.

   - <var translate="no">CONTROL_PLANE_SECURITY_GROUPS</var>: a securityGroupID created during
     the management service installation.
     You can customize this by adding any securityGroupIDs required to connect
     to the control plane.

   - <var translate="no">CONTROL_PLANE_IAM_PROFILE</var>: name of the AWS EC2 instance
     profile assigned to control plane replicas.

   - <var translate="no">ROOT_VOLUME_SIZE</var>: the size, in gibibyte (GiB), of your
     control plane root volumes.

   - <var translate="no">ROOT_VOLUME_TYPE</var> with the
     [EBS volume type](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-volume-types.html).
     For example, `gp3`.

   - <var translate="no">ROOT_VOLUME_IOPS</var> with the amount of provisioned IO operations per
     second (IOPS) for the volume. Only valid when `volumeType` is `GP3`. For
     more information, see
     [General Purpose SSD volumes (gp3)](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-volume-types.html#gp3-ebs-volume-type).

   - <var translate="no">ROOT_VOLUME_KEY</var> with the Amazon Resource Name of the
     [AWS KMS key](https://docs.cloud.google.com/kubernetes-engine/multi-cloud/docs/aws/previous-generation/how-to/prerequisites.md.txt#creating_a_kms_key) that
     encrypts your control plane instance root volumes.

   - <var translate="no">ETCD_VOLUME_SIZE</var>: the size of volumes used by etcd.

   - <var translate="no">ETCD_VOLUME_TYPE</var> with the
     [EBS volume type](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-volume-types.html).
     For example, `gp3`.

   - <var translate="no">ETCD_VOLUME_IOPS</var> with the amount of provisioned IO operations per
     second (IOPS) for the volume. Only valid when `volumeType` is `gp3`. For
     more information, see
     [General Purpose SSD volumes (gp3)](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-volume-types.html#gp3-ebs-volume-type).

   - <var translate="no">ETCD_VOLUME_KEY</var> with the Amazon Resource Name of the
     [AWS KMS key](https://docs.cloud.google.com/kubernetes-engine/multi-cloud/docs/aws/previous-generation/how-to/prerequisites.md.txt#creating_a_kms_key) that
     encrypts your control plane etcd data volumes.

   - <var translate="no">ARN_OF_KMS_KEY</var>: the AWS KMS key used to encrypt cluster
     Secrets.

   - <var translate="no">ANTHOS_CONNECT_NAME</var>: the
     [Connect](https://docs.cloud.google.com/anthos/multicluster-management/connect/overview)
     membership name used to register your cluster. The membership name
     must be unique. For example,
     `projects/YOUR_PROJECT/locations/global/memberships/CLUSTER_NAME`,
     where `YOUR_PROJECT` is your Google Cloud project
     and `CLUSTER_NAME` is a unique name in your project.
     This field is optional.

   - <var translate="no">YOUR_PROJECT</var>: your project ID.

   - <var translate="no">GCP_REGION</var>: the Google Cloud region where you want to store logs.
     Choose a region that is near the AWS region. For more information, see
     [Global Locations - Regions \& Zones](https://cloud.google.com/about/locations)
     --- for example, `us-central1`.

   - <var translate="no">ENABLE_LOGGING</var>: `true` or `false`, whether Cloud Logging is enabled on control plane nodes.

   - <var translate="no">ENABLE_MONITORING</var>: `true` or `false`, whether Cloud Monitoring is enabled on control plane nodes.

   - <var translate="no">WORKLOAD_IDENTITY_BUCKET</var>: the Cloud Storage bucket
     name containing your
     [workload identity](https://docs.cloud.google.com/kubernetes-engine/multi-cloud/docs/aws/previous-generation/how-to/workload-identity-cluster)
     discovery information. This field is optional.

4. Create one or more AWSNodePools for your cluster. Open a text
   editor and copy the following AWSCluster definition into a file
   named `custom-nodepools.yaml`.

   ```yaml
   apiVersion: multicloud.cluster.gke.io/v1
   kind: AWSNodePool
   metadata:
     name: NODE_POOL_NAME
   spec:
     clusterName: AWSCLUSTER_NAME
     version:  CLUSTER_VERSION # latest version is 1.25.5-gke.2100
     region: AWS_REGION
     subnetID: AWS_SUBNET_ID
     minNodeCount: MINIMUM_NODE_COUNT
     maxNodeCount: MAXIMUM_NODE_COUNT
     maxPodsPerNode: MAXIMUM_PODS_PER_NODE_COUNT
     instanceType: AWS_NODE_TYPE
     keyName: KMS_KEY_PAIR_NAME
     iamInstanceProfile: NODE_IAM_PROFILE
     proxySecretName: PROXY_SECRET_NAME
     rootVolume:
       sizeGiB: ROOT_VOLUME_SIZE
       volumeType: VOLUME_TYPE # Optional
       iops: IOPS # Optional
       kmsKeyARN: NODE_VOLUME_KEY # Optional 
   ```

   Replace the following:
   - <var translate="no">NODE_POOL_NAME</var>: a unique name for your AWSNodePool.
   - <var translate="no">AWSCLUSTER_NAME</var>: your AWSCluster's name. For example, `staging-cluster`.
   - <var translate="no">CLUSTER_VERSION</var>: a supported GKE on AWS Kubernetes [version](https://docs.cloud.google.com/kubernetes-engine/multi-cloud/docs/aws/previous-generation/concepts/versioning-and-upgrades.md.txt).
   - <var translate="no">AWS_REGION</var>: the same AWS region as your AWSCluster.
   - <var translate="no">AWS_SUBNET_ID</var>: an AWS subnet in the same region as your AWSCluster.
   - <var translate="no">MINIMUM_NODE_COUNT</var>: the minimum number of nodes in the node pool. See [Scaling user clusters](https://docs.cloud.google.com/kubernetes-engine/multi-cloud/docs/aws/previous-generation/how-to/scaling-user-cluster.md.txt) for more information.
   - <var translate="no">MAXIMUM_NODE_COUNT</var>: the maximum number of nodes in the node pool.
   - <var translate="no">MAXIMUM_PODS_PER_NODE_COUNT</var>: the maximum number of pods that GKE on AWS can allocate to a node.
   - <var translate="no">AWS_NODE_TYPE</var>: an [AWS EC2 instance type](https://docs.cloud.google.com/kubernetes-engine/multi-cloud/docs/aws/previous-generation/reference/instance-types.md.txt).
   - <var translate="no">KMS_KEY_PAIR_NAME</var>: the AWS KMS key pair assigned to each node pool worker.
   - <var translate="no">NODE_IAM_PROFILE</var>: the name of the AWS EC2 instance profile assigned to nodes in the pool.
   - <var translate="no">ROOT_VOLUME_SIZE</var>: the size, in gibibyte (GiB), of your control plane root volumes.
   - <var translate="no">VOLUME_TYPE</var>: the node's AWS [EBS volume type](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-volume-types.html). For example, `gp3`.
   - <var translate="no">IOPS</var>: the amount of provisioned IO operations per second (IOPS) for volumes. Only valid when `volumeType` is `gp3`.
   - <var translate="no">NODE_VOLUME_KEY</var>: the ARN of the AWS KMS key used to encrypt the volume. For more information, see [Using a customer managed CMK to encrypt volumes](https://docs.cloud.google.com/kubernetes-engine/multi-cloud/docs/aws/previous-generation/how-to/using-cmk.md.txt).
5. Apply the manifests to your management service.

   ```bash
   env HTTPS_PROXY=http://localhost:8118 \
     kubectl apply -f custom-cluster.yaml
   env HTTPS_PROXY=http://localhost:8118 \
     kubectl apply -f custom-nodepools.yaml
   ```

   > [!NOTE]
   > **Note:** Your cluster takes several minutes to start.

### Create a kubeconfig

While your user cluster starts, you can create a `kubeconfig` context for your
new user cluster. You use the context to authenticate to a user or management
cluster.

1. Use `anthos-gke aws clusters get-credentials` to generate a
   [`kubeconfig`](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/cluster-access-for-kubectl.md.txt#kubeconfig)
   for your user cluster in `~/.kube/config`.

   ```bash
   env HTTPS_PROXY=http://localhost:8118 \
     anthos-gke aws clusters get-credentials CLUSTER_NAME
   ```

   Replace <var translate="no">CLUSTER_NAME</var> with your cluster's name. For example,
   `cluster-0`.
2. Use `kubectl` to authenticate to your new user cluster.

   ```bash
   env HTTPS_PROXY=http://localhost:8118 \
     kubectl cluster-info
   ```

   If your cluster is ready, the output includes the URLs for Kubernetes
   components within your cluster.

## Viewing your cluster's status

The management service provisions AWS resources when you apply an
`AWSCluster` or `AWSNodePool`.

1. From your `anthos-aws` directory, use
   `anthos-gke` to switch context to your management service.

   ```sh
   cd anthos-aws
   anthos-gke aws management get-credentials
   ```

   > [!NOTE]
   > **Note:** `anthos-gke` uses the credentials you specified when you ran `aws configure`.

   <br />

2. To list your clusters, use `kubectl get AWSClusters`.

   ```bash
   env HTTPS_PROXY=http://localhost:8118 \
     kubectl get AWSClusters
   ```

   The output includes each cluster's name, state, age, version, and endpoint.

   For example, the following output includes only one `AWSCluster` named `cluster-0`:

       NAME        STATE          AGE     VERSION         ENDPOINT
       cluster-0   Provisioning   2m41s   1.25.5-gke.2100   gke-xyz.elb.us-east-1.amazonaws.com

### View your cluster's events

To see recent
[Kubernetes Events](https://www.bluematador.com/blog/kubernetes-events-explained)
from your user cluster, use `kubectl get events`.

1. From your `anthos-aws` directory, use
   `anthos-gke` to switch context to your management service.

   ```sh
   cd anthos-aws
   anthos-gke aws management get-credentials
   ```

   > [!NOTE]
   > **Note:** `anthos-gke` uses the credentials you specified when you ran `aws configure`.

   <br />

2. Run `kubectl get events`.

   ```bash
   env HTTPS_PROXY=http://localhost:8118 \
     kubectl get events
   ```

The output includes information, warning, and errors related to from your
management service.

### Deleting a user cluster

To delete a user cluster, perform the following steps:

1. From your `anthos-aws` directory, use
   `anthos-gke` to switch context to your management service.

   ```sh
   cd anthos-aws
   anthos-gke aws management get-credentials
   ```

   > [!NOTE]
   > **Note:** `anthos-gke` uses the credentials you specified when you ran `aws configure`.

   <br />

2. Use `kubectl delete` to delete the manifest containing your user clusters.

   ```bash
   env HTTPS_PROXY=http://localhost:8118 \
     kubectl delete -f CLUSTER_FILE
   ```

   Replace <var translate="no">CLUSTER_FILE</var> with the name of the manifest containing
   your AWScluster and AWSNodePool objects. For example, `cluster-0.yaml`.

### Deleting all user clusters

To delete all of your user clusters, perform the following steps:

1. From your `anthos-aws` directory, use
   `anthos-gke` to switch context to your management service.

   ```sh
   cd anthos-aws
   anthos-gke aws management get-credentials
   ```

   > [!NOTE]
   > **Note:** `anthos-gke` uses the credentials you specified when you ran `aws configure`.

   <br />

2. Use `kubectl delete` to delete your AWSNodePools and AWSClusters from your
   management service.

   ```bash
   env HTTPS_PROXY=http://localhost:8118 \
     kubectl delete AWSNodePool --all
   env HTTPS_PROXY=http://localhost:8118 \
     kubectl delete AWSCluster --all
   ```

For more information, see
[Uninstalling GKE on AWS](https://docs.cloud.google.com/kubernetes-engine/multi-cloud/docs/aws/previous-generation/how-to/uninstalling.md.txt).

## What's next

- Configure your identity provider with the
  [GKE Identity Service](https://docs.cloud.google.com/anthos/identity).

- [Launch your first workload](https://docs.cloud.google.com/kubernetes-engine/multi-cloud/docs/aws/previous-generation/quickstart.md.txt) on
  GKE on AWS.

- Create an externally facing deployment using a
  [load balancer](https://docs.cloud.google.com/kubernetes-engine/multi-cloud/docs/aws/previous-generation/how-to/loadbalancer.md.txt) or
  [Ingress](https://docs.cloud.google.com/kubernetes-engine/multi-cloud/docs/aws/previous-generation/how-to/ingress.md.txt).

- Read the specifications for the
  [AWSCluster](https://docs.cloud.google.com/kubernetes-engine/multi-cloud/docs/aws/previous-generation/reference/awscluster.md.txt) and
  [AWSNodePool](https://docs.cloud.google.com/kubernetes-engine/multi-cloud/docs/aws/previous-generation/reference/node-pools)
  Custom Resource Definitions.