This document describes how Managed Service for Apache Kafka manages storage
for a Kafka cluster, and how to configure disk capacity in a cluster. The
service automatically provisions disks when you create a cluster, but depending
on your Kafka workloads, you might want to allocate more disk capacity than the
default.

## Overview

Managed Service for Apache Kafka uses
[tiered storage](https://kafka.apache.org/37/operations/tiered-storage/).
Tiered storage combines two tiers of storage:

- A limited amount of fast *local storage*, in the form of pre-provisioned disks attached to the brokers.
- Effectively unlimited *remote storage*. Remote storage is less expensive and doesn't need to be pre-provisioned.

The tiered storage system lets you store large volumes of data with less risk
of running out of disk space. It can also cost less than using local storage
alone.

When a broker receives a message, it writes the message to a local segment file.
When the segment file reaches a maximum size or age, it is closed (or "rolled")
and moved to remote storage. At any time, there is a single open segment file
per partition, so the volume of local storage scales with the number of
partitions and the maximum size of the segment files. Brokers that handle more
partitions need more local storage space.

To set the size of the segment files, set the `log.roll.ms` and
`log.segment.bytes` broker configurations.

If you enable
[log compaction](https://docs.cloud.google.com/managed-service-for-apache-kafka/docs/edit-topic.md.txt#log-compaction),
then only local storage is used for that topic.

## Default disk configuration

By default, Managed Service for Apache Kafka provisions at least
100 GiB of disk capacity per vCPU for every broker. When you scale up a
cluster, the service ensures that brokers have at least 100 GiB per CPU. Disk
capacity per broker is never reduced, even if the number of vCPUs per broker
decreases.

To see how much disk capacity has been allocated, view the cluster's
[effective disk capacity](https://docs.cloud.google.com/managed-service-for-apache-kafka/docs/brokers/configure-disk-size.md.txt#view-effective-disk-capacity).

## Custom disk configuration

The [default disk configuration](https://docs.cloud.google.com/managed-service-for-apache-kafka/docs/brokers/configure-disk-size.md.txt#default-disk-configuration) provides a balance
of local storage that is safe and cost effective for most users. However, you
can also specify the local disk size by setting a *custom disk configuration*
when you create or update a cluster.

> [!IMPORTANT]
> **Important:** After you set a custom disk configuration, you can't re-enable the default disk configuration for that cluster.

You might choose to increase the disk size for the following reasons:

- **Log compaction with large key spaces** . If
  [log compaction](https://docs.cloud.google.com/managed-service-for-apache-kafka/docs/edit-topic.md.txt#log-compaction)
  is enabled on a topic, Kafka stores the latest message for each key on disk.
  A large key space can result in a large number of messages being kept on disk
  indefinitely.

- **Large number of partitions.** Each partition requires enough space to store
  a rolled segment, plus space for a new segment file while the broker copies
  the rolled segment file to remote storage. If you have more than 200
  partitions per vCPU, consider setting a custom storage configuration.

- **High load**. An overloaded broker might not have enough disk capacity to
  move rolled files to remote storage quickly enough, and therefore might
  require more local storage.

- **Out-of-disk errors.** If your cluster runs out of disk space on a broker,
  you can increase the disk capacity to make the cluster operational.

For more information, see
[Estimate the required disk size](https://docs.cloud.google.com/managed-service-for-apache-kafka/docs/plan-cluster-size.md.txt#required-disk-size).

### Disk configuration requirements

A custom disk configuration must meet the following requirements:

- The disk size per broker must be at least equal to the number of vCPUs on each
  broker multiplied by 100 GiB.

  For example, a cluster with 6 vCPUs and 3 brokers requires at least
  200 GiB per broker.
- The disk size per broker can't exceed 32768 GiB (32 TiB).

- Disk size per broker cannot decrease. If you update a cluster, the new disk
  size must be equal to or greater than the current
  [effective disk capacity](https://docs.cloud.google.com/managed-service-for-apache-kafka/docs/brokers/configure-disk-size.md.txt#view-effective-disk-capacity).

- If you scale up a cluster, the updated vCPU count and disk size must meet the
  previous requirements, taking into account that new brokers might be added.
  To calculate the number of brokers, see
  [Scaling the cluster](https://docs.cloud.google.com/managed-service-for-apache-kafka/docs/brokers.md.txt#scaling).

  For example, suppose you have a cluster with 3 vCPUs and 200 GiB of disks per
  broker. If you scale the cluster to 9 vCPUs, the minimum disk size is
  300 GiB (9 vCPUs / 3 brokers × 100 GiB). To meet this
  requirement, you would need to set the disk size along with the vCPU count in
  the same update request.

### Set a custom disk configuration

You can set a custom disk configuration when you create a cluster, or by
updating an existing cluster. The configuration applies to all brokers in the
cluster.

### Console

In the **Create Kafka cluster** or **Edit Kafka cluster** page, perform
the following steps:

1. Under **Capacity configuration** , click the
   **Enable custom local storage size** toggle to the on position.

   > [!IMPORTANT]
   > **Important:** After you click **Save**, this setting cannot be disabled again for this cluster.

2. In the **Custom local storage** field, enter the disk size per broker, in
   GiB. The value must meet the
   [Disk configuration requirements](https://docs.cloud.google.com/managed-service-for-apache-kafka/docs/brokers/configure-disk-size.md.txt#disk-configuration-requirements), or an
   error occurs.

3. Click **Save**.

### gcloud

Specify the `--broker-disk-size-gib` flag in the
`gcloud managed-kafka clusters create` or
`gcloud managed-kafka clusters update` command.

Create a cluster:

```bash
gcloud managed-kafka clusters create CLUSTER_ID \
  --location=LOCATION \
  --broker-disk-size-gib=DISK_SIZE

  # Other configruration flags
  ...
```

Update a cluster:

```bash
gcloud managed-kafka clusters update CLUSTER_ID \
  --location=LOCATION \
  --broker-disk-size-gib=DISK_SIZE
```

Replace the following:

- <var translate="no">CLUSTER_ID</var>: The name of the cluster.
- <var translate="no">LOCATION</var>: The location of the cluster.
- <var translate="no">DISK_SIZE</var>: The disk size per broker, in GiB. The value must meet the [Disk configuration requirements](https://docs.cloud.google.com/managed-service-for-apache-kafka/docs/brokers/configure-disk-size.md.txt#disk-configuration-requirements), or an error occurs.

For more information about creating and updating Kafka clusters, see the
following documents:

- [Create a Kafka cluster](https://docs.cloud.google.com/managed-service-for-apache-kafka/docs/create-cluster.md.txt)
- [Update a Kafka cluster](https://docs.cloud.google.com/managed-service-for-apache-kafka/docs/update-cluster.md.txt)

### View a custom disk configuration

To view a custom disk configuration on a Managed Service for Apache Kafka
cluster, perform the following steps.

### Console

1. In the Google Cloud console, go to the **Clusters** page.


   [Go to Clusters](https://console.cloud.google.com/managedkafka/clusters)

   <br />

2. From the list of clusters, click the cluster that you want to view.

3. Select the **Configurations** tab.

4. Under **Capacity** , if the cluster has a custom disk configuration, the
   **Custom local storage** row shows the disk size per broker. Otherwise, the
   row shows `Disabled`.

### gcloud

Run the
[`gcloud managed-kafka clusters describe`](https://docs.cloud.google.com/sdk/gcloud/reference/managed-kafka/clusters/describe)
command:

```bash
gcloud managed-kafka clusters describe CLUSTER_ID \
  --location=LOCATION
```

Replace the following:

- <var translate="no">CLUSTER_ID</var>: The name of the cluster.
- <var translate="no">LOCATION</var>: The location of the cluster.

In the returned cluster description, the `brokerCapacityConfig` field
shows the configuration:

```yaml
brokerCapacityConfig:
  diskSizeGib: '3000'
```

If you didn't set a custom disk configuration, this field is empty:

```yaml
brokerCapacityConfig: {}
```

## View the effective disk capacity

A cluster's *effective* disk capacity is how much disk space has been
provisioned per broker. To view the effective disk capacity for a cluster, run
the [`gcloud managed-kafka clusters describe`](https://docs.cloud.google.com/sdk/gcloud/reference/managed-kafka/clusters/describe)
command with the `--full` flag.

### Console

1. In the Google Cloud console, go to the **Clusters** page.


   [Go to Clusters](https://console.cloud.google.com/managedkafka/clusters)

   <br />

2. From the list of clusters, click the cluster that you want to view.

3. Select the **Configurations** tab.

4. Under **Capacity** , the **Effective local storage** row shows the disk size
   per broker.

### gcloud

```bash
gcloud managed-kafka clusters describe CLUSTER_ID --full \
  --location=LOCATION
```

Replace the following:

- <var translate="no">CLUSTER_ID</var>: The name of the cluster.
- <var translate="no">LOCATION</var>: The location of the cluster.

In the returned cluster description, the `effectiveCapacityConfig` field
contains the disk size per broker and the number of brokers:

```yaml
effectiveCapacityConfig:
  brokerCount: '3'
  brokerDiskSizeGib: '1024'
```

Total disk capacity equals `brokerCount` × `brokerDiskSizeGib`.

## Pricing

If your cluster uses [default disk configuration](https://docs.cloud.google.com/managed-service-for-apache-kafka/docs/brokers/configure-disk-size.md.txt#default-disk-configuration),
you are billed for 100 GiB per cluster vCPU, although the actual disk size
per broker might exceed this value.

After you set a [custom disk configuration](https://docs.cloud.google.com/managed-service-for-apache-kafka/docs/brokers/configure-disk-size.md.txt#custom-disk-configuration) on
a cluster, you are billed for the actual size of the disks across all of the
brokers.

## What's next

- [Plan the size of your Kafka cluster](https://docs.cloud.google.com/managed-service-for-apache-kafka/docs/plan-cluster-size.md.txt)
- [View a cluster](https://docs.cloud.google.com/managed-service-for-apache-kafka/docs/view-cluster.md.txt)
- [Create a cluster](https://docs.cloud.google.com/managed-service-for-apache-kafka/docs/create-cluster.md.txt)
- [Update a cluster](https://docs.cloud.google.com/managed-service-for-apache-kafka/docs/update-cluster.md.txt)

*Apache Kafka® is a registered
trademark of The Apache Software Foundation or its affiliates in the United
States and/or other countries.*