This document describes how to monitor the health of clients that produce or
consume data in your Managed Service for Apache Kafka cluster.

It's important to monitor client applications as part of your overall
reliability strategy. Metrics such as throughput, error rates, and consumer lag
can tell you whether client applications are experiencing reliability issues.
Problems might be caused by client configuration, uneven distribution of keys
across partitions, or cluster issues that affect only a specific partition.

## Server-side metrics

While it's useful to monitor client behavior directly, server-side metrics don't
require any additional instrumentation, and can help you to detect client-side
issues that affect reliability.

Server-side metrics are especially useful for detecting load imbalances across
brokers (hot brokers), and deviations in normal operations, such as spikes in
latency.

### Throughput

Monitor the following throughput metrics and compare them against your expected
throughput:

- **Message rate**, per broker and per topic.
- **Byte rate**, per broker and per topic.
- **Request rates**. A misconfigured client might flood brokers with a high rate of small requests (0-1000 bytes per request), reducing throughput.
- **Request latency**. Spikes in producer request latency might signal imbalanced load or a problem with client configuration.

Kafka offers throughput metrics per topic and for the cluster. These metrics
don't always have the same values when aggregated for all topics. Use an
aggregated metric for high-level monitoring and alerting, and look at
per-topic metrics when you troubleshoot throughput problems. Isolate any issues
to specific brokers.

### Request error rates

The [`topic_error_count`](https://docs.cloud.google.com/monitoring/api/metrics_gcp_i_o.md.txt#managedkafka/topic_error_count)
metric tracks the number of failed fetch and produce requests on the server
side. However, some classes of error aren't reflected in this metric. For
example:

- Misconfigured authorization settings might prevent a client from producing to
  a topic, without the error appearing in this metric.

- Cluster failures might prevent a broker from responding to requests at all.

For this reason, you should also monitor errors on the client side, including
request time-out errors.

### Consumer lag

Consumer lag measures how far behind a consumer client is from a particular
offset. Aggregating this metric by topic, partition, and broker is useful for
determining whether lag is due to specific brokers or partitions.

Consider creating an alert based on the maximum lag across the subset of topics
that are critical for your workload.

### Custom dashboard queries

We recommend creating custom dashboards and alerts to monitor these signals.
The following table shows [Prometheus Query Language (PromQL)](https://docs.cloud.google.com/monitoring/promql.md.txt) queries
that you can use to monitor client health.

| Signal | PromQL query |
|---|---|
| Throughput: Message rates per broker | ```promql sum by (resource_container, location, cluster_id, broker_index) ( rate( { "managedkafka.googleapis.com/message_in_count", monitored_resource="managedkafka.googleapis.com/Topic" }[${__interval}] ) ) ``` |
| Throughput: Message rates per broker per topic, for the largest 5 topics | ```promql topk(5, sum by (resource_container, location, cluster_id, broker_index, topic_id) ( rate( { "managedkafka.googleapis.com/message_in_count", monitored_resource="managedkafka.googleapis.com/Topic" }[${__interval}] ) ) ) ``` |
| Throughput: Bandwidth per topic and broker | ```promql sum by (resource_container, location, cluster_id, broker_index, topic_id) ( rate( { "managedkafka.googleapis.com/byte_in_count", monitored_resource="managedkafka.googleapis.com/Topic" }[${__interval}] ) ) ``` |
| Request rate | ```promql sum by (resource_container, location, cluster_id, request) ( rate( { "managedkafka.googleapis.com/request_count", monitored_resource="managedkafka.googleapis.com/Cluster", "request"="Produce" }[${__interval}] ) ) ``` |
| Request rate, cluster totals | ```promql sum by (resource_container, location, cluster_id, request) ( rate( { "managedkafka.googleapis.com/topic_request_count", monitored_resource="managedkafka.googleapis.com/Topic", "request"="Produce" }[${__interval}] ) ) ``` |
| Request latency | ```promql sum by (resource_container, location, cluster_id, broker_index, request) ( avg_over_time( { "managedkafka.googleapis.com/request_latencies", monitored_resource="managedkafka.googleapis.com/Cluster", "percentile"="95", "request"="Produce" }[${__interval}] ) ) ``` |
| Request error counts | ```promql sum by (resource_container, location, cluster_id, broker_index, request) ( rate( { "managedkafka.googleapis.com/topic_error_count", monitored_resource="managedkafka.googleapis.com/Topic" }[${__interval}] ) ) ``` |
| Consumer lag: Top 5 partitions by offset lag | ```promql topk(5, max by (resource_container, location, cluster_id, topic_id, partition_index) ( max_over_time( { "managedkafka.googleapis.com/offset_lag", monitored_resource="managedkafka.googleapis.com/TopicPartitionConsumerGroup" }[${__interval}] ) ) ) ``` |
| Consumer lag: Top 5 consumer groups by offset lag | ```promql topk(5, max by (resource_container, location, cluster_id, consumer_group_id) ( max_over_time( { "managedkafka.googleapis.com/offset_lag", monitored_resource="managedkafka.googleapis.com/TopicPartitionConsumerGroup" }[${__interval}] ) ) ) ``` |

## Client-side metrics

Sometimes client issues don't appear in server metrics. For example:

- If authentication or networking is misconfigured, messages might accumulate in
  the client's internal queues. As requests time out, the messages are
  re-enqueued and retried.

- If flow control is misconfigured, a client might produce more messages than
  the allocated number of threads can send. Although throughput might be
  consistent, a growing backlog of requests expire before they can be sent.

If your clients run on Compute Engine, Google Kubernetes Engine, or Cloud Run,
you can use [log-based metrics](https://docs.cloud.google.com/logging/docs/logs-based-metrics.md.txt) in
Cloud Monitoring to detect high error rates in logs. However, some Kafka
clients hide exceptions that lead to prolonged retries, unless you configure
higher log levels. Therefore, you should also monitor for increased request
latencies.

Java clients expose many metrics through Java Management Extensions (JMX). For
more information, see
[Monitoring](https://kafka.apache.org/37/operations/monitoring/) in the Apache Kafka
documentation. If possible, prioritize instrumenting your clients to report the
following metrics:

- Request error rates (`kafka.producer:type=producer-metrics,client-id="{client-id}"`)
- Request average latency (`kafka.producer:type=producer-metrics,client-id="{client-id}"`)

Send these metrics to a monitoring solution if possible. If you can connect to
the JMX ports on the machines that run the client instances, you can also read
these metrics interactively.

## Mitigations

If you see issues in your client applications, consider the following
mitigations:

- Look for imbalanced load across brokers (hot brokers). Make sure that
  [auto rebalancing](https://docs.cloud.google.com/managed-service-for-apache-kafka/docs/create-cluster.md.txt#rebalancing-config)
  is enabled in your cluster.

- If the request rate seems unusually high, check whether the client is sending
  a large number of small requests. Check the `max.request.size` and
  `batch.size` configurations on the producer.

- Check the client configurations for authentication, networking, and flow
  control.

- Excessive lag across all topics or partitions in a cluster might indicate the
  cluster is overloaded and should be scaled up.

- Excessive lag across all topics or partitions in a broker might indicate the
  broker is overloaded. Try improving the key distribution or
  [reassign partitions](https://docs.cloud.google.com/managed-service-for-apache-kafka/docs/brokers/assign-partitions.md.txt)
  to different brokers.

## What's next

- [Monitor a Kafka clusters](https://docs.cloud.google.com/managed-service-for-apache-kafka/docs/monitor-cluster.md.txt)
- [Monitor Kafka clusters for reliability](https://docs.cloud.google.com/managed-service-for-apache-kafka/docs/monitor-reliability.md.txt)
- [Create and manage custom dashboards](https://docs.cloud.google.com/monitoring/charts/dashboards.md.txt)

*Apache Kafka® is a registered
trademark of The Apache Software Foundation or its affiliates in the United
States and/or other countries.*