Troubleshoot Connect clusters and connectors

Preview

This product or feature is subject to the "Pre-GA Offerings Terms" in the General Service Terms section of the Service Specific Terms. Pre-GA products and features are available "as is" and might have limited support. For more information, see the launch stage descriptions.

This document provides troubleshooting guidance for Connect clusters and connectors.

Detached state

A Connect Cluster can be in the state of DETACHED. This state means that the underlying Kafka cluster has been deleted or is in the process of being deleted. Because Kafka Connect stores all state on Kafka, the Connect Cluster is permanetly stopped. A Connect Cluster cannot be re-attached, and this cluster should be deleted.

Troubleshoot a connector

If a connector fails to start up or fails while running, the operational state is Failed. A connector can also have errors while running; these appear in the connector logs.

To troubleshoot a connector, perform the following steps.

1. View the connector logs to identify the error. For more information, see View connector logs.

2. Identify the root cause of the error. The following topics describe specific errors that might occur for each connector type:

   • Troubleshoot a BigQuery sink connector
   • Troubleshoot a Cloud Storage sink connector
   • Troubleshoot a MirrorMaker 2.0 connector
   • Troubleshoot a Pub/Sub connector

3. If the connector is mis-configured, update the connector's configuration and then resume the connector. Transient failures might not require any changes to the configuration.

As a best practice, set the connector's task restart policy to restart with exponential backoff. This setting helps to avoid connector failures due to transient errors.

Connect cluster errors

This section describes errors that might occur in a Connect cluster.

Metadata topic has 'cleanup policy=delete'

The Connect cluster logs show an error similar to the following:

org.apache.kafka.common.config.ConfigException: Topic 'connect-offsets-
PROJECT_ID-LOCATION-CONNECT_CLUSTER_NAME-UNIQUE_ID' supplied via the
'offset.storage.topic' property is required to have 'cleanup policy=compact' to
guarantee consistency and durability of source connector offsets, but found the
topic currently has 'cleanup policy=delete'.

This error might occur for the following reasons:

• The Connect cluster's primary Kafka cluster was deleted and then recreated with the same name.
• The metadata topics created by the Connect cluster in the primary Kafka cluster were deleted.

To resolve this error, perform the following steps:

1. Locate the metadata topics in the primary Kafka cluster. The metadata topics are named with the following format:

   connect-status-PROJECT_ID-LOCATION-CONNECT_CLUSTER_NAME-UNIQUE_ID

   For example:

   connect-status-project1-us-central1-connect1-3c563900-a4e4-4eDe-88f7-73b16d6344d2

2. For each metadata topic, edit the topic and add the key-value pair cleanup.policy:compact to the Non-default parameters field. For more information, see Update a Kafka topic.

What's next

• Monitor a Connect cluster