Troubleshoot Bigtable subscriptions

Preview

This product or feature is subject to the "Pre-GA Offerings Terms" in the General Service Terms section of the Service Specific Terms. Pre-GA products and features are available "as is" and might have limited support. For more information, see the launch stage descriptions.

This document provides troubleshooting tips for Bigtable subscriptions.

Unable to create or update subscription

When you try to create or update a Bigtable subscription, you might encounter the following errors.

Permission denied error

If the Bigtable instance or table doesn't exist, or the configured service account doesn't have access to either, you receive a permission denied error.

To resolve the issue:

• Check that the instance and table exist. For more information, see the following documents:

  • Get instance information
  • View information about a table

• If they don't exist, create them and retry creating your subscription. For an example of this step, see Write from Pub/Sub to Bigtable.

• If you still get this error, check whether the service account has the correct permissions. For more information, see Grant IAM roles to the service account.

Missing column families

If the Bigtable table doesn't have the correct column families, you receive an INCOMPATIBLE_SCHEMA error.

This error can be caused by the following issues:

• The table doesn't have a data column family.
• Writing metadata is enabled, and the table doesn't have a pubsub_metadata column family.

To resolve the issue, create the necessary column families. For more information, see Add column families.

The subscription isn't writing data

If a Bigtable subscription is not writing data, check the subscription state by running the gcloud beta pubsub subscriptions describe command:

gcloud beta pubsub subscriptions describe SUBSCRIPTION_ID

Replace SUBSCRIPTION_ID with the name of the subscription.

The bigtableConfig.state field contains the state. If the subscription is ready to write messages to Bigtable, the state is ACTIVE. Otherwise, there is an error in the configuration that prevents the subscription from writing messages to the Bigtable table.

Configuration issues might include the following:

• The app profile isn't configured for single-cluster routing.

• Permission errors. See Permission denied error.

• The Bigtable table is missing a required column family. See Missing column families.

• The subscription can't write to the Bigtable table due to a message storage policy. This error occurs if enforceInTransit is True and the table location is not in an allowed region.

While a subscription is in an error state, it doesn't write messages to the Bigtable table. The messages remain in the subscription backlog; they aren't delivered to the dead-letter topic, if one is configured. Unacknowledged messages are retained for the subscription's message retention duration.

After the issue is resolved, the subscription eventually returns to a healthy state.

Increasing backlog

If your subscription state is ACTIVE but there is an increasing backlog of messages in the subscription, or messages are going to the subscription's dead letter topic, look for the following issues.

Publish rate exceeds capacity

If the rate of publishing messages to your topic exceeds the capacity of your Bigtable cluster, backlogs might build up in the subscription.

Message ordering comes with performance tradeoffs. Using Bigtable subscriptions with ordering might lead to lower throughput or higher message delivery latency. If you see performance problems with ordering enabled, make sure to use adequately granular ordering keys to avoid hot keys, or modify your application to not require ordering. For more information, see Considerations when using ordered messaging.

Monitor the performance of your cluster and consider increasing your cluster size. For more information, see the following topics in the Bigtable documentation:

• Monitoring Bigtable
• Understand performance

Cross-region traffic

If your application publishes to your topic from a different region than the Bigtable cluster, it takes longer for each message to get published, compared to publishing from the same region. The longer publishing time reduces the rate at which your subscription writes messages to Bigtable, reducing total throughput.

To minimize how long it takes for messages to get published, you should publish them in the same region as your Bigtable cluster.

What's next

• If you still have issues with your Bigtable subscription, see Getting support.