Create Bigtable subscriptions

This document describes how to create a Bigtable subscription in Pub/Sub. Bigtable subscriptions let you write Pub/Sub messages directly to a Bigtable table.

For more information about this subscription type, see Bigtable subscriptions.

Before you begin

The Bigtable table where you want to write Pub/Sub messages must already exist. For more information, see Create and manage tables.
The table must have a column family named data. You can add the column family when you create the table.
If you want to write message metadata to the table, the table must also have a column family named pubsub_metadata.

Required roles and permissions

To get the permissions that you need to create a Bigtable subscription, ask your administrator to grant you the Pub/Sub Editor (roles/pubsub.editor) IAM role on the project. For more information about granting roles, see Manage access to projects, folders, and organizations.

This predefined role contains the permissions required to create a Bigtable subscription. To see the exact permissions that are required, expand the Required permissions section:

Required permissions

The following permissions are required to create a Bigtable subscription:

pubsub.subscriptions.create on the project
pubsub.topics.attachSubscription on the topic

You might also be able to get these permissions with custom roles or other predefined roles.

Cross-project subscriptions

If you create a subscription in one project for a topic in another project, you must have pubsub.subscriptions.create permission on the project in which you are creating the subscription, and pubsub.topics.attachSubscription permission on the topic.

Grant IAM roles to the service account

Pub/Sub uses an Identity and Access Management (IAM) service account to access Google Cloud resources. To enable Pub/Sub to write to Bigtable, you must grant the Bigtable User (roles/bigtable.user) role to this service account.

By default, Pub/Sub uses the Pub/Sub service agent
(service-PROJECT_NUMBER@gcp-sa-pubsub.iam.gserviceaccount.com). You can also can specify a user-managed service account when you create the Bigtable subscription.

Use the Pub/Sub service agent

To grant the required role to the Pub/Sub service agent, perform the following steps. You can grant the role on the project or on the Bigtable instance that contains the table.

Project

In the Google Cloud console, go to the IAM page.

Go to IAM
Select Include Google-provided role grants.
Find the row for Cloud Pub/Sub Service Account and click Edit principal.
Click Add another role and select the Bigtable user role.

For more information, see Grant an IAM role by using the console.

Instance

In the Google Cloud console, go to the Bigtable > Instances page.

Go to Instances
In the Instances page, select the checkbox next to the name of the instance.
If the info panel isn't visible, click Show info panel.
Click Add principal, then enter the service account identifier, in the following format:

service-PROJECT_NUMBER@gcp-sa-pubsub.iam.gserviceaccount.com.
In the Assign roles list, select Bigtable User.
Click Save. The principal is granted the role on the resource.

Use a custom service account

If you grant the Bigtable User role to the Pub/Sub service agent, then any user with permission to create a subscription in your project can write to the Bigtable table. If you want to provide more granular permissions, configure a user-managed service account instead.

The following permissions are required to configure a user-managed service account to write to Bigtable:

The user-managed service account must have the Bigtable User role.
The Cloud Pub/Sub service account must have the iam.serviceAccounts.getAccessToken permission on the user-managed service account.
The user creating the subscription must have the iam.serviceAccounts.actAs permission on the user-managed service account.

When you create the subscription, specify the user-managed service account. For more information, see Create a Bigtable subscription.

Bigtable table writes

This section describes how a Bigtable subscription writes Pub/Sub messages to a table.

Message data

A Bigtable subscription writes the Pub/Sub message data to a column family named data.

Each row contains the data from one message.
The row key is a concatenation of the subscription ID, a message ID salt prefix, and the message ID, in the following format:

"projects/PROJECT_NUMBER/subscriptions/SUBSCRIPTION_ID#SALT_PREFIX#MESSAGE_ID"
The message data is written to a single column in the data column family. The column has an empty-string ("") column qualifier.
Data is written as a BYTES type.
The cell's timestamp is the message's publish time.

Message metadata

By default, the subscription doesn't write message metadata to the table. Optionally, you can enable writing metadata when you create a Bigtable subscription.

If writing metadata is enabled, the subscription writes the message metadata to a column family named pubsub_metadata. Before you create the subscription, ensure that your table has a column family with this name.

For each message, the subscription writes a new row with the following columns under the pubsub_metadata column family.

Column	Value	Data type
`subscription_name`	The name of the subscription	String
`message_id`	The message ID	String
`attributes`	A JSON object that contains the message attributes	String

Create a Bigtable subscription

To create a Bigtable subscription, use the gcloud beta pubsub subscriptions create command:

gcloud beta pubsub subscriptions create SUBSCRIPTION_ID \
  --topic=TOPIC_ID \
  --bigtable-table=projects/PROJECT_ID/instances/INSTANCE_ID/tables/TABLE_ID

Replace the following:

SUBSCRIPTION_ID: A name for the subscription.
TOPIC_ID: The name or ID of the topic to read from.
PROJECT_ID: The project ID.
INSTANCE_ID: The ID of the Bigtable instance that contains the table.
TABLE_ID: The ID of the table to write to. The table must have a column family named data.

The following flags are optional:

--bigtable-write-metadata: Specifies whether to write message metadata. If you omit this flag, the subscription doesn't write metadata. For more information, see Message metadata.
--bigtable-app-profile-id=APP_PROFILE: Specifies an app profile to use for Bigtable writes. The app profile must use single-cluster routing. If this flag isn't specified, the subscription uses the default app profile.
--bigtable-service-account-email: Specifies a service account to use when writing to Bigtable. If not specified, the subscription uses the Pub/Sub service agent. For more information, see Use a custom service account.

Example: Write from Pub/Sub to Bigtable

The following example shows how to create a Bigtable subscription, use the subscription to write Pub/Sub messages to a Bigtable table, and then query the table.

Create a new Bigtable instance named my-instance.

gcloud bigtable instances create my-instance --display-name="My instance" \
  --cluster-config=id=my-cluster-1,zone=ZONE,nodes=1

Replace ZONE with the zone where the cluster runs. Example: us-central1-b

Create a Bigtable table named table-1 with a column family named data.

gcloud bigtable instances tables create table-1 \
  --instance=my-instance \
  --column-families=data

Create a Pub/Sub topic named topic-1.
```
gcloud pubsub topics create topic-1
```

Create a Bigtable subscription that reads from the topic.

gcloud beta pubsub subscriptions create bigtable-sub \
  --topic=topic-1 \
  --bigtable-table=projects/PROJECT_ID/instances/my-instance/tables/table-1

Replace PROJECT_ID with your Google Cloud project ID.

Publish a message to the topic.
```
gcloud pubsub topics publish topic-1 --message='{"name":"Alice"}'
```
If the message is successfully published, the output is the message ID:
```
messageIds:
- 'MESSAGE_ID'
```

The Bigtable subscription writes the message data to the data column family. To query the data, perform the following steps:

In the Google Cloud console, go to the Bigtable Instances page.

Go to Instances
Click the instance named my-cluster-1.
In the navigation menu, click Bigtable Studio.
In the query editor, run the following SQL query.
```
SELECT _key, JSON_VALUE(CAST(data[''] AS STRING), '$.name') AS name FROM table-1;
```
The query result looks like the following:
```
+-------------------+
| _key      | name  |
+-----------|-------+
| "ROW_KEY" | Alice |
+-------------------+
```
The message data is stored as a BYTES type. In this example, the data is a JSON string, so the query can use GoogleSQL JSON functions to parse the data.

Monitor a Bigtable subscription

Cloud Monitoring provides a number of metrics to monitor subscriptions.

For a list of all the available metrics related to Pub/Sub and their descriptions, see the Monitoring documentation for Pub/Sub.

You can also monitor subscriptions from within Pub/Sub.

What's next

Learn more about Bigtable subscriptions.
Choose a subscription type.
Troubleshoot Bigtable subscriptions