This document explains how to create a Pub/Sub topic with Single Message Transforms (SMTs). Topic SMTs allow for lightweight modifications to message data and attributes directly within Pub/Sub. This feature enables data cleaning, filtering, or format conversion before the messages are published to the topic. To create a topic with SMTs, you can use the Google Cloud console, the Google Cloud CLI, the client library, or the Pub/Sub API. ## Before you begin - Learn about the [Pub/Sub service and its terminology](https://docs.cloud.google.com/pubsub/docs/pubsub-basics.md.txt). - Learn about [SMTs](https://docs.cloud.google.com/pubsub/docs/smts/smts-overview.md.txt). ### Required roles and permissions To get the permission that you need to create a topic with SMTs, ask your administrator to grant you the [Pub/Sub Editor](https://docs.cloud.google.com/iam/docs/roles-permissions/pubsub.md.txt#pubsub.editor) (`roles/pubsub.editor`) IAM role on your project. For more information about granting roles, see [Manage access to projects, folders, and organizations](https://docs.cloud.google.com/iam/docs/granting-changing-revoking-access.md.txt). This predefined role contains the `pubsub.topics.create` permission, which is required to create a topic with SMTs. You might also be able to get this permission with [custom roles](https://docs.cloud.google.com/iam/docs/creating-custom-roles.md.txt) or other [predefined roles](https://docs.cloud.google.com/iam/docs/roles-overview.md.txt#predefined). ## Create a topic with SMTs Before you create a topic with SMTs, review the documentation for [Properties of a topic](https://docs.cloud.google.com/pubsub/docs/create-topic.md.txt#properties_of_a_topic). To create a Pub/Sub with one or more SMTs, perform the following steps. You can enable up to 5 SMTs per topic. ### Console 1. In the Google Cloud console, go to the Pub/Sub **Topics** page. [Go to Topics](https://console.cloud.google.com/cloudpubsub/topic)
2. Click **Create topic**. 3. In the **Topic ID** field, enter an ID for your topic. For more information about naming topics, see the [naming guidelines](https://docs.cloud.google.com/pubsub/docs/pubsub-basics.md.txt#resource_names). 4. Under **Transforms** , click **Add a transform**. 5. Select the **Transform type** . For more information about the supported SMT types, see [Types of SMTs](https://docs.cloud.google.com/pubsub/docs/smts/smts-overview.md.txt#types_of_smts). 6. Set the configuration properties for the SMT. The set of properties depends on the type of SMT. For more information, see the documentation for that SMT type. 7. Optional. To validate the SMT, click **Validate** . If the SMT is valid, the message `"Validation passed"` is displayed. Otherwise, an error message is displayed. 8. To add another transform, click **Add a transform** and repeat the previous steps. To arrange the SMTs in a specific order, click **Move up** or **Move down** . To remove an SMT, click **Delete**. 9. Optional. To test an SMT on a sample message, perform the following steps: 1. Click **Test transforms**. 2. In the **Test transform** window, select the function that you want to test. 3. In the **Input message** window, enter a sample message. 4. To add an attribute to the message, click **Add an attribute** and enter the attribute's key and value. You can add multiple attributes. 5. Click **Test** . The result of applying the SMT on the message is displayed under **Output message**. 6. To close the **Test transforms** window, click **Close**. If you create more than one SMT, you can test the entire sequence of transforms as follows: 1. Test the first SMT in the sequence, as described in the previous steps. 2. Select the next SMT. The input message is pre-populated with the output message from the previous test. 3. Continue testing the SMTs in order, to make sure the entire sequence works as expected. 10. To create the topic, click **Create**. ### gcloud 1. In the Google Cloud console, activate Cloud Shell. [Activate Cloud Shell](https://console.cloud.google.com/?cloudshell=true) At the bottom of the Google Cloud console, a [Cloud Shell](https://docs.cloud.google.com/shell/docs/how-cloud-shell-works.md.txt) session starts and displays a command-line prompt. Cloud Shell is a shell environment with the Google Cloud CLI already installed and with values already set for your current project. It can take a few seconds for the session to initialize. 2. Create a YAML or JSON file that defines one or more SMTs. The YAML or JSON definition depends on the type of SMT. For more information, see [Types of SMTs](https://docs.cloud.google.com/pubsub/docs/smts/smts-overview.md.txt#types_of_smts). If the file includes more than one SMT, Pub/Sub executes them in the order listed. 3. Optional. To validate an SMT, run the [`gcloud pubsub message-transforms validate`](https://docs.cloud.google.com/sdk/gcloud/reference/pubsub/message-transforms/validate) command: ```bash gcloud pubsub message-transforms validate \ --message-transform-file=TRANSFORM_FILE ``` Replace the following: - TRANSFORM_FILE: The path to a YAML or JSON file that defines a single SMT. If you are creating multiple SMTs, you must validate them individually. 4. Optional. To test one or more SMTs on a sample Pub/Sub message, run the [`gcloud pubsub message-transforms test`](https://docs.cloud.google.com/sdk/gcloud/reference/pubsub/message-transforms/test) command: ```bash gcloud pubsub message-transforms test \ --message-transforms-file=TRANSFORMS_FILE \ --message=MESSAGE \ --attribute=ATTRIBUTES ``` Replace the following: - TRANSFORMS_FILE: The path to a YAML or JSON file that defines one or more SMTs. - MESSAGE: The body of the sample message. - ATTRIBUTES: Optional. A comma-separated list of message attributes. Each attribute is a key-value pair formatted as `KEY="VALUE"`. The command executes the SMTs in order, using the output from each SMT as the input to the next. The command outputs the results of each step. 5. To create the topic, run the [`gcloud pubsub topics create`](https://docs.cloud.google.com/sdk/gcloud/reference/pubsub/topics/create) command: ```bash gcloud pubsub topics create TOPIC_ID \ --message-transforms-file=TRANSFORMS_FILE ``` Replace the following: - TOPIC_ID: The ID or name of the topic you want to create. For guidelines on how to name a topic, see [Resource names](https://docs.cloud.google.com/pubsub/docs/pubsub-basics.md.txt#resource_names). The name of a topic is immutable. - TRANSFORMS_FILE: The path to a YAML or JSON file that defines one or more SMTs. ### C# Before trying this sample, follow the C# setup instructions in [Quickstart: Using Client Libraries](https://docs.cloud.google.com/pubsub/docs/create-topic-client-libraries). For more information, see the [Pub/Sub C# API reference documentation](https://cloud.google.com/dotnet/docs/reference/Google.Cloud.PubSub.V1/latest). ```c# using Google.Cloud.PubSub.V1; using System; public class CreateTopicWithSingleMessageTransformSample { public Topic CreateTopicWithSingleMessageTransform(string projectId, string topicId) { PublisherServiceApiClient publisher = PublisherServiceApiClient.Create(); var topicName = TopicName.FromProjectTopic(projectId, topicId); MessageTransform removeSSNTransform = new MessageTransform { JavascriptUdf = new JavaScriptUDF { FunctionName = "redactSsn", Code = "function redactSsn(message, metadata) {" + " const data = JSON.parse(message.data);" + " delete data['ssn'];" + " message.data = JSON.stringify(data);" + " return message;" + "}" }, }; Topic topic = publisher.CreateTopic(new Topic() { TopicName = topicName, MessageTransforms = { removeSSNTransform } }); Console.WriteLine($"Topic {topic.Name} created with SMT."); return topic; } } ```
### Java Before trying this sample, follow the Java setup instructions in [Quickstart: Using Client Libraries](https://docs.cloud.google.com/pubsub/docs/create-topic-client-libraries). For more information, see the [Pub/Sub Java API reference documentation](https://cloud.google.com/java/docs/reference/google-cloud-pubsub/latest/overview). ```java import com.google.api.gax.rpc.AlreadyExistsException; import com.google.cloud.pubsub.v1.TopicAdminClient; import com.google.pubsub.v1.JavaScriptUDF; import com.google.pubsub.v1.MessageTransform; import com.google.pubsub.v1.Topic; import com.google.pubsub.v1.TopicName; import java.io.IOException; public class CreateTopicWithSmtExample { public static void main(String... args) throws Exception { // TODO(developer): Replace these variables before running the sample. String projectId = "your-project-id"; String topicId = "your-topic-id"; createTopicWithSmtExample(projectId, topicId); } public static void createTopicWithSmtExample(String projectId, String topicId) throws IOException { TopicName topicName = TopicName.of(projectId, topicId); // UDF that removes the 'ssn' field, if present String code = "function redactSSN(message, metadata) {" + " const data = JSON.parse(message.data);" + " delete data['ssn'];" + " message.data = JSON.stringify(data);" + " return message;" + "}"; String functionName = "redactSSN"; JavaScriptUDF udf = JavaScriptUDF.newBuilder().setCode(code).setFunctionName(functionName).build(); MessageTransform transform = MessageTransform.newBuilder().setJavascriptUdf(udf).build(); try (TopicAdminClient topicAdminClient = TopicAdminClient.create()) { Topic topic = topicAdminClient.createTopic( Topic.newBuilder() .setName(topicName.toString()) // Add the UDF message transform .addMessageTransforms(transform) .build()); System.out.println("Created topic with SMT: " + topic.getName()); } catch (AlreadyExistsException e) { System.out.println(topicName + "already exists."); } } } ```
### Python Before trying this sample, follow the Python setup instructions in [Quickstart: Using Client Libraries](https://docs.cloud.google.com/pubsub/docs/create-topic-client-libraries). For more information, see the [Pub/Sub Python API reference documentation](https://docs.cloud.google.com/python/docs/reference/pubsub/latest.md.txt). ```python from google.cloud import pubsub_v1 from google.pubsub_v1.types import JavaScriptUDF, MessageTransform, Topic # TODO(developer) # project_id = "your-project-id" # topic_id = "your-topic-id" code = """function redactSSN(message, metadata) { const data = JSON.parse(message.data); delete data['ssn']; message.data = JSON.stringify(data); return message; }""" udf = JavaScriptUDF(code=code, function_name="redactSSN") transforms = [MessageTransform(javascript_udf=udf)] publisher = pubsub_v1.PublisherClient() topic_path = publisher.topic_path(project_id, topic_id) request = Topic(name=topic_path, message_transforms=transforms) topic = publisher.create_topic(request=request) print(f"Created topic: {topic.name} with SMT") ```
### Go The following sample uses the major version of the Go Pub/Sub client library (v2). If you are still using the v1 library, see [the migration guide to v2](https://github.com/googleapis/google-cloud-go/blob/main/pubsub/MIGRATING.md). To see a list of v1 code samples, see [the deprecated code samples](https://docs.cloud.google.com/pubsub/docs/samples.md.txt?language=golang&text=deprecated). Before trying this sample, follow the Go setup instructions in [Quickstart: Using Client Libraries](https://docs.cloud.google.com/pubsub/docs/create-topic-client-libraries). For more information, see the [Pub/Sub Go API reference documentation](https://pkg.go.dev/cloud.google.com/go/pubsub/v2). ```go import ( "context" "fmt" "io" "cloud.google.com/go/pubsub/v2" "cloud.google.com/go/pubsub/v2/apiv1/pubsubpb" ) // createTopicWithSMT creates a topic with a single message transform function applied. func createTopicWithSMT(w io.Writer, projectID, topicID string) error { // projectID := "my-project-id" // topicID := "my-topic" ctx := context.Background() client, err := pubsub.NewClient(ctx, projectID) if err != nil { return fmt.Errorf("pubsub.NewClient: %w", err) } defer client.Close() code := `function redactSSN(message, metadata) { const data = JSON.parse(message.data); delete data['ssn']; message.data = JSON.stringify(data); return message; }` transform := &pubsubpb.MessageTransform{ Transform: &pubsubpb.MessageTransform_JavascriptUdf{ JavascriptUdf: &pubsubpb.JavaScriptUDF{ FunctionName: "redactSSN", Code: code, }, }, } topic := &pubsubpb.Topic{ Name: fmt.Sprintf("projects/%s/topics/%s", projectID, topicID), MessageTransforms: []*pubsubpb.MessageTransform{transform}, } topic, err = client.TopicAdminClient.CreateTopic(ctx, topic) if err != nil { return fmt.Errorf("CreateTopic: %w", err) } fmt.Fprintf(w, "Created topic with message transform: %v\n", topic) return nil } ```
## What's next - [Update topic SMTs](https://docs.cloud.google.com/pubsub/docs/smts/update-topic-smt.md.txt) - [Delete topic SMTs](https://docs.cloud.google.com/pubsub/docs/smts/delete-topic-smt.md.txt)