The Spanner change streams to Cloud Storage template is a streaming pipeline
that streams Spanner data change records and writes them into a Cloud Storage bucket using
the Dataflow Portable Runner.

> [!NOTE]
> **Note:** The Spanner change streams connector doesn't support draining a job. To stop the pipeline, [cancel the job](https://docs.cloud.google.com/dataflow/docs/guides/stopping-a-pipeline#cancel) instead of draining it. Draining a job with the Spanner change streams connector can cause it to become stuck in a draining state. For more information, see [Draining](https://docs.cloud.google.com/spanner/docs/change-streams/use-dataflow#draining) in the Spanner documentation.

The pipeline groups Spanner change stream records into windows based on their timestamp,
with each window representing a time duration whose length you can configure with this template.
All records with timestamps belonging to the window are guaranteed to be in the window;
there can be no late arrivals. You can also define a number of output shards; the pipeline
creates one Cloud Storage output file per window per shard. Within an output file, records
are unordered. Output files can be written in either JSON or AVRO format,
depending on the user configuration.


Note that you can minimize network latency and network transport costs by running the
Dataflow job from the same region as your Spanner instance or
Cloud Storage bucket. If you use sources, sinks, staging file locations, or temporary
file locations that are located outside of your job's region, your data might be sent across
regions. See more about
[Dataflow regions](https://docs.cloud.google.com/dataflow/docs/concepts/regional-endpoints).

Learn more about [change streams](https://docs.cloud.google.com/spanner/docs/change-streams), [how to build change streams Dataflow pipelines](https://docs.cloud.google.com/spanner/docs/change-streams/use-dataflow), and [best practices](https://docs.cloud.google.com/spanner/docs/change-streams/use-dataflow#best_practices).

## Pipeline requirements

- The Spanner instance must exist prior to running the pipeline.
- The Spanner database must exist prior to running the pipeline.
- The Spanner metadata instance must exist prior to running the pipeline.
- The Spanner metadata database must exist prior to running the pipeline.
- The Spanner change stream must exist prior to running the pipeline.
- The Cloud Storage output bucket must exist prior to running the pipeline.

## Template parameters

### Required parameters

- **spannerInstanceId**: The Spanner instance ID to read change streams data from.
- **spannerDatabase**: The Spanner database to read change streams data from.
- **spannerMetadataInstanceId**: The Spanner instance ID to use for the change streams connector metadata table.
- **spannerMetadataDatabase**: The Spanner database to use for the change streams connector metadata table.
- **spannerChangeStreamName**: The name of the Spanner change stream to read from.
- **gcsOutputDirectory** : The path and filename prefix for writing output files. Must end with a slash. DateTime formatting is used to parse directory path for date \& time formatters. For example, `gs://your-bucket/your-path`.

### Optional parameters

- **spannerProjectId**: The ID of the Google Cloud project that contains the Spanner database to read change streams from. This project is also where the change streams connector metadata table is created. The default for this parameter is the project where the Dataflow pipeline is running.
- **spannerDatabaseRole** : The Spanner database role to use when running the template. This parameter is required only when the IAM principal who is running the template is a fine-grained access control user. The database role must have the `SELECT` privilege on the change stream and the `EXECUTE` privilege on the change stream's read function. For more information, see Fine-grained access control for change streams (<https://cloud.google.com/spanner/docs/fgac-change-streams>).
- **spannerMetadataTableName**: The Spanner change streams connector metadata table name to use. If not provided, a Spanner change streams metadata table is automatically created during pipeline execution. You must provide a value for this parameter when updating an existing pipeline. Otherwise, don't use this parameter.
- **startTimestamp** : The starting DateTime, inclusive, to use for reading change streams, in the format `Ex-2021-10-12T07:20:50.52Z`. Defaults to the timestamp when the pipeline starts, that is, the current time.
- **endTimestamp** : The ending DateTime, inclusive, to use for reading change streams. For example, `Ex-2021-10-12T07:20:50.52Z`. Defaults to an infinite time in the future.
- **spannerHost** : The Cloud Spanner endpoint to call in the template. Only used for testing. For example, `https://spanner.googleapis.com`. Defaults to: <https://spanner.googleapis.com>.
- **outputFileFormat** : The format of the output Cloud Storage file. Allowed formats are `TEXT` and `AVRO`. Defaults to `AVRO`.
- **windowDuration** : The window duration is the interval in which data is written to the output directory. Configure the duration based on the pipeline's throughput. For example, a higher throughput might require smaller window sizes so that the data fits into memory. Defaults to 5m (five minutes), with a minimum of 1s (one second). Allowed formats are: \[int\]s (for seconds, example: 5s), \[int\]m (for minutes, example: 12m), \[int\]h (for hours, example: 2h). For example, `5m`.
- **rpcPriority** : The request priority for Spanner calls. The value must be `HIGH`, `MEDIUM`, or `LOW`. Defaults to `HIGH`.
- **spannerChangeStreamTvfNameList**: Semicolon-separated list of Spanner Change Stream TVF names to query and union. Defaults to empty.
- **spannerDirectedReadOptions** : Directed read options for Cloud Spanner Change Stream reads. The value must be a JSON string format of DirectedReadOptions (e.g. {"includeReplicas":{"replicaSelections":\[{"location":"us-central1","type":"READ_ONLY"}\]}}). For more information, see Directed reads (<https://docs.cloud.google.com/spanner/docs/directed-reads>). Defaults to empty.
- **outputFilenamePrefix** : The prefix to place on each windowed file. For example, `output-`. Defaults to: output.
- **numShards**: The maximum number of output shards produced when writing. A higher number of shards means higher throughput for writing to Cloud Storage, but potentially higher data aggregation cost across shards when processing output Cloud Storage files. Defaults to: 20.

## Run the template

### Console

1. Go to the Dataflow **Create job from template** page.
[Go to Create job from template](https://console.cloud.google.com/dataflow/createjob)
2. In the **Job name** field, enter a unique job name.
3. Optional: For **Regional endpoint** , select a value from the drop-down menu. The default region is `us-central1`.


   For a list of regions where you can run a Dataflow job, see
   [Dataflow locations](https://docs.cloud.google.com/dataflow/docs/resources/locations).
4. From the **Dataflow template** drop-down menu, select the **Cloud Spanner change streams to Google Cloud Storage** template.
5. In the provided parameter fields, enter your parameter values.
6. Click **Run job**.

### gcloud

> [!NOTE]
> **Note:** To use the Google Cloud CLI to run flex templates, you must have [Google Cloud CLI](https://docs.cloud.google.com/sdk/docs/install) version 284.0.0 or later.

In your shell or terminal, run the template:

```bash
gcloud dataflow flex-template run JOB_NAME \
    --template-file-gcs-location=gs://dataflow-templates/VERSION/flex/Spanner_Change_Streams_to_Google_Cloud_Storage \
    --region REGION_NAME \
    --parameters \
spannerInstanceId=SPANNER_INSTANCE_ID,\
spannerDatabase=SPANNER_DATABASE,\
spannerMetadataInstanceId=SPANNER_METADATA_INSTANCE_ID,\
spannerMetadataDatabase=SPANNER_METADATA_DATABASE,\
spannerChangeStreamName=SPANNER_CHANGE_STREAM,\
gcsOutputDirectory=GCS_OUTPUT_DIRECTORY
```

Replace the following:

- `JOB_NAME`: a unique job name of your choice
- `VERSION`: the version of the template that you want to use

  You can use the following values:
  - `latest` to use the latest version of the template, which is available in the **non-dated** parent folder in the bucket--- [gs://dataflow-templates/latest/](https://console.cloud.google.com/storage/browser/dataflow-templates/latest)
  - the version name, like `2023-09-12-00_RC00`, to use a specific version of the template, which can be found nested in the respective dated parent folder in the bucket--- [gs://dataflow-templates/](https://console.cloud.google.com/storage/browser/dataflow-templates)

  > [!CAUTION]
  > **Caution:** The **latest** version of templates might update with breaking changes. Your production environments should use templates kept in the most recent **dated** parent folder to prevent these breaking changes from affecting your production workflows.

- `REGION_NAME`: the [region](https://docs.cloud.google.com/dataflow/docs/resources/locations) where you want to deploy your Dataflow job---for example, `us-central1`
- `SPANNER_INSTANCE_ID`: Cloud Spanner instance ID
- `SPANNER_DATABASE`: Cloud Spanner database
- `SPANNER_METADATA_INSTANCE_ID`: Cloud Spanner metadata instance ID
- `SPANNER_METADATA_DATABASE`: Cloud Spanner metadata database
- `SPANNER_CHANGE_STREAM`: Cloud Spanner change stream
- `GCS_OUTPUT_DIRECTORY`: File location for change streams output

### API

To run the template using the REST API, send an HTTP POST request. For more information on the
API and its authorization scopes, see
[`projects.templates.launch`](https://docs.cloud.google.com/dataflow/docs/reference/rest/v1b3/projects.templates/launch).

```json
POST https://dataflow.googleapis.com/v1b3/projects/PROJECT_ID/locations/LOCATION/flexTemplates:launch
{
   "launch_parameter": {
      "jobName": "JOB_NAME",
      "parameters": {
          "spannerInstanceId": "SPANNER_INSTANCE_ID",
          "spannerDatabase": "SPANNER_DATABASE",
          "spannerMetadataInstanceId": "SPANNER_METADATA_INSTANCE_ID",
          "spannerMetadataDatabase": "SPANNER_METADATA_DATABASE",
          "spannerChangeStreamName": "SPANNER_CHANGE_STREAM",
          "gcsOutputDirectory": "GCS_OUTPUT_DIRECTORY"
      },
      "containerSpecGcsPath": "gs://dataflow-templates/VERSION/flex/Spanner_Change_Streams_to_Google_Cloud_Storage",
   }
}
```

Replace the following:

- `PROJECT_ID`: the Google Cloud project ID where you want to run the Dataflow job
- `JOB_NAME`: a unique job name of your choice
- `VERSION`: the version of the template that you want to use

  You can use the following values:
  - `latest` to use the latest version of the template, which is available in the **non-dated** parent folder in the bucket--- [gs://dataflow-templates/latest/](https://console.cloud.google.com/storage/browser/dataflow-templates/latest)
  - the version name, like `2023-09-12-00_RC00`, to use a specific version of the template, which can be found nested in the respective dated parent folder in the bucket--- [gs://dataflow-templates/](https://console.cloud.google.com/storage/browser/dataflow-templates)

  > [!CAUTION]
  > **Caution:** The **latest** version of templates might update with breaking changes. Your production environments should use templates kept in the most recent **dated** parent folder to prevent these breaking changes from affecting your production workflows.

- `LOCATION`: the [region](https://docs.cloud.google.com/dataflow/docs/resources/locations) where you want to deploy your Dataflow job---for example, `us-central1`
- `SPANNER_INSTANCE_ID`: Cloud Spanner instance ID
- `SPANNER_DATABASE`: Cloud Spanner database
- `SPANNER_METADATA_INSTANCE_ID`: Cloud Spanner metadata instance ID
- `SPANNER_METADATA_DATABASE`: Cloud Spanner metadata database
- `SPANNER_CHANGE_STREAM`: Cloud Spanner change stream
- `GCS_OUTPUT_DIRECTORY`: File location for change streams output

## Template source code

### Java

    /*
     * Copyright (C) 2022 Google LLC
     *
     * Licensed under the Apache License, Version 2.0 (the "License"); you may not
     * use this file except in compliance with the License. You may obtain a copy of
     * the License at
     *
     *   http://www.apache.org/licenses/LICENSE-2.0
     *
     * Unless required by applicable law or agreed to in writing, software
     * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
     * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
     * License for the specific language governing permissions and limitations under
     * the License.
     */
    package com.google.cloud.teleport.v2.templates;

    import com.google.cloud.https://docs.cloud.google.com/java/docs/reference/google-cloud-core/latest/com.google.cloud.Timestamp.html;
    import com.google.cloud.spanner.https://docs.cloud.google.com/java/docs/reference/google-cloud-spanner-jdbc/latest/com.google.cloud.spanner.Options.html.https://docs.cloud.google.com/java/docs/reference/google-cloud-spanner-jdbc/latest/com.google.cloud.spanner.Options.RpcPriority.html;
    import com.google.cloud.teleport.metadata.https://docs.cloud.google.com/java/docs/reference/google-cloud-modelarmor/latest/com.google.cloud.modelarmor.v1.Template.html;
    import com.google.cloud.teleport.metadata.TemplateCategory;
    import com.google.cloud.teleport.v2.common.UncaughtExceptionLogger;
    import com.google.cloud.teleport.v2.options.SpannerChangeStreamsToGcsOptions;
    import com.google.cloud.teleport.v2.transforms.FileFormatFactorySpannerChangeStreams;
    import com.google.cloud.teleport.v2.utils.DurationUtils;
    import java.util.ArrayList;
    import java.util.Arrays;
    import java.util.https://docs.cloud.google.com/java/docs/reference/google-cloud-chronicle/latest/com.google.cloud.chronicle.v1.ExecuteDashboardQueryResponse.ColumnType.List.html;
    import java.util.stream.Collectors;
    import org.apache.beam.sdk.https://docs.cloud.google.com/java/docs/reference/google-cloud-aiplatform/latest/com.google.cloud.aiplatform.v1.Pipeline.html;
    import org.apache.beam.sdk.https://docs.cloud.google.com/java/docs/reference/google-cloud-firestore/latest/com.google.cloud.firestore.PipelineResult.html;
    import org.apache.beam.sdk.io.gcp.spanner.SpannerConfig;
    import org.apache.beam.sdk.io.gcp.spanner.SpannerIO;
    import org.apache.beam.sdk.options.PipelineOptionsFactory;
    import org.apache.beam.sdk.options.ValueProvider;
    import org.apache.beam.sdk.transforms.windowing.FixedWindows;
    import org.apache.beam.sdk.transforms.windowing.Window;
    import org.slf4j.Logger;
    import org.slf4j.LoggerFactory;

    /**
     * The {@link SpannerChangeStreamsToGcs} pipeline streams change stream record(s) and stores to
     * Google Cloud Storage bucket in user specified format. The sink data can be stored in a Text or
     * Avro file format.
     *
     * <p>Check out <a
     * href="https://github.com/GoogleCloudPlatform/DataflowTemplates/blob/main/v2/googlecloud-to-googlecloud/README_Spanner_Change_Streams_to_Google_Cloud_Storage.md">README</a>
     * for instructions on how to use or modify this template.
     */
    @Template(
        name = "Spanner_Change_Streams_to_Google_Cloud_Storage",
        category = TemplateCategory.STREAMING,
        displayName = "Cloud Spanner change streams to Cloud Storage",
        description = {
          "The Cloud Spanner change streams to Cloud Storage template is a streaming pipeline that streams Spanner data change records and writes them into a Cloud Storage bucket using Dataflow Runner V2.\n",
          "The pipeline groups Spanner change stream records into windows based on their timestamp, with each window representing a time duration whose length you can configure with this template. "
              + "All records with timestamps belonging to the window are guaranteed to be in the window; there can be no late arrivals. "
              + "You can also define a number of output shards; the pipeline creates one Cloud Storage output file per window per shard. "
              + "Within an output file, records are unordered. Output files can be written in either JSON or AVRO format, depending on the user configuration.\n",
          "Note that you can minimize network latency and network transport costs by running the Dataflow job from the same region as your Cloud Spanner instance or Cloud Storage bucket. "
              + "If you use sources, sinks, staging file locations, or temporary file locations that are located outside of your job's region, your data might be sent across regions. "
              + "See more about <a href=\"https://cloud.google.com/dataflow/docs/concepts/regional-endpoints\">Dataflow regional endpoints</a>.\n",
          "Learn more about <a href=\"https://cloud.google.com/spanner/docs/change-streams\">change streams</a>, <a href=\"https://cloud.google.com/spanner/docs/change-streams/use-dataflow\">how to build change streams Dataflow pipelines</a>, and <a href=\"https://cloud.google.com/spanner/docs/change-streams/use-dataflow#best_practices\">best practices</a>."
        },
        optionsClass = SpannerChangeStreamsToGcsOptions.class,
        flexContainerName = "googlecloud-to-googlecloud",
        documentation =
            "https://cloud.google.com/dataflow/docs/guides/templates/provided/cloud-spanner-change-streams-to-cloud-storage",
        contactInformation = "https://cloud.google.com/support",
        requirements = {
          "The Cloud Spanner instance must exist prior to running the pipeline.",
          "The Cloud Spanner database must exist prior to running the pipeline.",
          "The Cloud Spanner metadata instance must exist prior to running the pipeline.",
          "The Cloud Spanner metadata database must exist prior to running the pipeline.",
          "The Cloud Spanner change stream must exist prior to running the pipeline.",
          "The Cloud Storage output bucket must exist prior to running the pipeline."
        },
        streaming = true,
        supportsAtLeastOnce = true)
    public class SpannerChangeStreamsToGcs {
      private static final Logger LOG = LoggerFactory.getLogger(SpannerChangeStreamsToGcs.class);
      private static final https://docs.cloud.google.com/java/docs/reference/google-cloud-bigtable/latest/com.google.cloud.bigtable.common.Type.String.html USE_RUNNER_V2_EXPERIMENT = "use_runner_v2";

      public static void main(https://docs.cloud.google.com/java/docs/reference/google-cloud-bigtable/latest/com.google.cloud.bigtable.common.Type.String.html[] args) {
        UncaughtExceptionLogger.register();

        LOG.info("Starting Input Files to GCS");

        SpannerChangeStreamsToGcsOptions options =
            PipelineOptionsFactory.fromArgs(args).as(SpannerChangeStreamsToGcsOptions.class);

        run(options);
      }

      private static https://docs.cloud.google.com/java/docs/reference/google-cloud-bigtable/latest/com.google.cloud.bigtable.common.Type.String.html getProjectId(SpannerChangeStreamsToGcsOptions options) {
        return options.getSpannerProjectId().isEmpty()
            ? options.getProject()
            : options.getSpannerProjectId();
      }

      public static https://docs.cloud.google.com/java/docs/reference/google-cloud-firestore/latest/com.google.cloud.firestore.PipelineResult.html run(SpannerChangeStreamsToGcsOptions options) {
        LOG.info("Requested File Format is " + options.getOutputFileFormat());
        options.setStreaming(true);
        options.setEnableStreamingEngine(true);

        final https://docs.cloud.google.com/java/docs/reference/google-cloud-aiplatform/latest/com.google.cloud.aiplatform.v1.Pipeline.html pipeline = https://docs.cloud.google.com/java/docs/reference/google-cloud-aiplatform/latest/com.google.cloud.aiplatform.v1.Pipeline.html.create(options);

        // Get the Spanner project, instance, database, and change stream parameters.
        https://docs.cloud.google.com/java/docs/reference/google-cloud-bigtable/latest/com.google.cloud.bigtable.common.Type.String.html projectId = getProjectId(options);
        https://docs.cloud.google.com/java/docs/reference/google-cloud-bigtable/latest/com.google.cloud.bigtable.common.Type.String.html instanceId = options.getSpannerInstanceId();
        https://docs.cloud.google.com/java/docs/reference/google-cloud-bigtable/latest/com.google.cloud.bigtable.common.Type.String.html databaseId = options.getSpannerDatabase();
        https://docs.cloud.google.com/java/docs/reference/google-cloud-bigtable/latest/com.google.cloud.bigtable.common.Type.String.html metadataInstanceId = options.getSpannerMetadataInstanceId();
        https://docs.cloud.google.com/java/docs/reference/google-cloud-bigtable/latest/com.google.cloud.bigtable.common.Type.String.html metadataDatabaseId = options.getSpannerMetadataDatabase();
        https://docs.cloud.google.com/java/docs/reference/google-cloud-bigtable/latest/com.google.cloud.bigtable.common.Type.String.html changeStreamName = options.getSpannerChangeStreamName();

        // Retrieve and parse the start / end timestamps.
        https://docs.cloud.google.com/java/docs/reference/google-cloud-core/latest/com.google.cloud.Timestamp.html startTimestamp =
            options.getStartTimestamp().isEmpty()
                ? https://docs.cloud.google.com/java/docs/reference/google-cloud-core/latest/com.google.cloud.Timestamp.html.https://docs.cloud.google.com/java/docs/reference/google-cloud-core/latest/com.google.cloud.Timestamp.html#com_google_cloud_Timestamp_now__()
                : https://docs.cloud.google.com/java/docs/reference/google-cloud-core/latest/com.google.cloud.Timestamp.html.https://docs.cloud.google.com/java/docs/reference/google-cloud-core/latest/com.google.cloud.Timestamp.html#com_google_cloud_Timestamp_parseTimestamp_java_lang_String_(options.getStartTimestamp());
        https://docs.cloud.google.com/java/docs/reference/google-cloud-core/latest/com.google.cloud.Timestamp.html endTimestamp =
            options.getEndTimestamp().isEmpty()
                ? https://docs.cloud.google.com/java/docs/reference/google-cloud-core/latest/com.google.cloud.Timestamp.html.MAX_VALUE
                : https://docs.cloud.google.com/java/docs/reference/google-cloud-core/latest/com.google.cloud.Timestamp.html.https://docs.cloud.google.com/java/docs/reference/google-cloud-core/latest/com.google.cloud.Timestamp.html#com_google_cloud_Timestamp_parseTimestamp_java_lang_String_(options.getEndTimestamp());

        // Add use_runner_v2 to the experiments option, since Change Streams connector is only supported
        // on Dataflow runner v2.
        List<String> experiments = options.getExperiments();
        if (experiments == null) {
          experiments = new ArrayList<>();
        }
        if (!experiments.contains(USE_RUNNER_V2_EXPERIMENT)) {
          experiments.add(USE_RUNNER_V2_EXPERIMENT);
        }
        options.https://docs.cloud.google.com/java/docs/reference/google-cloud-dialogflow-cx/latest/com.google.cloud.dialogflow.cx.v3.ListExperimentsResponse.Builder.html#com_google_cloud_dialogflow_cx_v3_ListExperimentsResponse_Builder_setExperiments_int_com_google_cloud_dialogflow_cx_v3_Experiment_(experiments);

        https://docs.cloud.google.com/java/docs/reference/google-cloud-bigtable/latest/com.google.cloud.bigtable.common.Type.String.html metadataTableName =
            options.getSpannerMetadataTableName() == null
                ? null
                : options.getSpannerMetadataTableName();

        https://docs.cloud.google.com/java/docs/reference/google-cloud-bigtable/latest/com.google.cloud.bigtable.common.Type.String.html tvfNameListString = options.getSpannerChangeStreamTvfNameList();
        List<String> tvfNameList = null;
        if (tvfNameListString != null && !tvfNameListString.isEmpty()) {
          tvfNameList =
              Arrays.stream(tvfNameListString.https://docs.cloud.google.com/java/docs/reference/google-cloud-firestore/latest/com.google.cloud.firestore.pipeline.expressions.Expression.html#com_google_cloud_firestore_pipeline_expressions_Expression_split_com_google_cloud_firestore_pipeline_expressions_Expression_(";"))
                  .filter(name -> !name.https://docs.cloud.google.com/java/docs/reference/google-cloud-firestore/latest/com.google.cloud.firestore.pipeline.expressions.Expression.html#com_google_cloud_firestore_pipeline_expressions_Expression_trim__().isEmpty())
                  .collect(Collectors.toList());
        }

        final https://docs.cloud.google.com/java/docs/reference/google-cloud-spanner-jdbc/latest/com.google.cloud.spanner.Options.RpcPriority.html rpcPriority = options.getRpcPriority();
        SpannerConfig spannerConfig =
            SpannerConfig.create()
                .withHost(ValueProvider.StaticValueProvider.of(options.getSpannerHost()))
                .withProjectId(projectId)
                .withInstanceId(instanceId)
                .withDatabaseId(databaseId);
        // Propagate database role for fine-grained access control on change stream.
        if (options.getSpannerDatabaseRole() != null) {
          LOG.info("Setting database role on SpannerConfig: " + options.getSpannerDatabaseRole());
          spannerConfig =
              spannerConfig.withDatabaseRole(
                  ValueProvider.StaticValueProvider.of(options.getSpannerDatabaseRole()));
        }
        LOG.info("Created SpannerConfig: " + spannerConfig);
        https://docs.cloud.google.com/java/docs/reference/google-cloud-firestore/latest/com.google.cloud.firestore.Firestore.html#com_google_cloud_firestore_Firestore_pipeline__
            .apply(
                SpannerIO.https://docs.cloud.google.com/java/docs/reference/google-cloud-bigtable/latest/com.google.cloud.bigtable.data.v2.BigtableDataClient.html#com_google_cloud_bigtable_data_v2_BigtableDataClient_readChangeStream_com_google_cloud_bigtable_data_v2_models_ReadChangeStreamQuery_()
                    .withSpannerConfig(spannerConfig)
                    .withMetadataInstance(metadataInstanceId)
                    .withMetadataDatabase(metadataDatabaseId)
                    .withChangeStreamName(changeStreamName)
                    .withInclusiveStartAt(startTimestamp)
                    .withInclusiveEndAt(endTimestamp)
                    .withRpcPriority(rpcPriority)
                    .withMetadataTable(metadataTableName)
                    .withTvfNameList(tvfNameList)
                    .withDirectedReadOptions(options.getSpannerDirectedReadOptions()))
            .apply(
                "Creating " + options.getWindowDuration() + " Window",
                Window.into(FixedWindows.of(DurationUtils.parseDuration(options.getWindowDuration()))))
            .apply(
                "Write To GCS",
                FileFormatFactorySpannerChangeStreams.newBuilder().setOptions(options).build());

        return https://docs.cloud.google.com/java/docs/reference/google-cloud-firestore/latest/com.google.cloud.firestore.Firestore.html#com_google_cloud_firestore_Firestore_pipeline__.run();
      }
    }

## What's next

- Learn about [Dataflow templates](https://docs.cloud.google.com/dataflow/docs/concepts/dataflow-templates).
- See the list of [Google-provided templates](https://docs.cloud.google.com/dataflow/docs/guides/templates/provided-templates).

<br />

<br />