Cloud Storage Text to BigQuery pipeline is a streaming pipeline
that streams text files stored in Cloud Storage, transforms them using a
JavaScript user-defined function (UDF) that you provide, and appends the result to BigQuery.

The pipeline runs indefinitely and needs to be terminated manually via a
[cancel](https://docs.cloud.google.com/dataflow/docs/guides/stopping-a-pipeline#cancel) and not a
[drain](https://docs.cloud.google.com/dataflow/docs/guides/stopping-a-pipeline#drain), due to its use of the
`Watch` transform, which is a splittable `DoFn` that does not support
draining.

> [!NOTE]
> This template is impacted by [Beam Issue 18459](https://github.com/apache/beam/issues/18459) and will likely have memory issues on large buckets (observed on buckets with 5 million files). Here are some known workarounds:
>
> 1. Split the files into two or more buckets and use this template.
> 2. Create a custom template, possibly forked from this one, that streams file names through Pub/Sub and processes those files.

## Pipeline requirements

- Create a JSON file that describes the schema of your output table in BigQuery.
  Ensure that there is a top-level JSON array titled `fields` and that its
  contents follow the pattern `{"name": "COLUMN_NAME", "type": "DATA_TYPE"}`.
  For example:

  ```json
  {
    "fields": [
      {
        "name": "name",
        "type": "STRING"
      },
      {
        "name": "age",
        "type": "INTEGER"
      }
    ]
  }
  ```
- Create a JavaScript (`.js`) file with your UDF function that supplies the logic to transform the lines of text. Your function must return a JSON string.

  The following example splits each line of a CSV file, creates a JSON object
  with the values, and returns a JSON string:

  ```
  function process(inJson) {
    val = inJson.split(",");

    const obj = {
      "name": val[0],
      "age": parseInt(val[1])
    };
    return JSON.stringify(obj);
  }
  ```

## Template parameters

### Required parameters

- **inputFilePattern** : The gs:// path to the text in Cloud Storage you'd like to process. For example, `gs://your-bucket/your-file.txt`.
- **JSONPath** : The gs:// path to the JSON file that defines your BigQuery schema, stored in Cloud Storage. For example, `gs://your-bucket/your-schema.json`.
- **outputTable** : The location of the BigQuery table to use to store the processed data. If you reuse an existing table, it is overwritten. For example, `<PROJECT_ID>:<DATASET_NAME>.<TABLE_NAME>`.
- **javascriptTextTransformGcsPath** : The Cloud Storage URI of the `.js` file that defines the JavaScript user-defined function (UDF) you want to use. For example, `gs://your-bucket/your-transforms/*.js`.
- **javascriptTextTransformFunctionName** : The name of the JavaScript user-defined function (UDF) that you want to use. For example, if your JavaScript function code is `myTransform(inJson) { /*...do stuff...*/ }`, then the function name is `myTransform`. For sample JavaScript UDFs, see UDF Examples (<https://github.com/GoogleCloudPlatform/DataflowTemplates#udf-examples>) For example, `transform_udf1`.
- **bigQueryLoadingTemporaryDirectory** : Temporary directory for BigQuery loading process. For example, `gs://your-bucket/your-files/temp-dir`.

### Optional parameters

- **outputDeadletterTable** : Table for messages that failed to reach the output table. If a table doesn't exist, it is created during pipeline execution. If not specified, `<outputTableSpec>_error_records` is used. For example, `<PROJECT_ID>:<DATASET_NAME>.<TABLE_NAME>`.
- **useStorageWriteApiAtLeastOnce** : This parameter takes effect only if `Use BigQuery Storage Write API` is enabled. If enabled the at-least-once semantics will be used for Storage Write API, otherwise exactly-once semantics will be used. Defaults to: false.
- **useStorageWriteApi** : If true, the pipeline uses the BigQuery Storage Write API (<https://cloud.google.com/bigquery/docs/write-api>). The default value is `false`. For more information, see Using the Storage Write API (<https://beam.apache.org/documentation/io/built-in/google-bigquery/#storage-write-api>).
- **numStorageWriteApiStreams** : When using the Storage Write API, specifies the number of write streams. If `useStorageWriteApi` is `true` and `useStorageWriteApiAtLeastOnce` is `false`, then you must set this parameter. Defaults to: 0.
- **storageWriteApiTriggeringFrequencySec** : When using the Storage Write API, specifies the triggering frequency, in seconds. If `useStorageWriteApi` is `true` and `useStorageWriteApiAtLeastOnce` is `false`, then you must set this parameter.
- **pythonExternalTextTransformGcsPath** : The Cloud Storage path pattern for the Python code containing your user-defined functions. For example, `gs://your-bucket/your-function.py`.
- **javascriptTextTransformReloadIntervalMinutes** : Specifies how frequently to reload the UDF, in minutes. If the value is greater than 0, Dataflow periodically checks the UDF file in Cloud Storage, and reloads the UDF if the file is modified. This parameter allows you to update the UDF while the pipeline is running, without needing to restart the job. If the value is `0`, UDF reloading is disabled. The default value is `0`.

## User-defined function

This template requires a UDF that parses the input files, as described in
[Pipeline requirements](https://docs.cloud.google.com/dataflow/docs/guides/templates/provided/text-to-bigquery-stream#pipeline-requirements). The template calls
the UDF for every line of text in each input file. For more information about
creating UDFs, see
[Create
user-defined functions for Dataflow templates](https://docs.cloud.google.com/dataflow/docs/guides/templates/create-template-udf).

### Function specification

The UDF has the following specification:

- **Input**: a single line of text from an input file.
- **Output**: a JSON string that matches the schema of the BigQuery destination table.

## Run the template

### Console

1. Go to the Dataflow **Create job from template** page.
[Go to Create job from template](https://console.cloud.google.com/dataflow/createjob)
2. In the **Job name** field, enter a unique job name.
3. Optional: For **Regional endpoint** , select a value from the drop-down menu. The default region is `us-central1`.


   For a list of regions where you can run a Dataflow job, see
   [Dataflow locations](https://docs.cloud.google.com/dataflow/docs/resources/locations).
4. From the **Dataflow template** drop-down menu, select the **Cloud Storage Text to BigQuery (Stream)** template.
5. In the provided parameter fields, enter your parameter values.
6. Click **Run job**.

### gcloud

> [!NOTE]
> **Note:** To use the Google Cloud CLI to run flex templates, you must have [Google Cloud CLI](https://docs.cloud.google.com/sdk/docs/install) version 284.0.0 or later.

In your shell or terminal, run the template:

```bash
gcloud dataflow flex-template run JOB_NAME \
    --template-file-gcs-location gs://dataflow-templates/VERSION/flex/Stream_GCS_Text_to_BigQuery_Flex \
    --region REGION_NAME \
    --staging-location STAGING_LOCATION \
    --parameters \
javascriptTextTransformGcsPath=PATH_TO_JAVASCRIPT_UDF_FILE,\
javascriptTextTransformFunctionName=JAVASCRIPT_FUNCTION,\
JSONPath=PATH_TO_BIGQUERY_SCHEMA_JSON,\
inputFilePattern=PATH_TO_TEXT_DATA,\
outputTable=BIGQUERY_TABLE,\
outputDeadletterTable=BIGQUERY_UNPROCESSED_TABLE,\
bigQueryLoadingTemporaryDirectory=PATH_TO_TEMP_DIR_ON_GCS
```

Replace the following:

- `JOB_NAME`: a unique job name of your choice
- `REGION_NAME`: the [region](https://docs.cloud.google.com/dataflow/docs/resources/locations) where you want to deploy your Dataflow job---for example, `us-central1`
- `VERSION`: the version of the template that you want to use

  You can use the following values:
  - `latest` to use the latest version of the template, which is available in the **non-dated** parent folder in the bucket--- [gs://dataflow-templates/latest/](https://console.cloud.google.com/storage/browser/dataflow-templates/latest)
  - the version name, like `2023-09-12-00_RC00`, to use a specific version of the template, which can be found nested in the respective dated parent folder in the bucket--- [gs://dataflow-templates/](https://console.cloud.google.com/storage/browser/dataflow-templates)

  > [!CAUTION]
  > **Caution:** The **latest** version of templates might update with breaking changes. Your production environments should use templates kept in the most recent **dated** parent folder to prevent these breaking changes from affecting your production workflows.

- `STAGING_LOCATION`: the location for staging local files (for example, `gs://your-bucket/staging`)
- `JAVASCRIPT_FUNCTION`: the name of the JavaScript user-defined function (UDF) that you want to use


  For example, if your JavaScript function code is
  `myTransform(inJson) { /*...do stuff...*/ }`, then the function name is
  `myTransform`. For sample JavaScript UDFs, see
  [UDF Examples](https://github.com/GoogleCloudPlatform/DataflowTemplates#udf-examples).
- `PATH_TO_BIGQUERY_SCHEMA_JSON`: the Cloud Storage path to the JSON file containing the schema definition
- `PATH_TO_JAVASCRIPT_UDF_FILE`: the Cloud Storage URI of the `.js` file that defines the JavaScript user-defined function (UDF) you want to use---for example, `gs://my-bucket/my-udfs/my_file.js`
- `PATH_TO_TEXT_DATA`: the Cloud Storage path to your text dataset
- `BIGQUERY_TABLE`: your BigQuery table name
- `BIGQUERY_UNPROCESSED_TABLE`: the name of your BigQuery table for unprocessed messages
- `PATH_TO_TEMP_DIR_ON_GCS`: the Cloud Storage path to the temp directory

### API

To run the template using the REST API, send an HTTP POST request. For more information on the
API and its authorization scopes, see
[`projects.templates.launch`](https://docs.cloud.google.com/dataflow/docs/reference/rest/v1b3/projects.templates/launch).

```json
POST https://dataflow.googleapis.com/v1b3/projects/PROJECT_ID/locations/LOCATION/flexTemplates:launch
{
   "launch_parameter": {
      "jobName": "JOB_NAME",
      "parameters": {
       "javascriptTextTransformFunctionName": "JAVASCRIPT_FUNCTION",
       "JSONPath": "PATH_TO_BIGQUERY_SCHEMA_JSON",
       "javascriptTextTransformGcsPath": "PATH_TO_JAVASCRIPT_UDF_FILE",
       "inputFilePattern":"PATH_TO_TEXT_DATA",
       "outputTable":"BIGQUERY_TABLE",
       "outputDeadletterTable":"BIGQUERY_UNPROCESSED_TABLE",
       "bigQueryLoadingTemporaryDirectory": "PATH_TO_TEMP_DIR_ON_GCS"
      },
      "containerSpecGcsPath": "gs://dataflow-templates/VERSION/flex/Stream_GCS_Text_to_BigQuery_Flex",
   }
}
```

Replace the following:

- `PROJECT_ID`: the Google Cloud project ID where you want to run the Dataflow job
- `JOB_NAME`: a unique job name of your choice
- `LOCATION`: the [region](https://docs.cloud.google.com/dataflow/docs/resources/locations) where you want to deploy your Dataflow job---for example, `us-central1`
- `VERSION`: the version of the template that you want to use

  You can use the following values:
  - `latest` to use the latest version of the template, which is available in the **non-dated** parent folder in the bucket--- [gs://dataflow-templates/latest/](https://console.cloud.google.com/storage/browser/dataflow-templates/latest)
  - the version name, like `2023-09-12-00_RC00`, to use a specific version of the template, which can be found nested in the respective dated parent folder in the bucket--- [gs://dataflow-templates/](https://console.cloud.google.com/storage/browser/dataflow-templates)

  > [!CAUTION]
  > **Caution:** The **latest** version of templates might update with breaking changes. Your production environments should use templates kept in the most recent **dated** parent folder to prevent these breaking changes from affecting your production workflows.

- `STAGING_LOCATION`: the location for staging local files (for example, `gs://your-bucket/staging`)
- `JAVASCRIPT_FUNCTION`: the name of the JavaScript user-defined function (UDF) that you want to use


  For example, if your JavaScript function code is
  `myTransform(inJson) { /*...do stuff...*/ }`, then the function name is
  `myTransform`. For sample JavaScript UDFs, see
  [UDF Examples](https://github.com/GoogleCloudPlatform/DataflowTemplates#udf-examples).
- `PATH_TO_BIGQUERY_SCHEMA_JSON`: the Cloud Storage path to the JSON file containing the schema definition
- `PATH_TO_JAVASCRIPT_UDF_FILE`: the Cloud Storage URI of the `.js` file that defines the JavaScript user-defined function (UDF) you want to use---for example, `gs://my-bucket/my-udfs/my_file.js`
- `PATH_TO_TEXT_DATA`: the Cloud Storage path to your text dataset
- `BIGQUERY_TABLE`: your BigQuery table name
- `BIGQUERY_UNPROCESSED_TABLE`: the name of your BigQuery table for unprocessed messages
- `PATH_TO_TEMP_DIR_ON_GCS`: the Cloud Storage path to the temp directory

## Template source code

### Java

    /*
     * Copyright (C) 2018 Google LLC
     *
     * Licensed under the Apache License, Version 2.0 (the "License"); you may not
     * use this file except in compliance with the License. You may obtain a copy of
     * the License at
     *
     *   http://www.apache.org/licenses/LICENSE-2.0
     *
     * Unless required by applicable law or agreed to in writing, software
     * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
     * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
     * License for the specific language governing permissions and limitations under
     * the License.
     */
    package com.google.cloud.teleport.v2.templates;

    import com.google.api.client.json.https://docs.cloud.google.com/java/docs/reference/google-http-client/latest/com.google.api.client.json.JsonFactory.html;
    import com.google.api.services.bigquery.model.TableRow;
    import com.google.cloud.teleport.metadata.MultiTemplate;
    import com.google.cloud.teleport.metadata.Template;
    import com.google.cloud.teleport.metadata.TemplateCategory;
    import com.google.cloud.teleport.metadata.TemplateParameter;
    import com.google.cloud.teleport.v2.coders.FailsafeElementCoder;
    import com.google.cloud.teleport.v2.common.UncaughtExceptionLogger;
    import com.google.cloud.teleport.v2.options.BigQueryStorageApiStreamingOptions;
    import com.google.cloud.teleport.v2.templates.TextToBigQueryStreaming.TextToBigQueryStreamingOptions;
    import com.google.cloud.teleport.v2.transforms.BigQueryConverters.FailsafeJsonToTableRow;
    import com.google.cloud.teleport.v2.transforms.ErrorConverters.WriteStringMessageErrors;
    import com.google.cloud.teleport.v2.transforms.JavascriptTextTransformer.FailsafeJavascriptUdf;
    import com.google.cloud.teleport.v2.transforms.PythonExternalTextTransformer;
    import com.google.cloud.teleport.v2.transforms.PythonExternalTextTransformer.RowToStringFailsafeElementFn;
    import com.google.cloud.teleport.v2.utils.BigQueryIOUtils;
    import com.google.cloud.teleport.v2.utils.GCSUtils;
    import com.google.cloud.teleport.v2.utils.ResourceUtils;
    import com.google.cloud.teleport.v2.https://docs.cloud.google.com/java/docs/reference/google-http-client/latest/com.google.api.client.json.JsonToken.html#com_google_api_client_json_JsonToken_values__.FailsafeElement;
    import com.google.common.base.Strings;
    import com.google.common.collect.ImmutableList;
    import java.io.IOException;
    import org.apache.beam.sdk.Pipeline;
    import org.apache.beam.sdk.PipelineResult;
    import org.apache.beam.sdk.coders.CoderRegistry;
    import org.apache.beam.sdk.coders.StringUtf8Coder;
    import org.apache.beam.sdk.extensions.gcp.util.Transport;
    import org.apache.beam.sdk.io.TextIO;
    import org.apache.beam.sdk.io.gcp.bigquery.BigQueryIO;
    import org.apache.beam.sdk.io.gcp.bigquery.BigQueryIO.Write.CreateDisposition;
    import org.apache.beam.sdk.io.gcp.bigquery.BigQueryIO.Write.WriteDisposition;
    import org.apache.beam.sdk.io.gcp.bigquery.BigQueryInsertError;
    import org.apache.beam.sdk.io.gcp.bigquery.InsertRetryPolicy;
    import org.apache.beam.sdk.io.gcp.bigquery.WriteResult;
    import org.apache.beam.sdk.options.Default;
    import org.apache.beam.sdk.options.PipelineOptionsFactory;
    import org.apache.beam.sdk.options.ValueProvider.StaticValueProvider;
    import org.apache.beam.sdk.transforms.Flatten;
    import org.apache.beam.sdk.transforms.MapElements;
    import org.apache.beam.sdk.transforms.ParDo;
    import org.apache.beam.sdk.transforms.Watch.Growth;
    import org.apache.beam.sdk.https://docs.cloud.google.com/java/docs/reference/google-http-client/latest/com.google.api.client.json.JsonToken.html#com_google_api_client_json_JsonToken_values__.PCollection;
    import org.apache.beam.sdk.https://docs.cloud.google.com/java/docs/reference/google-http-client/latest/com.google.api.client.json.JsonToken.html#com_google_api_client_json_JsonToken_values__.PCollectionList;
    import org.apache.beam.sdk.https://docs.cloud.google.com/java/docs/reference/google-http-client/latest/com.google.api.client.json.JsonToken.html#com_google_api_client_json_JsonToken_values__.PCollectionTuple;
    import org.apache.beam.sdk.https://docs.cloud.google.com/java/docs/reference/google-http-client/latest/com.google.api.client.json.JsonToken.html#com_google_api_client_json_JsonToken_values__.TupleTag;
    import org.apache.beam.sdk.https://docs.cloud.google.com/java/docs/reference/google-http-client/latest/com.google.api.client.json.JsonToken.html#com_google_api_client_json_JsonToken_values__.TupleTagList;
    import org.apache.commons.lang3.StringUtils;
    import org.joda.time.Duration;
    import org.slf4j.Logger;
    import org.slf4j.LoggerFactory;

    /**
     * The {@link TextToBigQueryStreaming} is a streaming version of {@link TextIOToBigQuery} pipeline
     * that reads text files, applies a JavaScript UDF and writes the output to BigQuery. The pipeline
     * continuously polls for new files, reads them row-by-row and processes each record into BigQuery.
     * The polling interval is set at 10 seconds.
     *
     * <p>Check out <a
     * href="https://github.com/GoogleCloudPlatform/DataflowTemplates/blob/main/v2/googlecloud-to-googlecloud/README_Stream_GCS_Text_to_BigQuery_Flex.md">README</a>
     * for instructions on how to use or modify this template.
     */
    @MultiTemplate({
      @Template(
          name = "Stream_GCS_Text_to_BigQuery_Flex",
          category = TemplateCategory.STREAMING,
          displayName = "Cloud Storage Text to BigQuery (Stream)",
          description = {
            "The Text Files on Cloud Storage to BigQuery pipeline is a streaming pipeline that allows you to stream text files stored in Cloud Storage, transform them using a JavaScript User Defined Function (UDF) that you provide, and append the result to BigQuery.\n",
            "The pipeline runs indefinitely and needs to be terminated manually via a\n"
                + "    <a href=\"https://cloud.google.com/dataflow/docs/guides/stopping-a-pipeline#cancel\">cancel</a> and not a\n"
                + "    <a href=\"https://cloud.google.com/dataflow/docs/guides/stopping-a-pipeline#drain\">drain</a>, due to its use of the\n"
                + "    <code>Watch</code> transform, which is a splittable <code>DoFn</code> that does not support\n"
                + "    draining."
          },
          skipOptions = {
            "pythonExternalTextTransfromGcsPath",
            "pythonExternalTextTransformFunctionName"
          },
          optionsClass = TextToBigQueryStreamingOptions.class,
          flexContainerName = "googlecloud-to-googlecloud",
          documentation =
              "https://cloud.google.com/dataflow/docs/guides/templates/provided/text-to-bigquery-stream",
          contactInformation = "https://cloud.google.com/support",
          requirements = {
            "Create a JSON file that describes the schema of your output table in BigQuery.\n"
                + "    <p>\n"
                + "      Ensure that there is a top-level JSON array titled <code>fields</code> and that its\n"
                + "      contents follow the pattern <code>{\"name\": \"COLUMN_NAME\", \"type\": \"DATA_TYPE\"}</code>.\n"
                + "      For example:\n"
                + "    </p>\n"
                + "<pre class=\"prettyprint lang-json\">\n"
                + "{\n"
                + "  \"fields\": [\n"
                + "    {\n"
                + "      \"name\": \"location\",\n"
                + "      \"type\": \"STRING\"\n"
                + "    },\n"
                + "    {\n"
                + "      \"name\": \"name\",\n"
                + "      \"type\": \"STRING\"\n"
                + "    },\n"
                + "    {\n"
                + "      \"name\": \"age\",\n"
                + "      \"type\": \"STRING\"\n"
                + "    },\n"
                + "    {\n"
                + "      \"name\": \"color\",\n"
                + "      \"type\": \"STRING\",\n"
                + "      \"mode\": \"REQUIRED\"\n"
                + "    },\n"
                + "    {\n"
                + "      \"name\": \"coffee\",\n"
                + "      \"type\": \"STRING\",\n"
                + "      \"mode\": \"REQUIRED\"\n"
                + "    }\n"
                + "  ]\n"
                + "}\n"
                + "</pre>",
            "Create a JavaScript (<code>.js</code>) file with your UDF function that supplies the logic\n"
                + "    to transform the lines of text. Note that your function must return a JSON string.\n"
                + "    <p>For example, this function splits each line of a CSV file and returns a JSON string after\n"
                + "      transforming the values.</p>\n"
                + "<pre class=\"prettyprint\" suppresswarning>\n"
                + "function transform(line) {\n"
                + "var values = line.split(',');\n"
                + "\n"
                + "var obj = new Object();\n"
                + "obj.location = values[0];\n"
                + "obj.name = values[1];\n"
                + "obj.age = values[2];\n"
                + "obj.color = values[3];\n"
                + "obj.coffee = values[4];\n"
                + "var jsonString = JSON.stringify(obj);\n"
                + "\n"
                + "return jsonString;\n"
                + "}\n"
                + "</pre>"
          },
          streaming = true,
          supportsAtLeastOnce = true),
      @Template(
          name = "Stream_GCS_Text_to_BigQuery_Xlang",
          category = TemplateCategory.STREAMING,
          displayName = "Cloud Storage Text to BigQuery (Stream) with Python UDF",
          type = Template.TemplateType.XLANG,
          description = {
            "The Text Files on Cloud Storage to BigQuery pipeline is a streaming pipeline that allows you to stream text files stored in Cloud Storage, transform them using a Python User Defined Function (UDF) that you provide, and append the result to BigQuery.\n",
            "The pipeline runs indefinitely and needs to be terminated manually via a\n"
                + "    <a href=\"https://cloud.google.com/dataflow/docs/guides/stopping-a-pipeline#cancel\">cancel</a> and not a\n"
                + "    <a href=\"https://cloud.google.com/dataflow/docs/guides/stopping-a-pipeline#drain\">drain</a>, due to its use of the\n"
                + "    <code>Watch</code> transform, which is a splittable <code>DoFn</code> that does not support\n"
                + "    draining."
          },
          skipOptions = {
            "javascriptTextTransformGcsPath",
            "javascriptTextTransformFunctionName",
            "javascriptTextTransformReloadIntervalMinutes"
          },
          optionsClass = TextToBigQueryStreamingOptions.class,
          flexContainerName = "googlecloud-to-googlecloud-xlang",
          documentation =
              "https://cloud.google.com/dataflow/docs/guides/templates/provided/text-to-bigquery-stream",
          contactInformation = "https://cloud.google.com/support",
          requirements = {
            "Create a JSON file that describes the schema of your output table in BigQuery.\n"
                + "    <p>\n"
                + "      Ensure that there is a top-level JSON array titled <code>fields</code> and that its\n"
                + "      contents follow the pattern <code>{\"name\": \"COLUMN_NAME\", \"type\": \"DATA_TYPE\"}</code>.\n"
                + "      For example:\n"
                + "    </p>\n"
                + "<pre class=\"prettyprint lang-json\">\n"
                + "{\n"
                + "  \"fields\": [\n"
                + "    {\n"
                + "      \"name\": \"location\",\n"
                + "      \"type\": \"STRING\"\n"
                + "    },\n"
                + "    {\n"
                + "      \"name\": \"name\",\n"
                + "      \"type\": \"STRING\"\n"
                + "    },\n"
                + "    {\n"
                + "      \"name\": \"age\",\n"
                + "      \"type\": \"STRING\"\n"
                + "    },\n"
                + "    {\n"
                + "      \"name\": \"color\",\n"
                + "      \"type\": \"STRING\",\n"
                + "      \"mode\": \"REQUIRED\"\n"
                + "    },\n"
                + "    {\n"
                + "      \"name\": \"coffee\",\n"
                + "      \"type\": \"STRING\",\n"
                + "      \"mode\": \"REQUIRED\"\n"
                + "    }\n"
                + "  ]\n"
                + "}\n"
                + "</pre>",
            "Create a Python (<code>.js</code>) file with your UDF function that supplies the logic\n"
                + "    to transform the lines of text. Note that your function must return a JSON string.\n"
                + "    <p>For example, this function splits each line of a CSV file and returns a JSON string after\n"
                + "      transforming the values.</p>\n"
                + "<pre class=\"prettyprint\" suppresswarning>\n"
                + "import json\n"
                + "def transform(line): \n"
                + "  values = line.split(',')\n"
                + "\n"
                + "  obj = {\n"
                + "     'location' : values[0],\n"
                + "     'name' : values[1],\n"
                + "     'age' : values[2],\n"
                + "     'color' : values[3],\n"
                + "     'coffee' : values[4]\n"
                + "  }\n"
                + "  jsonString = JSON.dumps(obj);\n"
                + "\n"
                + "  return jsonString;\n"
                + "\n"
                + "</pre>"
          },
          streaming = true,
          supportsAtLeastOnce = true)
    })
    public class TextToBigQueryStreaming {

      private static final Logger LOG = LoggerFactory.getLogger(TextToBigQueryStreaming.class);

      /** The tag for the main output for the UDF. */
      private static final TupleTag<FailsafeElement<String, String>> UDF_OUT =
          new TupleTag<FailsafeElement<String, String>>() {};

      /** The tag for the dead-letter output of the udf. */
      private static final TupleTag<FailsafeElement<String, String>> UDF_DEADLETTER_OUT =
          new TupleTag<FailsafeElement<String, String>>() {};

      /** The tag for the main output of the json transformation. */
      private static final TupleTag<TableRow> TRANSFORM_OUT = new TupleTag<TableRow>() {};

      /** The tag for the dead-letter output of the json to table row transform. */
      private static final TupleTag<FailsafeElement<String, String>> TRANSFORM_DEADLETTER_OUT =
          new TupleTag<FailsafeElement<String, String>>() {};

      /** The default suffix for error tables if dead letter table is not specified. */
      private static final String DEFAULT_DEADLETTER_TABLE_SUFFIX = "_error_records";

      /** Default interval for polling files in GCS. */
      private static final Duration DEFAULT_POLL_INTERVAL = Duration.standardSeconds(10);

      /** Coder for FailsafeElement. */
      private static final FailsafeElementCoder<String, String> FAILSAFE_ELEMENT_CODER =
          FailsafeElementCoder.of(StringUtf8Coder.of(), StringUtf8Coder.of());

      private static final https://docs.cloud.google.com/java/docs/reference/google-http-client/latest/com.google.api.client.json.JsonFactory.html JSON_FACTORY = Transport.getJsonFactory();

      /**
       * Main entry point for executing the pipeline. This will run the pipeline asynchronously. If
       * blocking execution is required, use the {@link
       * TextToBigQueryStreaming#run(TextToBigQueryStreamingOptions)} method to start the pipeline and
       * invoke {@code result.waitUntilFinish()} on the {@link PipelineResult}
       *
       * @param args The command-line arguments to the pipeline.
       */
      public static void main(String[] args) {
        UncaughtExceptionLogger.register();

        // Parse the user options passed from the command-line
        TextToBigQueryStreamingOptions options =
            PipelineOptionsFactory.fromArgs(args)
                .withValidation()
                .as(TextToBigQueryStreamingOptions.class);
        BigQueryIOUtils.validateBQStorageApiOptionsStreaming(options);

        run(options);
      }

      /**
       * Runs the pipeline with the supplied options.
       *
       * @param options The execution parameters to the pipeline.
       * @return The result of the pipeline execution.
       */
      public static PipelineResult run(TextToBigQueryStreamingOptions options) {

        // Create the pipeline
        Pipeline pipeline = Pipeline.create(options);

        // Register the coder for pipeline
        FailsafeElementCoder<String, String> coder =
            FailsafeElementCoder.of(StringUtf8Coder.of(), StringUtf8Coder.of());

        CoderRegistry coderRegistry = pipeline.getCoderRegistry();
        coderRegistry.registerCoderForType(coder.getEncodedTypeDescriptor(), coder);

        // Determine if we are using Python UDFs or JS UDFs based on the provided options.
        boolean useJavascriptUdf = !Strings.isNullOrEmpty(options.getJavascriptTextTransformGcsPath());
        boolean usePythonUdf = !Strings.isNullOrEmpty(options.getPythonExternalTextTransformGcsPath());
        if (useJavascriptUdf && usePythonUdf) {
          throw new IllegalArgumentException(
              "Either javascript or Python gcs path must be provided, but not both.");
        }

        /*
         * Steps:
         *  1) Read from the text source continuously.
         *  2) Convert to FailsafeElement.
         *  3) Apply Javascript udf transformation.
         *    - Tag records that were successfully transformed and those
         *      that failed transformation.
         *  4) Convert records to TableRow.
         *    - Tag records that were successfully converted and those
         *      that failed conversion.
         *  5) Insert successfully converted records into BigQuery.
         *    - Errors encountered while streaming will be sent to deadletter table.
         *  6) Insert records that failed into deadletter table.
         */

        PCollection<String> sourceRead =
            pipeline.apply(
                TextIO.read()
                    .from(options.getInputFilePattern())
                    .watchForNewFiles(DEFAULT_POLL_INTERVAL, Growth.never()));
        PCollectionTuple transformedOutput;
        if (usePythonUdf) {
          transformedOutput =
              sourceRead
                  .apply(
                      "MapToRecord",
                      PythonExternalTextTransformer.FailsafeRowPythonExternalUdf
                          .stringMappingFunction())
                  .setRowSchema(PythonExternalTextTransformer.FailsafeRowPythonExternalUdf.ROW_SCHEMA)
                  .apply(
                      "InvokeUDF",
                      PythonExternalTextTransformer.FailsafePythonExternalUdf.newBuilder()
                          .setFileSystemPath(options.getPythonExternalTextTransformGcsPath())
                          .setFunctionName(options.getPythonExternalTextTransformFunctionName())
                          .build())
                  .apply(
                      ParDo.of(new RowToStringFailsafeElementFn(UDF_OUT, UDF_DEADLETTER_OUT))
                          .withOutputTags(UDF_OUT, TupleTagList.of(UDF_DEADLETTER_OUT)));

        } else {
          transformedOutput =
              pipeline

                  // 1) Read from the text source continuously.
                  .apply(
                      "ReadFromSource",
                      TextIO.read()
                          .from(options.getInputFilePattern())
                          .watchForNewFiles(DEFAULT_POLL_INTERVAL, Growth.never()))

                  // 2) Convert to FailsafeElement.
                  .apply(
                      "ConvertToFailsafeElement",
                      MapElements.into(FAILSAFE_ELEMENT_CODER.getEncodedTypeDescriptor())
                          .via(input -> FailsafeElement.of(input, input)))

                  // 3) Apply Javascript udf transformation.
                  .apply(
                      "ApplyUDFTransformation",
                      FailsafeJavascriptUdf.<String>newBuilder()
                          .setFileSystemPath(options.getJavascriptTextTransformGcsPath())
                          .setFunctionName(options.getJavascriptTextTransformFunctionName())
                          .setReloadIntervalMinutes(
                              options.getJavascriptTextTransformReloadIntervalMinutes())
                          .setSuccessTag(UDF_OUT)
                          .setFailureTag(UDF_DEADLETTER_OUT)
                          .build());
        }

        PCollectionTuple convertedTableRows =
            transformedOutput

                // 4) Convert records to TableRow.
                .get(UDF_OUT)
                .apply(
                    "ConvertJSONToTableRow",
                    FailsafeJsonToTableRow.<String>newBuilder()
                        .setSuccessTag(TRANSFORM_OUT)
                        .setFailureTag(TRANSFORM_DEADLETTER_OUT)
                        .build());

        WriteResult writeResult =
            convertedTableRows

                // 5) Insert successfully converted records into BigQuery.
                .get(TRANSFORM_OUT)
                .apply(
                    "InsertIntoBigQuery",
                    BigQueryIO.writeTableRows()
                        .withJsonSchema(GCSUtils.getGcsFileAsString(options.getJSONPath()))
                        .to(options.getOutputTable())
                        .withExtendedErrorInfo()
                        .withoutValidation()
                        .withCreateDisposition(CreateDisposition.CREATE_IF_NEEDED)
                        .withWriteDisposition(WriteDisposition.WRITE_APPEND)
                        .withFailedInsertRetryPolicy(InsertRetryPolicy.retryTransientErrors())
                        .withCustomGcsTempLocation(
                            StaticValueProvider.of(options.getBigQueryLoadingTemporaryDirectory())));

        // Elements that failed inserts into BigQuery are extracted and converted to FailsafeElement
        PCollection<FailsafeElement<String, String>> failedInserts =
            BigQueryIOUtils.writeResultToBigQueryInsertErrors(writeResult, options)
                .apply(
                    "WrapInsertionErrors",
                    MapElements.into(FAILSAFE_ELEMENT_CODER.getEncodedTypeDescriptor())
                        .via(TextToBigQueryStreaming::wrapBigQueryInsertError));

        // 6) Insert records that failed transformation or conversion into deadletter table
        PCollectionList.of(
                ImmutableList.of(
                    transformedOutput.get(UDF_DEADLETTER_OUT),
                    convertedTableRows.get(TRANSFORM_DEADLETTER_OUT),
                    failedInserts))
            .apply("Flatten", Flatten.pCollections())
            .apply(
                "WriteFailedRecords",
                WriteStringMessageErrors.newBuilder()
                    .setErrorRecordsTable(
                        StringUtils.isNotEmpty(options.getOutputDeadletterTable())
                            ? options.getOutputDeadletterTable()
                            : options.getOutputTable() + DEFAULT_DEADLETTER_TABLE_SUFFIX)
                    .setErrorRecordsTableSchema(ResourceUtils.getDeadletterTableSchemaJson())
                    .build());

        return pipeline.run();
      }

      /**
       * Method to wrap a {@link BigQueryInsertError} into a {@link FailsafeElement}.
       *
       * @param insertError BigQueryInsert error.
       * @return FailsafeElement object.
       * @throws IOException
       */
      static FailsafeElement<String, String> wrapBigQueryInsertError(BigQueryInsertError insertError) {

        FailsafeElement<String, String> failsafeElement;
        try {

          String rowPayload = JSON_FACTORY.https://docs.cloud.google.com/java/docs/reference/google-http-client/latest/com.google.api.client.json.JsonFactory.html#com_google_api_client_json_JsonFactory_toString_java_lang_Object_(insertError.getRow());
          String errorMessage = JSON_FACTORY.https://docs.cloud.google.com/java/docs/reference/google-http-client/latest/com.google.api.client.json.JsonFactory.html#com_google_api_client_json_JsonFactory_toString_java_lang_Object_(insertError.getError());

          failsafeElement = FailsafeElement.of(rowPayload, rowPayload);
          failsafeElement.setErrorMessage(errorMessage);

        } catch (IOException e) {
          throw new RuntimeException(e);
        }

        return failsafeElement;
      }

      /**
       * The {@link TextToBigQueryStreamingOptions} class provides the custom execution options passed
       * by the executor at the command-line.
       */
      public interface TextToBigQueryStreamingOptions
          extends TextIOToBigQuery.Options, BigQueryStorageApiStreamingOptions {
        @TemplateParameter.BigQueryTable(
            order = 1,
            optional = true,
            description = "The dead-letter table name to output failed messages to BigQuery",
            helpText =
                "Table for messages that failed to reach the output table. If a table doesn't exist, it is created during "
                    + "pipeline execution. If not specified, `<outputTableSpec>_error_records` is used.",
            example = "<PROJECT_ID>:<DATASET_NAME>.<TABLE_NAME>")
        String getOutputDeadletterTable();

        void setOutputDeadletterTable(String value);

        // Hide the UseStorageWriteApiAtLeastOnce in the UI, because it will automatically be turned
        // on when pipeline is running on ALO mode and using the Storage Write API
        @TemplateParameter.Boolean(
            order = 2,
            optional = true,
            parentName = "useStorageWriteApi",
            parentTriggerValues = {"true"},
            description = "Use at at-least-once semantics in BigQuery Storage Write API",
            helpText =
                "This parameter takes effect only if `Use BigQuery Storage Write API` is enabled. If"
                    + " enabled the at-least-once semantics will be used for Storage Write API, otherwise"
                    + " exactly-once semantics will be used.",
            hiddenUi = true)
        @Default.Boolean(false)
        @Override
        Boolean getUseStorageWriteApiAtLeastOnce();

        void setUseStorageWriteApiAtLeastOnce(Boolean value);
      }
    }

## What's next

- Learn about [Dataflow templates](https://docs.cloud.google.com/dataflow/docs/concepts/dataflow-templates).
- See the list of [Google-provided templates](https://docs.cloud.google.com/dataflow/docs/guides/templates/provided-templates).

<br />

<br />