# Cloud Pub/Sub to Cloud Storage template

Use the Managed Service for Apache Spark Cloud Pub/Sub to Cloud Storage template to extract
data from Pub/Sub to Cloud Storage.

## Use the template

Run the template using the gcloud CLI or Managed Service for Apache Spark
API.

### gcloud


Before using any of the command data below,
make the following replacements:

- <var class="edit" scope="PROJECT_ID" translate="no">PROJECT_ID</var>: Required. Your Google Cloud project ID listed in the [IAM Settings](https://console.cloud.google.com/iam-admin/settings/project).
- <var class="edit" scope="REGION" translate="no">REGION</var>: Required. [Compute Engine
  region](https://docs.cloud.google.com/compute/docs/regions-zones#available).
- <var class="edit" scope="SUBNET" translate="no">SUBNET</var>: Optional. If a subnet is not specified, the subnet in the specified REGION in the `default` network is selected.

  **Example:** `projects/PROJECT_ID/regions/REGION/subnetworks/SUBNET_NAME`
- <var class="edit" scope="TEMPLATE_VERSION" translate="no">TEMPLATE_VERSION</var>: Required. Specify `latest` for the latest template version, or the date of a specific version, for example, `2023-03-17_v0.1.0-beta` (visit [gs://dataproc-templates-binaries](https://console.cloud.google.com/storage/browser/dataproc-templates-binaries) or run `gcloud storage ls gs://dataproc-templates-binaries` to list available template versions).
- <var class="edit" scope="PUBSUB_SUBSCRIPTION_PROJECT_ID" translate="no">PUBSUB_SUBSCRIPTION_PROJECT_ID</var>: Required. The Google Cloud project ID listed in the [IAM Settings](https://console.cloud.google.com/iam-admin/settings/project) that contains the input Pub/Sub subscription to be read.
- <var class="edit" scope="SUBSCRIPTION" translate="no">SUBSCRIPTION</var>: Required. Pub/Sub subscription name.
- <var class="edit" scope="CLOUD_STORAGE_OUTPUT_BUCKET_NAME" translate="no">CLOUD_STORAGE_OUTPUT_BUCKET_NAME</var>: Required. Cloud Storage bucket name where output will be stored.

  **Note:** The output files will be stored in the `output/` folder inside the bucket.
- <var class="edit" scope="FORMAT" translate="no">FORMAT</var>: Required. Output data format. Options: `avro` or `json`.

  **Note:** If `avro`, you must add "`file:///usr/lib/spark/connector/spark-avro.jar`"
  to the `jars` gcloud CLI flag or API field.

  **Example (the `file://` prefix references a Managed Service for Apache Spark jar file):**
  `--jars=file:///usr/lib/spark/connector/spark-avro.jar,` \[ ... other jars\]
- <var class="edit" scope="TIMEOUT" translate="no">TIMEOUT</var>: Optional. Time in milliseconds before termination of stream. Defaults to **60000**.
- <var class="edit" scope="DURATION" translate="no">DURATION</var>: Optional. Frequency in seconds of writes to Cloud Storage. Defaults to **15 seconds**.
- <var class="edit" scope="NUM_RECEIVERS" translate="no">NUM_RECEIVERS</var>: Optional. Number of streams read from a Pub/Sub subscription in parallel. Defaults to **5**.
- <var class="edit" scope="BATCHSIZE" translate="no">BATCHSIZE</var>: Optional. Number of records to insert in one round trip into Cloud Storage. Defaults to **1000**.
- <var class="edit" scope="SERVICE_ACCOUNT" translate="no">SERVICE_ACCOUNT</var>: Optional. If not provided, the [default Compute Engine service account](https://docs.cloud.google.com/compute/docs/access/service-accounts.md.txt#default_service_account) is used.
- <var class="edit" scope="PROPERTY_NAME" translate="no">PROPERTY</var> and <var class="edit" scope="PROPERTY_VALUE" translate="no">PROPERTY_VALUE</var>: Optional. Comma-separated list of [Spark property](https://docs.cloud.google.com/managed-spark/docs/concepts/properties)=`value` pairs.
- <var class="edit" scope="LABEL" translate="no">LABEL</var> and <var class="edit" scope="LABEL_VALUE" translate="no">LABEL_VALUE</var>: Optional. Comma-separated list of `label`=`value` pairs.
- <var class="edit" scope="LOG_LEVEL" translate="no">LOG_LEVEL</var>: Optional. Level of logging. Can be one of `ALL`, `DEBUG`, `ERROR`, `FATAL`, `INFO`, `OFF`, `TRACE`, or `WARN`. Default: `INFO`.
- <var class="edit" scope="KMS_KEY" translate="no">KMS_KEY</var>: Optional. The Cloud Key Management Service key to use for encryption. If a key is not specified, data is [encrypted at rest](https://docs.cloud.google.com/docs/security/encryption/default-encryption.md.txt) using a Google-owned and Google-managed encryption key.

  **Example:** `projects/PROJECT_ID/regions/REGION/keyRings/KEY_RING_NAME/cryptoKeys/KEY_NAME`


Execute the

following

command:

#### Linux, macOS, or Cloud Shell

> [!NOTE]
> **Note:** Ensure you have initialized the Google Cloud CLI with authentication and a project by running either [gcloud init](https://docs.cloud.google.com/sdk/gcloud/reference/init); or [gcloud auth login](https://docs.cloud.google.com/sdk/gcloud/reference/auth/login.md.txt) and [gcloud config set project](https://docs.cloud.google.com/sdk/gcloud/reference/config/set.md.txt).

```bash
gcloud dataproc batches submit spark \
    --class=com.google.cloud.dataproc.templates.main.DataProcTemplate \
    --version="2.3" \
    --project="PROJECT_ID" \
    --region="REGION" \
    --jars="gs://dataproc-templates-binaries/TEMPLATE_VERSION/java/dataproc-templates.jar" \
    --subnet="SUBNET" \
    --kms-key="KMS_KEY" \
    --service-account="SERVICE_ACCOUNT" \
    --properties="PROPERTY=PROPERTY_VALUE" \
    --labels="LABEL=LABEL_VALUE" \
    -- --template=PUBSUBTOGCS \
    --templateProperty log.level="LOG_LEVEL" \
    --templateProperty pubsubtogcs.input.project.id="PUBSUB_SUBSCRIPTION_PROJECT_ID" \
    --templateProperty pubsubtogcs.input.subscription="SUBSCRIPTION" \
    --templateProperty pubsubtogcs.gcs.bucket.name="CLOUD_STORAGE_OUTPUT_BUCKET_NAME" \
    --templateProperty pubsubtogcs.gcs.output.data.format="FORMAT" \
    --templateProperty pubsubtogcs.timeout.ms="TIMEOUT" \
    --templateProperty pubsubtogcs.streaming.duration.seconds="DURATION" \
    --templateProperty pubsubtogcs.total.receivers="NUM_RECEIVERS" \
    --templateProperty pubsubtogcs.batch.size="BATCHSIZE"
```

#### Windows (PowerShell)

> [!NOTE]
> **Note:** Ensure you have initialized the Google Cloud CLI with authentication and a project by running either [gcloud init](https://docs.cloud.google.com/sdk/gcloud/reference/init); or [gcloud auth login](https://docs.cloud.google.com/sdk/gcloud/reference/auth/login.md.txt) and [gcloud config set project](https://docs.cloud.google.com/sdk/gcloud/reference/config/set.md.txt).

```bash
gcloud dataproc batches submit spark `
    --class=com.google.cloud.dataproc.templates.main.DataProcTemplate `
    --version="2.3" `
    --project="PROJECT_ID" `
    --region="REGION" `
    --jars="gs://dataproc-templates-binaries/TEMPLATE_VERSION/java/dataproc-templates.jar" `
    --subnet="SUBNET" `
    --kms-key="KMS_KEY" `
    --service-account="SERVICE_ACCOUNT" `
    --properties="PROPERTY=PROPERTY_VALUE" `
    --labels="LABEL=LABEL_VALUE" `
    -- --template=PUBSUBTOGCS `
    --templateProperty log.level="LOG_LEVEL" `
    --templateProperty pubsubtogcs.input.project.id="PUBSUB_SUBSCRIPTION_PROJECT_ID" `
    --templateProperty pubsubtogcs.input.subscription="SUBSCRIPTION" `
    --templateProperty pubsubtogcs.gcs.bucket.name="CLOUD_STORAGE_OUTPUT_BUCKET_NAME" `
    --templateProperty pubsubtogcs.gcs.output.data.format="FORMAT" `
    --templateProperty pubsubtogcs.timeout.ms="TIMEOUT" `
    --templateProperty pubsubtogcs.streaming.duration.seconds="DURATION" `
    --templateProperty pubsubtogcs.total.receivers="NUM_RECEIVERS" `
    --templateProperty pubsubtogcs.batch.size="BATCHSIZE"
```

#### Windows (cmd.exe)

> [!NOTE]
> **Note:** Ensure you have initialized the Google Cloud CLI with authentication and a project by running either [gcloud init](https://docs.cloud.google.com/sdk/gcloud/reference/init); or [gcloud auth login](https://docs.cloud.google.com/sdk/gcloud/reference/auth/login.md.txt) and [gcloud config set project](https://docs.cloud.google.com/sdk/gcloud/reference/config/set.md.txt).

```bash
gcloud dataproc batches submit spark ^
    --class=com.google.cloud.dataproc.templates.main.DataProcTemplate ^
    --version="2.3" ^
    --project="PROJECT_ID" ^
    --region="REGION" ^
    --jars="gs://dataproc-templates-binaries/TEMPLATE_VERSION/java/dataproc-templates.jar" ^
    --subnet="SUBNET" ^
    --kms-key="KMS_KEY" ^
    --service-account="SERVICE_ACCOUNT" ^
    --properties="PROPERTY=PROPERTY_VALUE" ^
    --labels="LABEL=LABEL_VALUE" ^
    -- --template=PUBSUBTOGCS ^
    --templateProperty log.level="LOG_LEVEL" ^
    --templateProperty pubsubtogcs.input.project.id="PUBSUB_SUBSCRIPTION_PROJECT_ID" ^
    --templateProperty pubsubtogcs.input.subscription="SUBSCRIPTION" ^
    --templateProperty pubsubtogcs.gcs.bucket.name="CLOUD_STORAGE_OUTPUT_BUCKET_NAME" ^
    --templateProperty pubsubtogcs.gcs.output.data.format="FORMAT" ^
    --templateProperty pubsubtogcs.timeout.ms="TIMEOUT" ^
    --templateProperty pubsubtogcs.streaming.duration.seconds="DURATION" ^
    --templateProperty pubsubtogcs.total.receivers="NUM_RECEIVERS" ^
    --templateProperty pubsubtogcs.batch.size="BATCHSIZE"
```

### REST


Before using any of the request data,
make the following replacements:

- <var class="edit" scope="PROJECT_ID" translate="no">PROJECT_ID</var>: Required. Your Google Cloud project ID listed in the [IAM Settings](https://console.cloud.google.com/iam-admin/settings/project).
- <var class="edit" scope="REGION" translate="no">REGION</var>: Required. [Compute Engine
  region](https://docs.cloud.google.com/compute/docs/regions-zones#available).
- <var class="edit" scope="SUBNET" translate="no">SUBNET</var>: Optional. If a subnet is not specified, the subnet in the specified REGION in the `default` network is selected.

  **Example:** `projects/PROJECT_ID/regions/REGION/subnetworks/SUBNET_NAME`
- <var class="edit" scope="TEMPLATE_VERSION" translate="no">TEMPLATE_VERSION</var>: Required. Specify `latest` for the latest template version, or the date of a specific version, for example, `2023-03-17_v0.1.0-beta` (visit [gs://dataproc-templates-binaries](https://console.cloud.google.com/storage/browser/dataproc-templates-binaries) or run `gcloud storage ls gs://dataproc-templates-binaries` to list available template versions).
- <var class="edit" scope="PUBSUB_SUBSCRIPTION_PROJECT_ID" translate="no">PUBSUB_SUBSCRIPTION_PROJECT_ID</var>: Required. The Google Cloud project ID listed in the [IAM Settings](https://console.cloud.google.com/iam-admin/settings/project) that contains the input Pub/Sub subscription to be read.
- <var class="edit" scope="SUBSCRIPTION" translate="no">SUBSCRIPTION</var>: Required. Pub/Sub subscription name.
- <var class="edit" scope="CLOUD_STORAGE_OUTPUT_BUCKET_NAME" translate="no">CLOUD_STORAGE_OUTPUT_BUCKET_NAME</var>: Required. Cloud Storage bucket name where output will be stored.

  **Note:** The output files will be stored in the `output/` folder inside the bucket.
- <var class="edit" scope="FORMAT" translate="no">FORMAT</var>: Required. Output data format. Options: `avro` or `json`.

  **Note:** If `avro`, you must add "`file:///usr/lib/spark/connector/spark-avro.jar`"
  to the `jars` gcloud CLI flag or API field.

  **Example (the `file://` prefix references a Managed Service for Apache Spark jar file):**
  `--jars=file:///usr/lib/spark/connector/spark-avro.jar,` \[ ... other jars\]
- <var class="edit" scope="TIMEOUT" translate="no">TIMEOUT</var>: Optional. Time in milliseconds before termination of stream. Defaults to **60000**.
- <var class="edit" scope="DURATION" translate="no">DURATION</var>: Optional. Frequency in seconds of writes to Cloud Storage. Defaults to **15 seconds**.
- <var class="edit" scope="NUM_RECEIVERS" translate="no">NUM_RECEIVERS</var>: Optional. Number of streams read from a Pub/Sub subscription in parallel. Defaults to **5**.
- <var class="edit" scope="BATCHSIZE" translate="no">BATCHSIZE</var>: Optional. Number of records to insert in one round trip into Cloud Storage. Defaults to **1000**.
- <var class="edit" scope="SERVICE_ACCOUNT" translate="no">SERVICE_ACCOUNT</var>: Optional. If not provided, the [default Compute Engine service account](https://docs.cloud.google.com/compute/docs/access/service-accounts.md.txt#default_service_account) is used.
- <var class="edit" scope="PROPERTY_NAME" translate="no">PROPERTY</var> and <var class="edit" scope="PROPERTY_VALUE" translate="no">PROPERTY_VALUE</var>: Optional. Comma-separated list of [Spark property](https://docs.cloud.google.com/managed-spark/docs/concepts/properties)=`value` pairs.
- <var class="edit" scope="LABEL" translate="no">LABEL</var> and <var class="edit" scope="LABEL_VALUE" translate="no">LABEL_VALUE</var>: Optional. Comma-separated list of `label`=`value` pairs.
- <var class="edit" scope="LOG_LEVEL" translate="no">LOG_LEVEL</var>: Optional. Level of logging. Can be one of `ALL`, `DEBUG`, `ERROR`, `FATAL`, `INFO`, `OFF`, `TRACE`, or `WARN`. Default: `INFO`.
- <var class="edit" scope="KMS_KEY" translate="no">KMS_KEY</var>: Optional. The Cloud Key Management Service key to use for encryption. If a key is not specified, data is [encrypted at rest](https://docs.cloud.google.com/docs/security/encryption/default-encryption.md.txt) using a Google-owned and Google-managed encryption key.

  **Example:** `projects/PROJECT_ID/regions/REGION/keyRings/KEY_RING_NAME/cryptoKeys/KEY_NAME`


HTTP method and URL:

```
POST https://dataproc.googleapis.com/v1/projects/PROJECT_ID/locations/REGION/batches
```


Request JSON body:

```
{
  "environmentConfig":{
    "executionConfig":{
      "subnetworkUri":"SUBNET",
      "kmsKey": "KMS_KEY",
      "serviceAccount": "SERVICE_ACCOUNT"
    }
  },
  "labels": {
    "LABEL": "LABEL_VALUE"
  },
  "runtimeConfig": {
    "version": "2.3",
    "properties": {
      "PROPERTY": "PROPERTY_VALUE"
    }
  },
  "sparkBatch":{
    "mainClass":"com.google.cloud.dataproc.templates.main.DataProcTemplate",
    "args":[
      "--template","PUBSUBTOGCS",
      "--templateProperty","log.level=LOG_LEVEL",
      "--templateProperty","pubsubtogcs.input.project.id=PUBSUB_SUBSCRIPTION_PROJECT_ID",
      "--templateProperty","pubsubtogcs.input.subscription=SUBSCRIPTION",
      "--templateProperty","pubsubtogcs.gcs.bucket.name=CLOUD_STORAGE_OUTPUT_BUCKET_NAME",
      "--templateProperty","pubsubtogcs.gcs.output.data.format=FORMAT",
      "--templateProperty","pubsubtogcs.timeout.ms=TIMEOUT",
      "--templateProperty","pubsubtogcs.streaming.duration.seconds=DURATION",
      "--templateProperty","pubsubtogcs.total.receivers=NUM_RECEIVERS",
      "--templateProperty","pubsubtogcs.batch.size=BATCHSIZE"
    ],
    "jarFileUris":[
      "file:///usr/lib/spark/connector/spark-avro.jar", "gs://dataproc-templates-binaries/TEMPLATE_VERSION/java/dataproc-templates.jar"
    ]
  }
}
```

To send your request, expand one of these options:

#### curl (Linux, macOS, or Cloud Shell)

> [!NOTE]
> **Note:** The following command assumes that you have logged in to the `gcloud` CLI with your user account by running [`gcloud init`](https://docs.cloud.google.com/sdk/gcloud/reference/init) or [`gcloud auth login`](https://docs.cloud.google.com/sdk/gcloud/reference/auth/login.md.txt) , or by using [Cloud Shell](https://docs.cloud.google.com/shell/docs.md.txt), which automatically logs you into the `gcloud` CLI . You can check the currently active account by running [`gcloud auth list`](https://docs.cloud.google.com/sdk/gcloud/reference/auth/list.md.txt).


Save the request body in a file named `request.json`,
and execute the following command:

```
curl -X POST \
     -H "Authorization: Bearer $(gcloud auth print-access-token)" \
     -H "Content-Type: application/json; charset=utf-8" \
     -d @request.json \
     "https://dataproc.googleapis.com/v1/projects/PROJECT_ID/locations/REGION/batches"
```

#### PowerShell (Windows)

> [!NOTE]
> **Note:** The following command assumes that you have logged in to the `gcloud` CLI with your user account by running [`gcloud init`](https://docs.cloud.google.com/sdk/gcloud/reference/init) or [`gcloud auth login`](https://docs.cloud.google.com/sdk/gcloud/reference/auth/login.md.txt) . You can check the currently active account by running [`gcloud auth list`](https://docs.cloud.google.com/sdk/gcloud/reference/auth/list.md.txt).


Save the request body in a file named `request.json`,
and execute the following command:

```
$cred = gcloud auth print-access-token
$headers = @{ "Authorization" = "Bearer $cred" }

Invoke-WebRequest `
    -Method POST `
    -Headers $headers `
    -ContentType: "application/json; charset=utf-8" `
    -InFile request.json `
    -Uri "https://dataproc.googleapis.com/v1/projects/PROJECT_ID/locations/REGION/batches" | Select-Object -Expand Content
```

You should receive a JSON response similar to the following:

```
{
  "name": "projects/PROJECT_ID/regions/REGION/operations/OPERATION_ID",
  "metadata": {
    "@type": "type.googleapis.com/google.cloud.dataproc.v1.BatchOperationMetadata",
    "batch": "projects/PROJECT_ID/locations/REGION/batches/BATCH_ID",
    "batchUuid": "de8af8d4-3599-4a7c-915c-798201ed1583",
    "createTime": "2023-02-24T03:31:03.440329Z",
    "operationType": "BATCH",
    "description": "Batch"
  }
}
```

<br />