# Hive to Cloud Storage template

Use the Managed Service for Apache Spark Hive to Cloud Storage template to extract
data from Hive to Cloud Storage.

## Use the template

Run the template using the gcloud CLI or Managed Service for Apache Spark
API.

### gcloud


Before using any of the command data below,
make the following replacements:

- <var class="edit" scope="PROJECT_ID" translate="no">PROJECT_ID</var>: Required. Your Google Cloud project ID listed in the [IAM Settings](https://console.cloud.google.com/iam-admin/settings/project).
- <var class="edit" scope="REGION" translate="no">REGION</var>: Required. [Compute Engine
  region](https://docs.cloud.google.com/compute/docs/regions-zones#available).
- <var class="edit" scope="TEMPLATE_VERSION" translate="no">TEMPLATE_VERSION</var>: Required. Specify `latest` for the latest template version, or the date of a specific version, for example, `2023-03-17_v0.1.0-beta` (visit [gs://dataproc-templates-binaries](https://console.cloud.google.com/storage/browser/dataproc-templates-binaries) or run `gcloud storage ls gs://dataproc-templates-binaries` to list available template versions).
- <var class="edit" scope="SUBNET" translate="no">SUBNET</var>: Optional. If a subnet is not specified, the subnet in the specified REGION in the `default` network is selected.

  **Example:** `projects/PROJECT_ID/regions/REGION/subnetworks/SUBNET_NAME`
- <var class="edit" scope="HOST" translate="no">HOST</var> and <var class="edit" scope="PORT" translate="no">PORT</var>: Required. Hostname or IP address and port of the source Hive database host.

  **Example:** `10.0.0.33`
- <var class="edit" scope="TABLE" translate="no">TABLE</var>: Required. Hive input table name.
- <var class="edit" scope="DATABASE" translate="no">DATABASE</var>: Required. Hive input database name.
- <var class="edit" scope="CLOUD_STORAGE_OUTPUT_PATH" translate="no">CLOUD_STORAGE_OUTPUT_PATH</var>: Required. Cloud Storage path where output will be stored.

  **Example:** `gs://templates/hive_to_cloud_storage_output`
- <var class="edit" scope="FORMAT" translate="no">FORMAT</var>: Optional. Output data format. Options: `avro`, `parquet`, `csv`, or `json`. Default: `avro`. **Note:** If `avro`, you must add `file:///usr/lib/spark/connector/spark-avro.jar` to the `jars` gcloud CLI flag or API field.

  **Example (the `file://` prefix references a Managed Service for Apache Spark jar file):**
  `--jars=file:///usr/lib/spark/connector/spark-avro.jar,` \[, ... other jars\]
- <var class="edit" scope="HIVE_PARTITION_COLUMN" translate="no">HIVE_PARTITION_COLUMN</var>: Optional. Column to partition Hive data.
- <var class="edit" scope="MODE" translate="no">MODE</var>: Required. Write mode for Cloud Storage output. Options: `append`, `overwrite`, `ignore`, or `errorifexists`.
- <var class="edit" scope="SERVICE_ACCOUNT" translate="no">SERVICE_ACCOUNT</var>: Optional. If not provided, the [default Compute Engine service account](https://docs.cloud.google.com/compute/docs/access/service-accounts.md.txt#default_service_account) is used.
- <var class="edit" scope="PROPERTY_NAME" translate="no">PROPERTY</var> and <var class="edit" scope="PROPERTY_VALUE" translate="no">PROPERTY_VALUE</var>: Optional. Comma-separated list of [Spark property](https://docs.cloud.google.com/managed-spark/docs/concepts/properties)=`value` pairs.
- <var class="edit" scope="LABEL" translate="no">LABEL</var> and <var class="edit" scope="LABEL_VALUE" translate="no">LABEL_VALUE</var>: Optional. Comma-separated list of `label`=`value` pairs.
- <var class="edit" scope="LOG_LEVEL" translate="no">LOG_LEVEL</var>: Optional. Level of logging. Can be one of `ALL`, `DEBUG`, `ERROR`, `FATAL`, `INFO`, `OFF`, `TRACE`, or `WARN`. Default: `INFO`.
- <var class="edit" scope="KMS_KEY" translate="no">KMS_KEY</var>: Optional. The Cloud Key Management Service key to use for encryption. If a key is not specified, data is [encrypted at rest](https://docs.cloud.google.com/docs/security/encryption/default-encryption.md.txt) using a Google-owned and Google-managed encryption key.

  **Example:** `projects/PROJECT_ID/regions/REGION/keyRings/KEY_RING_NAME/cryptoKeys/KEY_NAME`


Execute the

following

command:

#### Linux, macOS, or Cloud Shell

> [!NOTE]
> **Note:** Ensure you have initialized the Google Cloud CLI with authentication and a project by running either [gcloud init](https://docs.cloud.google.com/sdk/gcloud/reference/init); or [gcloud auth login](https://docs.cloud.google.com/sdk/gcloud/reference/auth/login.md.txt) and [gcloud config set project](https://docs.cloud.google.com/sdk/gcloud/reference/config/set.md.txt).

```bash
gcloud dataproc batches submit spark \
    --class=com.google.cloud.dataproc.templates.main.DataProcTemplate \
    --version="2.3" \
    --project="PROJECT_ID" \
    --region="REGION" \
    --jars="gs://dataproc-templates-binaries/TEMPLATE_VERSION/java/dataproc-templates.jar" \
    --subnet="SUBNET" \
    --kms-key="KMS_KEY" \
    --service-account="SERVICE_ACCOUNT" \
    --properties="spark.hadoop.hive.metastore.uris=thrift://HOST:PORT,PROPERTY=PROPERTY_VALUE" \
    --labels="LABEL=LABEL_VALUE" \
    -- --template=HIVETOGCS \
    --templateProperty log.level="LOG_LEVEL" \
    --templateProperty hive.input.table="TABLE" \
    --templateProperty hive.input.db="DATABASE" \
    --templateProperty hive.gcs.output.path="CLOUD_STORAGE_OUTPUT_PATH" \
    --templateProperty hive.gcs.output.format="FORMAT" \
    --templateProperty hive.partition.col="HIVE_PARTITION_COLUMN" \
    --templateProperty hive.gcs.save.mode="MODE"
```

#### Windows (PowerShell)

> [!NOTE]
> **Note:** Ensure you have initialized the Google Cloud CLI with authentication and a project by running either [gcloud init](https://docs.cloud.google.com/sdk/gcloud/reference/init); or [gcloud auth login](https://docs.cloud.google.com/sdk/gcloud/reference/auth/login.md.txt) and [gcloud config set project](https://docs.cloud.google.com/sdk/gcloud/reference/config/set.md.txt).

```bash
gcloud dataproc batches submit spark `
    --class=com.google.cloud.dataproc.templates.main.DataProcTemplate `
    --version="2.3" `
    --project="PROJECT_ID" `
    --region="REGION" `
    --jars="gs://dataproc-templates-binaries/TEMPLATE_VERSION/java/dataproc-templates.jar" `
    --subnet="SUBNET" `
    --kms-key="KMS_KEY" `
    --service-account="SERVICE_ACCOUNT" `
    --properties="spark.hadoop.hive.metastore.uris=thrift://HOST:PORT,PROPERTY=PROPERTY_VALUE" `
    --labels="LABEL=LABEL_VALUE" `
    -- --template=HIVETOGCS `
    --templateProperty log.level="LOG_LEVEL" `
    --templateProperty hive.input.table="TABLE" `
    --templateProperty hive.input.db="DATABASE" `
    --templateProperty hive.gcs.output.path="CLOUD_STORAGE_OUTPUT_PATH" `
    --templateProperty hive.gcs.output.format="FORMAT" `
    --templateProperty hive.partition.col="HIVE_PARTITION_COLUMN" `
    --templateProperty hive.gcs.save.mode="MODE"
```

#### Windows (cmd.exe)

> [!NOTE]
> **Note:** Ensure you have initialized the Google Cloud CLI with authentication and a project by running either [gcloud init](https://docs.cloud.google.com/sdk/gcloud/reference/init); or [gcloud auth login](https://docs.cloud.google.com/sdk/gcloud/reference/auth/login.md.txt) and [gcloud config set project](https://docs.cloud.google.com/sdk/gcloud/reference/config/set.md.txt).

```bash
gcloud dataproc batches submit spark ^
    --class=com.google.cloud.dataproc.templates.main.DataProcTemplate ^
    --version="2.3" ^
    --project="PROJECT_ID" ^
    --region="REGION" ^
    --jars="gs://dataproc-templates-binaries/TEMPLATE_VERSION/java/dataproc-templates.jar" ^
    --subnet="SUBNET" ^
    --kms-key="KMS_KEY" ^
    --service-account="SERVICE_ACCOUNT" ^
    --properties="spark.hadoop.hive.metastore.uris=thrift://HOST:PORT,PROPERTY=PROPERTY_VALUE" ^
    --labels="LABEL=LABEL_VALUE" ^
    -- --template=HIVETOGCS ^
    --templateProperty log.level="LOG_LEVEL" ^
    --templateProperty hive.input.table="TABLE" ^
    --templateProperty hive.input.db="DATABASE" ^
    --templateProperty hive.gcs.output.path="CLOUD_STORAGE_OUTPUT_PATH" ^
    --templateProperty hive.gcs.output.format="FORMAT" ^
    --templateProperty hive.partition.col="HIVE_PARTITION_COLUMN" ^
    --templateProperty hive.gcs.save.mode="MODE"
```

### REST


Before using any of the request data,
make the following replacements:

- <var class="edit" scope="PROJECT_ID" translate="no">PROJECT_ID</var>: Required. Your Google Cloud project ID listed in the [IAM Settings](https://console.cloud.google.com/iam-admin/settings/project).
- <var class="edit" scope="REGION" translate="no">REGION</var>: Required. [Compute Engine
  region](https://docs.cloud.google.com/compute/docs/regions-zones#available).
- <var class="edit" scope="TEMPLATE_VERSION" translate="no">TEMPLATE_VERSION</var>: Required. Specify `latest` for the latest template version, or the date of a specific version, for example, `2023-03-17_v0.1.0-beta` (visit [gs://dataproc-templates-binaries](https://console.cloud.google.com/storage/browser/dataproc-templates-binaries) or run `gcloud storage ls gs://dataproc-templates-binaries` to list available template versions).
- <var class="edit" scope="SUBNET" translate="no">SUBNET</var>: Optional. If a subnet is not specified, the subnet in the specified REGION in the `default` network is selected.

  **Example:** `projects/PROJECT_ID/regions/REGION/subnetworks/SUBNET_NAME`
- <var class="edit" scope="HOST" translate="no">HOST</var> and <var class="edit" scope="PORT" translate="no">PORT</var>: Required. Hostname or IP address and port of the source Hive database host.

  **Example:** `10.0.0.33`
- <var class="edit" scope="TABLE" translate="no">TABLE</var>: Required. Hive input table name.
- <var class="edit" scope="DATABASE" translate="no">DATABASE</var>: Required. Hive input database name.
- <var class="edit" scope="CLOUD_STORAGE_OUTPUT_PATH" translate="no">CLOUD_STORAGE_OUTPUT_PATH</var>: Required. Cloud Storage path where output will be stored.

  **Example:** `gs://templates/hive_to_cloud_storage_output`
- <var class="edit" scope="FORMAT" translate="no">FORMAT</var>: Optional. Output data format. Options: `avro`, `parquet`, `csv`, or `json`. Default: `avro`. **Note:** If `avro`, you must add `file:///usr/lib/spark/connector/spark-avro.jar` to the `jars` gcloud CLI flag or API field.

  **Example (the `file://` prefix references a Managed Service for Apache Spark jar file):**
  `--jars=file:///usr/lib/spark/connector/spark-avro.jar,` \[, ... other jars\]
- <var class="edit" scope="HIVE_PARTITION_COLUMN" translate="no">HIVE_PARTITION_COLUMN</var>: Optional. Column to partition Hive data.
- <var class="edit" scope="MODE" translate="no">MODE</var>: Required. Write mode for Cloud Storage output. Options: `append`, `overwrite`, `ignore`, or `errorifexists`.
- <var class="edit" scope="SERVICE_ACCOUNT" translate="no">SERVICE_ACCOUNT</var>: Optional. If not provided, the [default Compute Engine service account](https://docs.cloud.google.com/compute/docs/access/service-accounts.md.txt#default_service_account) is used.
- <var class="edit" scope="PROPERTY_NAME" translate="no">PROPERTY</var> and <var class="edit" scope="PROPERTY_VALUE" translate="no">PROPERTY_VALUE</var>: Optional. Comma-separated list of [Spark property](https://docs.cloud.google.com/managed-spark/docs/concepts/properties)=`value` pairs.
- <var class="edit" scope="LABEL" translate="no">LABEL</var> and <var class="edit" scope="LABEL_VALUE" translate="no">LABEL_VALUE</var>: Optional. Comma-separated list of `label`=`value` pairs.
- <var class="edit" scope="LOG_LEVEL" translate="no">LOG_LEVEL</var>: Optional. Level of logging. Can be one of `ALL`, `DEBUG`, `ERROR`, `FATAL`, `INFO`, `OFF`, `TRACE`, or `WARN`. Default: `INFO`.
- <var class="edit" scope="KMS_KEY" translate="no">KMS_KEY</var>: Optional. The Cloud Key Management Service key to use for encryption. If a key is not specified, data is [encrypted at rest](https://docs.cloud.google.com/docs/security/encryption/default-encryption.md.txt) using a Google-owned and Google-managed encryption key.

  **Example:** `projects/PROJECT_ID/regions/REGION/keyRings/KEY_RING_NAME/cryptoKeys/KEY_NAME`


HTTP method and URL:

```
POST https://dataproc.googleapis.com/v1/projects/PROJECT_ID/locations/REGION/batches
```


Request JSON body:

```
{
  "environmentConfig":{
    "executionConfig":{
      "subnetworkUri":"SUBNET",
      "kmsKey": "KMS_KEY",
      "serviceAccount": "SERVICE_ACCOUNT"
    }
  },
  "labels": {
    "LABEL": "LABEL_VALUE"
  },
  "runtimeConfig": {
    "version": "2.3",
    "properties": {
      "spark.hadoop.hive.metastore.uris":"thrift://HOST:PORT",
      "PROPERTY": "PROPERTY_VALUE"
    }
  },
  "sparkBatch":{
    "mainClass":"com.google.cloud.dataproc.templates.main.DataProcTemplate",
    "args":[
      "--template","HIVETOGCS",
      "--templateProperty","log.level=LOG_LEVEL",
      "--templateProperty","hive.input.table=TABLE",
      "--templateProperty","hive.input.db=DATABASE",
      "--templateProperty","hive.gcs.output.path=CLOUD_STORAGE_OUTPUT_PATH",
      "--templateProperty","hive.gcs.output.format=FORMAT",
      "--templateProperty","hive.partition.col=HIVE_PARTITION_COLUMN",
      "--templateProperty","hive.gcs.save.mode=MODE"
    ],
    "jarFileUris":[
      "file:///usr/lib/spark/connector/spark-avro.jar",
      "gs://dataproc-templates-binaries/TEMPLATE_VERSION/java/dataproc-templates.jar"
    ]
  }
}
```

To send your request, expand one of these options:

#### curl (Linux, macOS, or Cloud Shell)

> [!NOTE]
> **Note:** The following command assumes that you have logged in to the `gcloud` CLI with your user account by running [`gcloud init`](https://docs.cloud.google.com/sdk/gcloud/reference/init) or [`gcloud auth login`](https://docs.cloud.google.com/sdk/gcloud/reference/auth/login.md.txt) , or by using [Cloud Shell](https://docs.cloud.google.com/shell/docs.md.txt), which automatically logs you into the `gcloud` CLI . You can check the currently active account by running [`gcloud auth list`](https://docs.cloud.google.com/sdk/gcloud/reference/auth/list.md.txt).


Save the request body in a file named `request.json`,
and execute the following command:

```
curl -X POST \
     -H "Authorization: Bearer $(gcloud auth print-access-token)" \
     -H "Content-Type: application/json; charset=utf-8" \
     -d @request.json \
     "https://dataproc.googleapis.com/v1/projects/PROJECT_ID/locations/REGION/batches"
```

#### PowerShell (Windows)

> [!NOTE]
> **Note:** The following command assumes that you have logged in to the `gcloud` CLI with your user account by running [`gcloud init`](https://docs.cloud.google.com/sdk/gcloud/reference/init) or [`gcloud auth login`](https://docs.cloud.google.com/sdk/gcloud/reference/auth/login.md.txt) . You can check the currently active account by running [`gcloud auth list`](https://docs.cloud.google.com/sdk/gcloud/reference/auth/list.md.txt).


Save the request body in a file named `request.json`,
and execute the following command:

```
$cred = gcloud auth print-access-token
$headers = @{ "Authorization" = "Bearer $cred" }

Invoke-WebRequest `
    -Method POST `
    -Headers $headers `
    -ContentType: "application/json; charset=utf-8" `
    -InFile request.json `
    -Uri "https://dataproc.googleapis.com/v1/projects/PROJECT_ID/locations/REGION/batches" | Select-Object -Expand Content
```

You should receive a JSON response similar to the following:

```
{
  "name": "projects/PROJECT_ID/regions/REGION/operations/OPERATION_ID",
  "metadata": {
    "@type": "type.googleapis.com/google.cloud.dataproc.v1.BatchOperationMetadata",
    "batch": "projects/PROJECT_ID/locations/REGION/batches/BATCH_ID",
    "batchUuid": "de8af8d4-3599-4a7c-915c-798201ed1583",
    "createTime": "2023-02-24T03:31:03.440329Z",
    "operationType": "BATCH",
    "description": "Batch"
  }
}
```

<br />