This document describes how to configure the
[Cloud Storage connector](https://docs.cloud.google.com/dataproc/docs/concepts/connectors/cloud-storage) on a cluster to perform folder operations
and explains compatibility considerations for earlier connector versions.

When you use the Cloud Storage connector on a cluster with a
hierarchical namespace-enabled bucket, you enable [atomic folder renames](https://docs.cloud.google.com/storage/docs/rename-hns-folders.md.txt) in
your cluster configuration, which improves performance and consistency when
handling large folders for Hadoop, Spark, and Hive workloads. In buckets without
hierarchical namespace, a rename operation involves multiple object copy and
delete jobs, which can impact performance and consistency.

## Before you begin

To use features of hierarchical namespace buckets, use the following
Cloud Storage connector versions:

- 2.2.23 or later (if you are using version 2.x.x)
- 3.0.1 or later (if you are using version 3.x.x)

Earlier connector versions (3.0.0 and earlier than 2.2.23) have limitations. For
more information about the limitations, see [Compatibility with
Cloud Storage connector version 3.0.0 or versions earlier than
2.2.23](https://docs.cloud.google.com/storage/docs/hns-hadoop-workloads#backwards-compatibility-connector).

## Enable the Cloud Storage connector on a cluster

This section describes how to enable the Cloud Storage connector on a
Managed Service for Apache Spark cluster and a self-managed Hadoop cluster.

### Managed Service for Apache Spark

You can use the Google Cloud CLI to create a Managed Service for Apache Spark cluster and
enable the Cloud Storage connector to perform the folder
operations.

1. Create a Managed Service for Apache Spark cluster using the following command:

   <br />

   ```
     gcloud dataproc clusters create CLUSTER_NAME
     --properties=core:fs.gs.hierarchical.namespace.folders.enable=true,
     core:fs.gs.http.read-timeout=30000
     
   ```

   <br />

   Where:
   - `CLUSTER_NAME` is the name of the cluster. For example, `my-cluster`
   - `fs.gs.hierarchical.namespace.folders.enable` is used to enable the hierarchical namespace on a bucket.
   - `fs.gs.http.read-timeout` is the maximum time allowed, in
     milliseconds, to read data from an established connection. This is
     an optional setting.

     > [!NOTE]
     > **Note:** If you are using the Cloud Storage connector version 3.0.0 or a version earlier than 2.2.23, the configuration setting `fs.gs.hierarchical.namespace.folders.enable` is not supported and results in an error if included.

### Self-managed Hadoop

You can enable the Cloud Storage connector on your self-managed
Hadoop cluster to perform the folder operations.

1. Add the following to core-site.xml configuration file:

   <br />

   ```
       <property>
         <name>fs.gs.hierarchical.namespace.folders.enable</name>
         <value>true</value>
       </property>
       <property>
         <name>fs.gs.http.read-timeout</name>
         <value>30000</value>
       </property>
     
   ```

   <br />

   Where:
   - `fs.gs.hierarchical.namespace.folders.enable` is used to enable the hierarchical namespace on a bucket
   - `fs.gs.http.read-timeout` is the maximum time allowed, in
     milliseconds, to read data from an established connection. This is
     an optional setting.

     > [!NOTE]
     > **Note:** If you are using the Cloud Storage connector version 3.0.0 or a version earlier than 2.2.23, the configuration setting `fs.gs.hierarchical.namespace.folders.enable` is not supported and results in an error if included.

## Compatibility with Cloud Storage connector version 3.0.0 or versions earlier than 2.2.23

Using the Cloud Storage connector version 3.0.0 or versions earlier
than 2.2.23 or disabling folder operations for hierarchical namespace can lead
to the following limitations:

- **Inefficient folder renames** : Folder rename operations in Hadoop happen
  using object-level copy and delete operations, which is slower and less
  efficient than the dedicated `rename folder` operation.

- **Accumulation of empty folders**: Folders are not deleted
  automatically, leading to the accumulation of empty folders in your bucket.
  Accumulation of empty folders can have the following impact:

  - Increase storage costs if not deleted explicitly.
  - Slow down the list operations and increase the risk of list operation
    timeouts.

    > [!NOTE]
    > **Note:** To reduce the risk of list operation timeouts, configure the `fs.gs.http.read-timeout` timeout value to `30000` milliseconds. To configure timeout settings, see the [instructions to enable the Cloud Storage connector](https://docs.cloud.google.com/storage/docs/hns-hadoop-workloads.md.txt#enable-connector-on-cluster).

- **Compatibility issues**: Mixing the usage of earlier and later connector
  versions, or enabling and disabling folder operations, can lead to
  compatibility issues, when renaming folders. Consider the following scenario
  which uses a combination of connector versions:

  1. Use the Cloud Storage connector version earlier than 2.2.23 to
     perform the following tasks:

     1. Write objects under the folder `foo/`.
     2. Rename the folder `foo/` to `bar/`. The rename operation copies and deletes the objects under `foo/` but does not delete the empty `foo/` folder.
  2. Use the Cloud Storage connector version 2.2.23 with the
     folder operations settings enabled to rename the folder `bar/` to `foo/`.

  The connector version 2.2.23, with the folder operation enabled,
  detects the existing `foo/` folder, causing the rename operation to
  fail. The earlier connector version, did not delete the `foo/` folder as
  the folder operation was disabled.

## What's next

- [Create buckets with hierarchical namespace enabled](https://docs.cloud.google.com/storage/docs/create-hns-bucket.md.txt).
- [Create and manage folders](https://docs.cloud.google.com/storage/docs/create-folders.md.txt).