This page explains how to use the vSphere Container Storage Interface (CSI) storage
driver with Google Distributed Cloud clusters.

## Overview

Starting with Google Distributed Cloud version 1.5, the [vSphere CSI driver](https://vsphere-csi-driver.sigs.k8s.io/) is automatically
deployed in Google Distributed Cloud clusters. For information
about other CSI drivers, see [Installing a CSI driver](https://docs.cloud.google.com/anthos/clusters/docs/on-prem/1.7/how-to/install-csi-driver).

## Before you begin

Make sure you have the following prerequisites:

- vCenter and ESXi host both at version 6.7 update 3 or newer.

- vSphere `cns.searchable` [user account privileges](https://docs.cloud.google.com/anthos/clusters/docs/on-prem/1.7/how-to/vsphere-requirements-basic#vcenter-user-account-privileges).

If you are unsure whether your cluster meets the CSI prerequisites, run
`gkectl diagnose cluster` before deploying any CSI workloads.

Note that Google Distributed Cloud, does not support the [vSphere CSI volume expansion](https://vsphere-csi-driver.sigs.k8s.io/features/volume_expansion.html) feature.

## Confirm driver availability

When the vSphere CSI driver is installed in your user cluster, a StorageClass
called `standard-rwo` is created. To check that the driver is available
on all nodes, run the following command:

```
kubectl get csinode -o="custom-columns=NAME:metadata.name,DRIVERS:spec.drivers[].name"
```

You should see the following output:

```
NAME     DRIVERS
node-0   csi.vsphere.vmware.com
node-1   csi.vsphere.vmware.com
node-2   csi.vsphere.vmware.com
```

## Provisioning volumes

To provision volumes with the vSphere CSI driver, set the `storageClassName` field
of your [PersistentVolumeClaim](https://kubernetes.io/docs/reference/kubernetes-api/config-and-storage-resources/persistent-volume-claim-v1/) to `standard-rwo`.

## Setting the default storage class

To set `standard-rwo` as the default storage class, see [Default storage class](https://docs.cloud.google.com/anthos/clusters/docs/on-prem/1.7/how-to/default-storage-class).

## Creating additional StorageClasses

You can create additional vSphere CSI StorageClasses in your cluster using the
`datastoreurl` or `storagepolicyname` parameters.

The following example uses the `storagepolicyname` parameter:

```
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: gold
parameters:
  csi.storage.k8s.io/fstype: ext4
  storagepolicyname: "Gold Policy"
provisioner: csi.vsphere.vmware.com
```

To find the correct value for the `datastoreurl` parameter, run the following command:

```
govc datastore.info DATASTORE_NAME
```

where <var translate="no">DATASTORE_NAME</var> is the name of your [vSphere datastore](https://docs.cloud.google.com/anthos/clusters/docs/on-prem/1.7/how-to/admin-cluster-configuration-file#vcenter-datastore-field).

Use the value for <var translate="no">URL</var> from the output for `datastoreurl`. An example
output looks like the following:

```
Name:        DATASTORE_NAME
  Path:      /DATACENTER_NAME/datastore/DATASTORE_NAME
  Type:      VMFS
  URL:       ds:///vmfs/volumes/5d864fa0-7f2184d4-8eb4-3cfdfe0ce9a0/
  Capacity:  20479.8 GB
  Free:      10066.9 GB
```

## Volume cleanup

Volumes provisioned by the vSphere CSI driver are not deleted when you delete a
user cluster. You should delete all PVCs and
[StatefulSets](https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/) before
deleting the cluster. To delete your volumes, run the following command:

```
kubectl delete statefulsets,persistentvolumeclaims --namespace=MY_NAMESPACE --all
```

where <var translate="no">MY_NAMESPACE</var> is your Kubernetes [namespace](https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/).

If you deleted a user cluster without deleting its volumes, you can find the volumes in vCenter:

1. In vCenter, select a datastore.

2. Navigate to **Monitor** \> **Cloud Native Storage** \> **Container Volumes**.

The Kubernetes cluster name is shown in the **Details** section of each volume.

## Disabling vSphere CSI driver

To remove vSphere CSI support, add the following to your user cluster configuration:

```
# (Optional) Storage specification for the cluster
storage:
# To disable the vSphere CSI driver, set this to true. The value is set to false by default and # the vSphere CSI driver is enabled by default on vSphere version >= 6.7U3.
  vSphereCSIDisabled: true
```

## What's next

- [Read more about Google Distributed Cloud storage concepts](https://docs.cloud.google.com/anthos/clusters/docs/on-prem/1.7/concepts/storage)
- [Set a default StorageClass for your cluster](https://docs.cloud.google.com/anthos/clusters/docs/on-prem/1.7/how-to/default-storage-class)