This document explains how to use the vSphere Container Storage Interface (CSI)
driver with Google Distributed Cloud.
## Overview
The [vSphere CSI driver](https://vsphere-csi-driver.sigs.k8s.io/)
is deployed automatically when you install Google Distributed Cloud (software only) for
VMware. For information about using other CSI drivers, see
[Installing a CSI driver](https://docs.cloud.google.com/kubernetes-engine/distributed-cloud/vmware/docs/how-to/install-csi-driver.md.txt).
## vSphere requirements
To use the vSphere CSI driver, you must have:
- A Linux-based
[supported version of vCenter Server](https://docs.cloud.google.com/kubernetes-engine/distributed-cloud/vmware/docs/how-to/vsphere-requirements.md.txt)
- A [supported version of ESXI](https://docs.cloud.google.com/kubernetes-engine/distributed-cloud/vmware/docs/how-to/vsphere-requirements.md.txt)
### Volume expansion
- Offline [volume expansion](https://docs.cloud.google.com/kubernetes-engine/distributed-cloud/vmware/docs/concepts/storage.md.txt#volume_expansion)
requires both vCenter Server and ESXi to be version 7.0 Update 1 or later
- Online volume expansion requires both vCenter Server and ESXi to be version
7.0 Update 2 or later.
### Volume snapshot
Volume snapshot requires both vCenter Server and ESXi to be version 7.0 Update
3 or later.
### Cns.Searchable
Your vCenter user account must have the
[Cns.Searchable](https://techdocs.broadcom.com/us/en/vmware-cis/vsphere/vsphere/9-1/vsphere-security.html)
privilege on the root vCenter object.
If you are unsure whether your account meets the CSI prerequisites, run
`gkectl diagnose cluster` before deploying any CSI workloads.
## Verify that the driver is installed
When the vSphere CSI driver is installed in your user cluster, a StorageClass
called `standard-rwo` is created.
List the StorageClass objects:
```
kubectl --kubeconfig CLUSTER_KUBECONFIG get storageclass
```
Replace CLUSTER_KUBECONFIG with the path of your cluster kubeconfig
file.
Verify that `standard-rwo` appears in the output. For example:
```
standard kubernetes.io/vsphere-volume Delete Immediate false 33m
standard-rwo (default) csi.vsphere.vmware.com Delete Immediate true 32m
```
Verify that the vSphere CSI driver is installed:
```
kubectl --kubeconfig CLUSTER_KUBECONFIG get csinode \
-o="custom-columns=NAME:metadata.name,DRIVERS:spec.drivers[].name"
```
The output should indicate that the driver is available on all of your cluster
nodes. For example:
```
NAME DRIVERS
node-0 csi.vsphere.vmware.com
node-1 csi.vsphere.vmware.com
node-2 csi.vsphere.vmware.com
```
## Provisioning volumes
To provision volumes with the vSphere CSI driver, set the `storageClassName`
field of your
[PersistentVolumeClaim](https://kubernetes.io/docs/reference/kubernetes-api/config-and-storage-resources/persistent-volume-claim-v1/)
to `standard-rwo`.
## Setting the default storage class
To set `standard-rwo` as the default storage class, see
[Default storage class](https://docs.cloud.google.com/kubernetes-engine/distributed-cloud/vmware/docs/how-to/default-storage-class.md.txt).
## Using a snapshot to restore a volume
See
[Restore a volume from a snapshot](https://docs.cloud.google.com/kubernetes-engine/distributed-cloud/vmware/docs/how-to/vsphere-csi-restore-volume-snapshot.md.txt).
## Creating additional StorageClasses
You can create additional vSphere CSI StorageClasses in your cluster using the
`datastoreurl` or `storagepolicyname` parameters.
The following example uses the `storagepolicyname` parameter:
```
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: gold
parameters:
csi.storage.k8s.io/fstype: ext4
storagepolicyname: "Gold Policy"
provisioner: csi.vsphere.vmware.com
```
To find the correct value for the `datastoreurl` parameter, run the following
command:
```
govc datastore.info DATASTORE_NAME
```
Replace DATASTORE_NAME with the name of your
[vSphere datastore](https://docs.cloud.google.com/kubernetes-engine/distributed-cloud/vmware/docs/how-to/admin-cluster-configuration-file-latest.md.txt#vcenter-datastore-field).
Use the value for URL from the output for `datastoreurl`. An example
output looks like the following:
```
Name: DATASTORE_NAME
Path: /DATACENTER_NAME/datastore/DATASTORE_NAME
Type: VMFS
URL: ds:///vmfs/volumes/5d864fa0-7f2184d4-8eb4-3cfdfe0ce9a0/
Capacity: 20479.8 GB
Free: 10066.9 GB
```
## Volume cleanup
Volumes provisioned by the vSphere CSI driver are not deleted when you delete a
user cluster. You should delete all PVCs and
[StatefulSets](https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/) before
deleting the cluster. To delete your volumes, run the following command:
```
kubectl delete statefulsets,persistentvolumeclaims --namespace=MY_NAMESPACE --all
```
Replace MY_NAMESPACE with your Kubernetes
[namespace](https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/).
If you deleted a user cluster without deleting its volumes, you can find the
volumes in vSphere user interface:
1. Select a datastore.
2. Navigate to **Monitor** \> **Cloud Native Storage** \> **Container Volumes**.
3. In the **Details** section of each volume, you can see the Kubernetes cluster
name.
## Disabling vSphere CSI driver
To remove vSphere CSI support, add the following to your user cluster configuration:
```
# (Optional) Storage specification for the cluster storage:
# To disable the vSphere CSI driver, set this to true. The value is set to false by default and
# the vSphere CSI driver is enabled by default on vSphere version >= 6.7U3.
vSphereCSIDisabled: true
```
## What's next
- [Read more about Google Distributed Cloud storage concepts](https://docs.cloud.google.com/kubernetes-engine/distributed-cloud/vmware/docs/concepts/storage.md.txt)
- [Set a default StorageClass for your cluster](https://docs.cloud.google.com/kubernetes-engine/distributed-cloud/vmware/docs/how-to/default-storage-class.md.txt)