DWO should (optionally) delete shared PVCs when no workspaces are left that require it

[amisevsk]

Description

Currently, starting a workspace that uses the common or async storage-type results in the creation of a PVC in that workspace's namespace. This PVC exists outside that workspace's lifecycle and is shared with other workspaces in that namespace. However, even after all workspaces are deleted and the given k8s namespace is no longer used for workspaces, the PVC is left untouched.

When DWO detects that no workspaces require the common PVC anymore, it should delete that PVC to avoid leaving unused resources in the cluster.

Additional context

We would need to verify that this approach doesn't have too heavy of a performance cost, if e.g. garbage collection on the common PVC takes a long time and blocks starting new workspaces.

[AObuchow]

@amisevsk Do you know if there's currently an easy way to check how many workspaces are using a common PVC? Currently, I'd imagine we'd have to check the PVC's subdirectories for any "workspace" directories, which seems slow.

Perhaps we could add a count field to the CommonStorageProvisioner struct, which keeps track of the number of workspaces using the common PVC. Then when CleanupWorkspaceStorage is called, if the count hits 0, we mark the common PVC for deletion.

[amisevsk]

Keeping track of state in the operator is a no-go generally, as behaviour has to be idempotent (always does the same thing based on cluster state).

The easiest way to do it would be to do similar to what the async storage provisioner does to remove the async storage deployment (example here and here):

1. List all workspaces in the current namespace
2. Iterate through them checking storage class (noting that the default is common as well)
3. If the number of common + async workspaces that exist (started or stopped) is zero, delete common PVC instead of running cleanup job