Support automatically resizing workspace PVCs where supported.

[amisevsk]

Description

Since PR #827, it's possible to configure the PVC size used for both per-workspace and common PVCs. However, after a PVC is created, the storage size is not updated, and so the only way to reflect changes to the configuration is to first delete all DevWorkspaces in the namespace that use that PVC, allow the PVC to be cleaned up, and then recreate it by creating a new DevWorkspace (see #834).

Some persistent volume storage-classes support resizing PVCs. When this is possible, DWO should attempt to automatically increase PVC sizes if it detects that an existing PVC is smaller than the configured default.

This resize could potentially be performed on-demand, e.g. the next time a workspace is started. We do have to be a little careful as resizing potentially hundreds of PVCs could impact cluster stability/performance.

Additional context

Further notes:

• Since we currently don't update PVCs once they are created, there is a chance that the PVC has already been configured by users. Likely the implementation should only increase PVC size (if the default is less than the existing PVC's size, we do nothing).
• This feature has a potential performance impact, and it would be necessary to verify that updating the default storage size will not adversely impact performance/stability in large clusters (with potentially hundreds of affected PVCs)

[AObuchow]

So I've been working on this and the core functionality of expanding the PVC seems to work well.
However, dealing with ResourceQuota's adds a bit of complexity. Ideally, we want to be able to check that the expansion will succeed based on the available storage limits before we attempt the expansion. Otherwise, DWO will continuously attempt (and fail) to resize the PVC.

The problem is, we do not know ahead of time the name of the ResourceQuota object, only its namespace. Thus, the usual pattern of performing clusterAPI.Client.Get(clusterAPI.Ctx, namespaceName, resourceQuota) isin't possible since the NamespacedName requires the object name. Furthermore, doing a clusterAPI.Client.List(..) seems too inefficient as we only have the Namespace key to help us with the search (unless we document that cluster administrators must add a DWO-specific label to the ResourceQuota object) - there could be many objects/resources in the DWO namespace, and iterating through them to find the ResourceQuota could be very slow.

There is an API available to help list the available ResourceQuota in the namespace. However, it requires a kubernetes.Interface object, such as Clientset. In DWO, we are currently using the clusterAPI client from the controller-runtime. Thus we'd need to create a Clientset and pass it to the DWO storage provisioners (maybe globally, to prevent modifying the Provisioner interface).

CC: @ibuziuk

[ibuziuk]

This is where things with the automatic resizing become complicated. Basically, there could be different types of quotas applied to a cluster:

• per-user
• per-namespace

Since it is pretty hard at this point to handle this configuration properly on the DWC end, I would propose to implement this feature initially without taking into account the quota aspect + disable this feature by default, with the possibility of enabling it via DWOC flag.