VMDistributedCluster CR: orchestrate VMCluster upgrades

[vrutkovs]

Add a new CR - VMDistributedCluster - so that multiple VMClusters can be upgraded in an orchestrated fashion, ensuring the read VMAuth is disabled before upgrade and the VMAgent (if available) doesn't have pending bytes to send.

Fixes #1515

This CR can refer to VMClusters using one of two possible ways:

• Existing VMClusters can be referred to using ref property and changes applied using overrideSpec
• Entirely new VMClusters can be created with name and spec properties

Either way, settings in VMDistributedCluster would be applied to target VMClusters, overriding their existing settings if necessary.

Current implementation scope:

• VMDistributedCluster will create a VMAgent instance to proxy writes and vmauth LB to proxy reads
• VMDistributedCluster can create new VMCluster instances when name and spec are specified
• VMDistributedCluster can update existing VMCluster objects when ref and overrideSpec are set
• Before a cluster is updated, vmauth LB is updated to disable reads from this cluster
• VMClusters are updated one by one, waiting for them to change status to "operational" again
• Time to wait for the cluster to become ready can be configured
• After VMCluster update is complete, we're waiting for VMAgent to flush collected data again by checking its metrics
• VMAuth LB is updated to enable reads from this cluster
• Optionally, the controller can wait a configurable amount of time before proceeding to the next cluster
• Process is repeated for all remaining VMClusters

See #1515 (comment) for agreed limitations for v1alpha1 version:

• All objects must belong to the same namespace as VMDistributedCluster
• Referenced VMClusters are not being actively watched for changes, they only get reconciled periodically
• All objects must be referred to by name, label selectors are not supported
• Only VMClusters are supported, VMSingles are deferred for other versions
• Two delays are tweakable:
• • vmclusterWaitReadyDeadline
• • delay between zone updates
• No additional metric to indicate that the cluster is being upgraded to silence possible alerts

TODO:

• Add changelog entry
• Fix flaking tests
• Set ownerRefs to managed VMClusters
• Add high-level description of VMDistributedCluster and problem space
• Description-less CRD should be applied for development only. Rephrase descriptions in existing parts to make them fit for production
• Squash commits
  Keeping original commits for review as its useful to show how the feature was developed
• Update existing documentation to mention VMDistributedCluster and describe its target architecture and existing shortcomings

[AndrewChubatiuk]

initially thought distributed CR is needed for full distributed setup management, but looks like it only performs version upgrade. In this case just curious why we need different CRs for VM, VT and VL?

[vrutkovs]

Yes, so far we're focusing on upgrades - existing CRs provide sufficient flexibility IMO - and we didn't get a request for other actions so far.

In this case just curious why we need different CRs for VM, VT and VL?

VL and VT don't have agents (yet) so their specs would be different. However we can reuse the same approach and probably even some helper functions

[Haleygo]

Yes, so far we're focusing on upgrades - existing CRs provide sufficient flexibility IMO - and we didn't get a request for other actions so far.

I believe users would expect to modify the vmcluster spec value or apply extra flags to the vmclusters.
And since vmclusterSpec.ClusterVersion is optional, users could specify component versions inside vmclusterSpec which overrides the vmclusterSpec.ClusterVersion.

And currently, it seems VMDistributedCluster only covers a limited scenario where resources like vmcluster, vmuser, vmauth are defined and configured as needed.
Could you please provide an example of how to config them to achieve similar topology described in victoria-metrics-distributed chart? I expect VMDistributedCluster to be supported there when released.

[vrutkovs]

I believe users would expect to modify the vmcluster spec value or apply extra flags to the vmclusters.

Yup, setting generic overrideParams would be more flexible and, along with upgrades, would cover other maintenance tasks, i.e., adding replicas or setting flags