can't remove a ASD node when its OS crashes

[yongshengma]

I just got an issue. A node's OS (ubuntu 16) has completely crashed due to several times power off. I have no way to recover it at least now. Ovs remove node <ip> doesn't work in this case. I marked that node offline, moved vdisks to available nodes, shrinked vpool from that node and then tried to remove ASDs from backend. However, I got stuck in the last step. ASDs of that node are dark all the time and I have no chance to click remove . Moreover, I'm not sure if removing ASDs and their node in this way will work for the cluster. If it works, I suppose this node be re-installed and join back into current cluster as a new node.

[yongshengma]

This failed node has completely lost communication with other nodes. I did try to add a new one but it tried to connect the failed IP . It would be nice if ovs has a subcommand like drop for this case. This looks like a severe issue because it prevents me from recovering the cluster. Perhaps this is not trivial.

[yongshengma]

Say this is not trivial probably because data rebuild on live nodes have to be finished before failed node can be dropped. Otherwise there should be some way to show the progress of data rebuild so that no data loss will occur when failed node has to be removed.

[wimpers]

ovs remove node <ip> any specific error that popped up as that should work. Maybe related to #1905 ?

[JeffreyDevloo]

Hi Yongshengma

Reviewing the code of the Fargo repository:
There are indeed a few errors that might pop up when trying to remove the node. The code checks if all ASDs are remove-able. This is never the case since the node is permanently removed.

Most if not all corner cases around this topic were already fixed in a more recent version Enterprise Edition of the Framework. (https://github.com/openvstorage/framework-alba-plugin/releases/tag/1.11.2)
The repository to look for these ALBA related issues is https://github.com/openvstorage/framework-alba-plugin.

One small misconception to address: if your ALBA node goes offline, the maintenance agent will pick this up and already start repairing the fragments. The removal of the ASD-Manager is just to clean-up your GUI and letting ALBA know that those ASDs are permanently removed (ALBA will check the ASD state every once in a while to see if it is back up)
These repair tasks can be tracked with alba list-work. See alba list-work --help

You can check your remaining safety using the healthcheck (https://github.com/openvstorage/openvstorage-health-check) with ovs healthcheck alba disk-safety-test

Best regards

[wimpers]

After discussion with @JeffreyDevloo , closing this one down.