detect corrupted state in case app dies during middle of shuffile_migrate #2
Description
Currently, shuffile_migrate overwrites files in place when moving data. This is useful to conserve storage space, and it is necessary in the case of only being able to fit a single dataset in cache at a time.
However, in introduces a significant risk in that one has corrupt files during the time that shuffile_migrate is executing. If an application dies during that time, the files may be left in a corrupt state.
We should add logic to detect this condition and return an error to notify the user. If using er, the er library does its own tracking to detect this condition. However, we should add something for apps that may be using shuffile directly without going through the er component.