Reloading of corrupted cached files for PersistentDataset

[Phinnik]

I use PersistentDataset in my train loop. I needed to abort train script and when I run it again, one of cached files was corrupted because of abortion. PersistentDataset did not finish saving this cache file.
It threw an exception RuntimeError: Invalid magic number; corrupt file?

I would like to add a possibility to handle corrupt files. If cached file is corrupted, PersistenDataset will continue like there was no cached file and replace it with a good one.

I see, that if I add "except" block here, I can handle such behaviour.

May I take this ticket and make a PR?

[wyli]

sure please go ahead, and I think would be great to create a warning message if that error handling logic is triggered

[ishandutta0098]

@wyli here is a proposed try-except block to handle this:

if hashfile is not None and hashfile.is_file():  # cache hit
        try:
            return torch.load(hashfile)
        except PermissionError as e:
            if sys.platform != "win32":
                raise e
        except RuntimeError as e:
            if 'Invalid magic number; corrupt file' in str(e):
                print(f"Corrupt cache file detected: {hashfile}. Deleting and recomputing.")
                hashfile.unlink()
            else:
                raise e

[KumoLiu]

Hi @ishandutta0098, thanks for the proposal, would you like to create a PR?
Here is the contribution guide you could refer to.
Thanks!

[ishandutta0098]

Sure @KumoLiu please assign the issue to me, I will start working on it