should print a warning if file changed during backup

[enkore]

GNU tar(1) does this by stat(2)ing before and after reading from the file. If the ctime changed‡, a warning is printed.

From the mailing list thread about annoying stuff in Borg.

‡ this is platform dependent and needs to be handled different on Windows.

[ThomasWaldmann]

Hmm, isn't that a bit a questionable / superfluous feature?

Usually one knows whether one backups a changing or a not-changing set of files.

If you know the stuff is changing and you directly back it up without doing a snapshot (fs, lvm, ...), such warnings would be expected and you know it - so you usually either live with it (if harmless) or you just don't do that.

To implement this feature, we would have to do 2 stat() calls instead of 1. For many (new or changed) small files this might impact performance. The files cache helps a bit, as unchanged files will be quickly processed without reading the contents.

[enkore]

If the files cache is used (file_known_and_unchanged and all chunks presents) then this check doesn't make sense, since we never read from the file.

If we read from the file (chunker loop), then we can do it. I suspect the extra cost will be very low due to inode cache locality (stat - read stuff - stat); if it's a big file it's not a big difference either way.

Since we already have an open FD at that time we can stat through the FD - no directory traversal done.

[ThomasWaldmann]

Right, it is likely cached, so just 1 OS call overhead + a bit of python.

If we consider it superfluous, that doesn't matter though. ;)

[ThomasWaldmann]

As there was no work on this and we are close to RC phase, I am removing this from the milestone.

[ethchest]

This would also in many cases add a lot of output if funning with output/progress enabled. In many cases, snapshots are not really necessary but still, if a filesystem continues to be used, file will change, if only cache files. (which you could exclude, but that is also a lot of work. I noticed this already now when backing up /home - at least each app that has a cache or similar generated errors about files having been deleted.

[ThomasWaldmann]

@ethchest in case of dirs with files that do not need to be backed up, it is way better adding them to the exclude list than having a lot of warnings in each backup log and having unnecessary big backups due to backing them up. nowadays a lot of that is in /home/*/.cache/.

[ThomasWaldmann]

Do this after PR #4043 is merged.

[christophlehmann]

It would be very nice if this can go on 👍 In some environments (cloud server for example) we have to backup an active filesystem and sometimes the backup fails because of stat: [Errno 2] No such file or directory when the file were deleted during backup

[ThomasWaldmann]

@christophlehmann that is a different case.

In your case, the file vanished after borg listed the directory and before borg began processing the file.

So, borg noticed that and gave you a warning (not an error) and continued with the next file(s).

borg later will terminate with rc 1 for warning status (not with rc 2 as when an error occured).

This ticket is rather about a file having been modified after borg started processing it and before borg finished processing it. That means the file is backed up in an inconsistent state (which should raise a warning at least and could be avoided by doing a snapshot with lvm/zfs/...).

[christophlehmann]

@ThomasWaldmann Thanks for clarification!