Incremental repository check

[enkore]

E.g. borg check [--repository-only] --incremental. Checks all data since last borg check --incremental invocation. Saves last checked segment into the repository config, continues from there when called again.

Guess this could be extended to archives as well by storing the last --incremental timestamp in the manifest.

bounty: https://www.bountysource.com/issues/38004309-incremental-repository-check

[julian-klode]

This can even have an automatic time out as mentioned in #2216 so you can schedule partial checks running at each backup or so, and thus check your backup incrementally over the week.

[ThomasWaldmann]

full check (what we have now):

• borg check currently rebuilds a repo index in memory while it reads all objects from the repo. then it compares the rebuilt index to the on-disk index and shows differences (no --repair) or writes the rebuilt index to disk (--repair mode).
• while reading the objects (iter_objects(..., read_data=True)), crc32 for the headers+data is checked.
• if in repair mode and a segment file is defect, borg rebuilds the segment file on disk (dropping all invalid stuff).

partial check:

• as it is only partial and we must expect other borg operations modifying the repo in between partial checks, we can not build a repo index and thus not compare it to the on-disk index
• thus we can not support --repair in partial mode, of course
• just doing the crc32 check would be possible
• writing the segment number we reached (and where the next partial check needs to continue) will make a lot more write accesses to the config file (before: only init, config and key changes write to repo config). but we have code to safely update the config file.
• a (rough) time limit is easy to implement
• the validity of the repo index after such a check is unknown, so maybe only allow a partial check for --repository-only mode?

Q: does the archives check in no-repair full-check mode use the in-memory rebuilt index?
A: No, see rollback() call, sets self.index to None.