Massive backup speed increase with optional bypass of 'existing' files in backup

[jumper444]

As I watch backups using Borg it seems pretty apparent that the backup is laboriously going file by file to look for changes in data. I have about a 45G backup source with 45,000 files and it will take 15minutes or longer to run when I often only have 200 or so new/changed files over each backup (100MB or so).

Obviously rigorously reading ALL files for changes will never miss anything (and for a critical backup program that would be very important), but I would really like to see the following optional switch under CREATE command:

--quickcheck
(random idea for what to call switch)

When that switch is added to CREATE borg will NOT process the data of every file being backed up. Instead it will read the file attributes only (ls/dir, etc) and then skip processing any file when:

1. the archive already contains a file with the same name and same path, AND
2. the file size is the same, AND
3. the file date is the same

The inclusion of this switch for my backups would drop the backup time down 90% or more to maybe a minute or two. Yes, there are instances where a file is same name, location, size and date where somehow that file has changed, but they are extremely rare (unless malicious action is going on). And, besides, that's why it would be an optional switch if a person didn't want to take that risk.

Thoughts?

[jumper444]

This is akin to file compare programs whereby they usually have different options to use at user discretion:
-size
-date
-size and date
-full binary
(name is presumed to have to match for all options above)

So maybe if this switch idea is useful it would then have suboptions.
However just being conservative and having ALL of name, path, size, and date match would still massively cut my backup times.

[jumper444]

The use of PATH as a match has pros and cons.

Without its use a person on a multi-user (or non admin user) machine could create a file within their own directories with the same name, size, and date as some other file in the system and thus maliciously interfere (somewhat) with borg's processing when doing a 'quickcheck'.

With the use of PATH however you will spend time processing files which move location but are unchanged. The mitigating factor though is that files don't tend to move often (unless they are 'moving' since they are in randomly generated cache-type directories...which means they are probably unique anyway or excluded from backup), but when legit/normal files DO move they usually don't do so very. Once moved and a backup has processed all future backups to that archive will then have a record of the file at the new path new location. So the next time a 'quickcheck' backup is run it WILL find a matching path (in addition to name, file, size) and skip going forward.

My argument stands that PATH should be a criteria for 'quickcheck' but i wanted to mention this situation.

[ThomasWaldmann]

Please read the "files cache" related docs, also see --files-cache option and the FAQ.
I guess you'll see that quick check is already there and maybe you also find out, why it does not work for you.

[jumper444]

My apology. I did review the 'create' command looking for file comparison criteria and did come across the 'cache' switch. However I carelessly presumed cache meant the somewhat normal definition of only applying to how MUCH and WHERE duplicate-type data would be stored and used by a program, not that the cache setting would cover HOW file comparisons for changes would occur. I see that it is does however and I missed it.

However, after review it still does not explain or answer the question of why Borg is taking so long to find changed files. By my measure it can be done almost instantly. Allow me to explain:

The default setting of compare is: ctime,size,inode.

I can execute the command 'ls -ilcR backupFilesDirectory > out.txt' (recursive and with ctime, inode, size) on my entire backup directory and dump it to a file in under ten seconds.

Likewise, borg can pull the entire cache (which you indicate is what is used to perform comparisons for changes) from my repository in under ten seconds.
(aka...run 'borg delete --cache-only myRepo' then start a create and see how fast the cache comes back; takes seconds).

Therefore in about 20 seconds I can have on disk and memory the entire list of both repo files and the full information of files I'm about to backup on disk (inode, ctime, size). A simple script could compare the two in seconds and..boom...you have the exact list of whatever changed files need processing (in my case usually 100meg or so out of 45G). The backup could then proceed with only those files and it would be over in single minutes.

Instead, despite following the comparison criteria you outline borg uses, borg takes 15-20 minutes to laboriously scan through my backup files for some reason. Why? What is happening when I've already accounted for the comparison method within 20 seconds? Why is borg reading all these files when most are unchanged? Is it doing something else?

(PS: it may be an issue whereby with large backups both lists to compare (borg cache and backup target) cannot be kept in memory. That has multiple solutions and maybe you are using some, but they still don't seem to account for the speed diff: 1) keep only the borg cache memory resident, read and compare the new backup files line by line from disk; or 2) if BOTH cache lists are two big, then store both on disk sorted and read them line by line in order to find differences. But like I said, neither approach, if necessary, accounts for 15-20 minute scan or compare times.)

[jumper444]

PPS: if there is an argument about "we are scanning to pick up changes than happen DURING the backup process" then it works both ways. Yes, a "ls" and then processing the output will miss changes that happen to files after the ls, but before the backup ends.

However if borg scans then the opposite happens...it will pick up changes that happen to files near the end of the backup (10 or 15 minutes later) but then miss changes that happen to files that were already passed during the beginning minutes of the backup.

Unless borg uses a snapshot technology there is no way around missing either early or late changes during the backup. So if borg is indeed scanning file by file it has no obvious benefit other than making things MUCH slower vs just doing 'ls -R' and piping the output for processing within seconds and then conducting backup of changed files.

Maybe I'm wrong and missing something?

[ThomasWaldmann]

You are overlooking that borg is always creating a full backup archive of all input files (including the unchanged ones).

While doing that, it also makes sure that all chunks of a unchanged file actually are still present in the repository.

This is why it takes a bit longer, but it makes borg prune much easier if every archive is full and independent of other archives.

But, it does not need to read unchanged input files - if it does, either ctime, size or inode have changed / are not stable due to other reasons.

[jumper444]

While doing that, it also makes sure that all chunks of a unchanged file actually are still present in the repository.

How do you do this without reading the unchanged file from source to verify the chunks are present in archive?
And aren't you basically saying here the borg does TWO things for each backup:

1. a backup of new data (what I personally expect)
2. a chkdsk/fsck of everything it is reading or processing

??

IMO, step # 2 seems extra and not appropriate for a backup and is already present and handled by running the CHECK command;

I'm arguing for backup that borg should only do step # 1 which would simply/quickly compare files to what is in archive index/roster...then subsequently read/transfer/(check/fsck) data ONLY for new files or changed files. I am arguing that this extra 'sorta like fsck' step not occur for dramatic speed boost (and because it has its own separate command).

Maybe I'm missing something in your reply.

[ThomasWaldmann]

@jumper444 to detect whether a file is unchanged, borg uses the files cache (lookup via path, compares size,inode,ctime usually).

If everything matches, the file is unchanged and borg uses the chunks id list that is also stored in the files cache entry of that file.

The chunks id list is still correct, because the file content is unchanged.

Then borg checks whether all chunks from that list are still present in the repo. If not, borg will read and chunk the file as if it were changed. If all are present, borg will increment the refcount of these chunks, read the metadata for the file from the filesystem (stat, acls, xattrs, bsdflags) and create a new entry in the archive (that looks the same as if borg has read the file).

I already said that borg always creates a full backup. It is just not stupid and does not need to read file content which it already knows.

It's not a "fsck" and also not a "borg check". It is just checking the chunks from the cached chunksid list - they could be gone, if somebody has deleted all archives containing that file.

So, there's nothing inappropriate. For details, read the internals docs or the source.