Wishlist: diff command

[sergiocallegari]

One more enhancement idea (sorry if they are too many, this happens when you find a tool that you like).

Would it be possible to implement a diff command listing paths that are changed between two archives?

Now this requires mounting the repo and then running some recursive operations over the mounted archives that is rather expensive. Possibly, accessing the borg index this could be done much more efficiently.

[LegoChicken]

I agree this would be very useful. I like to know what has changed in my backup (typically by emailing it to myself). This is useful to check that something unexpected hasn't happened. For example, accidentally deleting files.

[ThomasWaldmann]

Just as a side note: you could run borg create -v ... then you get a "what has changed" listing right at backup time.

[LegoChicken]

Thanks. I'm not actively using borg at the moment, but am looking at replacing rdiff-backup. What does the -v output look like?

[ThomasWaldmann]

Like this:

U borg/__pycache__/__init__.cpython-34.pyc
A borg/__pycache__/_version.cpython-34.pyc
...
d borg/__pycache__

[jungle-boogie]

rsnapshot has a command: rsnapshot diff where is will give a result like the following:

Between /snapshots/alpha.1 and /snapshots/alpha.0:
9 were added, taking 169787 bytes
8 were removed, saving 81038 bytes

It would definitely be helpful if borg had something very similiar.

Being able to tell what changed is a very useful feature.

[DrupaListo-com]

you can emulate a "borg diff" via :
$ borg mount REPO mount_point
$ diff -qr mount_point/archive1 mount_point/archive2 >/tmp/diff.patch

[sergiocallegari]

Unfortunately you can only in some specific cases.

The suggestion is OK, if you can run the commands on the same machine where the backup is physically stored. If you need to diff from 'current status of machine' to 'some backup', this means that it is OK only if the backup and the current data are on the same machine (which is not a good idea). Even through a LAN, the proposal can end up generating so much traffic that it can be disturbing for anyone else on the same LAN (think 1-2 TB through the average data rate of 6-7 MB/s of an Ethernet 100 on exclusive use).

Note that often backups need to be kept away from the machine you back up to get the best safety (e.g. if your office is flooded or burned, you do not risk loosing the backup together with the main data). Furthermore, for best security your admin may give you borg only on a restricted access on the remote backup machine which may prevent you from running many other commands on it (e.g. no fuse mount).

Incidentally, the fact that backups are often remote is why in the past I was advocating for keeping some compression on by default on Borg, as it is in Attic. In most safety-first cases, borg gets to operate through the network, whose throughput ends up being the real scarce resource.

[DrupaListo-com]

@sergiocallegari here is another idea for the remote backup dest case:
use borg create --dry-run -v - ...
This will not backup, but give u list of changed/deleted files. Have not tested it myself but should work if u dump into some file that u inspect.

It is not a real diff, but maybe you dont need diff as in "THE diff command" ?

If we need real diffs, maybe for starters a new kind of "borg create -v" could be made like 'borg create --diff-output' that will imply --dry-run and will output diff file - of the type that 'diff -qr' emmits.

Makes sense as an idea ?
Either the -v or the 'diff -qr'-like outputs can be easily made inputs to some other script that will sort the diffs by size for example - to find biggest dirs / files that changed from last borg archive...

This sounds like another nice external tool that could be sth like a borg helper tool - part of the ecosystem... and someday it might even become 'borg diff'

Seems like what I propose - the "--dry-run -v " approach is quite simple to implement even as a borg command and wont bloat the codebase much? It is just syntactic sugar - "create --dry-run -v" -> "diff"
@ThomasWaldmann thoughts ?

PS. I toyed with the idea and here is a borg-diff one-line shell script that seems to work:

sudo borg create --dry-run -x -v $dest::root-tmp / 2>/tmp/diff.txt

Well NO, does not work actually - I thought the dash-prefixed list is the changed files but the dashes I got were indication of a dry-run ... ~!@#$%^&*

see https://borgbackup.readthedocs.org/en/stable/usage.html#item-flags

So the only workaround is to actually run the backup, store the -v output and then delete the archive if only a diff was the purpose... Quite an overkill imho.

We just need to allow the -v option to work for a dry-run too OR another -v analog that does that - maybe called --diff ...

Final thought: A real "borg create -v" and then "borg delete archive" seems better/faster/less-resources than a "borg mount" + diff -qr . So if we could have a better "create -v" we would all be happy.

[anarcat]

well, that's an interesting option. maybe we could coerce the --dry-run to output proper status flags. i had trouble with those as well in #445 and related, so i'm not sure how practical that would be, but worst case we rewrite a lighter version of create. that would seem like a easier objective than writing it from scratch...

[DrupaListo-com]

@anarcat - we could definetely take create and rip it off of it actually writing to the archive and using only the diff-related logic. Its always easy to destroy/remove/strip down things eh? :)

[Ape]

An efficient diff command would be really useful. I want to compare any two archives to see what files were changed, added, or removed. I can currently use diff -qr, but it is very slow with large files.

[infectormp]

+1

[s200999900]

+1