`--dry-run` does not show exclude tag file being present in the list of archived files as per `--keep-exclude-tags`

[Alexander-Shukaev]

Have you checked borgbackup docs, FAQ, and open Github issues?

Yes

Is this a BUG / ISSUE report or a QUESTION?

ISSUE

System information. For client/server mode post info for both machines.

Your borg version (borg -V).

borg 1.1.16

Operating system (distribution) and version.

Arch Linux

Hardware / network configuration, and filesystems used.

EXT4

Full borg commandline that lead to the problem (leave away excludes and passwords)

borg ... --dry-run ... --exclude-if-present .nobackup --keep-exclude-tags ...

Describe the problem you're observing.

The .nobackup file is not present in the list of output files to be backed up.

[ThomasWaldmann]

$ ls -laR testdir
testdir:
drwxr-xr-x  2 user user   4096 Jun 15 18:39 excl
drwxr-xr-x  2 user user   4096 Jun 15 18:39 incl

testdir/excl:
-rw-r--r-- 1 user user    0 Jun 15 18:39 .nobackup
-rw-r--r-- 1 user user    0 Jun 15 18:39 file1
-rw-r--r-- 1 user user    0 Jun 15 18:39 file2

testdir/incl:
-rw-r--r-- 1 user user    0 Jun 15 18:39 file1
-rw-r--r-- 1 user user    0 Jun 15 18:39 file2

$ borg create repo::archive testdir --list --dry-run --exclude-if-present .nobackup --keep-exclude-tags
- testdir/incl/file1
- testdir/incl/file2
- testdir/incl
x testdir/excl
- testdir

Without dry-run:

$ borg create repo::archive3 testdir --list --exclude-if-present .nobackup --keep-exclude-tags
A testdir/incl/file1
A testdir/incl/file2
d testdir/incl
A testdir/excl/.nobackup
x testdir/excl
d testdir

[ThomasWaldmann]

After fixing this, it would look like this:

$ borg create repo::archive testdir --list --dry-run --exclude-if-present .nobackup --keep-exclude-tags
- testdir/incl/file1
- testdir/incl/file2
- testdir/incl
- testdir/excl/.nobackup
x testdir/excl
- testdir

After fix without dry-run:

$ borg create repo::archive testdir --list --exclude-if-present .nobackup --keep-exclude-tags
A testdir/incl/file1
A testdir/incl/file2
d testdir/incl
A testdir/excl/.nobackup
x testdir/excl
d testdir

So one would see now that the .nobackup file would be backed up (no matter if with or without dry-run).

So guess that would fix it from a consistency standpoint.

Remaining issue: it still has to display x for testdir/excl as an indicator for an excluded directory.

But having a tag file in there means, that the testdir/excl directory item would be backuped also (because it is needed as a container for the .nobbackup file), so the x is not true for the directory object itself.

[Alexander-Shukaev]

I have to note that I don't even see x with the current version. I'm trying against remote repository though.

[Alexander-Shukaev]

There is one important detail that I wanted to test with this output. Maybe we can discuss it here. Usually, --keep-exclude-tags makes sense if this data will be pulled to another system and is also supposed to be backed up in the same manner, where that exclusion should not be forgotten. Though, let's say that to be absolutely sure, one would also add either --exclude '*/excl' or corresponding --exclude-from ..., while still having excl/.nobackup and --keep-exclude-tags. What would take precedence in this case? In my opinion, excl/.nobackup should still be backed up.

[ThomasWaldmann]

Yes, "not losing exclude-tagged dirs" is the idea. Not sure about precedence, just try it.

[Alexander-Shukaev]

Yes, "not losing exclude-tagged dirs" is the idea. Not sure about precedence, just try it.

Would like to follow up as I finally got to test this. It turns out that

1. --exclude or --exclude-from take precedence over --exclude-if-present and --keep-exclude-tags. Namely if a directory that has exclude tag file is also excluded via patterns, --keep-exclude-tags does not apply to it. Personally, I find this behavior unexpected, potentially leading to inconsistencies and issues with the backup. I think exclude tag files should be present no matter what. I will try to support my point further.
2. When one directory has exclude tag file and any child directory of it also has exclude tag file (recursively), only the "top-level" one is kept, while all the nested ones will be discarded. I'm of the same opinion about this behavior as in the first case.

Why is it important to keep all the exclude tag files recursively regardless of their level and exclude patterns? Because this provides a way to back up directory tree structures of any depth. For example, some directories contain log and/or cache files on different levels, which are not desired to be backed up, but the directories they reside in are parts of system installation packages and are tracked by package manager. As a result, excluding them from back up may produce inconsistent and/or unusable system. The only way to cut off nested exclude tag files should be --one-file-system as it is now as well.

What do you think of this? Thank you for your efforts.

I also have one more feature idea, not sure if it was discussed or offered before. How about having .borgignore files, which would behave similar to .gitignore. That is instead of specifying exclude patterns, one could put such files directly under directories where they should apply and control what is backed up in a fine-grained way.

[ThomasWaldmann]

Well, if you exclude / exclude-norecurse a directory via pattern, it does not even look for exclude tags.

Can you give a specific example of a directory tree where your nested-tagfiles backup would make sense?

BTW, this should be in a new ticket, it is unrelated to the topic of this issue.