excluded directory still accessed, causing backup to fail on missing permissions

[mbunkus]

This is a regression in borg 1.1.0.

I'm running backups as root while regular users have some directories mounted via FUSE. root is not allowed to access those directories. Therefore I exclude them from being backed up with --exclude /path/to/fuse-mount-point.

In borg 1.0.9 this used to work fine. The directories were not backed up, borg did not emit an error message, and barring any other errors borg exited with code 0 signalling success.

In borg 1.1.0 it still tries to access that mount point. That fails, and borg emits an error message "Permission denied". The whole borg run exits with error code 1 in that case signalling an error.

Here's how to reproduce:

1. Create a repository
2. Set up a tree to back up with some files and a directory to be used as the FUSE mount point
3. Configure FUSE not to allow access to mounted directories by other users (in /etc/fuse.conf: option user_allow_other must not be active)
4. Mount a file system as a regular user on said mount point, e.g. with sshfs
5. Try to create a borg backup as root while excluding said mount point

Here's an example I've just run:

[0 mbunkus@chai-latte ~] mkdir -p tmp/test/{tree,repo}
[0 mbunkus@chai-latte ~] cd ~/tmp/test/tree
[0 mbunkus@chai-latte ~/tmp/test/tree] borg init -e none ~/tmp/test/repo
[0 mbunkus@chai-latte ~/tmp/test/tree] echo chunky > bacon.txt
[0 mbunkus@chai-latte ~/tmp/test/tree] echo foo > bar.txt
[0 mbunkus@chai-latte ~/tmp/test/tree] mkdir fuse
[0 mbunkus@chai-latte ~/tmp/test/tree] sshfs sweet-chili: fuse
[0 mbunkus@chai-latte ~/tmp/test/tree] ls -l
total 12
-rw-rw-r-- 1 mbunkus Domain Users    7 Oct 23 12:41 bacon.txt
-rw-rw-r-- 1 mbunkus Domain Users    4 Oct 23 12:43 bar.txt
drwxr-xr-x 1    1000        10002 2468 Oct 23 10:38 fuse/
[2 mbunkus@chai-latte ~/tmp/test/tree] sudo borg create --exclude ~/tmp/test/tree/bar.txt --exclude ~/tmp/test/tree/fuse $HOME/tmp/test/repo::test ~/tmp/test/tree
Warning: Attempting to access a previously unknown unencrypted repository!
Do you want to continue? [yN] y
/home/mbunkus/tmp/test/tree: stat: [Errno 13] Permission denied: '/home/mbunkus/tmp/test/tree/fuse'
[1 mbunkus@chai-latte ~/tmp/test/tree] sudo borg list $HOME/tmp/test/repo::test
drwxrwxr-x mbunkus Domain Users        0 Mon, 2017-10-23 12:43:30 home/mbunkus/tmp/test/tree
-rw-rw-r-- mbunkus Domain Users        7 Mon, 2017-10-23 12:41:55 home/mbunkus/tmp/test/tree/bacon.txt
[0 mbunkus@chai-latte ~/tmp/test/tree]

The test shows that the --exclude statements are correct. The first exclusion of the file bar.txt is handled correctly by borg; the file isn't present in the archive. The second exclusion isn't handled correctly as the error message shows.

[infectormp]

did you try 1.1.1 version?

[mbunkus]

I wasn't aware that 1.1.1 had been released. Sorry about that.

A quick re-run of my procedure shown above with 1.1.1 shows that the problem still exists in 1.1.1.

[ThomasWaldmann]

Likely this is due to some changes in 1.1 branch - we added the option that you can exclude some directory and include some stuff below it again - for that, the excluded dir needs to be traversed and files/dirs encountered matched against the other patterns.

IIRC this was supposed to be a --patterns (new) thing only, so the behaviour change of the already existing --exclude was unintended (and unknown yet).

For a quick workaround, you could use --patterns with the pattern that unconditionally excludes (not recursing into the exclusion). See the docs for details.

[mbunkus]

I've read the whole section "borg help patterns" in the documentation, but I'm not sure I really understand which pattern you mean. I tried --pattern '- /home/…, but that didn't change anything, just like --pattern '- pf:/home/…:

[0 mbunkus@chai-latte ~/tmp/test/tree] sudo ~/borg-1.1.1 create --pattern "- $HOME/tmp/test/tree/bar.txt" --pattern "- $HOME/tmp/test/tree/fuse" $HOME/tmp/test/repo::test2 ~/tmp/test/tree
/home/mbunkus/tmp/test/tree: stat: [Errno 13] Permission denied: '/home/mbunkus/tmp/test/tree/fuse'
[1 mbunkus@chai-latte ~/tmp/test/tree] sudo ~/borg-1.1.1 create --pattern "- $HOME/tmp/test/tree/bar.txt" --pattern "- pf:$HOME/tmp/test/tree/fuse" $HOME/tmp/test/repo::test3 ~/tmp/test/tree
/home/mbunkus/tmp/test/tree: stat: [Errno 13] Permission denied: '/home/mbunkus/tmp/test/tree/fuse'
[1 mbunkus@chai-latte ~/tmp/test/tree] sudo ~/borg-1.1.1 create --pattern "- $HOME/tmp/test/tree/bar.txt" --pattern "- pp:$HOME/tmp/test/tree/fuse" $HOME/tmp/test/repo::test4 ~/tmp/test/tree
/home/mbunkus/tmp/test/tree: stat: [Errno 13] Permission denied: '/home/mbunkus/tmp/test/tree/fuse'
[1 mbunkus@chai-latte ~/tmp/test/tree]

[ThomasWaldmann]

Hmm, is the norecurse variant "!" of exclude (instead of recursing variant "-") not documented?

[mbunkus]

I'd say it isn't:

[0 mbunkus@chai-latte ~] ./borg-1.1.1 help patterns | grep -F '!'
    matching any single character specified, including ranges, and '[!...]'
[0 mbunkus@chai-latte ~]

However, using ! doesn't change anything either:

[0 mbunkus@chai-latte ~] sudo ~/borg-1.1.1 create --pattern "- $HOME/tmp/test/tree/bar.txt" --pattern "! pp:$HOME/tmp/test/tree/fuse" $HOME/tmp/test/repo::test6 ~/tmp/test/tree
/home/mbunkus/tmp/test/tree: stat: [Errno 13] Permission denied: '/home/mbunkus/tmp/test/tree/fuse'
[1 mbunkus@chai-latte ~] sudo ~/borg-1.1.1 create --pattern "- $HOME/tmp/test/tree/bar.txt" --pattern "! $HOME/tmp/test/tree/fuse" $HOME/tmp/test/repo::test7 ~/tmp/test/tree
/home/mbunkus/tmp/test/tree: stat: [Errno 13] Permission denied: '/home/mbunkus/tmp/test/tree/fuse'
[1 mbunkus@chai-latte ~]

[ThomasWaldmann]

Looks like it does a stat() (which fails) on the excluded dir before deciding to not recurse into it.

[ThomasWaldmann]

BTW, rc 1 means "warning", 2 is "error".

For "borg create" failing to read some items is not an error, it will just log that and continue with the backup (that will eventually reach its normal end). Only some major, fatal issue will give rc 2.

[mbunkus]

That's good to know, thanks (how did I overlook this?). I'll adjust my scripts accordingly.

[ThomasWaldmann]

Note:

• the --exclude option internally creates a "!" (exclude-norecurse) type of exclusion, so at least that fits past behaviour.
• yes, 1.0 did the stat() after the matcher.

[ThomasWaldmann]

There's even one more issue:

the stat() in 1.1 is done outside of the "try"-block, so an exception there is catched by the exception handler only layer higher in the recursion and reports not the problematic path, but its parent directory.

See above: it complains about "tree", not about "tree/fuse".

1.0: stat() inside try-block.