When a exclude-norecurse pattern ends with a trailing slash, Borg still recurses one level below the excluded path

[c0d3z3r0]

Have you checked borgbackup docs, FAQ, and open Github issues?

Yes. #3209 might be related.

Is this a BUG / ISSUE report or a QUESTION?

ISSUE

System information. For client/server mode post info for both machines.

Your borg version (borg -V).

1.2.2

Operating system (distribution) and version.

Arch Linux

Hardware / network configuration, and filesystems used.

n.a.

How much data is handled by borg?

n.a.

Full borg commandline that lead to the problem (leave away excludes and passwords)

see below

Describe the problem you're observing.

When a exclude-norecurse pattern ends with a trailing slash, Borg still recurses one level below the excluded path. The exclude still works correct, but Borg should skip recursion.

# normal exclude works as expected
(venv) c0d3@z3r0:[~]# rm -rf repo; borg init -e none repo; venv/bin/borg -p create --pattern '- re:^test/' --list repo:: /test                                                  
x /test/Downloadx 0 N test                                                                                                                                                                    
x /test/Downloadx/owh
x /test/blub
d /test

# without trailing slash: works as expected
(venv) c0d3@z3r0:[~]# rm -rf repo; borg init -e none repo; venv/bin/borg -p create --pattern '! re:^test' --list repo::test /test                                                 
x /test

# trailing slash, regex: Borg recurses (exclude still working correct, though)
(venv) c0d3@z3r0:[~]# rm -rf repo; borg init -e none repo; venv/bin/borg -p create --pattern '! re:^test/' --list repo::test /test
x /test/Downloadx 0 N test                                                                                                                                                                    
x /test/blub
d /test

# same for sh:, just for comparison
(venv) c0d3@z3r0:[~]# rm -rf repo; borg init -e none repo; venv/bin/borg -p create --pattern '! sh:test' --list repo::test /test                                                  
x /test
(venv) c0d3@z3r0:[~]# rm -rf repo; borg init -e none repo; venv/bin/borg -p create --pattern '! sh:test/' --list repo::test /test                                                  
x /test/Downloadx 0 N test                                                                                                                                                                    
x /test/blub
d /test

[ThomasWaldmann]

The problem might be that you did not really exclude /test, because the trailing slash in the exclude-norecurse pattern did not match.

Not sure about the best behaviour here though. The main idea behind exclude-norecurse (!) was to not recurse into excluded dirs searching for stuff that might be included again (like possible with the normal exclude pattern -).

Reasons for this can be:

• efficiency (like "i am sure there is nothing to back up in there")
• strange effects happening when recursing into there (automounter mountpoints)
• special filesystems with no user data (like /proc /sys ...)

[c0d3z3r0]

The problem might be that you did not really exclude /test, because the trailing slash in the exclude-norecurse pattern did not match.

Probably yes, but why?

From the patterns man page:

A directory exclusion pattern can end either with or without a slash ('/').
If it ends with a slash, such as some/path/, the directory will be
included but not its content. If it does not end with a slash, such as
some/path, both the directory and content will be excluded.

I don't see that happening here.

Btw. it seems odd to me (but might be right) that Borg first recurses, then checks the directory itself. I would expect it to be the other way round.

Not sure about the best behaviour here though. The main idea behind exclude-norecurse (!) was to not recurse into excluded dirs searching for stuff that might be included again (like possible with the normal exclude pattern -).

Reasons for this can be:

• efficiency (like "i am sure there is nothing to back up in there")
• strange effects happening when recursing into there (automounter mountpoints)
• special filesystems with no user data (like /proc /sys ...)

Sure, I got that, but it doesn't seem to work as my example shows.

[jdchristensen]

I think the patterns man page is explaining this correctly. A pattern ending in a slash does not match the directory, so if you have "some/path/" as an exclusion pattern, the directory "some/path" is not excluded. But that pattern matches things in that directory, so they are excluded. I think it is good that borg is consistent about this, with the trailing slash matching the contents of a directory but not the directory itself. So, as far as I can see, all of the examples you posted are consistent with each other.

[c0d3z3r0]

But that pattern matches things in that directory, so they are excluded.

Oh yeah, when reading it like this it indeed makes sense to me!

However, Borg still goes down the tree at least one level. I haven't checked the code, yet, but maybe it's possible to let Borg feed directories with trailing slashes into the matcher to really prevent recursion completely. Not sure, though, if that would lead to other side-effects

[ThomasWaldmann]

I had a look at the code in archiver.py (search for scandir) and it looks like a cosmetic problem to me that is not easy to fix.

It does the scandir because the pattern did not match yet on the directory (because of the trailing slash not being matched), then it recurses, but then the pattern matches and thus it lists all contained files as x.