add include patterns

[leo-b]

Sometimes the ability to not only exclude patterns but also explicitly specify includes can be useful.

I propose a new optional pattern prefix '+' that may be used with --exclude or in --exclude-from files to define include-patterns. (Just like include/exclude pattern rules in rsync.)

These patterns may be used to explicitly include files in a backup that would otherwise be excluded by a following pattern.

Some use cases:

• exclude docs but backup pdfs:

  --exclude +/data/docs/pdf --exclude /data/docs/
  

  (maybe --include /data/docs/pdf should be introduced to improve readability.)

• use a common exclude list deployed to all backup clients but override it on some specific client:

  global.inclexcl:

  /tmp
  *.tmp
  /scratch
  

  local.inclexcl:

  +/scratch
  /home/dir_to_exclude
  

  $ borg create --exclude-from local.inclexcl --exclude-from global.inclexl backup /

  equivalent:

  global.inclexcl:

  -sh:/tmp
  -fm:*.tmp
  -fm:/scratch
  

  local.inclexcl:

  +fm:/scratch
  -/home/dir_to_exclude
  

(Actually the same approach is already used by the --keep-tag-files option.)

[PlasmaPower]

Filenames can start with + though.

[leo-b]

Filenames can start with + though.

Use -+file or pm:+file to escape and exclude a file named +file.
It's just like you have to escape files named xx:file:

Explicit style selection is necessary when a non-default style is desired or when the desired pattern starts with two alphanumeric characters followed by a colon (i.e. aa:something/*).

[PlasmaPower]

@leo-b It still would be a breaking change, although probably nobody's excluding a file starting with +.

[leo-b]

@PlasmaPower I agree it should be noted in the release notes for academic purposes... ;-)

[ThomasWaldmann]

@leo-b don't you find giving include patterns with --exclude weird somehow?

[enkore]

That weirdness is shared with some other programs as well (.*ignore files are notorious for this).

I'm not sure if this is really needed. It seems to me like a fairly rare use case and since both exclude patterns and path patterns support regexpes the functionality is already there (although more complex to use).

[leo-b]

@ThomasWaldmann as suggested: we could introduce an --include option to make it more intuitively.

Of course you can use regex negative lookahead assertions to except files from exclusion but those regexes will become really complicated and unreadable.

Besides my second usecase (deploying a global default exlude-list with the ability to override it with a per-client filter-file) wouldn't be possible with existing functionality.

[leo-b]

Other tools with include/exclude syntax:

• rsync: https://download.samba.org/pub/rsync/rsync.html
  (see INCLUDE/EXCLUDE PATTERN RULES)
• rdiff-backup: http://www.nongnu.org/rdiff-backup/examples.html#exclude
• duplicity: http://duplicity.nongnu.org/duplicity.1.html#sect9
• IBM Tivoli Storage Client: (patterns are evaluates from bottom up here)
  http://www.ibm.com/support/knowledgecenter/SSTG2D_7.1.0/com.ibm.itsm.client.doc/t_cfg_crtinclexcl.html

... and probably many more. :-)

[ovizii]

Just curios if I can include certain file types right now, aka run borg like this:

/usr/local/sbin/borg create -s -C zlib,6 -v $REPOSITORY::hostname-date +%Y-%m-%d /var/lib/vz/dump/vzdump-lxc-102-*

I'm interesting if this can work:
vzdump-lxc-102-*

[enkore]

Unquoted star-patterns and other glob patterns like in your example are interpreted by the shell

[ovizii]

@enkore thanks. is there some more explanation somewhere? couldn't find anything in the docs: https://borgbackup.readthedocs.io but maybe I am searching for the wrong terms?

[enkore]

These are documented in the shell manual e.g.

http://wiki.bash-hackers.org/syntax/pattern
https://www.gnu.org/software/bash/manual/html_node/Pattern-Matching.html
http://zsh.sourceforge.net/Intro/intro_2.html

[piotrturski]

i also vote for this feature. another possible syntax is not to make -e more complicated but add another one -i / --include (or similar) with same patterns semantic as -e and let them be written in order:

-e ~/docs        # exclude docs \
-i ~/docs/pdf      # but retain pdfs \
-i ~/docs/pictures  # retain also pictures \
-e ~/docs/pdf/useless-pdf    # but exclude useless pdfs

then processing should be easy. just go from bottom to top and do first action that matches

[piotrturski]

another possibility would be to allow delegation of file selection to external program. like taking them from pipe or doing something similar to find -exec