windows: backup and extract ADS

[enkore]

Alternate Data Streams (ADS) are the Windows equivalent of resource forks (and/or extended attributes, depending on platform and the moon-phase when you're asking the question ;)

Support for these shall be added to the borg.xattr module.

[ThomasWaldmann]

isn't there the big difference that a ADS can contain lots of data? size like the regular file contents?

[enkore]

How large xattrs can be depends completely on the system and FS. Linux has a limit of 64K here, I think some file systems on Linux have further restrictions. On the other Hand, OSX resource forks (which are read through the xattr API as well #51) have no length limit [1], while Solaris has no limits at all (#1337)[2]. Not sure what limits the *BSDs have.

Typical examples of ADS aren't very large.

A different way of implementing this would be storing them as separate items (: is illegal in Windows file names, while legal in *nix). That would create a lot of clobber when extracted elsewhere, though.

[1]: More research: In HFS+ resource forks and extended attributes are technically the same thing, and resource forks are accessed through the xattr API as we already know. But while a resource fork has file-like limits on data length, xattrs on OSX are limited to 4K in length (the same limit applies on extN on Linux).

[2] Note that Solaris seems to be the only OS where xattrs/forks/ADS have separate metadata on their own.

[Anakonda]

In theory ADS can store any data that file can - if you open a file it opens ADS without name: EG filename is filename::$DATA.
I would try storing them as seperate files. Though when trying to extract file:stream created in non-windows file would have to be created too/first.

[enkore]

Hm, maybe just introduce an Item attribute like "subitems" with a list of items that are encapsulated by the parent item. Then an ADS would just be a regular item in that list, with chunked contents (so no problems with large ADS). This approach would also allow storing what ZFS/Solaris does.

I think there was also the idea mentioned in another ticket of just having alternate/extra data streams as a first-class Item citizen. It was either about resource forks or about sparse files. Not sure which one :D

---

Another interesting restriction: While ADS per se can have the same size as the data/main stream of a file, it seems ADS transported over SMB cannot exceed 64K.

SMB 3.0 supports list of streams less than or equal to 64K.

[ThomasWaldmann]

Doesn't that mean that the LIST ("directory") of streams can't be more than 64K, but the stream itself is unbounded?

[enkore]

Hm, didn't read it like this (yesterday), but seems more likely (today) :)

SMB 3.0 supports list (of streams) less than or equal to 64K.

[enkore]

More research

• When using ntfs(-3g) xattr's are mapped to ADS, specifically the user. namespace is (everything else is empty)

  • I.e. the ADS "asdf" becomes the xattr "user.asdf" on Linux

  • ADS larger than 64K are not directly accessible on Linux (Argument list too long)

  • An ADS larger than 64K can be listed, but not retrieved.

  • There is a special mount option that makes them available as regular files:

    streams_interface=value
    This option controls how the user can access Alternate Data Streams (ADS) or in other words, named data streams. It can be set to, one of none, windows or xattr. If the option is set to none, the user will have no access to the named data streams. If it’s set to windows (not possible with lowntfs-3g), then the user can access them just like in Windows (eg. cat file:stream). If it’s set to xattr, then the named data streams are mapped to xattrs and user can be manipulated by using {get,set}fattr utilities. The default is xattr on Linux, none on other OSes.

• NTFS EA are not accessible on Linux. They are used since Windows 7 by the CSC / offline file services.

[tk7r]

@enkore : MacOSX xattr is NOT limited to 4K. Quite contrary.
I have tested it on MacOSX 10.7 and could set more than 100 KByte as an attribute. (limited by the command line size limit)
In my tests on MacOSX 10.11 i have found a file "in the wild" with more than 300KByte as an attribute.
And i have written a little c program to get the xattr limit from MacOSX itself. It's limited to 2^31 Byte per file on MacOSX 10.11.

[enkore]

Hm, interesting. Seems that there is no reliance on sources there. According to the man page it should be....

The maximum supported size of extended attribute can be found out using pathconf(2) with _PC_XATTR_SIZE_BITS option.

[tk7r]

I'll attach my test program. rename the file to have a ".c" extension; compile with
$ gcc -o test_resource_fork_size test_resource_fork_size.c
and run it
test_resource_fork_size.txt

[tk7r]

I ran test_resource_fork_size on MacOSX 10.7 : Limit is 2^18 Byte there. = 256KByte