BTRFS subvolumes treated as external filesystems

[aardbol]

Have you checked borgbackup docs, FAQ, and open Github issues?

Yes

Is this a BUG / ISSUE report or a QUESTION?

BUG

System information. For client/server mode post info for both machines.

Desktop computer that backups to slave HDD

Your borg version (borg -V).

1.1.6

Operating system (distribution) and version.

Antergos up-to-date, rolling release distro

Hardware / network configuration, and filesystems used.

BTRFS

How much data is handled by borg?

200GB

Full borg commandline that lead to the problem (leave away excludes and passwords)

borg create --one-file-system ...
I can't remember the exact full command exactly because it was in a script that I wrote, which I can't recover due to data loss.

I remember that I used the above in combination with exclude /tmp and exclude /var/cache/pacman/pkg
I also used encryption and zstd compression, level 6

Describe the problem you're observing.

Borg skipped all my btrfs subvolumes during bavkups due to the --one-file-system argument. I thought it was skipping external filesystems only as is described in the documentation. Well it now seems that it skipped even btrfs subvolumes like my /home folder in all my backups.

I was surprised to find it out now because the total amount of back-up size, not taking into account compression and deduplication, was around the size of my partition, so I thought it was a complete back-up. Unfortunately now I found out that in all those back-ups, the /home folder was ignored...

Note that btrfs subvolumes are also handled by fstab, so they are mounted at boot time

Can you reproduce the problem? If so, describe how. If not, describe troubleshooting steps you took before opening the issue.

Yes, use --one-file-system

[mlbarrow]

First, I am sorry to hear that you lost data. I have been where you are and it is quite an awful experience.

That being said, understand that btrfs subvolumes are, in fact, separate filesystems. For example, you'll notice that you can't remove a subvolume using rm. Is there a reason why you were using the --one-file-system argument for this particular backup job?

There was discussion on #2141 some time back to get the documentation where it is currently. Do you have a recommendation of what would make it more clear?

Also, you should regularly perform test restores to make sure your backups are working properly. At a minimum, do a borg list repo::archive to see if it contains what you think it should. There could be other issues going on (hardware issues, filesystem corruption, something changing files during backup process) that could make it difficult/impossible to recover above and beyond neglecting to back up files in the first place.

[ThomasWaldmann]

This is the line in the code deciding about the --one-file-system property:

https://github.com/borgbackup/borg/blob/1.1.6/src/borg/archiver.py#L588

This is the doc string:

https://github.com/borgbackup/borg/blob/1.1.6/src/borg/archiver.py#L3199

I don't think this is a borg bug, it rather seems that btrfs subvolumes being different devices / filesystems was an unexpected property of these for you and you accidentally excluded them.

Feel free to do a PR if you have an idea about how to improve the docs or the code.

[aardbol]

You are right, I should've properly done backups, which means that I also should have done restore test from time to time. Unfortunately, I just based the reliability of the backup on the total size of the backup, the manual checks in the mounts and the reliability of the drive where the backups were on. So now I'm learning it the hard way that I should've this exact check. I still can't understand that /home was skipped though, because most of the files where in that folder and I noticed Borg also parsing that folder, based on the difference of the number of files and the total size between last backup and the current one at that time.

The reason that I used --one-file-system was to skip external mounts from being backed up in the the same job, mounts that are in /mnt and /run/media. That was a more efficient way than manually excluding all folders.

I think BTRFS subvolumes should be included with --one-file-system to avoid mistakes like that again for other people, because as far as I'm convinced off, in practise I think most of use would consider them as part of the main filesystem, just like /home could reside on a second drive instead of all / on one drive. Subvolumes are only separated from the root filesystem (and then mounted to it at boot time) to be able to exclude them from the snapshots to prevent data loss during rollbacks or to give them different mount options for performance, stability of reliability reasons.

I'm going to look into the code which you provided links for and try to PR a change myself, hoping that it would be supported.

[level323]

I'm going to look into the code which you provided links for and try to PR a change myself, hoping that it would be supported.

I would oppose such a change. To my mind (and workflow) the way borg currently handles --one-file-system is the best and most consistent. A change that treated btrfs subvolumes as a "special case" would require writing btrfs-filesystem-specific code, which itself is a hint that this is probably not the right direction to go.

Further, if you're going to propose this change for btrfs, what about other snapshotting filesystems like ZFS?

Also, BTRFS subvolumes are more flexible than you indicate. For example, subvolumes can be nested. They can be nested at arbitrary positions in the filesystem tree. However (and arguably providing further clues that the current approach taken by the borg code is correct) is the fact that creating a snapshot of a subvol that itself contains other subvols will not snapshot those child subvols.

Numerous machines I administer run on (and boot from) a single btrfs partition with various subvols (e.g. os1, os2, os1-home, os2-home, user-documents, etc). Backing up multiple subvols in a single borg command simply requires listing the path of each subvol in borg create. One can use the --one-file-system argument in this case and borg will correctly traverse within source subvol(s) specified on the borg command line but not traverse into any filesystems mounted within that subvol, which is what I believe you want. So all you need to do to achieve the effect you originally intended is to change the arguments you supply to borg that define the backup source(s).

Alternatively, for maximum data consistency when backing up a running system, one can snapshot the subvols to be backed up, have borg backup those snapshots, then delete the snapshots. In this case, the snapshots will not contain any mounted filesystems within them, so --one-file-system is not even needed in this case.

All the best

[aardbol]

level323

I don't think you should consider this as being a special case by implementing specific code to handle this differently as how it's currently done. I think one should look at this as the whole BTRFS partition being one filesystem, as the subvolumes (hence "sub"volume) are also part of that same filesystem (and partition), but are connected to it with different features for reasons already mentioned or separated from it (behind the scenes) on purpose to be able to create scopes for snapshots so that rollbacks don't cause data loss of data that shouldn't be part of that rollback, like logs or data in /home when you just want to rollback a system upgrade that caused corruption.

This specific (the latter) use of snapshots is one of the most commonly used. SUSE (the BTRFS experts) does it and documents it very well on why specific subvolumes are created, Ubuntu does it with the / and /home subvolumes (they don't even implement a single mount option in their setup) and Synology does it for that reason (and only providing very limited supported mount options).

Also, BTRFS subvolumes are more flexible than you indicate. For example, subvolumes can be nested. They can be nested at arbitrary positions in the filesystem tree. However (and arguably providing further clues that the current approach taken by the borg code is correct) is the fact that creating a snapshot of a subvol that itself contains other subvols will not snapshot those child subvols.

This should have been clear that I meant this by saying /home is a subvolume in my / path. But comparing behaviour of BTRFS snapshots to backups is where you are incorrectly mixing two different things together. It's the same like saying RAID is also a backup. It isn't, it helps you against hardware failure of a limited number of drives. Snapshots protect your from unwanted changes of that same filesystem. Backups, if done correctly, protect you from both of these and even more.

If you look at the os.path.ismount(path) function in Python, it should behave correctly (the case I'm defending) with BTRFS subvolumes, because they are the same inode and treated as such by this function, so it shouldn't be hard to implement. This post to confirm my case more: https://unix.stackexchange.com/questions/345471/btrfs-same-inode-number. Both of these rather confirms the behaviour of Borg being illogical rather than your way of working or comparison between snapshots and backups to be correct.

[ThomasWaldmann]

Are you saying os.path.ismount(your_btrfs_subvolume_mountpoint) returns False?

[ThomasWaldmann]

https://github.com/python/cpython/blob/v3.7.0/Lib/posixpath.py#L190

[level323]

@IsaakGroepT

But comparing behaviour of BTRFS snapshots to backups is where you are incorrectly mixing two different things together.

Either you misunderstood me or I was unclear, because I'm absolutely aware of the fundamental difference between snapshots and backups. I think this confusion started because you weren't clear in your original post that you were using nested snapshots. I assumed that you were using a snapshot structure like Ubuntu does (subvols not nested and hanging off the btrfs root directory) and therefore you were backing up a live/running system that also had various mount points within the subtrees of each btrfs subvolume and you were using --one-file-system to prevent borg traversing other non-btrfs mounts (such as /proc, /dev and perhaps network shares etc). Hopefully that helps you to understand my reasoning behind suggesting you backup temporary snapshots of your subvolumes, because I thought you were backing up a live/running system.

Now you've clarified that you're using nested subvolumes, I better understand the issue you're wanting to discuss.

In any case, my central point stands - you can list each subvolume individually on the boirg create command line and borg will dutifully back up all the files in each subvolume. If you want to automate this further it shouldn't be difficult at all because btrfs subv list ..... is there to help you script an automated way to scoop up backups across all your subvolumes, even if you you're regularly creating/moving/renaming them. See also man btrfs-subvolume

So via btrfs subvol list .... the functionality you desire (backing up all subvolumes) is already possible and can be fully automated with a few extra lines of scripting.

In any event, if the present borg docs are unclear, then by all means let's clarify them so that no one else misses this important point.

All the best

[aardbol]

Are you saying os.path.ismount(your_btrfs_subvolume_mountpoint) returns False?

Unfortunately it doesn't seem to be so simple. I tested it and my subvolume was detected as a mount by that function

[plattrap]

An observation is that the BTRFS subvolumes show a device major number of 0. Whether this is guaranteed or just coincidence I have no idea.

Perhaps add a flag --risky-treat-btrfs-as-one-device and change:

btrfs_risky = btrfs_risky_flag and st.st_dev & 0xff00 == 0 and restrict_dev & 0xff00 == 0
recurse = restrict_dev is None or st.st_dev == restrict_dev or btrfs_risky

This would treat all subvolumes as if they are the same filesystem as the start.

[cmurf]

Btrfs subvolumes are separate file trees, they aren't really a separate file system. Subvolumes share all the other trees: root, csum, extent, uuid, chunk, dev, etc. On the other hand, the inode numbering starts over in each subvolume, so a given inode number is not unique on a Btrfs volume where it is unique on a Btrfs subvolume.

Anyway, it's an understandable point of confusion for the user as well as for development. You probably wouldn't really want a backup to, by default, consider all subvolumes (which means including snapshots as they are just a pre-populated subvolume) on the file system for backups as it would lead to a lot of unnecessary duplication of data. The reality is Btrfs volumes can be sufficiently more complicated that treating it as if it's any other file system, and not have a least Btrfs specific warnings to the user for potentially ambiguous situations, is going to end up in misaligned expectations.

[RonnyPfannschmidt]

how about something else - instead of trying to figure out a way to handle this add 2 new tools

a) a warning that will print file-system boundaries that borg will not traverse into
b) a way to include such boundaries into the backup set (for smaller amounts of subvolumes the normal inclusion mechanism of listing paths may suffice, for other setups, different mechanisms may be required anyway

[aardbol]

@cmurf I don't think duplication is an issue to take into account, because Borg has a proper deduplication system that handles this.

I think a warning would be a satisfying way to improve this situation, next to improving the documentation to mention this behaviour specifically.
An extra option to exclude external storage devices and include subvolumes of the same filesystem would be very interesting too as an efficient way for more complex btrfs filesystems.

Since the ismount() function was not interesting, here's Python code that can tell you if a volume is a btrfs subvolume: https://github.com/efficiosoft/btrfs-backup/blob/master/btrfs_backup/util.py. The license is compatible

[level323]

@IsaakGroepT: @cmurf's point about files in snapshots being assigned different inode numbers would mean that for a btrfs volume that has multiple subvolume snapshots (as is the case for users who use tools like snapper) a version of borg that traversed multiple subvols/snapshots would have a long initial run, re-reading large numbers of files that were actually identical but borg couldn't tell because the inode numbers were different. Subsequent runs would benefit from the cache though, as expected.

[cmurf]

@IsaakGroepT: I don't know what "proper" means in the context of deduplication, and I don't know how Borg's deduplication works. If I point borg at a path omitting --one-file-system I expect borg will have to read the contents of every file in that path in order to know whether files are duplicates. If I have 100 GiB of data, and in that same path 10 snapshots of that data, then borg is going to end up reading 1000GiB. That borg deduplicates the backup has no bearing on whether it has to read duplicated data due to the existence of snapshots. Consider an even more extreme case 1000GiB of data, snapshot 10 times. Now borg (or pretty much any tool including rsync) is going to read 10TiB in order to know only 1TiB is unique. That's gonna take days. So yes duplicated data resulting from snapshotting is highly relevant.

And therefore I think it would be entirely sane for borg developers to use --one-file-system on Btrfs volumes by default, with a warning that it's doing so. Separate documentation can explain the rationalization for doing this as, due to snapshots Btrfs volumes can appear to be huge even when 99% of the files point to shared extents. Why bother unnecessarily reading all of that data just to deduplicate it in the backup? That kind of volume replication is available with "Btrfs seed sprout" and more selectively with "Btrfs send receive" using Btrfs user space tools, but of course it requires a Btrfs destination. Open question if libbtrfs offers some API that would help borgbackup deduplicate on read, where shared extents in snapshots are turned into hardlinks on destination, and if not, what that would look like but I'd think both rsync and borg would benefit.

@level323 You definitely cannot infer much of anything about files and their inode numbers on Btrfs. The inode numbers in a snapshot created with btrfs sub snap command will match that of the orginal subvolume (this is why snapshot creation is so fast, there's almost no metadata writes with snapshot creation); whereas if I use cp --reflink, the inode number assigned depends on the destination subvolume and what the next available inode number is. If I use find -inum for inode on a Btrfs volume, I may get back a lot of different files and even dirs, and maybe some identical files that share the same data extents. It's just not knowable without looking at the extent addresses to see if they're shared or not.

With one exception, you can assume that an inode number is not used more than once in a given subvolume. The exception is subvolumes themselves always have inode number 256.

Another gotcha pertains to mounting subvolume with the -o subvol, or -o subvolid mount option. Behind the scenes these are bind mounts. So, depending on how things are assembled, that might pose curious behaviors, not least of which is it's possible snapshots are entirely invisible (not in any mount path).