restore big files/block devices with considering already present data

[Lanozavr]

I use borg to make backups of my LVM volumes. It works fine at backup creation time but not as good when I do 'borg extract'. It pulls many gigabytes of data from remote borg repo server although I still have an almost unchanged device in place.

It would be nice to have an option to not just extract data from a borg repo but to also check against an existing file/block device, if it already has the required data and only retrieve changed data from the remote repo.

[ThomasWaldmann]

Yeah, that would be good.

Notable:

• when writing to a block device, one likely would use borg extract --stdout ... > /dev/blkxxx to directly extract to the block device and when doing that, borg does not even know where the data is going / what it could compare against
• for similar reasons borg create --stdin ... might be used at backup time - in this case we might not have the blockdevice name inside the archive
• this is the simpler part of a more complex "missing feature": that of bringing a local data set into an archived state (here, simple: 1 file == a block device, more complex case: a filesystem with existing directories and existing files)
• borg create already has --read-special to indicate that special files should actually have their content read, not just the device file archived. in that case, we have blockdevice names (relative though) in the archive, but at extraction time, desired device names could be different

[ThomasWaldmann]

If A is archived data stream and F is the fs data stream:

• iterate over A chunks list to get chunkid_A, size
• read size bytes from F, compute chunkid_F
• if chunkid_A == chunkid_F, we already have the correct data, next...
• if not, read chunk from A, write to F (seek to correct pos first), next...
• at the end of A, truncate F to the same size.

[alighazi288]

@ThomasWaldmann Based on your comment above, I think I understand the solution and I have outlined a potential approach.

Would it be okay if I work on this issue? Any guidance or specific considerations you have would be greatly appreciated!

[ThomasWaldmann]

@alighazi288 sure, feel free to work on this in master branch.

Don't use block devices, but just normal files with multiple chunks. To test, use chunker settings that create rather small chunks (e.g. 512 bytes), so that we don't need very big files to test this.

Maybe it is simplest to even use the "fixed" chunker when starting to work on tests for this. As the block size is fixed, it is easy to produce a file with specific block content and then do changes in known blocks and check how it behaves.

[alighazi288]

Thank you so much for the information! Also, any particular reason for working directly in master instead of a feature branch?