Replication (Copying) of archives between repositories

[verygreen]

It looks like ability to merge several repositories is a good feature to have.

I started several repos one per node fearing the "long rebuild" mentioned in the documentation, when multiple nodes backup into a single place. Now, after some testing I see that the rebuild is not as bad as feared and might even go away eventually with "borgception".
I imagine the repos themselves have plenty of shared data that would benefit from some extra deduplication too. Also I do want to preserve my accumulated history (and avoid the many-many-hours penalty of creating a new backup).

Since the chunks are stored separately (?), it seems the task should be doable to merge one repo into another - just referencing the deduplicated chunks and hardlinking the ones not present.

This can take a form of "borg merge newrepo oldrepo1 oldrepo2 ...." or possibly "borg merge targetrepo sourcerepo".

In any case source repos don't need to have anything modified, the chunks that are needed could be hardlinked in place to the newrepo.

A bit of a challenge is how to avoid archive name clashes - possible with e.g. bulk-rename of all names in a repo with a given prefix, I guess (could be a separate command and make the merge fail early if there are any identical names before doing anything).

Once the merge is complete the user then would be able to remove repos that become unnecessary.

Teh constraints are: to make the merge succesful, same compression should be used in the repos, and also they should be unencrypted, I imagine?

I just tried to see how off-mark I am with some of my repos with the help of the hardlinks tool and the duplication is indeed wide:

# hardlink -c -n -vv /backups/borg/intelbox /backups/borg/intelbox2 /backups/borg/rack3
Would link /backups/borg/rack3/data/0/790 to /backups/borg/intelbox2/data/0/1450, would save 5401840
Would link /backups/borg/rack3/data/0/410 to /backups/borg/intelbox2/data/0/1017, would save 6035747
Would link /backups/borg/rack3/data/0/443 to /backups/borg/intelbox2/data/0/1050, would save 5516112
Would link /backups/borg/rack3/data/0/852 to /backups/borg/intelbox2/data/0/400, would save 5262756
Would link /backups/borg/rack3/data/0/569 to /backups/borg/intelbox2/data/0/1176, would save 5379274
Would link /backups/borg/rack3/data/0/572 to /backups/borg/intelbox2/data/0/1179, would save 5795358
Would link /backups/borg/rack3/data/0/519 to /backups/borg/intelbox2/data/0/1126, would save 5409308
Would link /backups/borg/rack3/data/0/890 to /backups/borg/intelbox2/data/0/1529, would save 6084170
Would link /backups/borg/rack3/data/0/561 to /backups/borg/intelbox2/data/0/1168, would save 6233735
Would link /backups/borg/rack3/data/0/897 to /backups/borg/intelbox2/data/0/1536, would save 5558533
Would link /backups/borg/rack3/data/0/650 to /backups/borg/intelbox2/data/0/1242, would save 5375076
...
Directories 9
Objects 6187
IFREG 6178
Comparisons 172
Would link 172
Would save 998854656

Does this make sense at all?

Would an alternative approach be better (i.e. have ability to specify other "look-into" repos to hardlink to for chunks present there, but not yet in the one we are creating an archive in, shared chunk repos, ...).
Should I just hardlink identical files across the reporitories and just repeat the comparison from time to time for more hardlinking as a poor's man saving?
Any other ideas?

[RonnyPfannschmidt]

chunks are stored in a key-value format that does contain multiple actions per segment file (insert/replace/delete)

it seems a new storage format might be necessary - or at least some extensive logic

[enkore]

I considered adding --target to borg recreate with support for different repositories. It wouldn't need that much additional logic, but one would still need to go through everything and check in what repo refs must be increased/decreased etc.

In any case source repos don't need to have anything modified, the chunks that are needed could be hardlinked in place to the newrepo.

This approach would only work for unencrypted repos; for encrypted repos all keys are different and hence the same chunks would have different names, and copying an archive between repos gives it a different fingerprint.

[RonnyPfannschmidt]

there would be a need to reencrypt unless the key is the same

[verygreen]

@enkore Yes, going through every chunk of one of the source repos is necessary to find which ones are identical to the ones in the target repo and to also add the ones that are not (and change references where data is referred to).

@RonnyPfannschmidt A good first step would be to have it working with unencrypted repos first, I imagine, then reencryption added separately (is this really common to have multiple nodes backup into the same encrypted repository with the same key?)

[ThomasWaldmann]

@verygreen i backup several machines into same encrypted repo.

[enkore]

(is this really common to have multiple nodes backup into the same encrypted repository with the same key?)

Depends on the setup and security requirements.

A good first step would be to have it working with unencrypted repos first, I imagine, then reencryption added separately

It's on the same abstraction layer, i.e. implementing it for unencrypted repos would mean strange code and lots of special-casing; doing it right the first time is easier. The current recreate code in master has laid much of the groundwork down for this.

I don't have a TODO for myself, but if I had one, then it would look something like this for Borg at this time:

• hashindex endianness stuff, overflow stuff
• Investigate FEC implementation paths (and relation to flexible compression)
• --target for recreate
• Investigate ways to get rid of csize in a compatible way
• Write a better spec for DEKs (partially done, some secret gist)

[verygreen]

So, I've been reading the documentation and this bit caught my attention:

Segments are built locally, and then uploaded. Those files are strictly append-only and modified only once.

So are they modified (append-only) or not? Because if they are that means I cannot just hardlink chunks between several repos.

[enkore]

What's meant is that a segment is opened for writing, then some data is written into it (appended). When it's full or a commit takes place it's closed, and will never be opened for writing again ("modified only once") (compaction just deletes them).

The segment files are basically a transaction log. I.e. what's actually in there are sequences of

• PUT <key> <data>
• DELETE <key>
• COMMIT

A segment file can contain hundreds of chunks, in no easily predicted order (they are written as they are made up, so depend on files seen etc.)

Compaction goes ahead and "realizes" DELETEs. I.e. when compaction knows about a DELETE it will actually remove the PUT that added that data and write the other data from that segment into a new segment, then the old segment is deleted.