repo: to upgrade or not to upgrade...

[ThomasWaldmann]

borg helium milestone (current master branch) will be reached quicker than expected and there will be some major changes:

• new kdf (argon2) to improve key security - key upgrade for existing keys is available (but the AES encryption key will stay the same, of course, for existing repos)
• new crypto (AEAD, session keys, faster, safer and easier, no counter management issues any more) - as it is now, this will only be available for new repos.
• new repository v2 format (safer and faster, see also repository format conversion for existing repos #6512) - as it is now, borg master branch does not work with v1 repos and creates v2 repos.

the new crypto and the new repo format have some major advantages (and the old code / old repos have some major disadvantages), so it would be good to get everything to the new stuff and also get rid of the old stuff ASAP.

the question is how we get there, considering:

• users having big repos and possibly tight free space
• borg developer resources being limited

[ThomasWaldmann]

Options

do a breaking change (borg 2.0 would not read, write or upgrade old repos):

• least development effort (no need to write a complicated upgrader)
• we could use developer time to maybe do some more breaking changes while we are at it (e.g. a repo index with csize, leading to additional speed improvements and cleanups)
• release quicker, work on other nice features afterwards (e.g. multithreading)
• users could use export-tar/import-tar (using borg-pax tar format) to copy archives into new repos (would need a borg-pax backport to 1.2.x, but that is not much effort), needs 100% more space while doing that.
• alternatively, users can run old and new borg in parallel, slowly phasing out the old repo while building up the new repo. might increase space usage by 100% until the old repo is gone.
• this would be very clean considering the produced state, all legacy would be gone (the code has still support for defects from attic)

stay compatible (release as borg 1.3)

• keep quite some legacy code for now, so borg is able to at least read old repos.
• more complex code, when would we get rid of the legacy code?
• there is no in-place crypto-upgrader yet
• there is no in-place repo v1 -> v2 upgrader yet
• any in-place upgrader has a management issue, because upgrades can be interrupted, leaving a half-converted repo behind. tracking what's converted and what not can use a lot of RAM (e.g. have a set(converted_chunks)).
• in general, such upgraders can be complex and time consuming to write, would likely delay the release and all else.

[FabioPedretti]

I'd vote for the "breaking change" approach.
If one still need to manage an old repo he can easily keep using 1.1/1.2.

[bill-myers]

Any solution that doesn't support migration or requires twice the disk space to migrate because the repository data file format changes and restartable in-place migration is not supported is obviously completely unacceptable (since there may not be enough disk space to do the migration).

So if you want a new repository format you need to either support reading data files in the current format forever, or provide proper upgrade code that can convert file by file and can handle repositories in a partially upgraded state by resuming the upgrade properly, and ideally provide both of those features.

My suggestion is to:

1. Refactor the code so that there is a clean interface implemented by both the old and new format code, make sure the old and new code implementing the interface is in separate files and keep the old code forever
2. Support mixed repositories in any combination (including multiple data file formats and multiple encryption methods)
3. Optionally drop support for creating data in the old format
4. Optionally provide a command to convert everything to the new format, which must work incrementally so that the extra disk space required does not depend on the size of the repository and must not corrupt the repository if the power plug is yanked at any point
5. Optionally provide a command-line option to disable the old code, to prevent rollback attacks

[ThomasWaldmann]

@bill-myers see #6703 for the current approach.

That supports migration, but you'll need a new repo for it (and the space for the new repo).

Everybody not having the space (yet) can also just continue using the old borg version until the space situation at the current repo location improves or until stuff is moved to another location anyway.

An in-place migration likely won't happen due to the complexity of the changes, see also #6602.

[fantasya-pbem]

I support the BC 2.0 version. We should not sacrifice faster, cleaner and less complex code to backwards compatibility.

[BillKen]

Can anything be done to easily make borg1.x and borg2 coexist? (namespacing?) I realise skilled admins will probably be able to install both in different paths but having builtin support will make distro packaging for users who need both during transition easier.
BillK

[ThomasWaldmann]

@BillKen this is being discussed in #6602.

[ThomasWaldmann]

guess we can close this one and continue there.