borg2: replacing "append-only"

[ThomasWaldmann]

In 2.0.0b16, the remainders of "--append-only" were removed.

Some features like append-only repositories rely on a server-side component that enforces them (because that shall only be controllable server-side, not client-side).

So, that can only work, if such a server-side component exists, which is the case for borg 1.x ssh: repositories (but not for borg 1.x non-ssh: repositories).

For borg2, we currently have:

• fs repos
• sftp: repos
• rclone: repos (enabling many different cloud providers)
• s3/b3: repos
• ssh: repos using client/server rpc code similar as in borg 1.x

So, only for the last method we have a borg server-side process that could enforce some features, but not for any of the other repo types.

Also, a-o depended on how borg 1.x dealt with the segment-files (always appending new stuff at the end, transactional behavior with commit or rollback). borg2 neither uses segment files nor transactions.

For an "append-only replacement" (let's call it simply "no-delete" for now) the current idea is that this should not be done within borg, but solved by a not-granted repo object delete/overwrite permission enforced by the storage.

borg create could then use credentials that miss permission to delete/overwrite existing repo objects in archives/* and data/*, so nothing can get removed or damaged in the repo, but a new backup archive can be added.

borg delete and borg prune would need permissions to do the soft-delete in archives/*.

borg compact would use credentials that include permission to delete.

borg repo-compress would need overwrite permissions (because the chunk ID is based on H(plaintext) and stays the same if data gets compressed differently).

[ThomasWaldmann]

Related, key-based idea:

• Feature request: restricted keys (append only key, delete only key, etc) #4125

Related (old) tickets:

• Borg serve read-only mode #4756
• Write-only mode (do not read anything from the remote) #4071
• Audit an append-only mode repo to make sure the client was well behaved #2251
• Implement a "safe" append-/readonly-mode #1772
• Ability to have true write-only repos #1165
• More borg serve restrictions for key control #1164

These old tickets sometimes referred to the borg 1.x repo (segments, append-only, transactions, etc.).

borg2 / master branch (as of 2025-05) is now much simpler:

• the borg 1.x manifest contained a list of archives, requiring a read-modify-write operation (which required locking). borg2 has the list of archives as a directory, does not require locking and multiple borg create can run in parallel and just add their archive pointers to archive/.
• create only adds new objects to the repo (if an ID is not yet present). this is true for data/ (chunks) as well as for archives/ (it adds a new file pointing to the new archive metadata chunk).
• prune and delete only soft-delete (rename) archive pointers below archives/.
• check and compact are the only operations requiring a lock and which may delete unused or defect objects in the repo.

[ThomasWaldmann]

I added some tooling/infrastructure for working on this:

• borgstore now has kind of an "access.log" (it just outputs it at DEBUG level), so one can easily see what borg is doing in the store.
• borgstore now has a permissions system in the posixfs backend, with which borg can be tested in full permissions, read-only permissions and no-delete/no-overwrite permissions stores.
• currently this stuff is only in borgstore master branch, not released yet

For borg, see #8837 and #8844.

For now mostly for testing, experimenting, analyzing and debugging, no pretty error msgs yet.

export BORG_REPO_PERMISSIONS=all  # or read-only or no-delete
borg repo-create ...

export BORG_REPO_PERMISSIONS=no-delete
borg create ...
borg check ...

export BORG_REPO_PERMISSIONS=write-only
borg create ...

export BORG_REPO_PERMISSIONS=read-only
borg repo-list ...
borg list ...
borg extract ...

[ThomasWaldmann]

Guess this is done (as far as the permissions are concerned, some cli options stuff todo, see #8840).

[rovo89]

As far as I remember, "append-only" had often been misinterpreted, in that it didn't delete anything right now, but could flag some things for deletion which will silently be deleted during the next run without the flag. So let me ask if the following scenario is supported:

• Let CLIENT be any machine where data is originally created, modified and deleted, e.g. photos.
• Let OFFSITE be a Linux server in a remote location, like a friend's Raspberry Pi with some disks attached.
• OFFSITE is meant to have a backup in case CLIENT suffers a complete hardware failure, or worse: a ransomware attack.
• OFFSITE isn't totally trusted, so nobody with access to it should be able to see the files in the repo, and also OFFSITE shouldn't be allowed to access CLIENT (which would allow them to see the original files and potentially even more).
• On the other hand, CLIENT shouldn't be trusted too much either, especially if under attack, so it should only be able to push new daily backups to OFFSITE, but mustn't be able to manipulate or delete existing backups, neither immediately nor scheduled.
• Regular deletion of older backups shall be done manually or by a cronjob(*) on OFFSITE itself.
• With that, a ransomware attack on CLIENT couldn't destroy the backups on OFFSITE because the only command that CLIENT can execute is to send a new backup. And even if the new backup contains the maliciously encrypted files or is empty, the data can safely be restored to the state before the attack.

Is that what you have in mind, and what should be supported now (pending CLI options)?

---

(*) I assume a cronjob would need the repo key, so it would also be able to decrypt all files. Not sure if there would be a better way. Maybe allow a second command in authorized_keys of OFFSITE with predefined parameters for the pruning command, so CLIENT would still need to bring the repo key, but wouldn't be able to delete anything newer than 30 days?