[feature request] Write bbolt databases to two disk locations synchronously

[callebtc]

Background

Probably the easiest way to enable much a safer backup solution for most users is to allow them to use two databases on two different physical drives that are synchronously written to (i.e. second one is a copy of the first, reading is done from the first).

[yyforyongyu]

This is essentially building a distributed data storage, won't say it's easy. How about using etcd?

[kristapsk]

This is essentially building a distributed data storage

CLN just allows to configure two SQLite databases. Or use replication with PostgreSQL.

[guggero]

I think SQLite offers this as a feature natively, while bbolt doesn't. So this should be built into #6570 instead and not be retro-fitted onto the Bolt database (given that etcd and PostgreSQL exists as an option too).

[Roasbeef]

Yep, the main options today are etcd or postgres given they're already distributed databases themselves. It's possible to create a system where one mirrors the channel.db on the fly, but that can be error prone. sqlite has a native WAL mode that effectively gives this feature, as you can replicate that, though you'd need to deal w/ a data loss window, slower writes, and also a larger database as tradeoffs.

[Roasbeef]

In terms of just a local database, your main options are various RAID options, and filesystems like zfs and btrfs that natively handle things like this a and are extremely reliable.

Closing this for now since we offer two replicated DB types, and once sqlite is added (scheduled for 0.16) users can experiment with various WAL modes (with all their tradeoffs) locally which give users various options, but ofc you'd need a proper distributed database and effectively synchronous replication to be 100% safe.

[Roasbeef]

Also re sqlite something like this (https://litestream.io/) could be used, but users need to be aware that there's a short period of time (I think it's 5 seconds or so, but maybe configurable), where if their node goes down, they can lose data as everything isn't fully replicated.