Channel state redundancy for production backends

[GamerSg]

I am starting to see Lightning wallets which backs up every state change to the cloud, which allows a complete recovery in the event the device is lost/unusable. While this is fine for consumer level wallets, service providing backends with higher volumes may need something more reliable/private. If i am to build a production service over lightning, i cannot rely on "shutdown node and hope other nodes will close my channels honestly" when there is an outage or disk failure/corruption.

My idea is to simply add an optional WAL(write-ahead) log file which is a technique used by many enterprise grade databases to commit updates to the DB to a log file before actually updating the actual tablespace on disk. In the event of a crash, the database is able to fully recover from a recent backup of the tablespace and the WAL file. Ideally, the WAL log file should be on a different disk(s) to reduce probability of total failure due to disk issues.

To be even more secure, multiple log files in different disks can be specified.

[GamerSg]

More info about WAL
https://en.wikipedia.org/wiki/Write-ahead_logging

[cdecker]

We are definitely looking into ways to better safeguard c-lightning nodes against dataloss. Some of the features that are currently being discussed are off-site backups (like the ones that you describe) and watchtowers. Since these solutions can be very diverse, and dependent on your preferences, your available resources and the rest of your stack, I don't think there is a one-size-fits-all solution here.

This is one of the reasons we have started the work on plugins. Plugins allow you to implement a quick and simple customization of the lightnignd daemon without relying on us to merge it into the repo (it might not fit others).

In particular for the redundant storage of the backends I think you'll find that the plugin hooks mechanism that is currently under review (see PR #2237) useful. The hooks are a way to synchronously execute logic before lightningd continues with its work. If we add a hook after each commitment, you can save that to any available backup store, and make sure that this state is stored safely, before sending the commitment to your peer, i.e., locking it in.

We don't currently have the necessary hooks, but once we have narrowed down the necessary information it should be straight forward to expose that information to a backup plugin.

[GamerSg]

That sounds great!
Once the plugin hooks are in place, i might write the plugin.

[ZmnSCPxj]

We use sqlite3 to store channel state, which has strong mitigations against data loss already. Sqlite3 defaults to a rollback log instead of a writeahead log, and I believe we use default. Difference between rollback vs writeahead seems to be mostly performance and what it needs from the filesystem, in terms of data loss protection I believe they are similar. For replication of the db across multiple devices I believe mdadm should work for most uses.

For my own opinion, replication of the data at the filesystem level is better as sqlite3 handles dataloss better than anything we can build on top.

For a real production environment, you may be better off investing in a server with ECC memory, multiple hdd, ZFS with dedup off (we would be mostly storing cryptographic nonces, unlikely to be duplicated) and with replication. That would likely be more reliable.

[gdistasi]

A lot of "I think", "likely", "I believe" in the last post...

That would likely more reliable than... what? Backups cannot be made reliably on the same machine, which could, for instance, explode or be stolen.

What about the hooks for the plugin?

[cdecker]

We now have the db_write hook (docs) that allows you to synchronously backup changes before they are written to the DB. That, in addition with the postgres DB backend for more enterprisey deployments, should be a suitable solution for this issue I think.

[ZmnSCPxj]

Given PostGres and db_write support, and the backup plugin, can we consider closing this issue?