deleteFailedPayment offline?

[nicolasburtey]

Background

Following this issue: #5297 I'm trying to clean up the db so that lnd doesn't make 1h to start.

I have run the command deleteFailedPayment to try to clean up the bbolt database. following deleteFailedPayment rpc, the node has been stuck for 2h, and the db size hasn't change during this time (looking at a watch du). After the 2 hours I have restarted the node and it is still working, but the db size hadn't changed.

would it be possible to run this command offline (with chantools maybe)?
Maybe the implementation of the deleteFailedPayment is just not made for db that are 6GB+ in size?

The node currently run in production so it's hard to make test on it.

Your environment

• version of lnd: 0.12.1
• which operating system (uname -a on *Nix). linux / gcloud
• version of btcd, bitcoind, or other backend. bitcoind

[alexbosworth]

Don't run this command on 0.12.1! it will delete all your payments not just failed ones

[guggero]

Also, deleting contents of a bbolt DB does not reduce its file size. Only after compaction is the reserved free space actually freed up. I guess because it was one huge transaction to delete all payments it was rolled back (or not committed) when you aborted. Chantools doesn't have that functionality yet, no.

[nicolasburtey]

Also, deleting contents of a bbolt DB does not reduce its file size.

it should actually increases it right?
before the issue #5297 db was at 1GB, now is 6GB. so I would expect the db to increase to ~11GB after the payment deletion

this could be handy to have this in chantools? I guess it could do the operation faster than if this is run inside lnd? I don't know where the bottleneck lies in this operation (CPU, disk access or something else)

[nicolasburtey]

I guess because it was one huge transaction to delete all payments it was rolled back (or not committed) when you aborted. Chantools doesn't have that functionality yet, no.

I don't know much about bbolt. it uses transaction to be able to atomically make a change in the db across different key/pair?

[nicolasburtey]

Don't run this command on 0.12.1! it will delete all your payments not just failed ones

ok. I though 0.13 was only adding the option to deleted failed htlc payment (instead of "global" failed payment). would not have been a bug issue on my side because payment upon completion are stored in another db but still good to know.

[guggero]

I don't think the file size should increase by a lot during deletion. As far as I understand the records are just marked as deleted in the internal allocation table. Possibly it took so long because it tried to delete all payments in one transaction which is very memory intensive and probably degrades in performance with the number of pending deletes. Perhaps it would be good if this could be done in chunks (or just more often in the future).

I'm hesitant to add features like that to chantools as they could run out of sync with lnd depending on the version that's used. I can also only pin one version of lnd. So users of older nodes would run into the problem that their DB is migrated to that lnd version if they use chantools for anything other than read-only operations.

[guggero]

I think the proper solution would be to add batching to the deletion so not everything happens in a huge DB transaction.

In the meantime, to fix your particular situation: Could you try adding the failed_htlcs_only: true flag (so basically calling the RPC with {failed_payments_only: true, failed_htlcs_only: true} first?
That removes less data and might complete in a reasonable time. And after that call succeeds, try again with only the failed_payments_only flag.

In the meantime I'm going to look into adding batched deletes.

[nicolasburtey]

the main hurdle I think is that this command is working on lnd 0.13 only? installing a pre-released version in production with a single point of failure (we don't have a redundant lnd just yet in our infra) seems risky

[guggero]

Ah yes, I keep forgetting that...

Okay, I went ahead and created a version of the DeletePayments method that does batched deletes. This'll be in v0.13.1-beta (hopefully) for future users (probably won't make it into 0.13.0 itself).

I also went ahead and back-ported this onto a 0.12.1-beta branch and added it as an offline command to chantools (see new release https://github.com/guggero/chantools/releases/tag/v0.8.6).

TBH I don't know what is riskier at this point, running 0.13 or chantools against your mainnet node 😅
I did test this new chantools deletepayments method on a fresh 0.12.1-beta node and it seems to work...

If you're willing to give this a try, here is what I would recommend:

1. Stop lnd
2. Create a backup of the channel.db
3. Run chantools deletepayments --failedonly --channeldb <channel.db> (drop the --failedonly depending on your needs). NOTE: This will only work for lnd v0.12.1-beta!
4. Run chantools compactdb ...
5. Start lnd again. If you see any errors, please DM me on Slack and DO NOT just revert to using the backup channel.db.
6. If you don't see any errors and lnd starts fine, please destroy the backup of the channel.db to make sure it isn't used by accident.

[nicolasburtey]

closing this issue now that chantools have an option for it