unconfirmed channels are not included in channel backup when using gRPC

[rbrooklyn]

Background

Unable to recover after a hardware failure

Your environment

• LND version 0.8.1
• Linux 5.3.12-1-default Fix name typo in README #1 SMP Thu Nov 21 07:21:43 UTC 2019 (a6f6081) x86_64 x86_64 x86_64 GNU/Linux
• bitcoin 0.18.1 running on a remote host on the LAN
• LND was originally installed on another computer which has had a hardware failure.
• channel.backup file was downloaded from RTL client. However I believe it uses LND's backup files

Steps to reproduce and actual behaviour

Channel opened last night. channel.backup file taken and saved away from computer hosting LND. 24 word seed kept safe. The channel.backup file was taken before the channel was confirmed on-chain, but after I initiated the channel opening process.

At some point overnight, the hardware failed.

Set up LND on my desktop computer. Ran command to restore wallet: ./lncli create --multi_file=channel.backup. Enter in 24 word recovery seed. Allow LND to do its thing.

See the following in the log:

2019-12-10 20:17:35.689 [INF] LNWL: RECOVERY MODE ENABLED -- rescanning for used addresses with recovery_window=2500
2019-12-10 20:17:36.340 [INF] LNWL: Seed birthday surpassed, starting recovery of wallet from height=607115 hash=00000000000000000006de43c6f3ef4761652f2ab4b72e99cacf3958dfd82f95 with recovery-window=2500
2019-12-10 20:18:21.234 [INF] LNWL: Scanning 446 blocks for recoverable addresses
2019-12-10 20:19:55.940 [INF] LNWL: Recovered 1 external addrs at height=607437 hash=00000000000000000006fc1ff25bdefa30bf14f06f768d4f88f3323152f69de6
2019-12-10 20:19:55.940 [INF] LNWL: Found 1 spends from watched outpoints at height=607437 hash=00000000000000000006fc1ff25bdefa30bf14f06f768d4f88f3323152f69de6
2019-12-10 20:20:22.169 [INF] LNWL: Recovered addresses from blocks 607115-607560
2019-12-10 20:20:22.481 [INF] LNWL: Started rescan from block 0000000000000000000fa8e058b44e7b61b613eb961bb5a9a0fbd07df5d4a567 (height 607560) for 1 address
2019-12-10 20:20:23.450 [INF] LNWL: Catching up block hashes to height 607561, this might take a while
2019-12-10 20:20:23.453 [INF] LNWL: Done catching up block hashes
2019-12-10 20:20:23.453 [INF] LNWL: Finished rescan for 1 address (synced to block 00000000000000000008fd994b5f8310177b40c2e8cd3b997ad81e35dab0dd4b, height 607561)
2019-12-10 20:20:24.017 [INF] LTND: Chain backend is fully synced (end_height=607561)!

Which I assume is on-chain scanning. However, it doesn't appear that LND is making any attempt to recover channel balance.

Verifying channel backup gives blank response:

./lncli verifychanbackup --multi_file="channel.backup"

{

}

Asking to manually run the restore again produces no output at all and no response from LND.
./lncli restorechanbackup --multi_file=channel.backup

Are funds safu?

[wpaulino]

I recommend trying out v0.8.2-beta-rc2 since there are some SCB fixes that might be helpful.

Do you have any other logs after the ones you pasted above? Channel recovery doesn't happen until after this point.

[rbrooklyn]

Thanks for the advice.

I removed ~/.lnd and started the process again with 0.8.2-beta-rc2.

Everything beyond what I pasted in the original report seems to just be normal LND activity from what I can see.

I have attached the full log here.

lnd.log

[Roasbeef]

There's no indication that any recovery was initiated in those logs. I'd try using the regular lncli restorechanbackup commands.

[Roasbeef]

I'd also check the listchannels and pendingchannels commands, if the channels were re-inserted, then they'll be present there. Also ultimately, recovering the channels actually entails closing them out on chain and sweeping those funds on chain into your wallet.

[rbrooklyn]

Hi Roasbeef,

Unfortunately, restorechanbackup doesn't do anything. I run it, I get no response at all from lncli and lnd itself doesn't respond in the logs.

Both listchannels and pendingchannels show nothing of interest:

./lncli listchannels
{
"channels": [
]
}

./lncli pendingchannels
{
"total_limbo_balance": "0",
"pending_open_channels": [
],
"pending_closing_channels": [
],
"pending_force_closing_channels": [
],
"waiting_close_channels": [
]
}

I'm not sure if I can increase debug logging somewhere to help with this? Currently LND simply doesn't react at all when I run the restorechanbackups command.

Luckily I am able to email the node owner. If I am able to establish a dialogue with him, how can I make it as easy as possible to say "force close X channel please" as I can't identify the channel at my end. I've already emailed and given the txid, but I think that would involve them trawling through logs, and they might just ignore it.

[guggero]

How many channels should be in the channel.backup? If there are more than a few dozen, the lncli restorechanbackup process can take a while (multiple minutes). I recommend setting the --debuglevel=debug then you should see some progress in the logs.
The channels to restore should end up in waiting_close_channels of pendingchannels. Otherwise the restorechanbackup command didn't work properly.

Finally, the node will only start to negotiate channel closes with the peers after the graph is synced. That can take a few hours, look out for the synced_to_graph flag of lncli getinfo.

[rbrooklyn]

Just 1 channel. It was the first and only channel this node had ever opened.

I left lnd running overnight. No movement on my coins, sadly. Even when running the recovery again this morning there is still no reaction from lncli or LND.

A replacement part for the failed node is coming tomorrow. If I use the same software and environment as before I guess I stand a better chance of things working, so I'll try that and restore from my seed words. That way if the channel is forced close from the other side I'll get my funds back.

But I really want to know what is wrong. I have the backup, yet LND seems content on ignoring it.

[Roasbeef]

Try to re-issue the restorechanbackup command. Watch the logs when you do so, ideally on trace, and starting with --noboostrap.

[rbrooklyn]

Hi Roasbeef,

Nothing happens. Literally nothing. I run the restorechanbackup command, lncli returns nothing, lnd does nothing, even with the debug level set to trace and --nobootstrap set on startup.

However, the peer has got back in contact with me. He found the channel and force closed it and it's been confirmed for several hours now. So presumably that's progress. However, when deleting ~/.lnd and creating a new wallet with my seed words, LND still doesn't pick up the funds, despite the transaction being very deeply confirmed now.

Looking at the logs, it identifies me moving the money into the wallet with the right block height, it identifies me moving the money out of the wallet to fund the channel with the right block height, but then it proudly declares that it's fully caught up and no funds are seen. This is with and without giving the channel backup file; same result both times. Very confusing.

Surely if the peer has forced closed I should be able to recover funds now? I'm making the assumption that the peer closed the channel fairly and didn't transmit a punishment transaction (as presumably that would involve a timelock of some description, which doesn't seem to be the case here). Seems odd that LND isn't picking up the funds being returned, unless I'm misunderstanding something about this whole process.

[wpaulino]

However, when deleting ~/.lnd and creating a new wallet with my seed words, LND still doesn't pick up the funds, despite the transaction being very deeply confirmed now.

If the channel was opened prior to v0.8.0-beta, then the funds don't go to a wallet address and they need to be swept.

lncli returns nothing

Returns nothing as if it hangs or it returns you to the CLI prompt?

If you increase the RPC log level on your bitcoind node, do you see any activity?

[rbrooklyn]

If the channel was opened prior to v0.8.0-beta, then the funds don't go to a wallet address and they need to be swept.

This node was running 0.8.1 at the time the channel was created. It's only ever run 0.8.0 or above and I think I didn't even create the wallet until 0.8.1.

Returns nothing as if it hangs or it returns you to the CLI prompt?

CLI prompt. Nothing on LND, nothing on bitcoind. Literally as if I did nothing at all.

[guggero]

How big (in bytes) is your channel.backup file? I suspect it doesn't contain any channels for some reason. That would explain the behavior. Did you make nightly backups of that file? Maybe you find one that has a bigger file size?

[rbrooklyn]

That would make sense. You might be on to something there. File size is 45 bytes. So not empty but very small. I don't know what the normal size of these files should be.

I made the channel using the RTL client. It advised I made a backup, which is the file it gave me. I did make the backup before the channel's funding transaction was confirmed, which is maybe something to do with it?

In any case, now the channel has been force closed by the other party, the SCB file shouldn't factor in (assuming I'm understanding this correctly), so how can I access the funds? I still have my 24 words, which I know is correct as LND perfectly described the transaction into the wallet and out of the wallet to fund the channel. So it's there somewhere I'm assuming.

[guggero]

Ok, we have found the problem then. 45 bytes is only the binary wrapper, that's not enough for a channel. The RPC call that gives you the backups (which RTL uses) doesn't include unconfirmed channels. I'm going to change the title of this issue so we know what the problem is.
I'm going to create a fix as soon as I can.

In the mean time, you can follow the instructions of my blog post: https://medium.com/@guggero/did-you-lose-funds-on-the-lightning-network-because-of-a-disk-crash-8971b6a92494

The problem is (as @wpaulino explained) that you need an information (the commit_point) from the other node to be able to create the tweaked key that the funds now reside on after the remote peer force-closed. The seed is unfortunately not enough for this (until we have the "tweakless" channels which are only available if both nodes are on 0.8.0 or later).

You should be able to only do the first part of the described steps that creates the fake backup file and then use that file with your current node that is already synced. Feel free to contact me on Slack if you need help with it.