LND does not recover limbo balance from Borked channels

[pvorangecrush]

Background

I had 4 force closes that got "ChanStatusBorked". The limbo_balance which should be owed to me just sits in a V0_P2WSH output. I can't find any evidence that LND is tracking these outputs and will attempt to claim them into the wallet. For one of these closures it has been 2041 confirmations, so I think csv_delay has likely been passed. To me it seems LND is not trying to claim these funds. How do I do so? Are the funds lost for all time?

Your environment

• LND: "v0.15.5-beta",
• Linux: "19.0-35-generic 36~22.04.1-Ubuntu"
• bitcoind: v24.99.0-78aee0fe2ca7

Steps to reproduce

All channels that have channel status ChanStatusBorked fail to claim limbo funds to wallet.

ChannelPoint: d6e99309184e39e44b2c645d2f092e639ba9ff8fd4b571199131a40bde0fad5d:0
ClosingTx: 1e71b62453757b32246fcfe07e257e055572f675295143d3657dcf05f3bcbbfe

ChannelPoint: 2a150dea324cc6ae0da58d7d4e69417402bac0a0dc36bb07410ca776ec987da6:1
ClosingTx: 22fdea129652378ed6c12045754e692ca3a5472eb614efc59b434be9d86d2f22

ChannelPoint: af25ab94051503e0a44f0672af2bf6547859b771b0a73315dc2e410ec19f9c0a:2
ClosingTx: 45e034087408f1156bb86ede9d056cd9224e148d39764c68fe84bc9162c16e69

ChannelPoint: 739db084c4093385e993c745e72f1b21e886a6f61388532851d51e8b3cbddcb0:0
ClosingTx: 78b8c3c58e210f9fd73db6aa544d7f6d4887602c738b257d3509149be46a8044

Expected behaviour

Limbo funds should be spendable in wallet.

Actual behaviour

Limbo funds are not spendable in wallet.

[guggero]

The funds are definitely not lost. The main question is why they got into the borked state in the first place. Do you remember why or when exactly it happened? Did you experience data corruption or other issues with your channel.db?

Also, how do these channels show up in your lnd? Are they in the lncli pendingchannels output?
What side issued the force close?

Can you also check your log please? It could be that lnd is attempting to sweep the funds but is unable to because of the fee rate (or mempool size).

[pvorangecrush]

They are all four in my pending channels:
2 were force closed by remote and 2 by local.
Of note, I was having this issue, #6434 but resolved that issue just now by upgrading to v0.16.0-beta

The channels became borked in different ways:
This one says "INITIATOR_LOCAL" and my local closing txid was confirmed to blockchain. The channel partner was offline for an extended period and lnd intiated the force close
ChannelPoint: d6e99309184e39e44b2c645d2f092e639ba9ff8fd4b571199131a40bde0fad5d:0
ClosingTx: 1e71b62453757b32246fcfe07e257e055572f675295143d3657dcf05f3bcbbfe

This one says "INITIATOR_REMOTE" but my local closing txid was confirmed to blockchain not the remote close transaction. I attempted cooperative close, but then accidently closed the running "lncli closechannel" . After that, I initiated a --force close
ChannelPoint: 2a150dea324cc6ae0da58d7d4e69417402bac0a0dc36bb07410ca776ec987da6:1
ClosingTx: 22fdea129652378ed6c12045754e692ca3a5472eb614efc59b434be9d86d2f22

This one says "INITIATOR_LOCAL" and my local closing txid was confirmed to blockchain. The channel partner was offline for an extended period and lnd intiated the force close
ChannelPoint: af25ab94051503e0a44f0672af2bf6547859b771b0a73315dc2e410ec19f9c0a:2
ClosingTx: 45e034087408f1156bb86ede9d056cd9224e148d39764c68fe84bc9162c16e69

This one says "INITIATOR_REMOTE" and my local txid is listed in closing_txid but the remote closing tx was accepted to the blockchain (I listed the remote closing txid below). The channel was incorrectly set as active, though the closing transaction had been published by the peer days earlier when my node was offline for 2 hours. I restarted LND and LND then recognized a force close had taken place but adopted the "Borked" status.
ChannelPoint: 739db084c4093385e993c745e72f1b21e886a6f61388532851d51e8b3cbddcb0:0
ClosingTx: 78b8c3c58e210f9fd73db6aa544d7f6d4887602c738b257d3509149be46a8044

In all cases the correct limbo balance was moved to an address that is not listed in my lncli listunspent. Could this be related to the bug where I had spent UTXOs in my "listunspent" caused by zero change transactions? Its like the wallet just doesn't claim anything from these channels. The outputs equal to the limbo balance from the closing tx's remain unspent for now.

[guggero]

I think the INITIATOR_* value just indicates who opened the channel initially.

Hmm, none of these actions would explain the "borked" status. Quite weird.

was moved to an address that is not listed in my lncli listunspent.

That is normal, if your node force closes then those funds are time locked and need to be sweeped into a non-script wallet address. Can you check your logs by grepping for one of the closing transactions please? Anything with the SWPR prefix should be interesting.

[pvorangecrush]

The odd part is that it is trying to sweep an output:
22fdea129652378ed6c12045754e692ca3a5472eb614efc59b434be9d86d2f22:1

that has already been spent here:
6348df063362b20feca73266f8246b2394cae912af0f7d60a8d9e6d55ccc800f

2023-03-29 12:05:08.045 [INF] SWPR: Sweep request received: out_point=22fdea129652378ed6c12045754e692ca3a5472eb614efc59b434be9d86d2f22:1, witness_type=CommitmentAnchor, relative_time_lock=0, absolute_time_lock=0, amount=0.0000033 BTC, params=(fee=144 blocks, force=true, exclusive_group=0xc009520c98) 2023-03-29 12:05:08.047 [INF] SWPR: Sweep request received: out_point=a152b779301dd0431c2052c65ca7e98af6f4df0d9f5847fe38a0904683baba47:1, witness_type=CommitmentAnchor, relative_time_lock=0, absolute_time_lock=0, amount=0.0000033 BTC, params=(fee=144 blocks, force=true, exclusive_group=0xc009520c98)
This outpoint a152b779301dd0431c2052c65ca7e98af6f4df0d9f5847fe38a0904683baba47:1 is the remote_txid for the same channel closing which was never published to blockchain.

[pvorangecrush]

Looking at another forced close channel from the list, it seems to see that the closing transaction exists, and then subscribes for confirmations but doesn't seem to notice the anchor outputs have been swept from the closing transactions, so it goes about the business of doing so itself and fails because the outputs have already been spent.

2023-03-29 12:05:07.095 [INF] CNCT: Re-publishing force close tx(45e034087408f1156bb86ede9d056cd9224e148d39764c68fe84bc9162c16e69) for channel af25ab94051503e0a44f0672af2bf6547859b771b0a73315dc2e410ec19f9c0a:2
2023-03-29 12:05:07.095 [ERR] BTCN: Broadcast attempt failed: -27: Transaction already in block chain
2023-03-29 12:05:07.095 [INF] LNWL: Inserting unconfirmed transaction 45e034087408f1156bb86ede9d056cd9224e148d39764c68fe84bc9162c16e69
2023-03-29 12:05:07.105 [INF] CNCT: Re-publishing force close tx(b637390f27a9cc1e6ee1ee633a7954776a2ed326d5602ac601f3fdbb99fc4ea8) for channel 739db084c4093385e993c745e72f1b21e886a6f61388532851d51e8b3cbddcb0:0
2023-03-29 12:05:07.105 [INF] NTFN: New confirmation subscription: conf_id=7, txid=45e034087408f1156bb86ede9d056cd9224e148d39764c68fe84bc9162c16e69, num_confs=6 height_hint=783067
2023-03-29 12:05:07.106 [ERR] BTCN: Broadcast attempt failed: -25: bad-txns-inputs-missingorspent
2023-03-29 12:05:07.106 [INF] LNWL: Inserting unconfirmed transaction b637390f27a9cc1e6ee1ee633a7954776a2ed326d5602ac601f3fdbb99fc4ea8
2023-03-29 12:05:07.118 [INF] NTFN: Dispatching 6 confirmation notification for txid=1e71b62453757b32246fcfe07e257e055572f675295143d3657dcf05f3bcbbfe

[pvorangecrush]

Also, it does look like the fee preference is too low, so it is not attempting publishing (at whence point might it recognize that the outputs were previously swept??)

2023-03-29 13:34:28.146 [WRN] SWPR: Skipping input 78b8c3c58e210f9fd73db6aa544d7f6d4887602c738b257d3509149be46a8044:1: fee preference resulted in invalid fee rate 514 sat/kw, minimum is 522 sat/kw
2023-03-29 13:34:28.148 [WRN] SWPR: Skipping input a152b779301dd0431c2052c65ca7e98af6f4df0d9f5847fe38a0904683baba47:1: fee preference resulted in invalid fee rate 514 sat/kw, minimum is 522 sat/kw
2023-03-29 13:34:28.149 [WRN] SWPR: Skipping input 1e71b62453757b32246fcfe07e257e055572f675295143d3657dcf05f3bcbbfe:0: fee preference resulted in invalid fee rate 514 sat/kw, minimum is 522 sat/kw
2023-03-29 13:34:28.151 [WRN] SWPR: Skipping input 22fdea129652378ed6c12045754e692ca3a5472eb614efc59b434be9d86d2f22:1: fee preference resulted in invalid fee rate 514 sat/kw, minimum is 522 sat/kw
2023-03-29 13:34:28.152 [WRN] SWPR: Skipping input f2436b4616ee0b19a32b522597afe31f6ce499b79ba2dc5abd3564796de6e76b:0: fee preference resulted in invalid fee rate 514 sat/kw, minimum is 522 sat/kw
2023-03-29 13:34:28.154 [WRN] SWPR: Skipping input b637390f27a9cc1e6ee1ee633a7954776a2ed326d5602ac601f3fdbb99fc4ea8:1: fee preference resulted in invalid fee rate 514 sat/kw, minimum is 522 sat/kw
2023-03-29 13:34:28.155 [INF] SWPR: Sweep candidates at height=783075 with fee_rate=7446 sat/kw, yield 0 distinct txns
2023-03-29 13:34:28.155 [INF] SWPR: Sweep candidates at height=783075 with fee_rate=7446 sat/kw, yield 0 distinct txns
2023-03-29 13:34:28.155 [INF] SWPR: Sweep candidates at height=783075 with fee_rate=7446 sat/kw, yield 0 distinct txns
2023-03-29 13:34:28.162 [INF] HSWC: Garbage collected 163 shared secret hashes at height=783075

[pvorangecrush]

Following the resolution of issue #7420 I have re-published the channel open and channel close transactions with lncli wallet publishtx <hex of transaction>, which will hopefully free the sweeper up. I did the same for all the anchor sweeps downstream of the force close transactions in the hope that the sweeper will recognize those transactions as spent as well.

5 hours later, I still can't clear the pending sweeps. It continues to think the anchors can be swept despite the fact they have already been swept by another party days ago.

@guggero Could this be causing it not to claim the limbo balance?

[guggero]

Okay, so good to know that lnd is still attempting to sweep. So it is indeed a mempool/fee issue. What size is your bitcoind's mempool allowance? (-maxmempool)? And are you using a non-default value in lnd for --bitcoind.estimatemode?

Can you try upgrading to 0.16.0 that was just released? That improved some things around transaction broadcasting. If that still doesn't help, you can try to rescan your unspent outputs by starting with --reset-wallet-transactions.

[pvorangecrush]

@guggero Yes, I did the upgrade with 0.16.0 and --reset-wallet-transactions which removed two ghost UTXOs which might have held up some sweep transactions. The fees were low last night, so the sweeper was creating transactions, but it still doesn't seem to get that the anchors are already swept by a 3rd party. It keeps including these spent anchors in sweep transactions and bitcoind rejects them as already spent. All of these anchors are sitting in my pending sweeps. This causes new anchor sweeps to fail, so they will get grabbed by someone else, get stuck in my pending sweeps, and so it seems this process will continue forever locking me out of all future channel fund sweeping attempts. What is the expected LND path to resolution when someone nips in a steals your anchor sweep?

[guggero]

Hmm, I'm not sure how you can convince the sweeper to forget those swept anchors. We'll take a look at it.
In the meantime, you could use chantools sweeptimelockmanual to sweep them yourself, as a workaround.

[pvorangecrush]

@guggero

To get the value for --timelockaddr you must look up the channel's funding output on chain, then follow it to the force close output. The time locked address is always the one that's longer (because it's P2WSH and not P2PKH).

It is not clear what this means as all the output in the closing transaction were P2WSH. Do you mean the unspent output address of the closing transaction where the limbo funds are? I have tried this many times, and it never finds any keys, so I think I am doing it wrong.

Also, I went through some of the chantools workflow starting with my channels.db, and chantools --fromsummary ./results/<summary-file-created-in-last-step>.json rescueclosed --channeldb ./results/compacted.db recovered zero keys which seems quite odd despite the fact that summary from the prior step noted there were potential funds recoverable. Starting to feel like I'm never recovering these funds.

Is the problem LND, the channels.db, or the LND wallet? I can see the closing transactions for the locally initialized force closes in my channel.db dump. Why would LND create a closing transaction to an address it doesn't have the keys to? How to I create the key that can unlock these addresses? The csv_delay is well past for 3/4 of these closes.

[guggero]

Oh, I just realized I never updated chantools to work with anchor channels... That's probably why things don't work as expected. Also, I'd be very careful when doing things with the channel DB directly. That could have bad impact on any online channels. So please don't run any random commands. You will be able to recover the funds, don't worry. Just a bit of patience is needed.

[pvorangecrush]

Yes, I’ve only been exploring a csv dump of an offline copy I made of the channel.db to see what information is there. My node is still online. Is there some guide or spec for force close anchor transactions? Most of the information I find through GitHub or Google searches is woefully outdated so I must be looking in the wrong places.