Lost anchor: stuck sweep transaction (unconfirmed balance)

[C-Otto]

Background

After a force-close of an anchor channel my node was unable to recover the anchor output in time (it was swept by someone else, and the state is "LOST" in lncli pendingchannels). I didn't check, but I think this happened at a time with high on-chain fees, and the close tx was already confirmed (so grabbing the anchor doesn't help with that).

Now, a few days after my node published the now double-spent sweep transaction, I still see funds as "unconfirmed_balance" (I don't have anything else pending, and the mempool was just emptied). I believe that lnd still waits for the sweep transaction to confirm, which will never happen.

I'd like lnd to give up and "unblock" my funds.

Your environment

• 0.14.1-beta
• bitcoind v22 (pruned)

Steps to reproduce

1. Open anchor channel
2. Force-Close anchor channel
3. Wait for sweep transaction to be created
4. Steal anchor before anchor is recovered using sweep transaction

Expected behaviour

Sweep transaction is cancelled (forgotten), unspent transaction inputs can be used as inputs for other transactions.

Actual behaviour

Inputs are listed as "unconfirmed balance".

[C-Otto]

To make matters worse, I think lnd happily used this (unconfirmed) sweep tx output as an input for a new channel (I specified --min_confs 0 out of habit). The open TX is not broadcast, this is in my logs: [ERR] FNDG: Unable to broadcast funding tx xxx for ChannelPoint(xxx:1): transaction rejected: output already spent.

Please let me know what I can do with the "pending open" channel. Is it safe to abandon the channel, as the open TX cannot be confirmed, ever?

More confusingly, I also think the 100k reserve for anchor channels is broken due to this (I have lots of anchor channels).

Before I attempted to open a channel:

{
    "total_balance": "8431982",
    "confirmed_balance": "7868584",
    "unconfirmed_balance": "563398",
    "account_balance": {
        "default": {
            "confirmed_balance": "7868584",
            "unconfirmed_balance": "563398"
        }
    }
}

After:

{
    "total_balance": "12530",
    "confirmed_balance": "12530",
    "unconfirmed_balance": "0",
    "account_balance": {
        "default": {
            "confirmed_balance": "12530",
            "unconfirmed_balance": "0"
        }
    }
}

[C-Otto]

Related to #386

[ziggie1984]

Related to this Topic, should we make the Anchor Sweep Transaktion (non-RBF Signaling) in that case nobody could RBF the funds even after they are broadcasted or does this imply even more logic for handling funds in case there are no other outputs left in the wallet. Might increase the complexity of the wallet so I understand why we RBF the Anchor Sweep in the first place but just bringing it up here as an idea.

[Roasbeef]

I didn't check, but I think this happened at a time with high on-chain fees, and the close tx was already confirmed (so grabbing the anchor doesn't help with that).

We've seen a script attempt to more aggressively sweep anchor outputs, they'd claimed $70 or so worth when I last checked. Right now we always use the smallest fee rate possible when attempting to sweep anchor outputs. You'll be able to set the anchor fee rate after this PR: #6025

Re that balance snapshot, when UTXOs are locked, they don't show up in that base balance output. We should likely add another field there though, so it's a bit easier to keep track of things.

The open TX is not broadcast, this is in my logs: [ERR] FNDG: Unable to broadcast funding tx xxx for ChannelPoint(xxx:1): transaction rejected: output already spen

This PR fixes that behavior: #5158

[C-Otto]

Great! I think the only missing bit is to fail the sweep transaction (detect the double spend), i.e. release the funds that are used as another input alongside the anchor?

[zerofeerouting]

To add more context - today I had the same issue. Did not do anything extraordinary except opening one channel with openchannel --node_key xxx --local_amt 4000000 --sat_per_vbyte 1 --min_confs 0. This channel got opened without any issues.

Today I tried opening another channel the same way, resulting in the channel showing as pending, my wallet balance being very different and the logs showing the following:

2022-02-08 13:28:36.276 [INF] FNDG: Initiating fundingRequest(local_amt=0.05 BTC (subtract_fees=false), push_amt=0 mSAT, chain_hash=xxx, peer=xxx, min_confs=0)
2022-02-08 13:28:36.281 [INF] CHFD: Performing funding tx coin selection using 253 sat/kw as fee rate
2022-02-08 13:28:36.721 [WRN] CHFD: Unable to find funding output for shim intent: unable to create witness script, no funding keys
2022-02-08 13:28:36.731 [INF] FNDG: Target commit tx sat/kw for pendingID(xx): 2299
2022-02-08 13:28:36.731 [INF] FNDG: Dust limit for pendingID(xx): 0.00000354 BTC
2022-02-08 13:28:36.731 [INF] FNDG: Starting funding workflow with xxx.onion:9735 for pending_id(xxx), committype=anchors-zero-fee-second-level
2022-02-08 13:28:37.392 [INF] FNDG: Recv'd fundingResponse for pending_id(xxx)
2022-02-08 13:28:37.412 [INF] FNDG: pendingChan(xxx): remote party proposes num_confs=2, csv_delay=600
2022-02-08 13:28:37.412 [INF] FNDG: Generated ChannelPoint(xxx:1) for pending_id(xxx)
2022-02-08 13:28:37.918 [INF] FNDG: Broadcasting funding tx for ChannelPoint(xxx:1): yyyyyyyy
2022-02-08 13:28:37.919 [INF] LNWL: Inserting unconfirmed transaction xxx
2022-02-08 13:28:37.920 [ERR] LNWL: Notifying unmined tx notification (xxx) while creating notification for blocks
2022-02-08 13:28:37.929 [INF] LNWL: Removed invalid transaction: (*wire.MsgTx)(0xc06c33e0c0)({
2022-02-08 13:28:37.930 [ERR] FNDG: Unable to broadcast funding tx yyyyyyy for ChannelPoint(xxx:1): transaction rejected: output already spent

[ziggie1984]

@Roasbeef not sure but I think this problem will affect way more people having to rescan their wallets if they figure out that their anchor was swept. Because I was sweeping a lot of Anchor Outputs and effectively also RBF existing Anchor-Sweeps which means way more people have a pending tx in their lnd wallet which will never be confimed.

[zerofeerouting]

Just an update for anyone affected by this. To get everything in order again you must reset your wallet balance so that LND completely rescans the blockchain for transactions.

1. Restart LND and reindex wallet transactions

Insert this line in [Application Options]

reset-wallet-transactions=true

and restart LND. Make sure to remove this line afterwards, otherwise LND will do this every time on startup.

It will take a while, depending on your hardware and bitcoind setup. Only do this if you're not running a pruned bitcoind - then you're in uncharted territory (i.e. I don't know if it will work - it probably won't).

2. Abandon the affected channels

If you do have pending channels that used the already spent anchor outputs (where the funding transaction was rejected by the network) you can abandon those channels to make them disappear from the pending_channels list. Make sure that these are really "phantom" channels, don't abandon any channels that are just waiting for confirmations.

[Roasbeef]

In practice I don't think this should require doing a full rescan. With the way the sweeper works, when it tries to publish something invalid, it'll just periodically back off and retry again, possibly with a distinct output set. We also watch for when that output is spent and then stop sweeping and signal back to the caller. So if no unconfirmed dependent spends exist, then it should just go away.

When you do a funding attempt, all the inputs are locked, which rn doesn't show up in the normal balance call. If we detect an invalid spend, then the transaction is removed from the wallet, and eventually the locked funds will become available again. What we don't do rn is automatically remove that transaction from the pending channels list, which is fixed in this PR.

So I think the missing gap here is:

• When the sweeper encounters an output that was spent, and not by us, then it should report back to the wallet to remove that transaction.

With the way the wallet interacts rn with RBF transaction, it doesn't know which one will actually confirm (the mempool is a murky place), so it's possible a user attempts to spend one of those outputs as zero conf. When/if a valid transaction eventually confirms the wallet state will be cleaned up, but only if it's actually one of our spends (need to verify this).

[zerofeerouting]

@Roasbeef - it also looks like LND does not check if a channel funding transaction is valid (neither a local one, nor a remote channel funding transaction).

I was wondering how it's possible to have pending opens for weeks (iniitiated by the remote party) where the funding transaction can't be found in the mempool. It seems that those peers tried to open a channel with an invalid funding transaction and my node not even realizing that the channel opening was basically a double-spend that should have been rejected right away.