[bug]: sweeper keeps trying to sweep invalid anchor output until locking up all on-chain funds on node

[Horndev]

Background

LND trying to sweep invalid output, locking up all wallet funds.

Your environment

• version of lnd: LND 0.15.4-beta
• which operating system 18.04-Ubuntu
• neutrino

Steps to reproduce

force close channel https://mempool.space/tx/c8b3d28d1273e6e4d6ea0de25b4c27bc89ca14718ce7c97a31b07da73a027185

Expected behaviour

If anchor transaction was swept already, it should be detected and lnd should stop trying to sweep it.

c8b3d28d1273e6e4d6ea0de25b4c27bc89ca14718ce7c97a31b07da73a027185:1 in the above tx.

Actual behaviour

Repeated sweeps are created that are invalid. A different unused UTXO is selected each time but is not accepted into mempool since the output is already spent. More and more transactions are created to try and sweep until no more UTXO are available.

Here is a screenshot of the situation in RTL: None of those transactions exist in mempool.

Here are some of those raw tx from listchaintxs...

90eda27443517657c971d26a7cb099f9b7b1fc89bcc9ca6160b7e974b044e5d3 : 020000000001028571023aa77db0317ac9e78c7114ca89bc274c5be20dead6e4e673128dd2b3c80100000000000000002a770181af76a3dc17f2e071f2500c87c879ce7e2fe4d42e77b6dbde5cae78c0010000000000000000012f8b200000000000225120009be33bb1c1ff4de6ca3de20ba6f570c1dd72e5bf74484d3ad8649b4d9a509c0247304402200d65ae25fb7e781ced657c1af3258eca8bee2e939dab49808a0fa386766995c502204aa458d75242e33a492ff8f80004f2c6708fb4410fbde4a713f75e09a8d50ec501282103fbf21f752192b0022491a0478b5a4c94ff1534e7328fb9884e8e88675d29e6f8ac736460b26802483045022100b6adc1c1bbaa04387c1fcc5aba482ecb7815f39bba1e7fc7791b082bc800bab8022026e6c0f8e3aac9cec46d70c9620b1800c9eeb5d82f4ae530dce16c80187b57f00121032059e6a56281f5ebf3b7579b08ece1766cb531923182d53dbe4c15f3fbd1a1a6b7d50b00

482870fe60e65ef17d6d2ca8facef0dcc8e528c21c3ed932262347d463fca18e : 020000000001028571023aa77db0317ac9e78c7114ca89bc274c5be20dead6e4e673128dd2b3c8010000000000000000402a839bc36b61c6714c9bc076670ca0ebcb3d72d05c9237691483c128a6a7d4000000000000000000010ace020000000000225120ca2f32bfb44c79fda9efbb474076ed20f4824c0bf8317d06b4a65dc0b42eb2ad02483045022100e4f2feb3c0d274b0351a565cf460e477e61e7ae21400ffb52cb85ff8477709dd02205066f5e3578e2cc5a348f8e25926a5359d1f1aaa5cc6913331b064ff207d75d001282103fbf21f752192b0022491a0478b5a4c94ff1534e7328fb9884e8e88675d29e6f8ac736460b26801409be7cb57b719e235e1bb59fb04b4839e6264b70ab82064444b460497aea1c62415c2fcc2150ba5c70daf9eae3766b82764a12d1e3ca68c906d18edd0869c89f4b2d50b00

53085820a65121739be6ffb067aedbfff10ddf7f6883b61e4631b8dc2c5adfad: 020000000001028571023aa77db0317ac9e78c7114ca89bc274c5be20dead6e4e673128dd2b3c801000000000000000048fefaff6b8845d25a47275a3b1293c708d35d87a22663bb54b25f5f381eb00b0100000000000000000168ae1f00000000002251204e714f6ecc8ccaf307b1c47c51b61e50f85fb48db6e5d98a3eb5905c0e4641820247304402207c098b7ddc4d09b7f0f72265c630a6f24c86bd9f56d2f0821df7b937535bf49c022046bc5c11751d88336bcb5038a9b1f60e195c6024a77995242a62a0517c0c603401282103fbf21f752192b0022491a0478b5a4c94ff1534e7328fb9884e8e88675d29e6f8ac736460b2680140dd305a449aec63b772f08679501835f58efeeccbc2f85613848afb3b0e8f1ad60f81fb39958969feeea872e26e7d8f890b631a66e406999ae047c24c7b118cb3abd50b00

You can see each of these is trying to sweep that same outpoint. Here is what it looks like in listchaintxs:

"previous_outpoints": [
                {
                    "outpoint": "c8b3d28d1273e6e4d6ea0de25b4c27bc89ca14718ce7c97a31b07da73a027185:1",
                    "is_our_output": false
                },
                {
                    "outpoint": "1e6f48e78bd44dd2374c470e6106ed31069d37455ff86a376cf99890d32886c2:0",
                    "is_our_output": true
                }
            ]

[Horndev]

This seems to be related to #6274 but the issue exists in my node now, which already includes that merge.

maybe related to #6367 #6434

** I should note that I have already run --reset-wallet-transactions and the issue persists.

[positiveblue]

@Horndev I think you are right and it is the same problem as in #6434

The fix has been already merged but you will have to wait until the next release to get it. Until then --reset-wallet-transactions won't fix your problem

Sorry I did it understand it wrong.

[Horndev]

Ok, but this is a real problem that makes the node unusable now.

I don't think the sweeper should be locking up the entire balance. At minimum, a reserve should be left behind. Not to mention that it makes it impossible to open new channels, and any UTXOs freed from closed channels are instantly locked up also. It's a pretty crippling issue for me.

[positiveblue]

What is the block height in lncli getinfo? Is the node synced?

As you say c8b3d28d1273e6e4d6ea0de25b4c27bc89ca14718ce7c97a31b07da73a027185:1 is the input of all those txs.
This may have been the first one 19h ago which has not been confirmed yet 961c8bdb0d0c9af9e6aefe119512db92fc4c08bd6e78114f8ead85bab36298fe

Do you see it in your logs/when you call the rpc list txs?

[Horndev]

    "block_height": 775620,
    "block_hash": "000000000000000000058918f6d4a070443ef9f112eaabb858e15958f44814ca",
    "best_header_timestamp": "1675885337",
    "synced_to_chain": true,
    "synced_to_graph": true,

yes, that input is in all of those txs.

Here it is in the logs:

2023-02-07 23:55:21.964 [INF] SWPR: Creating sweep transaction 961c8bdb0d0c9af9e6aefe119512db92fc4c08bd6e78114f8ead85bab36298fe for 2 inputs (c8b3d28d1273e6e4d6ea0de25b4c27bc89ca14718ce7c97a31b07da73a027185:1 (CommitmentAnchor), c35d6ffc8c428ee391bbdfb726dd982502dfe88178c7d023e43e6a7ae3bc6b37:0 (WitnessKeyHash)) using 253 sat/kw, tx_weight=767, tx_fee=0.00000194 BTC, parents_count=0, parents_fee=0 BTC, parents_weight=0
2023-02-07 23:55:21.975 [INF] LNWL: Inserting unconfirmed transaction 961c8bdb0d0c9af9e6aefe119512db92fc4c08bd6e78114f8ead85bab36298fe

It is no longer in listchaintxs after running --reset-wallet-transactions. I had to re-scan since all of my UTXOs were already locked, and after rescanning, they are again getting locked in sweeps.

It's almost like there is some kind of race condition causing the entire wallet balance to get frozen up.

[positiveblue]

Are they ALL locked? A CPFP spending form that tx bumping the fee (if possible) may unlock this situation.

[Horndev]

Yes - that's what happens, all of my UTXOs get locked up. I would have to send some more funds on-chain to wallet and try to beat the sweeper to do a CPFP. I don't know if I can even pull it off. (probably not a good idea, and I do not want to pull out cold funds to mess around here)

[positiveblue]

What about your balance from the closed channel. Was it a force close? https://mempool.space/tx/c8b3d28d1273e6e4d6ea0de25b4c27bc89ca14718ce7c97a31b07da73a027185

[Horndev]

Yes it was a force close. that UTXO is still locked, although at this point it's only 25 more blocks to maturity. At that point my guess is that it will create another invalid sweep transaction that it is going to get used in. joy.

[Horndev]

I wish there was a way that I could manually add a transaction into the wallet, since the 961c8bdb0d0c9af9e6aefe119512db92fc4c08bd6e78114f8ead85bab36298fe transaction is not visible to the wallet. If I could do that, I would try to re-craft that transaction with same inputs but higher fee as it does have RBF.

[Horndev]

I think I did something... https://mempool.space/tx/56bf762416c7506b20a163927a85f5aedf981d94a3a6484454a33e28da0d1699

wallet bumpfee --sat_per_vbyte=15 --force c8b3d28d1273e6e4d6ea0de25b4c27bc89ca14718ce7c97a31b07da73a027185:1

"pending_sweeps": [
                {
                        "outpoint": "c8b3d28d1273e6e4d6ea0de25b4c27bc89ca14718ce7c97a31b07da73a027185:1",
                        "witness_type": "COMMITMENT_ANCHOR",
                        "amount_sat": 330,
                        "sat_per_vbyte": 15,
                        "broadcast_attempts": 6,
                        "next_broadcast_height": 775625,
                        "requested_sat_per_vbyte": 15,
                        "requested_conf_target": 0,
                        "force": true
                },

*** Fingers crossed that this will release my wallet locked UTXOs...

[positiveblue]

It just got confirmed, that should unlock the rest of your utxos

[Horndev]

Yes! It unlocked those UTXOs. but there is another one now. It seems that I somehow created a CPFP on one of the invalid txns, and it's now trying to sweep that. Ugh.

        "pending_sweeps": [
                {
                        "outpoint": "d4a349044f3da93e9852b439a424dd1ab170ac71435703740b11396cd4945234:0",
                        "witness_type": "UNKNOWN_WITNESS",
                        "amount_sat": 2947918,
                        "sat_per_vbyte": 15,
                        "broadcast_attempts": 5,
                        "next_broadcast_height": 775636,
                        "requested_sat_per_vbyte": 15,
                        "requested_conf_target": 0,
                        "force": false
                }
        ]