[bug]: Unknown Unconfirmed Balance

[WalterSmuts]

Background

I folks, am running LND as a router. I have unconfirmed balance and not really sure what's happening. Haven't made any channel opens or blockchain payments since way before this issue came up. So I think it must be related to lnd closing a channel.

Here's the relevant data:

$ lncli walletbalance | grep uncon
    "unconfirmed_balance": "1051070",
            "unconfirmed_balance": "1051070"
$ lncli listunspent --unconfirmed_only
{
	"utxos": [
		{
			"address_type": 4,
			"address": "bc1pgptzpw0jr635psy7z4wzm5e74sqlmf96wfqe0uzd9jvpwjjw0adqnek4jc",
			"amount_sat": 1051070,
			"pk_script": "5120405620b9f21ea340c09e155c2dd33eac01fda4ba724197f04d2c98174a4e7f5a",
			"outpoint": "3bc2fa79c945da5f5d457954a2235678ad3e1c1b3a3d7cbd831cff14984cab97:0",
			"confirmations": 0
		}
	]
}
$ lncli pendingchannels
{
    "total_limbo_balance": "0",
    "pending_open_channels": [
    ],
    "pending_closing_channels": [
    ],
    "pending_force_closing_channels": [
    ],
    "waiting_close_channels": [
    ]
}

The relevant transaction doesn't appear on public trackers. Any idea what happened and how to fix?

Your environment

• version of lnd

$ lnd --version
lnd version 0.15.2-beta commit=v0.15.2-beta

• which operating system (uname -a on *Nix)

$ uname -a
Linux ip-172-31-22-39 5.18.12-1-ec2 #1 SMP PREEMPT_DYNAMIC Sat, 16 Jul 2022 21:17:41 +0000 x86_64 GNU/Linux

(Modified arch linux)

• version of btcd, bitcoind, or other backend
  (neutrino backend)

• any other relevant environment details
  Happy to grep some logs if you can give pointers. The ERR logs are a bit noisy. Here's the tail end:

Nov 02 07:22:59 [REDACTED_IP] lnd[265812]: 2022-11-02 07:22:59.278 [ERR] HSWC: Unhandled error while reforwarding htlc settle/fail over htlcswitch: UnknownNextPeer
Nov 02 07:23:04 [REDACTED_IP] lnd[265812]: 2022-11-02 07:23:04.825 [ERR] HSWC: UnknownNextPeer
Nov 02 07:23:04 [REDACTED_IP] lnd[265812]: 2022-11-02 07:23:04.825 [ERR] HSWC: Unhandled error while reforwarding htlc settle/fail over htlcswitch: UnknownNextPeer
Nov 02 07:46:13 [REDACTED_IP] lnd[265812]: 2022-11-02 07:46:13.491 [ERR] HSWC: insufficient bandwidth to route htlc
Nov 02 07:46:13 [REDACTED_IP] lnd[265812]: 2022-11-02 07:46:13.491 [ERR] HSWC: Unhandled error while reforwarding htlc settle/fail over htlcswitch: insufficient bandwidth to route htlc
Nov 02 07:46:18 [REDACTED_IP] lnd[265812]: 2022-11-02 07:46:18.812 [ERR] HSWC: insufficient bandwidth to route htlc
Nov 02 07:46:18 [REDACTED_IP] lnd[265812]: 2022-11-02 07:46:18.812 [ERR] HSWC: Unhandled error while reforwarding htlc settle/fail over htlcswitch: insufficient bandwidth to route htlc
Nov 02 07:52:31 [REDACTED_IP] lnd[265812]: 2022-11-02 07:52:31.461 [ERR] HSWC: insufficient bandwidth to route htlc
Nov 02 07:52:31 [REDACTED_IP] lnd[265812]: 2022-11-02 07:52:31.461 [ERR] HSWC: Unhandled error while reforwarding htlc settle/fail over htlcswitch: insufficient bandwidth to route htlc
Nov 02 07:58:37 [REDACTED_IP] lnd[265812]: 2022-11-02 07:58:37.418 [ERR] LTND: Unable to lookup witness: no witnesses

[ellemouton]

Hi @WalterSmuts! Thanks for opening the issue :)

On first glance, this looks very similar to #6367, #6231, #6498, #6434

Could you perhaps grep your logs for the transaction ID? (3bc2fa79c945da5f5d457954a2235678ad3e1c1b3a3d7cbd831cff14984cab97)

[WalterSmuts]

$ journalctl -u lnd | grep 3bc2fa79c945da5f5d457954a2235678ad3e1c1b3a3d7cbd831cff14984cab97
Oct 24 10:36:20 [REDACTED_IP] lnd[265812]: 2022-10-24 10:36:20.526 [INF] SWPR: Creating sweep transaction 3bc2fa79c945da5f5d457954a2235678ad3e1c1b3a3d7cbd831cff14984cab97 for 2 inputs (1223529f41409200453b3452a5db8ff4de0145ac8be222100566d3c209a2c3f4:0 (CommitmentAnchor), 67426e4cbf40a685597450c68f29eb4f7a4d9979eb8eb529a2bb7acdf222153d:0 (TaprootPubKeySpend)) using 1803 sat/kw, tx_weight=725, tx_fee=0.00002501 BTC, parents_count=1, parents_fee=0.00001128 BTC, parents_weight=1288
Oct 24 10:36:26 [REDACTED_IP] lnd[265812]: 2022-10-24 10:36:26.733 [INF] LNWL: Inserting unconfirmed transaction 3bc2fa79c945da5f5d457954a2235678ad3e1c1b3a3d7cbd831cff14984cab97

[ellemouton]

interesting. Is your channel with outpoint 67426e4cbf40a685597450c68f29eb4f7a4d9979eb8eb529a2bb7acdf222153d:0 still listed as open in your listchannels output?

Do you recall trying to close this channel? Could you maybe also grep a few logs from above the ones you just posted for this channels outpoint (and perhaps for the channel ID)

[WalterSmuts]

interesting. Is your channel with outpoint 67426e4cbf40a685597450c68f29eb4f7a4d9979eb8eb529a2bb7acdf222153d:0 still listed as open in your listchannels output?

Nope, don't think so.

lncli listchannels  | grep 67426e4cbf40a685597450c68f29eb4f7a4d9979eb8eb529a2bb7acdf222153d | wc
      0       0       0

$ lncli closedchannels  | grep 67426e4cbf40a685597450c68f29eb4f7a4d9979eb8eb529a2bb7acdf222153d | wc
      1       2     101

Do you recall trying to close this channel?

Definitely not personally. Could be peer-initiated or a force-close by lnd. Atleast that's what I think.

Could you maybe also grep a few logs from above the ones you just posted for this channels outpoint (and perhaps for the channel ID).

Not sure which channel you're referring to.

[ellemouton]

Alright - chatted to the OP offline & we found that the associated channel has been closed but the sweep tx did not broadcast correctly it seems and that is what is showing up in LND.

@WalterSmuts, you are running with a neutrino back-end correct? Would be useful to see some more logs -> my suspicion is that LND gave the tx to neutrino and then neutrino didnt broadcast it properly...

Perhaps would help to turn on debug logs & restart so that the tx is given to neutrino again for broadcast

[ellemouton]

The OP sent me some logs after turning on debug level. Can confirm that:

1. LND re-sends the tx to neutrino LNWL: Successfully rebroadcast unconfirmed transaction
2. Neutrino sends an INV with the tx to a bunch of peers BTCN: Sending inv
3. The peers to ask the OP's node for the tx BTCN: Received getdata
4. The OP's node sends the tx BTCN: Sending tx

However, the TX is still not appearing in the mempool.

Perhaps the peers are getting the tx and then rejecting it? I have asked the OP to send me the raw hex of the tx so I can try submit it so bitcoind myself to see what error is returned

[ellemouton]

Alrighty! getting closer to the issue:

Tried to submit the tx to the mempool and got:

error code: -25
error message:
bad-txns-inputs-missingorspent

The output from listchaintxns shows the following:

  "label": "0:sweep",
            "previous_outpoints": [
                {
                    "outpoint": "1223529f41409200453b3452a5db8ff4de0145ac8be222100566d3c209a2c3f4:0",
                    "is_our_output": false
                },
                {
                    "outpoint": "67426e4cbf40a685597450c68f29eb4f7a4d9979eb8eb529a2bb7acdf222153d:0",
                    "is_our_output": true
                }
            ]

The first tx is not a valid tx

[ellemouton]

miiiight have to do with the fact that someone else swept the anchor output which then invalidated our own sweep of the anchor: https://blockstream.info/tx/078c3357606d7d959e9e5d3b70409971880c384886511c0b5d333b6604ea521d?input:12

[ellemouton]

related: #6241 & #6274

[WalterSmuts]

Issue still persisting...

miiiight have to do with the fact that someone else swept the anchor output which then invalidated our own sweep of the anchor: https://blockstream.info/tx/078c3357606d7d959e9e5d3b70409971880c384886511c0b5d333b6604ea521d?input:12

@Roasbeef I believe #6274 is supposed to solve this issue. Any idea why it's not working?