walletbalance shows wrong confirmed balance, channels stuck as pending

[TrabbipG]

Background

lncli walletbalance is showing confirmed balance which I don't have. lnd lets me open a channel with those funds, but obviously failing to send out the opening transactions because those outputs are already spent. Channels remain pending for over 1 month now, lncli walletbalance is still showing a wrong balance.

Your environment

• version of lnd
  0.4.2
• which operating system (uname -a on *Nix)
  4.9.0-6-amd64 Fix name typo in README #1 SMP Debian 4.9.88-1+deb9u1 (2018-05-07) x86_64 GNU/Linux
• version of btcd, bitcoind, or other backend
  bitcoind (0.16.1)
• any other relevant environment details

Steps to reproduce

In my case my walletbalance shows
{ "total_balance": "139197", "confirmed_balance": "139197", "unconfirmed_balance": "0" }
Although I only really have about 12000 sat. lnd allows me to open a channel with those non existing funds, those channels show up as pending

{
    "total_limbo_balance": "0",
    "pending_open_channels": [
        {
            "channel": {
                "remote_node_pub": "0374ecf61ed6c1208c42339f47decde2bc0c4393ac95f07827b3471e939d7eb961",
                "channel_point": "69a75bfae6642276efc1bbbf3d8860e3710cc6894c0cfaa438b4987418837427:1",
                "capacity": "110000",
                "local_balance": "109819",
                "remote_balance": "0"
            },
            "confirmation_height": 0,
            "commit_fee": "181",
            "commit_weight": "600",
            "fee_per_kw": "250"
        },
        {
            "channel": {
                "remote_node_pub": "0374ecf61ed6c1208c42339f47decde2bc0c4393ac95f07827b3471e939d7eb961",
                "channel_point": "835672640218607009922e594baba1d3fdad67fb181aab44e84852051d5361bf:0",
                "capacity": "116252",
                "local_balance": "115528",
                "remote_balance": "0"
            },
            "confirmation_height": 0,
            "commit_fee": "724",
            "commit_weight": "600",
            "fee_per_kw": "1000"
        }
    ],
    "pending_closing_channels": [
    ],
    "pending_force_closing_channels": [
    ],
    "waiting_close_channels": [
    ]
}

but the opening transactions obviously never goes out

log says:

2018-05-31 18:51:13.139 [ERR] FNDG: Unable to complete reservation sign complete: Transaction rejected: output already spent
2018-05-31 18:51:13.139 [ERR] FNDG: Failing funding flow: Transaction rejected: output already spent ((*lnwire.Error)(0xc42e7021c0)({
2018-05-31 18:51:13.139 [ERR] RPCS: unable to open channel to NodeKey(0374ecf61ed6c1208c42339f47decde2bc0c4393ac95f07827b3471e939d7eb961): Transaction rejected: output already spent

2018-06-03 23:22:22.708 [ERR] FNDG: Unable to complete reservation sign complete: Transaction rejected: output already spent
2018-06-03 23:22:22.708 [ERR] FNDG: Failing funding flow: Transaction rejected: output already spent ((*lnwire.Error)(0xc4220b07c0)({
2018-06-03 23:22:22.708 [ERR] RPCS: unable to open channel to NodeKey(0374ecf61ed6c1208c42339f47decde2bc0c4393ac95f07827b3471e939d7eb961): Transaction rejected: output already spent

The weird thing about it is that the balance disappears as confirmed balance from lncli walletbalance, after trying to open a channel with those funds, but after some time (I guess hours) it reappears as confirmed balance while the channel is still pending.

Expected behaviour

lncli walletbalance not showing spent outputs as confirmed

Actual behaviour

lncli walletbalance showing funds which were already spent as confirmed, reappearing after trying to open a channel with those "non existent" funds while the channel stays pending.

When trying to send an onchain transaction

lncli sendcoins --addr 34bFuLRhXgD8En6k3pgokDaqHck6yzpUmL --amt 130000 --sat_per_byte 1
[lncli] rpc error: code = Unknown desc = insufficient funds available to construct transaction

[wpaulino]

Do you have an idea of how these outputs were spent in the first place? If they were part of a funding transaction that never confirmed within our timeout (288 blocks), then we'll mark the channel as funding canceled, but we currently don't handle being able to reuse those outputs again. I'm currently working on a way to fix this. If you still have your seed around, you can start up a new node and restore. You should make sure you close all your channels beforehand though as the seed won't recover the funds in your channels.

[Roasbeef]

Can you try to update to the current master? Since 0.4.2 a few bugs have been fixed to ensure that we detect "output already spent"-like errors from the chain backend.

Are you using the bitcoind backend?

[TrabbipG]

@wpaulino I must have used them for funding other channels. It's a bit hard for me to keep track of my overall balance because the channelbalance seems to shift up or down, depending on the mempool I assume. I would say that there was no transaction which didn't confirm in time, but I can't be 100% sure. Is there a way to find that out?

I do have my seed, so starting from scratch is an option if there's no other way currently.

@Roasbeef I'm using bitcoind. I also just updated my node, but it didn't help. Walletbalance now shows

 lncli walletbalance
{
    "total_balance": "216074",
    "confirmed_balance": "216074",
    "unconfirmed_balance": "0"
}

without deposit or channel closing. Pending channels are also still here.

Also it says this at bootup, maybe it helps. It's been over a month now and it still waits for confirmations?

2018-07-03 16:59:09.436 [INF] FNDG: Waiting for funding tx (69a75bfae6642276efc1bbbf3d8860e3710cc6894c0cfaa438b4987418837427) to reach 3 confirmations
2018-07-03 16:59:09.436 [INF] NTFN: New confirmation subscription: txid=69a75bfae6642276efc1bbbf3d8860e3710cc6894c0cfaa438b4987418837427, numconfs=3
2018-07-03 16:59:42.898 [INF] RPCS: [listchannels] fetched 14 channels from DB
2018-07-03 17:03:15.814 [INF] NTFN: New confirmation subscription: txid=835672640218607009922e594baba1d3fdad67fb181aab44e84852051d5361bf, numconfs=3
2018-07-03 17:03:15.814 [INF] FNDG: Waiting for funding tx (835672640218607009922e594baba1d3fdad67fb181aab44e84852051d5361bf) to reach 3 confirmations

[Roasbeef]

So it's the case that those two funding transactions were just never broadcast. As a result, they should be removed from the set of pending channels. Your balance is now correct, but there needs to be logic to double spend those inputs (or just delete those pending channels), @wpaulino is working on a patch to resolve this.

[TrabbipG]

@Roasbeef Yes, those funding transactions never got broadcasted because those outputs were already spent, but the walletbalance before trying to open those two channels was already wrong.

Weirdly the balance changed now after trying to send an onchain transaction.

lncli walletbalance
{
    "total_balance": "216074",
    "confirmed_balance": "216074",
    "unconfirmed_balance": "0"
}

lncli sendcoins --addr 37DKFw4pXXEoBgvTbv5r9119UzdNVqUvxa --amt 216074 --sat_per_byte 5
[lncli] rpc error: code = Unknown desc = insufficient funds available to construct transaction
lncli sendcoins --addr 37DKFw4pXXEoBgvTbv5r9119UzdNVqUvxa --amt 200000 --sat_per_byte 5
[lncli] rpc error: code = Unknown desc = -25: Missing inputs
lncli sendcoins --addr 37DKFw4pXXEoBgvTbv5r9119UzdNVqUvxa --amt 150000 --sat_per_byte 5
[lncli] rpc error: code = Unknown desc = insufficient funds available to construct transaction

now it shows this

lncli walletbalance
{
    "total_balance": "13884",
    "confirmed_balance": "2699",
    "unconfirmed_balance": "11185"
}

total_balance could actually be the correct one now, I'm just confused why it shows an unconfirmed_balance , because obviously no transaction was broadcasted, listchaintxns also doesn't show anything. Assuming the unconfirmed_balance will be confirmed, it seems like updating to the current master branch and trying to send an onchain transaction might have fixed this.

Is there a way for me to delete those pending channels or do I have to wait for the patch you mentioned?

[alexbosworth]

I'm seeing the same issue - the possibly related factor I noticed is that by tracking down a transaction referenced as being rebroadcast on startup it has a fee that is 1 satoshi off from a 1 sat/vbyte fee and therefore doesn't get confirmed

[alexbosworth]

I tracked down the unconfirmed balance in my case to a spend of a received HTLC output or HTLC success that uses a fee 1 sat below the min relay fee

[alexbosworth]

Debugging by looking at the transaction it forms and broadcasts I can see that one of the inputs it's selecting has already been spent into a channel. So I guess the issue is that it thinks one of its coins is not spent but it has been spent.

[jpentland]

I have the same issue on commit: a43d02f

$ ncli pendingchannels
{
"total_limbo_balance": "164076",
"pending_open_channels": [
],
"pending_closing_channels": [
],
"pending_force_closing_channels": [
{
"channel": {
"remote_node_pub": "03b439ff61f39f557fdea45129230f4f0c22b6625fcbbdba0dee52cfc56eed5e4a",
"channel_point": "a710f5a009d9721adf876b92a54b602ed066dd342c13dd3fba4be14f14870bee:1",
"capacity": "166325",
"local_balance": "164076",
"remote_balance": "0"
},
"closing_txid": "b8c76bce5f40a371921c01e073529b9b408cde9b71655b28ddefeb7e4c1896e1",
"limbo_balance": "164076",
"maturity_height": 0,
"blocks_til_maturity": 0,
"recovered_balance": "0",
"pending_htlcs": [
]
}
],
"waiting_close_channels": [
]
}

This channel was force closed by the remote host (not sure why). Checking a block explorer shows that the closing_txid was never confirmed, and in fact a conflicting transaction was confirmed instead:

• Original TX (btc.com)
• Final TX (btc.com)

edit: I also updated to the lastest commit (b0288d4) but the channel didn't change.

Also I am using bitcoind as the backend.

[wpaulino]

@jpentland, your issue sounds a lot like #1341. #1554 attempts to resolve that. Note that you won't be able to clear the channel from your set of pending force closed channels until the forgetchannel PR is merged.

[juscamarena]

I'm having this same issue using btcd as the backend.

2018-07-20 14:47:38.784 [INF] LNWL: Inserting unconfirmed transaction 4199c61b057b404f28d19a0366dbfbb52eb5c6766459bff1d2a5caefc489bd42
2018-07-20 14:47:38.942 [INF] DISC: Broadcasting batch of 8 new announcements
2018-07-20 14:47:39.131 [ERR] FNDG: Unable to complete reservation sign complete: Transaction rejected: output already spent
2018-07-20 14:47:39.131 [INF] FNDG: Cancelling funding reservation for node_key=03cb7983dc247f9f81a0fa2dfa3ce1c255365f7279c8dd143e086ca333df10e278, chan_id=47f54fda08198f60a786ad45b1a5692ddef5f214305c8484aa3bd068f8e4966a

When opening a channel.