[bug]: force close transactions disappeared

[yaslama]

Background

Our node force closed a channel but it seems that the fees were too low and the commitment type is STATIC_REMOTE_KEY, so we need to wait for the transaction to be confirmed. The problem is that after a while, the transaction disappears from listchaintxns and is not re-broadcasted.
When I try to force close again the channel, I obtain the error: "channel is already in the process of being force closed".

Your environment

• version of lnd: 0.15.4
• which operating system (uname -a on *Nix): linux
• version of btcd, bitcoind, or other backend: bitcoind 22.0
• any other relevant environment details

Steps to reproduce

Force close a channel which was created without anchors and when the fees were very low. Wait for a while.
Check that the channel appears in lncli pendingchannels | jq '.waiting_close_channels'
Check that when trying to close it again, the following error is returned: channel is already in the process of being force closed
Check if lncli listchaintxns contains the closing tx

Expected behaviour

The closing tx has to stay in the listchaintxns and be broadcasted periodically.

Actual behaviour

lncli listchaintxns doesn't contain the closing tx

[Roasbeef]

Do you see the transaction rebroadcast on start up?

As part of 0.16, we'll now continually rebroadcast transactions in the background: #7448. In versions before that, we either rely on the full node to rebroadcast, or for rebroadcast on start up.

[yaslama]

Do you see the transaction rebroadcast on start up?

I still didn't restart the node. What you say is that when it's not re-broadcasted, it also disappears from listchaintxns?

[Roasbeef]

I think what happened is:

• You broadcast the tx, it falls out of the mempool due to the fee rate being too low, lnd removes the tx as it got an "unmatched backend error"
• We restarted and broadcast, and the same happens again

The core fix that needs to happen here is described here: #4616

We should catch that error and proceed. However I'm not sure that's the best option, as then a user would have their transaction silently not be broadcasted (not rebroadcasted either).

To do that, we need to be able to reliably catch that error for all chain backends: https://github.com/btcsuite/btcwallet/blob/master/wallet/wallet.go#L3791-L3795.

Note that for neutrino, we don't get any sort of rejection feedback anymore from bitcoind as they removed the reject method from their full node implementation. As a result, for neutrino we don't have a good way of knowing that a transaction won't propagate due to being too low fee (other than looking at the fee rate in the past blocks).

[yaslama]

Thanks @Roasbeef for your detailed answer! In addition to your proposed change, what about tagging/marking this tx instead of removing it, so we can have the ability to at least re-broadcast them manually?

[guggero]

#7513 will add the raw transaction hex to the log when publishing it fails, so it can be published manually at a later stage.

[yaslama]

#7513 will add the raw transaction hex to the log when publishing it fails, so it can be published manually at a later stage.

The problem is that the logs are discarded quickly (for instance for privacy reasons) and the tx is lost.