Last sweep tx is constantly rebroadcasted on startup

[roeierez]

Background

As a follow up to this #7599 (comment)
The last sweep tx is persisted and always rebraodcated on startup. In case this tx double spends another tx it causes the user balance to stay in unconfirmed state.

Your environment

• LND v15.3
• mobile android/iOS using neutrino

Steps to reproduce

We have seen this in support cases on closed channels.

Expected behaviour

We expect the double spent transaction to be removed and not constitently being rebroadcasted.

Actual behaviour

Sweep tx always rebroadcasted.

In general since Btcwallet has a mechnism to rebroadcast unmined transactions so do you see any reason to persist the last sweep tx and republish it at startup by the sweeper? https://github.com/lightningnetwork/lnd/blob/master/sweep/sweeper.go#L344

[guggero]

IIUC was fixed in 0.16.2, see #7448, #7599

[Roasbeef]

@guggero this is different, the sweeper will always try to rebroadcast the last sweep transaction each time. It doesn't need to though, as the wallet will pick that up, and any other sweeps are assumed to be re-offered on start up.

[Roasbeef]

What we need to do is actually start to remove the last week txn from the sweeper's DB, and also remove it here if/when we get that error:

lnd/sweep/sweeper.go

Lines 343 to 365 in bbbf7d3

	// Retrieve last published tx from database.
	lastTx, err := s.cfg.Store.GetLastPublishedTx()
	if err != nil {
	return fmt.Errorf("get last published tx: %v", err)
	}
	// Republish in case the previous call crashed lnd. We don't care about
	// the return value, because inputs will be re-offered and retried
	// anyway. The only reason we republish here is to prevent the corner
	// case where lnd goes into a restart loop because of a crashing publish
	// tx where we keep deriving new output script. By publishing and
	// possibly crashing already now, we haven't derived a new output script
	// yet.
	if lastTx != nil {
	log.Debugf("Publishing last tx %v", lastTx.TxHash())
	// Error can be ignored. Because we are starting up, there are
	// no pending inputs to update based on the publish result.
	err := s.cfg.Wallet.PublishTransaction(lastTx, "")
	if err != nil && err != lnwallet.ErrDoubleSpend {
	log.Errorf("last tx publish: %v", err)
	}
	}

As is, it tries to make minimal assumptions about what will actually be re-broadcasted, but we have a rebroadcast assumption at a higher level now.

[Roasbeef]

We also never delete any of the past sweep txid's as well. This is safe to do as we only need them to dtect our own sweeps, but after we move a conflict, then we can remove them.

[roeierez]

Any update on this? We keep getting support cases from our (Breez) users that are unable to spend their funds.

[saubyk]

@Roasbeef does the pr #7746 help address this issue?

[ziggie1984]

@Roasbeef does the pr #7746 help address this issue?

No it does not.

I try give a short overview.

What is important to point out, that this only is a problem for neutrino nodes. Normal nodes will remove this double spend tx immediately not blocking any funds.
This ties to #7800 as well.

Basic problem is that somehow the Bitcoind Backend does not send any failure in case we are broadcasting a double spend when using the BIP157 I think. This is one happens to the bitcoind backend:

2023-07-24T08:24:12Z [net] got inv: witness-tx e111cd8f28b4fa2a0439baca5585f442b5b32674bd15c183404fb9a5889a77b5  new peer=14
2023-07-24T08:24:14Z [net] Requesting tx e111cd8f28b4fa2a0439baca5585f442b5b32674bd15c183404fb9a5889a77b5 peer=14
2023-07-24T08:24:14Z [mempool] stored orphan tx e111cd8f28b4fa2a0439baca5585f442b5b32674bd15c183404fb9a5889a77b5 (mapsz 1 outsz 2)
2023-07-24T08:24:14Z [mempoolrej] e111cd8f28b4fa2a0439baca5585f442b5b32674bd15c183404fb9a5889a77b5 from peer=14 was not accepted: bad-txns-inputs-missingorspent

It will not report an error because it first stores it as orphan therefore not replying with a reject message to the lnd node. And only later not accepting it but this is never send to the client (the lnd node) did something change here in the bitcoind backend. I have the feeling that all transactions are accepted by the bitcoind backend now with the current design and nothing gets rejected at all?

Solving this problem includes 2 fixes:

We could just not republish the last transaction as referenced in this issue here:
https://github.com/lightningnetwork/lnd/blob/master/sweep/sweeper.go#L344

This would not completely fix the issue because part of the transactions are already in the "unminded tx-store" see explanation in #7800.

So we need both parts of the solution. If nobody is already working on this one, I can do that.

Still two questions I have:

1. Is the behaviour of bitcoind correct here in regards of BIP157. Why is bitcoind not rejecting this transactions in the first place (Will test with btcd later and see whether it reports correct)

2. Before removing the last-tx broadcasting I need to understand this comment by Joost referenced here:
   

   lnd/sweep/sweeper.go

   Lines 349 to 355 in bbbf7d3

   	// Republish in case the previous call crashed lnd. We don't care about
   	// the return value, because inputs will be re-offered and retried
   	// anyway. The only reason we republish here is to prevent the corner
   	// case where lnd goes into a restart loop because of a crashing publish
   	// tx where we keep deriving new output script. By publishing and
   	// possibly crashing already now, we haven't derived a new output script
   	// yet.

   
   I do not really understand this corner case, so I hesitate to remove this behaviour in the first place.

[ziggie1984]

Update regarding BTCD as a backend for the BIP157 connection:

When using btcd instead of bitcoind the double spend is recognized and the transaction removed, so it could be a short term fix to connect all breez wallets to btcd backends ? @roeierez how do breez clients decide which BIP157 peer they choose, is it hard coded or based on discovery ?

[kingonly]

Update regarding BTCD as a backend for the BIP157 connection:

When using btcd instead of bitcoind the double spend is recognized and the transaction removed, so it could be a short term fix to connect all breez wallets to btcd backends ? @roeierez how do breez clients decide which BIP157 peer they choose, is it hard coded or based on discovery ?

No discovery. We have a configuration that users can change https://doc.breez.technology/Connecting-a-Bitcoin-node-supporting-BIP-157.html