[bug]: Outgoing HTLC stuck "In Flight" 7,000+ blocks past expiry without Arbitrator force-closing

[TrezorHannes]

Pre-Submission Checklist

• I have searched the existing issues and believe this is a new bug.
• I am not asking a question about how to use lnd, but reporting a bug (otherwise open a discussion).

LND Version

v0.20.1-beta (commit 848b72c)

LND Configuration

[sweeper]
# Duration of the sweep batch window. The sweep is held back during the batch
# window to allow more inputs to be added and thereby lower the fee per input.
# sweeper.batchwindowduration=1000h
# sweeper.batchwindowduration=10m
; The max fee rate in sat/vb which can be used when sweeping funds. Setting
; this value too low can result in transactions not being confirmed in time,
; causing HTLCs to expire hence potentially losing funds.
sweeper.maxfeerate=1000

; The conf target to use when sweeping non-time-sensitive outputs. This is
; useful for sweeping outputs that are not time-sensitive, and can be swept at
; a lower fee rate.
sweeper.nodeadlineconftarget=1008

; An optional config group that's used for the automatic sweep fee estimation.
; The Budget config gives options to limits ones fee exposure when sweeping
; unilateral close outputs and the fee rate calculated from budgets is capped
; at sweeper.maxfeerate. Check the budget config options for more details.
sweeper.budget=5000

[sweeper.budget]

; The amount in satoshis to allocate as the budget to pay fees when sweeping
; the to_local output. If set, the budget calculated using the ratio (if set)
; will be capped at this value.
sweeper.budget.tolocal=10000

; The ratio of the value in to_local output to allocate as the budget to pay
; fees when sweeping it.
sweeper.budget.tolocalratio=0.1

; The amount in satoshis to allocate as the budget to pay fees when CPFPing a
; force close tx using the anchor output. If set, the budget calculated using
; the ratio (if set) will be capped at this value.
; sweeper.budget.anchorcpfp=

; The ratio of a special value to allocate as the budget to pay fees when 
; CPFPing a force close tx using the anchor output. The special value is the
; sum of all time-sensitive HTLCs on this commitment subtracted by their
; budgets.
; sweeper.budget.anchorcpfpratio=0.5

; The amount in satoshis to allocate as the budget to pay fees when sweeping a
; time-sensitive (first-level) HTLC. If set, the budget calculated using the
; ratio (if set) will be capped at this value.
; sweeper.budget.deadlinehtlc=

; The ratio of the value in a time-sensitive (first-level) HTLC to allocate as
; the budget to pay fees when sweeping it.
; sweeper.budget.deadlinehtlcratio=0.3

; The amount in satoshis to allocate as the budget to pay fees when sweeping a
; non-time-sensitive (second-level) HTLC. If set, the budget calculated using
; the ratio (if set) will be capped at this value.
; sweeper.budget.nodeadlinehtlc=

; The ratio of the value in a non-time-sensitive (second-level) HTLC to
; allocate as the budget to pay fees when sweeping it.
sweeper.budget.nodeadlinehtlcratio=0.1

Backend Version

Bitcoin Core RPC client version v29.0

Backend Configuration

n/A

OS/Distribution

Linux debian-nuc 6.1.0-43-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.162-1 (2026-02-08) x86_64 GNU/Linux

Bug Details & Steps to Reproduce

Environment:

• Version: v0.20.1-beta (commit 848b72c)
• Network: Mainnet
• Peer: 027831fdda22a3fdb95ac7f4a6348467720ba073edb61317c7bc965315c9e5bbe1 (Edelweiss)

Description:
An outgoing payment has been stuck "In Flight" since February 2026. The HTLC expiration height is 936,152, while the current block height is 943,886. Despite being >7,700 blocks past expiry, LND has not initiated a force-close.

The logs show a persistent loop of LND attempting to "Resume" the payment followed by "htlcswitch shutting down" or "router shutting down" during restarts, which seems to reset the arbitration timer or logic.

Key Log Snippets:

• 2026-03-06 [ERR] CRTR: Payment 8f610... with status=In Flight failed: router shutting down
• 2026-03-11 [ERR] RPCS: [/lnrpc.Lightning/DeletePayment]: payment '8f610...' has inflight HTLCs and therefore cannot be deleted: payment is in transition
• 2026-03-28 [INF] CRTR: Resuming HTLC attempt 56133813 for payment 8f610...

Actual Behavior:
Channel remains active: true. lncli listchannels shows unsettled_balance: 201477 and the expired HTLC. Manual attempts to delete the payment fail with "payment is in transition."

Expected Behavior

The ChannelArbitrator should have identified the expired HTLC and broadcast a commitment transaction to protect the funds once the grace period passed.

Note that I did not force-close the channel yet, intentionally - so we could debug together in case this is hard to replicate.

Debug Information

zgrep "8f61081a639f1ff1ceef913aee0b89ce359724d1f0d51485b109ddbf598871b7" ~/.lnd/logs/bitcoin/mainnet/lnd.log.*.gz
/home/admin/.lnd/logs/bitcoin/mainnet/lnd.log.3398.gz:2026-02-09 17:34:52.990 [WRN] CRTR: Attempt 56133810 for payment 8f61081a639f1ff1ceef913aee0b89ce359724d1f0d51485b109ddbf598871b7 failed: TemporaryChannelFailure(update=(*lnwire.ChannelUpdate1)(0xc019011970)({
/home/admin/.lnd/logs/bitcoin/mainnet/lnd.log.3398.gz:2026-02-09 17:38:42.409 [INF] RRPC: Payment stream 8f61081a639f1ff1ceef913aee0b89ce359724d1f0d51485b109ddbf598871b7 canceled
/home/admin/.lnd/logs/bitcoin/mainnet/lnd.log.3400.gz:2026-02-26 21:14:56.843 [ERR] CRTR: Error collecting result for attempt 56133813 in payment 8f61081a639f1ff1ceef913aee0b89ce359724d1f0d51485b109ddbf598871b7: htlcswitch shutting down
/home/admin/.lnd/logs/bitcoin/mainnet/lnd.log.3400.gz:2026-02-26 21:23:07.853 [INF] CRTR: Resuming payment 8f61081a639f1ff1ceef913aee0b89ce359724d1f0d51485b109ddbf598871b7
/home/admin/.lnd/logs/bitcoin/mainnet/lnd.log.3400.gz:2026-02-26 21:23:07.972 [INF] CRTR: Resuming HTLC attempt 56133813 for payment 8f61081a639f1ff1ceef913aee0b89ce359724d1f0d51485b109ddbf598871b7
/home/admin/.lnd/logs/bitcoin/mainnet/lnd.log.3400.gz:2026-02-26 23:16:50.765 [INF] RRPC: Payment stream 8f61081a639f1ff1ceef913aee0b89ce359724d1f0d51485b109ddbf598871b7 canceled
/home/admin/.lnd/logs/bitcoin/mainnet/lnd.log.3401.gz:2026-03-06 10:41:09.399 [ERR] CRTR: Error collecting result for attempt 56133813 in payment 8f61081a639f1ff1ceef913aee0b89ce359724d1f0d51485b109ddbf598871b7: htlcswitch shutting down
/home/admin/.lnd/logs/bitcoin/mainnet/lnd.log.3401.gz:2026-03-06 10:41:09.700 [ERR] CRTR: Payment 8f61081a639f1ff1ceef913aee0b89ce359724d1f0d51485b109ddbf598871b7 with status=In Flight failed: router shutting down
/home/admin/.lnd/logs/bitcoin/mainnet/lnd.log.3401.gz:2026-03-06 10:41:09.708 [ERR] CRTR: Resuming payment 8f61081a639f1ff1ceef913aee0b89ce359724d1f0d51485b109ddbf598871b7 failed: router shutting down
/home/admin/.lnd/logs/bitcoin/mainnet/lnd.log.3401.gz:2026-03-06 11:06:51.302 [INF] CRTR: Resuming payment 8f61081a639f1ff1ceef913aee0b89ce359724d1f0d51485b109ddbf598871b7
/home/admin/.lnd/logs/bitcoin/mainnet/lnd.log.3401.gz:2026-03-06 11:06:51.307 [INF] CRTR: Resuming HTLC attempt 56133813 for payment 8f61081a639f1ff1ceef913aee0b89ce359724d1f0d51485b109ddbf598871b7
/home/admin/.lnd/logs/bitcoin/mainnet/lnd.log.3402.gz:2026-03-11 17:34:12.064 [INF] RPCS: [DeletePayment] payment_identifier=8f61081a639f1ff1ceef913aee0b89ce359724d1f0d51485b109ddbf598871b7, failed_htlcs_only=false
/home/admin/.lnd/logs/bitcoin/mainnet/lnd.log.3402.gz:2026-03-11 17:34:12.070 [ERR] RPCS: [/lnrpc.Lightning/DeletePayment]: payment '8f61081a639f1ff1ceef913aee0b89ce359724d1f0d51485b109ddbf598871b7' has inflight HTLCsand therefore cannot be deleted: payment is in transition
/home/admin/.lnd/logs/bitcoin/mainnet/lnd.log.3404.gz:2026-03-28 22:16:10.047 [INF] CRTR: Resuming payment 8f61081a639f1ff1ceef913aee0b89ce359724d1f0d51485b109ddbf598871b7
/home/admin/.lnd/logs/bitcoin/mainnet/lnd.log.3404.gz:2026-03-28 22:16:10.108 [INF] CRTR: Resuming HTLC attempt 56133813 for payment 8f61081a639f1ff1ceef913aee0b89ce359724d1f0d51485b109ddbf598871b7


lncli trackpayment 8f61081a639f1ff1ceef913aee0b89ce359724d1f0d51485b109ddbf598871b7
+-------------------------------------+--------------+--------------+--------------+-------+----------+---------------------+---------------------------------------------------------------------+
| HTLC_STATE                          | ATTEMPT_TIME | RESOLVE_TIME | RECEIVER_AMT | FEE   | TIMELOCK | CHAN_OUT            | ROUTE                                                               |
+-------------------------------------+--------------+--------------+--------------+-------+----------+---------------------+---------------------------------------------------------------------+
| TEMPORARY_CHANNEL_FAILURE @ 3rd hop |        0.098 |       70.506 | 201471       | 4.827 |   936252 | 1015825598867636232 | Edelweiss🏔️->God Bless ₿itcoin->speed3->LQWD-Australia->HODLmeTight |
| IN_FLIGHT                           |       70.560 |            - | 201471       | 6.648 |   936152 | 1015825598867636232 | Edelweiss🏔️->BTC_USA->antistate->LQWD-Australia->HODLmeTight        |
+-------------------------------------+--------------+--------------+--------------+-------+----------+---------------------+---------------------------------------------------------------------+
Amount + fee:   0 + 0 sat
Payment hash:   8f61081a639f1ff1ceef913aee0b89ce359724d1f0d51485b109ddbf598871b7
Payment status: IN_FLIGHT
^C[lncli] rpc error: code = Canceled desc = context canceled
admin@debian-nuc:~$ lncli listpeers | grep -B 2 -A 30 "027831fdda22a3fdb95ac7f4a6348467720ba073edb61317c7bc965315c9e5bbe1"
        },
        {
            "pub_key": "027831fdda22a3fdb95ac7f4a6348467720ba073edb61317c7bc965315c9e5bbe1",
            "address": "10.8.1.1:42270",
            "bytes_sent": "1168",
            "bytes_recv": "1004",
            "sat_sent": "41299869",
            "sat_recv": "41541530",
            "inbound": true,
            "ping_time": "-1",
            "sync_type": "PASSIVE_SYNC",
            "features": {
                "0": {
                    "name": "data-loss-protect",
                    "is_required": true,
                    "is_known": true
                },
                "5": {
                    "name": "upfront-shutdown-script",
                    "is_required": false,
                    "is_known": true
                },
                "7": {
                    "name": "gossip-queries",
                    "is_required": false,
                    "is_known": true
                },
                "8": {
                    "name": "tlv-onion",
                    "is_required": true,
                    "is_known": true
                },
                "12": {
--
                {
                    "timestamp": "1775463787",
                    "error": "disconnecting 027831fdda22a3fdb95ac7f4a6348467720ba073edb61317c7bc965315c9e5bbe1@10.8.1.1:35908, reason: server: DisconnectPeer called"
                }
            ],
            "flap_count": 14153,
            "last_flap_ns": "1775463788257531398",
            "last_ping_payload": ""
        },
        {
            "pub_key": "022387c7a6b5ebbf53960b56e88fae0ed0aa8cc1fa58158492b135b703c9a2cf97",
            "address": "85.209.52.121:9735",
            "bytes_sent": "217312508",
            "bytes_recv": "23680016",
            "sat_sent": "45681142",
            "sat_recv": "46459919",
            "inbound": false,
            "ping_time": "23433",
            "sync_type": "PASSIVE_SYNC",
            "features": {
                "0": {
                    "name": "data-loss-protect",
                    "is_required": true,
                    "is_known": true
                },
                "5": {
                    "name": "upfront-shutdown-script",
                    "is_required": false,
                    "is_known": true
                },
                "7": {
                    "name": "gossip-queries",
                    "is_required": false,
admin@debian-nuc:~$ lncli listchannels | grep -i "dc3b96ceec73a60f69cfe4ff78d9bf6ce959db60df859146ddfb9c679c4f722f" -B 5 -A 40
        },
        {
            "active": true,
            "remote_pubkey": "027831fdda22a3fdb95ac7f4a6348467720ba073edb61317c7bc965315c9e5bbe1",
            "channel_point": "27724f9c679cfbdd469185df60db59e96cbfd978ffe4cf690fa673ecce963bdc:8",
            "chan_id": "dc3b96ceec73a60f69cfe4ff78d9bf6ce959db60df859146ddfb9c679c4f722f",
            "scid": "1015825598867636232",
            "scid_str": "923888x1601x8",
            "capacity": "3000000",
            "local_balance": "40183",
            "remote_balance": "2757330",
            "commit_fee": "350",
            "commit_weight": "1288",
            "fee_per_kw": "270",
            "unsettled_balance": "201477",
            "total_satoshis_sent": "41299869",
            "total_satoshis_received": "41541530",
            "num_updates": "177346",
            "pending_htlcs": [
                {
                    "incoming": false,
                    "amount": "201477",
                    "hash_lock": "8f61081a639f1ff1ceef913aee0b89ce359724d1f0d51485b109ddbf598871b7",
                    "expiration_height": 936152,
                    "htlc_index": "71447",
                    "forwarding_channel": "0",
                    "forwarding_htlc_index": "0",
                    "locked_in": true
                }
            ],
            "csv_delay": 360,
            "private": false,
            "initiator": false,
            "chan_status_flags": "ChanStatusDefault",
            "local_chan_reserve_sat": "30000",
            "remote_chan_reserve_sat": "30000",
            "static_remote_key": false,
            "commitment_type": "ANCHORS",
            "lifetime": "731253",
            "uptime": "36",
            "close_address": "",
            "push_amount_sat": "0",
            "thaw_height": 0,
            "local_constraints": {
                "csv_delay": 360,
                "chan_reserve_sat": "30000",

Environment

• LND on baremetal via systemd
• LNDg accessing LND via gRPC from remote via wireguard tunnel
• litloop running on the same machine, but not integrated. Accessing via rpcmiddleware.enable=true

[Roasbeef]

/home/admin/.lnd/logs/bitcoin/mainnet/lnd.log.3400.gz:2026-02-26 21:14:56.843 [ERR] CRTR: Error collecting result for attempt 56133813 in payment 8f61081a639f1ff1ceef913aee0b89ce359724d1f0d51485b109ddbf598871b7: htlcswitch shutting down

It looks like the daemon is trying to shutdown, but something is stopping it. Do you have a profile? Did a restart resolve the issue?

[Roasbeef]

Do you have log fragments for the CNCT or HSWC sub systems?

[TrezorHannes]

It looks like the daemon is trying to shutdown, but something is stopping it. Do you have a profile? Did a restart resolve the issue?

Hey @Roasbeef - thank you for coming back to me so quickly. no profile unfortunately. I have done 2-3 restarts meanwhile, and the bug happened when I still ran LND v0.19.3-beta. I've upgraded 2 weeks ago and restarted, but the shutdown process is not re-triggered as far as I can see in the logs.

So I'm afraid even changing the profile or debug level won't give us much more.
Hence:

Do you have log fragments for the CNCT or HSWC sub systems?

Only debuglevel=info - sorry.

[Roasbeef]

Only debuglevel=info - sorry.

That should be fine, so you grep'd for those two strings and found nothing?

You can use lncli debuglevel to bump up CNCT to at least debug. On that log level, every block it'll print out the decision to go to chain or not:

lnd/contractcourt/channel_arbitrator.go

Lines 1856 to 1892 in 167f03b

	log.Debugf("ChannelArbitrator(%v): checking commit chain actions at "+
	"height=%v, in_htlc_count=%v, out_htlc_count=%v",
	c.cfg.ChanPoint, height,
	len(htlcs.incomingHTLCs), len(htlcs.outgoingHTLCs))
	actionMap := make(ChainActionMap)
	// First, we'll make an initial pass over the set of incoming and
	// outgoing HTLC's to decide if we need to go on chain at all.
	haveChainActions := false
	for _, htlc := range htlcs.outgoingHTLCs {
	// We'll need to go on-chain for an outgoing HTLC if it was
	// never resolved downstream, and it's "close" to timing out.
	//
	// TODO(yy): If there's no corresponding incoming HTLC, it
	// means we are the first hop, hence the payer. This is a
	// tricky case - unlike a forwarding hop, we don't have an
	// incoming HTLC that will time out, which means as long as we
	// can learn the preimage, we can settle the invoice (before it
	// expires?).
	toChain := c.shouldGoOnChain(
	htlc, c.cfg.OutgoingBroadcastDelta, height,
	)
	if toChain {
	// Convert to int64 in case of overflow.
	remainingBlocks := int64(htlc.RefundTimeout) -
	int64(height)
	log.Infof("ChannelArbitrator(%v): go to chain for "+
	"outgoing htlc %x: timeout=%v, amount=%v, "+
	"blocks_until_expiry=%v, broadcast_delta=%v",
	c.cfg.ChanPoint, htlc.RHash[:],
	htlc.RefundTimeout, htlc.Amt, remainingBlocks,
	c.cfg.OutgoingBroadcastDelta,
	)
	}

I've upgraded 2 weeks ago and restarted, but the shutdown process is not re-triggered as far as I can see in the logs.

So the logs above are a post restart state?

[ziggie1984]

whats your value for: payments-expiration-grace-period= ?

[TrezorHannes]

@Roasbeef

That should be fine, so you grep'd for those two strings and found nothing?

I only zgrep'ed for the Payment Hash zgrep "8f61081a639f1ff1ceef913aee0b89ce359724d1f0d51485b109ddbf598871b7" which goes back to 9th of Feb. I've shared all what my gzipped logs have wrt this hash.

You can use lncli debuglevel to bump up CNCT to at least debug. On that log level, every block it'll print out the decision to go to chain or not:

Will do now and report back

So the logs above are a post restart state?

Partially. I restarted the first time aprox 2-3 weeks after the blockheight expiration moved into negative.
So my hunch is:

• 02-09 something broke
• 02-26 first restart
• 03-06 second restart
• 03-28 third restart (full reboot) with upgrade from LND 19 to current version

@ziggie1984

whats your value for: payments-expiration-grace-period= ?

Ouch - is that it? Because it's an outgoing, no risk, no sweep?

cat .lnd/lnd.conf | grep payments-expiration-grace-period
payments-expiration-grace-period=10000h

[TrezorHannes]

Doing a restart with payments-expiration-grace-period=500h now and will report back.

Restart finished, this didn't go on chain:

grepcat 8f61081a639f1ff1ceef913aee0b89ce359724d1f0d51485b109ddbf598871b7
2026-04-06 10:23:19.357 [INF] RRPC: Payment stream 8f61081a639f1ff1ceef913aee0b89ce359724d1f0d51485b109ddbf598871b7 canceled
2026-04-06 23:12:10.102 [ERR] CRTR: Error collecting result for attempt 56133813 in payment 8f61081a639f1ff1ceef913aee0b89ce359724d1f0d51485b109ddbf598871b7: htlcswitch shutting down
2026-04-06 23:43:04.123 [INF] CRTR: Resuming payment 8f61081a639f1ff1ceef913aee0b89ce359724d1f0d51485b109ddbf598871b7
2026-04-06 23:43:04.125 [INF] CRTR: Resuming HTLC attempt 56133813 for payment 8f61081a639f1ff1ceef913aee0b89ce359724d1f0d51485b109ddbf598871b7

I've just changed lncli debuglevel --level=CNCT=debug and restart a second time tomorrow

[ziggie1984]

no restart needed, you should see the logs for this particular channel as soon as a new block comes in.

Could you also get the logs from your channel peer to understand why he is not failing this HTLC ?

[TrezorHannes]

no restart needed, you should see the logs for this particular channel as soon as a new block comes in.

Could you also get the logs from your channel peer to understand why he is not failing this HTLC ?

I've asked him. Will let you know

[TrezorHannes]

channelpoint_grep-CNCT=debug.log

@ziggie1984 uploaded log, grepped the channel-point, not the payment hash.

Most telling pattern:

2026-04-07 20:08:40.731 [DBG] CNCT: ChannelArbitrator(27724f9c679cfbdd469185df60db59e96cbfd978ffe4cf690fa673ecce963bdc:8): received new block: height=944077, processing...
2026-04-07 20:08:40.731 [INF] CNCT: ChannelArbitrator(27724f9c679cfbdd469185df60db59e96cbfd978ffe4cf690fa673ecce963bdc:8) no close event
2026-04-07 20:08:40.731 [DBG] CNCT: ChannelArbitrator(27724f9c679cfbdd469185df60db59e96cbfd978ffe4cf690fa673ecce963bdc:8): attempting state step with trigger=chainTrigger from state=StateDefault at height=944077
2026-04-07 20:08:40.731 [DBG] CNCT: ChannelArbitrator(27724f9c679cfbdd469185df60db59e96cbfd978ffe4cf690fa673ecce963bdc:8): examining active HTLCs in block 944077, confCommitSet: nil
2026-04-07 20:08:40.733 [DBG] CNCT: ChannelArbitrator(27724f9c679cfbdd469185df60db59e96cbfd978ffe4cf690fa673ecce963bdc:8): checking commit chain actions at height=944077, in_htlc_count=0, out_htlc_count=1
2026-04-07 20:08:40.733 [DBG] CNCT: ChannelArbitrator(27724f9c679cfbdd469185df60db59e96cbfd978ffe4cf690fa673ecce963bdc:8): no actions for chain trigger, terminating
2026-04-07 20:08:40.733 [DBG] CNCT: ChannelArbitrator(27724f9c679cfbdd469185df60db59e96cbfd978ffe4cf690fa673ecce963bdc:8): terminating at state=StateDefault
2026-04-07 20:09:33.279 [DBG] CNCT: ChainWatcher(27724f9c679cfbdd469185df60db59e96cbfd978ffe4cf690fa673ecce963bdc:8) received blockbeat 944078

[ziggie1984]

ok I know where the problem is, the time always referes to startup time, hence when you set it to 500h, it will force-close after 500hours after restart.

[TrezorHannes]

I'll close this and reopen, if the FC doesn't get in 500hrs post reboot.

Thanks for looking into this