[bug]: Payment stuck in IN_FLIGHT

[cjc21]

Background

There is an increasing amount of payments getting stuck in IN_FLIGHT for a long time. The only way to resolve them is to restart the node it seems. We currently have 6 outstanding IN_FLIGHT payments.

Your environment

0.19.2-beta commit=v0.19.2-beta
Ubuntu x86_64
Bitcoin Core version v0.21.1

Steps to reproduce

| HTLC_STATE | ATTEMPT_TIME | RESOLVE_TIME | RECEIVER_AMT | FEE | TIMELOCK | CHAN_OUT | ROUTE

example 1:
| IN_FLIGHT | 59.107 | - | 200000 | 1.2 | 908068 | |

example 2:
| IN_FLIGHT | 36.310 | - | 194380.5 | 1.194 | 908063 | |

Expected behaviour

Status should be updated when payment has failed.

Actual behaviour

Payments continue to be stuck in IN_FLIGHT

[ziggie1984]

Can you share some logs, for a specific payment hash which is "stuck"

[cjc21]

@ziggie1984 I will try my best to share the relevant logs for one instance that happened about a week ago.

| HTLC_STATE                          | ATTEMPT_TIME | RESOLVE_TIME | RECEIVER_AMT | FEE   | TIMELOCK | CHAN_OUT           |
| IN_FLIGHT                           |       36.310 |            - | 194380.5     | 1.194 |   908063 | 961105104226942977 |

There are about 22 failed payment attempts, including just a few of them here. Let me know if you want the full list.

2025-07-30 16:20:59.305 [WRN] CRTR: Attempt 261637 for payment 7829e2602ef11e34c472dfaee0703d5125ac7cf914fc04f3572f24e4e55ce93b failed: TemporaryChannelFailure(update=(*lnwire.ChannelUpdate1)(0xc00a318840)({
 Signature: (lnwire.Sig) {
  bytes: ([64]uint8) (len=64 cap=64) {
   00000000  01 51 6d 5f 22 14 15 71  55 01 30 cb 21 dc 67 ed  |.Qm_"..qU.0.!.g.|
   00000010  81 aa 69 9a a4 71 d6 45  a8 cd fb ce ce 1e 4a 79  |..i..q.E......Jy|
   00000020  0f 3b 78 87 bc ae 56 19  f9 8f c3 63 07 48 69 bc  |.;x...V....c.Hi.|
   00000030  9e 1c 12 fb cf 1a 66 33  bf 88 08 d7 e1 70 18 67  |......f3.....p.g|
  },
  sigType: (lnwire.sigType) 0
 },
 ChainHash: (chainhash.Hash) (len=32 cap=32) 000000000019d6689c085ae165831e934ff763ae46a2a6c172b3f1b60a8ce26f,
 ShortChannelID: (lnwire.ShortChannelID) 854922:256:1,
 Timestamp: (uint32) 1753885974,
 MessageFlags: (lnwire.ChanUpdateMsgFlags) 00000001,
 ChannelFlags: (lnwire.ChanUpdateChanFlags) 00000000,
 TimeLockDelta: (uint16) 144,
 HtlcMinimumMsat: (lnwire.MilliSatoshi) 1000 mSAT,
 BaseFee: (uint32) 1000,
 FeeRate: (uint32) 1,
 HtlcMaximumMsat: (lnwire.MilliSatoshi) 198000000000 mSAT,
 ExtraOpaqueData: (lnwire.ExtraOpaqueData) {
 }
})
)@2
2025-07-30 16:21:02.692 [WRN] CRTR: Attempt 261638 for payment 7829e2602ef11e34c472dfaee0703d5125ac7cf914fc04f3572f24e4e55ce93b failed: TemporaryChannelFailure(update=(*lnwire.ChannelUpdate1)(0xc00772f810)({
 Signature: (lnwire.Sig) {
  bytes: ([64]uint8) (len=64 cap=64) {
   00000000  d7 87 f8 0a dc 06 aa 34  7d 28 bc f4 45 7b 1f 04  |.......4}(..E{..|
   00000010  85 50 df 46 13 63 0d 0b  08 52 b9 dd 5f 3a 30 17  |.P.F.c...R.._:0.|
   00000020  36 c3 76 43 2b cb ad 01  2d 97 d8 2e 16 21 72 14  |6.vC+...-....!r.|
   00000030  42 56 8c 76 a1 e8 4e 1b  3d 3f 24 ff e1 0f 64 b3  |BV.v..N.=?$...d.|
  },
  sigType: (lnwire.sigType) 0
 },
 ChainHash: (chainhash.Hash) (len=32 cap=32) 000000000019d6689c085ae165831e934ff763ae46a2a6c172b3f1b60a8ce26f,
 ShortChannelID: (lnwire.ShortChannelID) 879724:226:0,
 Timestamp: (uint32) 1753869304,
 MessageFlags: (lnwire.ChanUpdateMsgFlags) 00000001,
 ChannelFlags: (lnwire.ChanUpdateChanFlags) 00000000,
 TimeLockDelta: (uint16) 40,
 HtlcMinimumMsat: (lnwire.MilliSatoshi) 1000 mSAT,
 BaseFee: (uint32) 1000,
 FeeRate: (uint32) 1,
 HtlcMaximumMsat: (lnwire.MilliSatoshi) 15840000000 mSAT,
 ExtraOpaqueData: (lnwire.ExtraOpaqueData) {
 }
})
)@4
2025-07-30 16:21:03.500 [WRN] CRTR: Attempt 261639 for payment 7829e2602ef11e34c472dfaee0703d5125ac7cf914fc04f3572f24e4e55ce93b failed: TemporaryChannelFailure(update=(*lnwire.ChannelUpdate1)(0xc0136e4370)({
 Signature: (lnwire.Sig) {
  bytes: ([64]uint8) (len=64 cap=64) {
   00000000  fe 0d 21 3f bc ba 7f ef  d2 95 8e 41 e8 69 9b 49  |..!?.......A.i.I|
   00000010  d6 28 4e 91 4e 1a 12 a8  d1 58 06 ea 10 d9 56 8f  |.(N.N....X....V.|
   00000020  0e f4 1b db 89 a6 b3 80  22 ff 14 69 fa 37 e1 e3  |........"..i.7..|
   00000030  63 32 60 23 72 37 e7 48  61 e0 d2 76 d3 3d bb d7  |c2`#r7.Ha..v.=..|
  },
  sigType: (lnwire.sigType) 0
 },
 ChainHash: (chainhash.Hash) (len=32 cap=32) 000000000019d6689c085ae165831e934ff763ae46a2a6c172b3f1b60a8ce26f,
 ShortChannelID: (lnwire.ShortChannelID) 872226:1997:0,
 Timestamp: (uint32) 1753873273,
 MessageFlags: (lnwire.ChanUpdateMsgFlags) 00000001,
 ChannelFlags: (lnwire.ChanUpdateChanFlags) 00000000,
 TimeLockDelta: (uint16) 80,
 HtlcMinimumMsat: (lnwire.MilliSatoshi) 1000 mSAT,
 BaseFee: (uint32) 1000,
 FeeRate: (uint32) 1,
 HtlcMaximumMsat: (lnwire.MilliSatoshi) 16533000000 mSAT,
 ExtraOpaqueData: (lnwire.ExtraOpaqueData) {
 }
})
)@2
2025-07-30 16:21:05.173 [WRN] CRTR: Attempt 261640 for payment 7829e2602ef11e34c472dfaee0703d5125ac7cf914fc04f3572f24e4e55ce93b failed: TemporaryChannelFailure(update=(*lnwire.ChannelUpdate1)(0xc0136e5e40)({
 Signature: (lnwire.Sig) {
  bytes: ([64]uint8) (len=64 cap=64) {
   00000000  7a c2 19 84 bb 92 81 8a  bb cc 3a bd 14 cb a2 1d  |z.........:.....|
   00000010  a4 d5 69 ac 15 71 7c b0  c8 b9 51 65 c0 b0 b3 d7  |..i..q|...Qe....|
   00000020  4b 4f c2 41 93 ff 6c db  fa d2 c6 63 1f 4c 52 54  |KO.A..l....c.LRT|
   00000030  19 cf d5 72 99 2f c1 b5  e9 1f 57 4b 70 b1 42 8f  |...r./....WKp.B.|
  },
  sigType: (lnwire.sigType) 0
 },
 ChainHash: (chainhash.Hash) (len=32 cap=32) 000000000019d6689c085ae165831e934ff763ae46a2a6c172b3f1b60a8ce26f,
 ShortChannelID: (lnwire.ShortChannelID) 829576:458:0,
 Timestamp: (uint32) 1753879944,
 MessageFlags: (lnwire.ChanUpdateMsgFlags) 00000001,
 ChannelFlags: (lnwire.ChanUpdateChanFlags) 00000000,
 TimeLockDelta: (uint16) 80,
 HtlcMinimumMsat: (lnwire.MilliSatoshi) 1000 mSAT,
 BaseFee: (uint32) 0,
 FeeRate: (uint32) 0,
 HtlcMaximumMsat: (lnwire.MilliSatoshi) 9900000000 mSAT,
 ExtraOpaqueData: (lnwire.ExtraOpaqueData) {
 }
})
)@3

A few final logs related to this payment hash

2025-07-30 16:21:58.378 [WRN] CRTR: Payment attempt not completed before context timeout, id=7829e2602ef11e34c472dfaee0703d5125ac7cf914fc04f3572f24e4e55ce93b
2025-07-30 16:21:58.459 [ERR] CRTR: Payment 7829e2602ef11e34c472dfaee0703d5125ac7cf914fc04f3572f24e4e55ce93b with status=In Flight failed: payment has failure reason
2025-07-30 16:21:58.459 [ERR] CRTR: Payment 7829e2602ef11e34c472dfaee0703d5125ac7cf914fc04f3572f24e4e55ce93b failed: payment has failure reason
2025-07-30 16:21:58.459 [ERR] CRTR: Error collecting result for attempt 261651 in payment 7829e2602ef11e34c472dfaee0703d5125ac7cf914fc04f3572f24e4e55ce93b: payment lifecycle exiting

Quick question: the timelock for this payment is 908063. Considering we are way past that block and lnd is still reporting that the payment is in_flight - can we infer that something is "wrong" here? (what I meant by "stuck")

[ziggie1984]

Quick question: the timelock for this payment is 908063. Considering we are way past that block and lnd is still reporting that the payment is in_flight - can we infer that something is "wrong" here? (what I meant by "stuck")

Does this payment at least resolve after restart ? But yeah this "stuck" means that the payment still needs to get the report from the network switch, however if you don't see any HTLC on your channel related to this hash, you can be sure that there are no "stuck" live HTLC, but just the payment db status not yet updated.

[cjc21]

Yes it did resolve after a restart

Amount + fee:   0 + 0 sat
Payment hash:   7829e2602ef11e34c472dfaee0703d5125ac7cf914fc04f3572f24e4e55ce93b
Payment status: FAILED, reason: FAILURE_REASON_TIMEOUT

[ziggie1984]

Try restricting the number of shards when doing a payment until a fix is out and maybe increase the timeout for a payment as well.