when a funding TX takes a long time to confirm, channels get stuck in syncing

[viaj3ro]

It's not a big deal but those channels have to be "unstuck" manually via restart or disconnect once the funding TX is confirmed. They don't come online automatically.

In the past, those channels sometimes got stuck for good and had to be force closed. Not sure if this is still an issue though.

[t-bast]

Did this happen for channels with lnd nodes?

Being stuck in syncing is completely independent of long confirmation of funding txs (at least on the eclair side). To get out of the syncing state we wait for the remote party to send a ChannelReestablish, I've heard reports that lnd doesn't always send them when the funding tx is not confirmed.

Can you share the logs of around the time the funding tx was confirmed? Please filter on one specific channel before sending the logs.

[viaj3ro]

it happened during the last fee spike and unfortunately I'm not 100% sure anymore which channels were affected. Will prepare logs and share here as soon as I have the issue again.

[cfromknecht]

To get out of the syncing state we wait for the remote party to send a ChannelReestablish, I've heard reports that lnd doesn't always send them when the funding tx is not confirmed.

I can confirm that LND doesn't send ChannelReestablish if the remains unbroken after funding, but they should always be sent upon reconnection even if the funding transaction hasn't confirmed. According to this comment the user suspects that the connection remained open the entire time. We should be able to confirm/deny this via @viaj3ro's logs, and continue triaging from there

[t-bast]

It's likely that the connection was kept alive the whole time, especially since @viaj3ro mentioned it working again after a forced disconnect/reconnect.
@cfromknecht if you find something in the flow that eclair does wrong (if we're not sending a message that we should, and that would unblock the situation) do let me know!

[cfromknecht]

One question I have from your prior comments, does eclair wait for ChannelReestablish to move out of syncing even if the connection is unbroken after funding flow? I'm guessing not, but just want to double check so I have an accurate mental model :P

[t-bast]

Good point, I see where you're getting at. If the connection was never broken, we don't need a ChannelReestablish to bring the channel up once the funding tx is confirmed.

It's only after a disconnection that we require receiving a ChannelReestablish to be able to bring the channel up (and get out of the syncing state).

[cfromknecht]

Okay cool, that is also my understanding of lnd's behavior so at least we agree on what the behavior should be 👍

[viaj3ro]

found the logs for this:

lightningnetwork/lnd#4368 (comment)

[t-bast]

Closing this issue (see lightningnetwork/lnd#4368) for the full details.

[viaj3ro]

Unfortunately I was finally able to reproduce this bug.

I have 4 channels that are stuck in syncing and can't get unstuck. They were all opened on the 7th, 8th and 9th of January 2021 with a fee of 3 sats/byte and got their first confirmation yesterday.

I opened 2 more channels at the time, one with 11 s/b and one with 7 s/b fee. The former got a confirmation within 24 hours and the latter after 10 days on January 17th. Both didn't get stuck. I might have had to manually disconnect and reconnect to actually bring them online though. (see lightningnetwork/lnd#4368)

The 4 that are stuck are most likely going to cost me a huge amount in force closure fees and might take months until I finally get my funds back. Any ideas, how to avoid this and get them online instead? If not, this will be quite painful, hence worth figuring out what went wrong.

My first guess would be that there didn't happen a restart of eclair until January 17th when the last of the first two channels got confirmed, which might explain why it didn't get stuck even though it took 10 days until confirmation.

What's the best way to investigate further?

[t-bast]

The issue there is that your channels took too long to confirm so the fundee "forgot" them.

There must be a limit after which a fundee will forget about a channel that wasn't confirmed, otherwise it's an easy DoS vector (because it costs resources to store information about a channel and check on-chain confirmations). Right now this limit is 5 days in eclair, and probably approximately the same thing in lnd. Since your channels took more than 2 weeks to confirm, your peer deleted the data stored on their side. Unfortunately that means they probably don't have their funding private keys anymore, so can't help you do a mutual close (or restore the channel).

I agree that this is not acceptable. But there must be some time limit enforced. I have an idea that could fix this, but it requires a spec change (and a code change in all implementations). Let me know if you think that would work for you, and if it does I'll start working on a spec PR and discuss it with other implementers:

• When opening a channel, you tell your peer the deadline before which you'll get it confirmed (for example, 1440 blocks from now)
• The fundee checks that deadline, and if they find it acceptable, tell you to move forward and open the channel: otherwise if they think the deadline is too long, they will tell you straight away and you'll stop the channel open flow
• Your node broadcasts the funding transaction, and uses CPFP if needed to ensure it confirms before the deadline you and your peer agreed on
• Your peer will keep the channel information on his side until the deadline is reached

That means you'll sometimes need to pay more fees than you'd like to get your funding tx confirmed through CPFP, but that gives you a guarantee that as long as you get it confirmed before the deadline, your peer will remember it.