Background
We have had several nodes that fail to start up when using Neutrino. This seems to have started with 0.17.0 I believe because we haven't experienced this previously. Essentially after a restart the node sits at RPC_ACTIVE indefinitely and never reaches SERVER_ACTIVE. When running a getinfo call, it reports as synced_to_chain: false. I inspected the Neutrino instances on all affected nodes and neutrino is reporting as synced and has at least 8 peers. In the logs I can see that the node seems to be syncing the chain because it continuously spits out:
2023-11-07 13:56:50.367 [INF] BTCN: Processed 1 block in the last 26m56.33s (height 815717, 2023-11-07 13:56:31 +0000 UTC)
2023-11-07 13:30:04.036 [INF] BTCN: Verified 1 filter header in the last 10m37.3s (height 815716, 2023-11-07 13:29:46 +0000 UTC)
2023-11-07 13:29:54.033 [INF] BTCN: Attempting to fetch set of un-checkpointed filters at height=815716, hash=000000000000000000043a2be219746d6751b79cc75139aaabc402f7ca7a3b6f
However, looking at closer to the restart time, it seems like it's stuck rescanning the chain. Filtering only on the LNWL logs I can see this:
2023-11-05 07:34:47.189 [WRN] LNWL: Query(0) from peer 91.134.145.202:8333 failed, rescheduling: did not get response before timeout
2023-11-05 07:34:39.188 [WRN] LNWL: Query(0) from peer 70.68.70.171:8333 failed, rescheduling: did not get response before timeout
2023-11-05 07:34:35.188 [WRN] LNWL: Query(0) failed with error: did not get response before timeout. Timing out.
2023-11-05 07:34:35.188 [WRN] LNWL: Query(0) from peer 5rljfc3fdgnhpsvuju3jsblsdktyjnpzedr264plemli2f6hjv4wutyd.onion:8333 failed, rescheduling: did not get response before timeout
2023-11-05 07:32:07.051 [INF] LNWL: Started rescan from block 00000000000000000000a6d3415f09ab5439ad2dd262c7840a4dded9be41271c (height 814836) for 14 addresses
2023-11-05 07:32:05.335 [INF] LNWL: The wallet has been unlocked without a time limit
2023-11-05 07:32:01.650 [INF] LNWL: Starting web API fee estimator
2023-11-05 07:32:01.138 [INF] LNWL: Opened wallet
And nothing meaningful happens after that. So it seems like with Neutrino LND gets stuck syncing the chain and never finishes. I don't know of a way to get more info on the status of re-syncing.
Your environment
- version of
lnd: 0.17.0
- which operating system (
uname -a on *Nix): amazon linux 2
- version of
btcd, bitcoind, or other backend: neutrino
- any other relevant environment details
Steps to reproduce
I don't know of a way to reliably reproduce this. It seems fairly random when it happens.
Expected behaviour
LND starts up like normal and is able to continue operations
Actual behaviour
LND gets stuck starting and sits for days without being able to do anything.
Output
$ lncli neutrino status
{
"active": true,
"synced": true,
"block_height": 815717,
"block_hash": "0000000000000000000219e6da7c862f6c10bf236fbde945640789ee7657e3d6",
"peers": [
"lvsf4gflp5drwy3sfvqk4lklltb4ul4uehyepvvpfvqncn6gnue5abid.onion:8333",
"4h62mftc5bo3lz5dluty3s42todzd4jjmagbbeqbbccmunmaa7h6ocyd.onion:8333",
"vpgtduxbksexfp5twhzlzmtzayxci6ui4axgp6xaoayg5o6l242v2xqd.onion:8333",
"78.203.56.219:8333",
"62.117.216.176:8333",
"svbdvvsvpvjiotkspmwas6vghpb3jwa2piuos7kixoqy2ar7xbwnz4yd.onion:8333",
"35.163.36.246:8333",
"4q6xkttr2vvycn6e4uw55feflsq3oyrmazn7tfz45yxni73yhjj42jid.onion:8333"
]
}
$ lncli getinfo
{
"version": "0.17.0-beta commit=lightning-terminal-v0.11.0-alpha",
"commit_hash": "8d7c53b2d1ccab6abc41a01b7c61e874cf194542",
"identity_pubkey": "<redacted>",
"alias": "<redacted>",
"color": "#ff5000",
"num_pending_channels": 0,
"num_active_channels": 0,
"num_inactive_channels": 6,
"num_peers": 0,
"block_height": 815717,
"block_hash": "0000000000000000000219e6da7c862f6c10bf236fbde945640789ee7657e3d6",
"best_header_timestamp": "1699365391",
"synced_to_chain": false,
"synced_to_graph": false,
"testnet": false,
"chains": [
{
"chain": "bitcoin",
"network": "mainnet"
}
],
"uris": [
],
...
}
Background
We have had several nodes that fail to start up when using Neutrino. This seems to have started with 0.17.0 I believe because we haven't experienced this previously. Essentially after a restart the node sits at
RPC_ACTIVEindefinitely and never reachesSERVER_ACTIVE. When running agetinfocall, it reports assynced_to_chain: false. I inspected the Neutrino instances on all affected nodes and neutrino is reporting as synced and has at least 8 peers. In the logs I can see that the node seems to be syncing the chain because it continuously spits out:However, looking at closer to the restart time, it seems like it's stuck rescanning the chain. Filtering only on the
LNWLlogs I can see this:And nothing meaningful happens after that. So it seems like with Neutrino LND gets stuck syncing the chain and never finishes. I don't know of a way to get more info on the status of re-syncing.
Your environment
lnd: 0.17.0uname -aon *Nix): amazon linux 2btcd,bitcoind, or other backend: neutrinoSteps to reproduce
I don't know of a way to reliably reproduce this. It seems fairly random when it happens.
Expected behaviour
LND starts up like normal and is able to continue operations
Actual behaviour
LND gets stuck starting and sits for days without being able to do anything.
Output