[bug]:LND in litd Fails to Start Graph Sync with bitcoind Backend after migration from neutrino

[racarr202]

Background

After migrating a node from a neutrino backend to a bitcoind backend within a litd environment, the LND sub-server fails to become fully operational.

The node starts, the wallet can be unlocked, and ZMQ connections to bitcoind are successful (evidenced by "Received new block notification" log entries in lnd.log). However, the channel graph sync never begins. The DISC subsystem shows no activity on startup, and lncli getinfo perpetually reports synced_to_graph: false. Additionally, any peer connection attempts fail with lnd.log indicating server is still in the process of starting

The problem is immediately resolved by switching the LND backend configuration from bitcoind back to neutrino.

Your environment

version of lnd: 0.19.1-beta (integrated in litd 0.15.0)
which operating system: ubuntu 24.04
version of btcd, bitcoind, or other backend: v28.1.knots20250305

Steps to reproduce

Configure litd to run an integrated LND node with neutrino backend.
Switch from neutrino server to bitcoind server providing valid RPC credentials and ZMQ addresses.

Expected behaviour

After the wallet is unlocked, LND should begin syncing the channel graph by making RPC calls to bitcoind. The logs should show activity from the DISC subsystem. After some time, lncli getinfo should report synced_to_graph: true, and the node should become fully operational.

Actual behaviour

After the wallet is unlocked, the LND sub-server receives new block notifications via ZMQ but never starts the graph sync. The logs show no startup activity from the DISC subsystem. lncli getinfo perpetually shows synced_to_graph: false. Any attempts to use network features like lncli connect fail with the error rpc error: code = Unknown desc = server is still in the process of starting. The node never becomes fully operational with the bitcoind backend.

[guggero]

I think this is somewhat expected and should eventually fix itself. With Neutrino, the gossiped channels are never validated on-chain, because that would take way too long by fetching all blocks from the network (see routing.assumechanvalid and neutrino.validatechannels configuration items for more explanation). But now with the bitcoind backend, the node can look up each channel funding transaction in the chain and validate it.
That just takes quite a while, depending on how fast your disk and CPU are.

You can either wait or turn on routing.assumechanvalid=true when switching to bitcoind (it's turned on by default with the Neutrino backend).

[racarr202]

I'm not surprised by your answer but I am suspicious something is amiss.

I've changed lnd.bitcoin.node=bitcoind and provided all the necessary related settings (rpcuser, rpcpass, zmqpubrawblock, zmqpubrawtx).
I've also set debuglevel=NTFR=debug,NTFN=info to try and see more of what's happening. I get lots of info from NTFN but only the below from NTFR. If lnd was indeed getting lots of validated block info from bitcoind wouldn't that be seen in the logs?

[DBG] NTFR: ChainNotifier RPC server successfully registered with root gRPC server
[DBG] NTFR: ChainKit RPC server successfully registered with root gRPC server
[DBG] NTFR: ChainNotifier REST server successfully registered with root REST server
[DBG] NTFR: ChainKit REST server successfully registered with root REST server

As an FYI, per your suggestion, when I set lnd.routing.assumechanvalid=true, the node starts, indicates "synced_to_chain": true, "synced_to_graph": true, and my channels all showing as active.

[guggero]

I'd try with BTCN=debug, but I'm not sure there is much logging around this. Or perhaps CRTR=debug since the router is doing the validation.

[racarr202]

The log says this, [INF] CRTR: FilteredChainView starting which sounds about right.

I'll just leave it for a day or so and see if it comes good.

[racarr202]

I'm pretty confident something is ammis.

When I set my debug=rpc in bitcoin.conf and monitor the bitcoind debug logs I am not getting any regular RPC requests from lnd as part of this synchronisation process.

I have proven that RPC calls do get through though as when I manually running lncli chain getblock this correctly appears in bitcoind's logs as [rpc] ThreadRPCServer method=getblock user=bitcoinrpc.

When I set lnd.routing.assumechanvalid=true I get a few hundred RPC calls, mostly getblock and getblockhash when the node starts up then a call to uptime and getpeerinfo once a minute.

[Roasbeef]

If you get us a goroutine profile, then we should be able to see what's going on.

I'd also double check that you have the correct zmq+rpc authentication details for the bitcoind node.

[racarr202]

If you get us a goroutine profile, then we should be able to see what's going on.

goroutine.prof.txt

I'd also double check that you have the correct zmq+rpc authentication details for the bitcoind node.

When I run lncli chain getblock I see the following at the bitcoind end [rpc] ThreadRPCServer method=getblock user=bitcoinrpc. As for zmq, I'm regularly getting new block notificaitons from the NTFN subsystem. Given this, is there any chance my zmq or rpc details could be incorrect?

[Roasbeef]

I see that the node is waiting on a block from bitcoind, as it's now going through to fetch every single block to validate:

goroutine 949 [chan receive]:
github.com/lightningnetwork/lnd/routing/chainview.(*BitcoindFilteredChainView).FilterBlock(0xc000e93000, 0xc00d5aefa0)
	github.com/lightningnetwork/lnd@v0.19.1-beta/routing/chainview/bitcoind.go:231 +0x129
github.com/lightningnetwork/lnd/graph.(*Builder).syncGraphWithChain(0xc0005bf3e0)
	github.com/lightningnetwork/lnd@v0.19.1-beta/graph/builder.go:421 +0x731
github.com/lightningnetwork/lnd/graph.(*Builder).Start(0xc000d14b28?)
	github.com/lightningnetwork/lnd@v0.19.1-beta/graph/builder.go:258 +0x3ae
github.com/lightningnetwork/lnd.(*server).Start.func1()
	github.com/lightningnetwork/lnd@v0.19.1-beta/server.go:2379 +0x1d71
sync.(*Once).doSlow(0xc00064b888?, 0xc00110ff00?)
	sync/once.go:76 +0xb4
sync.(*Once).Do(...)
	sync/once.go:67
github.com/lightningnetwork/lnd.(*server).Start(0xc00064b888, {0x4a74638, 0xc000d7d110})
	github.com/lightningnetwork/lnd@v0.19.1-beta/server.go:2235 +0xb6
github.com/lightningnetwork/lnd.Main.func12()
	github.com/lightningnetwork/lnd@v0.19.1-beta/lnd.go:761 +0x28
created by github.com/lightningnetwork/lnd.Main in goroutine 201
	github.com/lightningnetwork/lnd@v0.19.1-beta/lnd.go:760 +0x3ac5

[racarr202]

I'm happy to close this issue as it's now resolved on my end.

After some more debugging on my end, the hang appears to be correlated with the blocksonly=1 setting in my bitcoin.conf.

Observations:

The Problem: My node would consistently hang on the initial graph sync after migrating from Neutrino. A goroutine dump confirmed a deadlock state.

The Change: The only effective change I made was setting blocksonly=0 in bitcoin.conf.

The Result: After that change, the deadlock was gone and the node synced to the graph successfully.

Suggestion:
Based on this, it seems lnd may not be compatible with a bitcoind backend running in blocksonly mode. To help others, it might be worth investigating if lnd can detect this configuration on startup and provide a warning, rather than silently hanging.

[saubyk]

Closing the issue, as the problem has been resolved per the above comment.