[bug]: Memory leak when running bitcoind backend in polling mode

[litch]

Background

We're working on running against bitcoind backends for various reasons, and are noticing a pretty severe memory leak when running against bitcoind with rpcpolling.

I've ensured we're running bitcoin 25 (following discussion around this PR).

I've tried running a slow polling frequency (1 minute) and the leak is just slower with the longer duration. I haven't dug too deeply into the code to debug but it feels like it's just not cleaning up connections after each request.

I'll have to do some digging to get a profile if you need it, but @alexbosworth mentioned this is a (kind of) known issue and to just ping you here.

Your environment

• version of lnd: 0.17.3

• which operating system (uname -a on *Nix): Linux 00ea31e9-92e7-433e-a41f-032e4267b9aan-7c975b685c-lgsfk 5.4.253-167.359.amzn2.aarch64 Fix name typo in README #1 SMP Tue Aug 15 21:40:30 UTC 2023 aarch64 aarch64 aarch64 GNU/Linux

• version of btcd, bitcoind, or other backend: bitcoind 25.1

• any other relevant environment details

Steps to reproduce

lit.conf
lnd.bitcoin.node=bitcoind
# Bitcoind
lnd.bitcoind.rpchost=bitcoind-mainnet.bitcoind.svc.cluster.local:8332
lnd.bitcoind.rpcuser=<>
lnd.bitcoind.rpcpass=<>
lnd.bitcoind.zmqpubrawblock=
lnd.bitcoind.zmqpubrawtx=
lnd.bitcoind.rpcpolling=true
lnd.bitcoind.blockpollinginterval=5s
lnd.bitcoind.txpollinginterval=5s

Expected behaviour

Memory consumption should be stable.

Actual behaviour

[Roasbeef]

Can you get a memory profile? Then we'll see exactly where the memory is going. Both svg and flame graph would be helpful.

You also want to reduce that mempool poll interval you have, as otherwise, it'll fetch the entire mempool every 5 seconds. Today the mempool can swell up to 1 GB or so depending on your settings, so it can cause memory to spike.

Compared to zmq, polling uses more memory, as for unconfirmed transactions, it needs to fetch the mempool to detect those. For zmq, all newly added txns are sent directly over the socket.

[gkrizek]

I was able to replicate and get a heap dump. I'm attaching lnd-rpcpolling-1.dump and lnd-rpcpolling-2.dump (in a zip). The first one was taken shortly after start up. The second one was taken ~8 hours later when memory was obviously higher. This was taken with lnd.bitcoind.blockpollinginterval=5s and lnd.bitcoind.txpollinginterval=5s to make the run up go faster for the sake of debugging.

lnd-rpcpolling.zip

[gkrizek]

@Roasbeef, that makes sense that polling uses more memory because of the mempool. Does LND hang on to the full mempool in between each interval? I would think on each interval LND gets the mempool from the RPC, processes it, then releases it. Which I believe would make memory spike for the duration of the processing, but then drop back down again.

For example, if lnd.bitcoind.txpollinginterval=10m, then every 10 minutes we'd see an increase in memory usage. But return to a baseline after processing. Then spike again at the next interval.

[Roasbeef]

. Does LND hang on to the full mempool in between each interval?

Only the txids, but it needs to read the entire thing (decode all the txns, etc) each interval. Polling needs to do this as otherwise there's no sort of ws notification type system to detect transactions. This was why zmq was added in the first place.

Here's the relevant code if you're curious: https://github.com/btcsuite/btcwallet/blob/5df09dd4335865dde2a9a6d94a26a7f5779af825/chain/mempool.go#L423-L426

https://github.com/btcsuite/btcwallet/blob/5df09dd4335865dde2a9a6d94a26a7f5779af825/chain/bitcoind_rpc_events.go#L269-L299

processes it, then releases it.

Go is a GC language, there's no explicit release (unless you force a manual GC), just eventual garbage collection.

[Roasbeef]

Second profile shows what we'd expect: it's spending memory decoding transactions and sending RPCs to bitcoind to have the transactions be decoded. Looking at the memory usage there, it doesn't seem to be totally unexpected. I think we can close this issue now? zmq can be lower memory as bitcoind is what needs to keep this notification state. You can lower memory by reducing the polling interval, with the tradeoff of hearing about zero conf transactions later.

btcd v0.24 ended up significantly optimizing wire decoding. 0.18 will have this dep updated, so we'll see much lower memory usage there, and less CPU optimization as well.

Are you running with the default mempool size?

[litch]

Yes we're using default mempool size.

Lowering the polling interval causes a somewhat slower memory leak that still consumes all memory until being OOM killed. It just takes days instead of hours. I don't exactly understand why the polling process should leak memory whereas the notification version doesn't. I can think of a few implementations of the strategy that would leak memory, but none that should.

We're investigating using bitcoind/polling because of (remembered?) performance issues with bitcoind/zmq (a bitcoind node can't support as many LND instances as you'd expect/want), but will re-investigate if this is indeed a wontfix.

If 0.18 has a slower memory leak, that's cool, but as of now bitcoind/polling is totally broken, so maybe it should at least be documented as such. (or maybe this discussion will serve as documentation as people try to use it)

[ellemouton]

@litch - could you grab a profile for us too? It might show us something we aren't seeing from the profiles sent by @gkrizek

[Roasbeef]

It just takes days instead of hours. I don't exactly understand why the polling process should leak memory whereas the notification version doesn't. I can think of a few implementations of the strategy that would leak memory, but none that should.

Re polling vs zmq: with zmq, the state is all within the bitcoind node. The client just needs to decode a small amount of information from the socket, and then funnel that elsewhere. It only needs to read something when there's new data. For polling, it needs to fetch the entire mempool and decode the entire thing, then maintain some in-memory maps to be able to properly dispatch notifications. It also needs to issue an RPC request for each new transaction that was added since the last delta, as just with the txid's it can't know if something credits a wallet addr to send a notification. We use JSON-RPC 2.0 for that, so we can send batch requests, in the worst case, we'd try to fetch every single transactions (first time booting up), but afterwards, it should only be that new delta.

Can you get a memory profile at the point where things seem to be overflowing? Seems the above profile was taken at an earlier moment, showing just 200 MB or so on the heap. At this point it isn't clear to me that it is actually a memory leak. If it was, we'd see run away memory allocation in the profile. My current conclusion is that with a very low polling interval (as noted you can safely increase it to minutes), a lot of garbage starts to build up which lingers more than it should.

We're investigating using bitcoind/polling because of (remembered?) performance issues with bitcoind/zmq (a bitcoind node can't support as many LND instances as you'd expect/want), but will re-investigate if this is indeed a wontfix.

So with polling, you'll also run into a similar limit. With zmq, you had to configure the high water mark setting as otherwise, messages would start to be dropped at a certain point. With polling, bitcoind still has a fixed number of rpcthreads, if more requests are outstanding than the number of threads (also see rpcworkqueue as well, which is similar to the zmq high water mark), it may just start to drop requests.

If you can give us a profile that shows some run away memory usage, then we can certainly find the issue and optimize it. So far the profiles haven't necessarily shown that yet:

[gkrizek]

@Roasbeef, I can take another profile when it's much higher in memory usage. Thanks for the extra info in how ZMQ vs RPC is used.

Additional question, Does the txpollinterval only affect polling the mempool? What are the implications of setting this really high (let's say 24h, which I think would be basically useless)? How does that affect LND/would that cause problems? Thinking in comparison to things like CoreLN which only uses RPCs of bitcoind, but has no concept of the mempool iirc.

[Roasbeef]

Additional question, Does the txpollinterval only affect polling the mempool?

Yes it's only used for zero conf transactions. If you rely on zero conf transactions for the node, then you'll want to have that be set to a a value on the order of minutes-ish. We added this to replicate the behavior of zmq, as many users (LSPs, companies, etc) still rely on zero conf channel funding.

What are the implications of setting this really high (let's say 24h, which I think would be basically useless)? How does that affect LND/would that cause problems? Thinking in comparison to things like CoreLN which only uses RPCs of bitcoind, but has no concept of the mempool iirc.

Setting it to a extremely high value is a tradeoff:

• You won't need to fetch the mempool at all, so your memory usage will be lower.
• We enhanced the mempool scanning to protect against some of the recently disclosed pinning vectors. Classes of interactions similar to pinning my be inadvertently arising more frequently today due to the current mempool conditions.

With that said, I think you should be ok setting it to something on the order of 10s of minutes.

What values are you using today? With those values, how much memory usage at peak are you observing?