Bitcore Node is quite Memory greedy

[wzrdtales]

Hey together,

I'm currently evaluating bitcore node to build transaction services on it. Beside that it was a good moment to setup a bockexplorer with your awesome insight on it. Thanks for this already

I'm running the node now for some time and I see that after some days the node is exploding in terms of RAM usage.

ps aux

USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
bt2       6225 36.5 47.9 14957252 11849604 ?   Ssl  Okt13 2235:36 bitcore-node 

You see it is using 11GiB of RAM actually I needed to cap the memory usage for this process to 12GiB as the other processes do need RAM themselves too.

My question now is, is this behavior intended? I do not see such a memory usage using a normal bitcoind node with indexing enabled. It is not a problem if the process uses 12GiB of RAM, but as soon as this is the case, bitcore node gets incredibly slow. It literally is just unavailable on a new block for about 20 to 40 seconds, it maxes out a single cpu core while working on this block, I normally expect that it should also use all available cores. This is the reason I actually need to restart the node from time to time.

If there is some known issue with this, please let me know, couldn't find anything. And also if you need more information please just ask.

[braydonf]

Hi @wzrdtales

Thanks for sharing your experience. I've also had a few issues with the memory increasing, and I believe it's related to the mempool. We've updated the bindings to use Bitcoin Core 11.1 which should help.

[wzrdtales]

@braydonf awesome, I will give it a shot :)

[wzrdtales]

I'm running now the newest master since yesterday

USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
bt2      18472 12.4 15.3 6750812 3795672 pts/4 Sl+  Okt19 131:48 bitcore  

Lets wait some more days and watch this.

[wzrdtales]

@braydonf Nope this did not help, something is wrong there with bitcore node:

USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
bt2      18472 29.9 52.8 15464388 13079492 pts/4 Rl+ Okt19 1159:29 bitcore 

It reached the memory cap again and also it is compared to the other full node I'm running with indexing enabled very slow when a new blocks enters. the full node needs 1 to 5 seconds the bitcore-node node needs over 30 seconds minimum for 1 new block

[braydonf]

We'll take another look at this. I think there are some optimizations that we can make with how we're creating mempool address indexes for faster lookups.

[braydonf]

I'm convinced that this is mempool index related. Debugging by capturing heap allocations and inspecting objects with a running node shows that most objects are related to the mempool indexes. And with another node I've noticed this behavior:

1. The memory usage will be around ~3.8GB
2. A new block will enter
3. There will be about 20 seconds of inactivity: https://github.com/bitpay/bitcore-node/blob/master/src/libbitcoind.cc#L1444
4. The memory will drop below 1GB: https://github.com/bitpay/bitcore-node/blob/master/lib/services/address/index.js#L260
5. The memory will then slowly climb back up to ~3.8GB as the index is rebuilt asycronously: https://github.com/bitpay/bitcore-node/blob/master/lib/services/address/index.js#L266

As discussed with @pnagurny I think we can solve the 20 seconds of inactivity by having an event from bitcoind when a transaction leaves the mempool ( included in PR: #352 ).This will also be useful later when updating to use new mempool limiting features.

To reduce how much memory is being used we can store things more efficiently with binary encoding and also use leveldb to keep the additional indexes on disk, that can be cached in memory if needed.