Understanding sharp mem usage with a disabled cache

[papandreou]

I'm seeing some weird memory usage patterns on a production system that does on-demand image processing after we switched to sharp recently. I found #349 and #260 which seemed very similar, and was happy that the matter seemed to be concluded to everyone's satisfaction.

As far as I can tell, there are still problems, though. I changed @janaz's test setup (thank you very much for putting that online) to use the latest sharp master commit, the latest libvips master commit, disable sharp's cache via sharp.cache(false), and run more iterations of the test: https://github.com/papandreou/sharp-mem-usage/tree/fiddleAround

A make tests run in my fork yielded this output: https://gist.github.com/papandreou/27fde30202fe778f44dbaff9d7a39f63

The RSS starts at 33 MB, raises to 42 MB after the first iteration, and adds about 10 MB per iteration for a bit, slows down, plateaus at 275 MB for about 30 iterations, then a full mark-sweep garbage collection of the JavaScript heap brings it down to 177 MB, the pattern repeats itself a few times, and then it starts a steady increase and approaches 500 MB near the end of the 500 iterations.

... There is something going on :). I don't know exactly how to interpret the results, but I would expect the RSS to come down to about the initial level after a full mark-sweep GC.

The outfit branch shows the same trend.

[lovell]

Hello, I wonder if what we're seeing here is the effect of memory fragmentation?

Given this is Linux, are you able to try with a different memory allocator? If so, add jemalloc to the installed packages then use LD_PRELOAD=/path/to/lib/libjemalloc.so.1 node --trace-gc test to run the tests within the container.

It's probably also worth testing with file-based output via toFile instead of toBuffer to see if this alters the memory behaviour. This will help confirm/deny the involvement of Node Buffer objects.

[papandreou]

Thanks for looking into it :)

With jemalloc it's still basically the same trend: https://gist.github.com/papandreou/ef7543f1b79f7d8857c84d0632ec143c -- it ends up with an RSS of 428 MB.

With file-based input, on the other hand: https://gist.github.com/papandreou/b71e0f336bf709e743c0889a3db96cb6 -- it's much more stable and ends up at 59 MB after 500 iterations.

Switching back to buffer-based input, but only reading the file once instead of at the start of every iteration shows the same low overall memory footprint as the file-based input: https://gist.github.com/papandreou/818a6ffd729697ed6359f9abbdc7100d

It seems like we're getting close, but I don't know exactly what to conclude? Either node.js has a memory fragmentation problem, or sharp is leaking buffers, but it's less visible when there's only a single, big buffer in play?

[papandreou]

Reading the file once, but creating a copy of the buffer before each iteration (image => Buffer.concat([image])) makes the RSS climb again.

[papandreou]

Seems like it does depend on the node.js version:

0.10.45:

example before iteration #1 { rss: 35733504, heapTotal: 10324992, heapUsed: 5621720 }
example before iteration #2 { rss: 48336896, heapTotal: 11344896, heapUsed: 5808192 }
[...]
example before iteration #100 { rss: 60399616, heapTotal: 11344896, heapUsed: 5716080 }

0.12.14:

example before iteration #1 { rss: 44343296, heapTotal: 12835584, heapUsed: 6608768 }
example before iteration #2 { rss: 54784000, heapTotal: 12835584, heapUsed: 6851904 }
[...]
example before iteration #100 { rss: 63778816, heapTotal: 17029888, heapUsed: 6777864 }

io.js 2.5.0:

example before iteration #1 { rss: 43737088, heapTotal: 14423040, heapUsed: 6252816 }
example before iteration #2 { rss: 56307712, heapTotal: 14423040, heapUsed: 6496480 }
[...]
example before iteration #100 { rss: 66166784, heapTotal: 18550784, heapUsed: 7190832 }

io.js 3.0.0:

example before iteration #1 { rss: 49037312, heapTotal: 11327232, heapUsed: 6800992 }
[...]
example before iteration #99 { rss: 481263616, heapTotal: 17518848, heapUsed: 6402736 }
example before iteration #100 { rss: 485507072, heapTotal: 17518848, heapUsed: 6423008 }

io.js 3.3.1:

example before iteration #1 { rss: 50425856, heapTotal: 11327232, heapUsed: 6976832 }
example before iteration #2 { rss: 67518464, heapTotal: 11327232, heapUsed: 7215952 }
[...]
example before iteration #100 { rss: 486883328, heapTotal: 17518848, heapUsed: 6503456 }

4.4.4:

example before iteration #1 { rss: 51679232, heapTotal: 11536480, heapUsed: 6891464 }
example before iteration #2 { rss: 68751360, heapTotal: 11536480, heapUsed: 7790792 }
[...]
example before iteration #100 { rss: 230236160, heapTotal: 9489248, heapUsed: 6491544 }

5.11.1:

example before iteration #1 { rss: 51593216, heapTotal: 10619424, heapUsed: 6905552 }
example before iteration #2 { rss: 69070848, heapTotal: 11651360, heapUsed: 7795328 }
[...]
example before iteration #100 { rss: 230211584, heapTotal: 10624032, heapUsed: 6702000 }

So the problem got introduced with io.js 3.0.0. I guess that makes the Buffer-on-top-of-UintArray8 reimplementation the most likely culprit: https://github.com/nodejs/node/blob/master/doc/changelogs/CHANGELOG_IOJS.md#2015-08-04-version-300-rvagg -- or maybe sharp hasn't correctly adapted to that change.

[lovell]

Thanks for running the extra tests on the various versions of Node/io.js. The implementation of Buffer within Node has quite an influence on the results!

The GCs that occur due to "external memory allocation limit reached" appear to reduce the RSS by a much greater value when using jemalloc, which suggests this is at least partly a memory fragmentation problem.

sharp uses Nan::NewBuffer to create the Buffer object, providing a custom callback to guarantee we free the memory in the same shared library as it was malloc'd (to keep Windows happy).

You can track the GC of sharp-created Buffer objects by adding some printf-style "debugging" to the FreeCallback to show when backing data is freed.

[papandreou]

For completeness -- it seems like node.js 6.2.0 improves on this quite a bit:

6.2.0

example before iteration #1 { rss: 52207616, heapTotal: 12578816, heapUsed: 7073760 }
example before iteration #2 { rss: 69861376, heapTotal: 12578816, heapUsed: 8019312 }
[...]
example before iteration #100 { rss: 117493760, heapTotal: 10481664, heapUsed: 6657384 }

[lovell]

Thank you for this thorough testing @papandreou !

Looking through recent Node commits, I think the change in V8 API used when cleaning up Buffer objects as part of nodejs/node#5204 might have improved things.

[papandreou]

I guess that fully explains the symptoms originally observed by @janaz. It might make sense to discourage the use of io.js 3.x and node.js 4.x + 5.x in the README. It's of course unfortunate that the current node.js LTS series has memory fragmentation issues, so it might also make sense to ping about this, in case they aren't already aware.

The really funny thing is that the issue I'm having occurs with node.js 0.10, so it's also back to the drawing board for me :)

[lovell]

nodejs/node#5204 has the dont-land-on-v4.x label, which I suspect means that the version of V8 in Node v4 doesn't support the API being used, or at least that the change is considered major enough to break other things. Not sure what we can do here, other than your most excellent suggestion of documenting it :)

[papandreou]

Ok, I think I've found another problem that occurs with node.js 0.10 as well -- the memory utilization pattern changes drastically when I change the test script to use stream-based input (which my actual app does):

With Buffer-based input: http://pastebin.com/Ny0xyD4m
With stream-based input: http://pastebin.com/dhYwM4kK

This is the difference between the two test runs: papandreou/sharp-mem-usage@5c97369

This looks rather serious as well, any ideas?

[lovell]

Using Stream-based input usually results in the creation of a lot of small-but-temporary Buffer objects, which might exacerbate any memory fragmentation.

Is this causing the process to crash?

Have you tried experimenting with V8's "new space" ("semi space" in recent versions) and "old space" heap settings?