Possible memory leak?

[pajtai]

I have a definite memory leak in my app, and I'm trying to narrow it down.

The smallest bit of code that I can comment out to remove the leak seems to be (I've reduce the sharp calls to be as minimal as possible):

                                new sharp(bodyBuffer)
                                    .resize(width - 0, height)
                                    .toBuffer(resizedFileName)
                                    .then(function() {
                                        res.send('ok');
                                    })
                                    .catch(function(error) {
                                        console.log('errrrr', error);
                                    });

If I replace the above with

res.send('ok');

The leak goes away.

Is there anything I should be closing after the resize? I was thinking about maybe setting this.options.bufferIn to null at some point? One thing I've noticed is that the 'finish' event is never fired... so if I do:

                         var s     =  new sharp(bodyBuffer)
                                    .resize(width - 0, height);
                       s.on('finish',function() { console.log('done'});

the console log never fires.... but the images are definitely getting resized properly.

Thanks for any suggestions.

[lovell]

Hi Peter, are you measuring the presence of a suspected leak by watching RSS or via a tool such as valgrind? There is an example of using the latter at https://github.com/lovell/sharp/blob/master/test/leak/leak.sh

Is this when running on a memory-constrained or swap-less VM? If so, and you've not already done so, are you able to tune the --max_old_space_size and --max_new_space_size (replaced with --max_semi_space_size in iojs) v8 options?

Events, e.g. error, are currently fired only when Streaming data out. Is this behaviour you'd expect by default for all output?

[pajtai]

I'm measuring the memory usage locally by using htop and watching the memory (RES) of the node process. On the server I'm measuring the memory using pm2 list and the "memory" column. The memory starts at around 140MB (normal for the ap) and by doing multiple page requests I can easily get it to top 200MB with the code above. I've tested whether it's just delayed gc by using the --expose-gc flag and running global.gc() before each res.send('ok'), but that didn't help.

This is not in a VM. It showed up both for a mac and Ubuntu 14.04.

I might try out the leak test you referenced above, and see if I can create a reproducible test.

Regarding the events, for the real usage I pipe the output of toBuffer to the response and put it on s3, and it's been working fine - in this case there would be events, since I'm piping out?... so unless there's some cleanup I need to do that involves events, I wouldn't need it.

new sharp(bodyBuffer)
    .resize(width - 0, height)
    .toBuffer(resizedFileName)
    .then(function(buffer) {
        var bufferStream = new stream.PassThrough();
        bufferStream.end(buffer);
        // First send the response
        bufferStream.pipe(res);

        // Then put to s3
        s3.putObjectAsync({
            Bucket: amazon.bucket,
            ACL : 'public-read',
            Key : urlTail,
            ContentType : mime.lookup(urlTail),
            Body:buffer
        })

[lovell]

libvips' operation cache will be included in the resident memory as reported by htop. This is outside the V8-managed heap and has a default max size of 100MB, which would easily account for the 60MB diff you're seeing.

If you're not already doing so you can modify its size using the sharp.cache method.

[pajtai]

Thanks for pointing that out. Was not aware of that, so it's very helpful. I'll investigate some more and try and confirm whether it's really just the libvip operation cache.

[pajtai]

Thanks again for the tip about the operation cache, Lovell. After playing around with it some more I think that's all it was.