Resource leak that brings down the Hub

[Splaktar]

Last night, the Hub stopped responding to all requests. Antonio Bertucci let us know about it in the Gitter chat about 5 hours ago.

When I logged into the console for the Hub and attached to the tmux session, I found the following issue:
nodejs/node-v0.x-archive#9409

There is a work around in that thread, but I haven't had time to try it. Also we've only seen this issue once, so it may be a fluke. If it happens again, and the work around fixes it, we can be a bit more confident.

There is a PR for the issue but it was never merged and now appears to be targetted at an abandoned repo of Node.js.

[Splaktar]

Happened again last night:

[10298] GET / - 17524 ms - 2296
assert.js:86
throw new assert.AssertionError({
        ^
AssertionError: Resource leak detected.
at removeWorker (cluster.js:346:9)
at ChildProcess.<anonymous> (cluster.js:366:34)
at ChildProcess.g (events.js:199:16)
at ChildProcess.emit (events.js:110:17)
at Process.ChildProcess._handle.onexit (child_process.js:1074:12)

[Splaktar]

Had to restart the hub again tonight.

[Splaktar]

Went down again a couple hours ago, working on a fix...

[tasomaniac]

I happened 3 times in 10 days. Waiting for 4 or more days makes sense.