Connections closed by node stay permanently in FIN_WAIT2

Hi

I love using node :)

Normally when node closes a connection (using socket.end()), the following packets are exchanged:
node > client [AF]
client > node [A]
client > node [AF]
node > client [A]

However, sometimes the client does not respond well, and I get only these packets:
node > client [AF]
client > node [A]

With node, this results in connections permanently (not temporarily) in FIN_WAIT2 (netstat -nop):
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name Timer
tcp 0 0 XX.XX.XX.XX:80 XX.XX.XX.XX:17863 FIN_WAIT2 26375/node off (0.00/0/0)
tcp 0 0 XX.XX.XX.XX:80 XX.XX.XX.XX:17832 FIN_WAIT2 26375/node off (0.00/0/0)
[very long list]
tcp 0 0 XX.XX.XX.XX:80 XX.XX.XX.XX:17992 FIN_WAIT2 26375/node off (0.00/0/0)
tcp 0 0 XX.XX.XX.XX:80 XX.XX.XX.XX:17851 FIN_WAIT2 26375/node off (0.00/0/0)

The problem here is that the timer is set to off, so the socket stays indefinitely in this state, eventually exhausting the server.

I wanted to know why I see this problem only with node (and not with nginx and perl POE). I read the following interesting information (source http://alldunne.org/2010/07/tcp-connection-is-permanently-in-fin_wait2-state/):
"When programming sockets in C, it’s useful to know an important difference between the shutdown (either READ or WRITE) and close of a socket.
Shutting down the socket will cause the connection to go into the FIN_WAIT2 state while the TCP/IP stack waits for the connection to be closed at the other end of the connection. But if the other side doesn’t shutdown/close the connection, then the connection will remain in the FIN_WAIT2 state indefinitly (assuming of course that your process hasn’t been terminated).
If your process where to close the connection, rather than call shutdown on it, then the connection will fall into FIN_WAIT2 as before, however, the TCP/IP stack will time the connection out after a certain period."

Having read this, maybe node is shutting down the socket instead of closing it and causes the socket to exist forever, when it was not closed well on the client side.

Best regards

[bnoordhuis]

I've never seen that behavior but it sounds plausible. We sometimes shut down the socket before closing it. Maybe there's a code path somewhere that shuts down but fails to close the file descriptor.

What do you see when you run strace -re shutdown,close -p $(pidof node)?

Thank you for your feedback. I will provide you this info next week, because my co-programmer went on holiday today.

If I understand the strace below well, node only does a shutdown when calling socket.end(). If the assumption is true, that "the connection will remain in the FIN_WAIT2 state indefinitly" when doing a shutdown instead of a close, then a simple DOS attack can exhaust a server (calling a node server many times, let node disconnect, but with iptables blocking sending a return FIN).

strace -e shutdown,close -p 19751

Process 19751 attached - interrupt to quit
close(23) = 0
shutdown(77, 1 /* send /) = 0
--- SIGPIPE (Broken pipe) @ 0 (0) ---
close(77) = 0
shutdown(77, 1 / send /) = 0
shutdown(90, 1 / send /) = 0
shutdown(111, 1 / send /) = 0
shutdown(127, 1 / send /) = 0
shutdown(155, 1 / send /) = 0
shutdown(142, 1 / send /) = 0
shutdown(156, 1 / send */) = 0
^CProcess 19751 detached

A temporary workaround is to enable keep-alive when the connection was opened, because then the FIN_WAIT2 will timeout after 2 hours. Normally it should be 1 minute, but this is better than infinite.

[bnoordhuis]

Do you see corresponding close() calls, i.e. a close(42) some time after shutdown(42)? I guess the answer must be yes or you'd run out of file descriptors eventually.

No shutdown after close of visa versa:

Process 30244 attached - interrupt to quit
16:09:34.489649 close(176)              = 0
16:10:23.627035 close(172)              = 0
16:12:14.821779 shutdown(54, 1 /* send */) = 0
16:12:16.361595 close(54)               = 0
16:12:19.475044 close(190)              = 0
16:12:44.836160 shutdown(109, 1 /* send */) = 0
16:12:46.961546 close(109)              = 0
16:13:20.848573 shutdown(173, 1 /* send */) = 0
16:14:20.869405 shutdown(184, 1 /* send */) = 0
16:15:24.906139 shutdown(172, 1 /* send */) = 0
16:18:39.997952 shutdown(195, 1 /* send */) = 0
16:18:41.538199 close(195)              = 0
16:19:18.377569 close(203)              = 0
16:21:38.181774 close(28)               = 0
16:22:09.202133 close(214)              = 0
16:24:45.386578 close(168)              = 0
16:24:57.199691 close(225)              = 0
16:25:42.173282 shutdown(216, 1 /* send */) = 0
16:25:55.816329 close(81)               = 0
16:25:58.082459 close(224)              = 0
16:26:44.191287 shutdown(28, 1 /* send */) = 0
16:27:14.094205 close(193)              = 0
16:27:43.219924 shutdown(223, 1 /* send */) = 0
16:29:14.252537 shutdown(226, 1 /* send */) = 0
^C

The result in netstat is:
#netstat -otn | grep -v ESTA
Active Internet connections (w/o servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       Timer
tcp        0      1 XX.XX.XX.XX:2001     XX.XX.XX.XX:5896     FIN_WAIT1   on (25.26/5/0)
tcp        0      1 XX.XX.XX.XX:2001     XX.XX.XX.XX:4965      FIN_WAIT1   on (3.56/11/0)
tcp        0      1 XX.XX.XX.XX:2001     XX.XX.XX.XX:4927      FIN_WAIT1   on (83.44/6/0)
tcp        0      1 XX.XX.XX.XX:2001     XX.XX.XX.XX:8133      FIN_WAIT1   on (7.18/2/0)
tcp        0      1 XX.XX.XX.XX:2001     XX.XX.XX.XX:7563      FIN_WAIT1   on (84.27/5/0)
tcp        0      1 XX.XX.XX.XX:2001     XX.XX.XX.XX:6799     FIN_WAIT1   on (63.97/12/0)
tcp        0      1 XX.XX.XX.XX:2001     XX.XX.XX.XX:7900     FIN_WAIT1   on (67.56/11/0)

[bnoordhuis]

I see at least a few matching shutdown/close syscalls and, realistically speaking, it must happen or you'd run out of file descriptors before too long (unless you set ulimit -n extremely high).

Do you perhaps have a test case I can try?

I have set the open files descriptors very high indeed (I am handling close to 100.000 M2M devices). This is the perfect test case for node, because with mobile devices everything can go wrong, like disappearing without closing the connection (resulting in FIN_WAIT1), not replying to a FIN from the server (resulting in FIN_WAIT2).

The last list I sent to you is the strace from the start of the node application for 15 minutes and the netstat is taken at the end of the 15 minute sample.

Is there a way to check the number of opened file descriptors in node, so I can check there is a "leak"?

I guess three test cases would be:

1. Client connects to node server, client is physically disconnected from the connection, node server tries to send something.
2. Client connects to node server, client is physically disconnected from the connection, node server tries to disconnect with socket.end(). This results in a FIN_WAIT1.
3. Client connects to node server, client has iptables set in such a way that a packet with FIN is never sent out, node server tries to disconnect. This results in a FIN_WAIT2.

[bnoordhuis]

Is there a way to check the number of opened file descriptors in node

Yes, ls /proc/$(pidof node)/fd/

Can you check with sysctl net.ipv4.tcp_fin_timeout what the FIN timeout is set to? If it's not 60 seconds or less, please lower it and see if the problem goes away.

Today node crashed because I was out of file descriptors. I noticed that there is no leak. The number of fd is as expected.

I noticed that if keepalive is set, it takes tcp_keepalive_time instead of tcp_fin_timeout when in FIN_WAIT2. So by default it is 7200 seconds (2 hours) even if tcp_fin_timeout is set to the default 60 seconds. If however keepalive is NOT set, then the socket goes into FIN_WAIT2 infinitely when the socket is not closed properly from the client side. This happens when node does first shutdown, then close. In my opinion this is a bug in node which can be exploited easily by a DOS attack.

Today node crashed (100% CPU) when it ran out of fd. The reason is that a lot of M2M devices that I manage don't return a FIN when node closes the connection. I tuned some parameters in /etc/sysctl and turned on socket keepalive. Now I have about 10.000 sockets in the FIN_WAIT2 state, waiting to time out. So this is a good workaround, but still sensitive for DOS attacks.

[bnoordhuis]

Today node crashed because I was out of file descriptors. I noticed that there is no leak. The number of fd is as expected.

Okay, thanks. That means it's not specifically a node bug (by which I mean that any TCP server is vulnerable to this particular weakness of the protocol, not just node.)

I'm not sure how or even if we should fix this. The shutdown(SHUT_WR) that node does (half-close, stop sending but don't stop receiving just yet) is there to give the other end a chance to properly shut down the connection. The problem with doing a full close is that lots of client applications don't handle that well at all if it's unexpected.

"Well, that's a bug in the client application," you say (and you'd be right). Yes, but you don't have to sift through the myriads of bug reports that changes like these generate. :-)

I half-kid, obviously, but you see the dilemma - it's going to break stuff that worked before (even if we make it opt-in - because there's always someone.)

Thanks for your feedback.

I am not an expert in how to exactly 'end' a connection, but when I opened this issue I referred to one of the sources I found, saying you should always close a socket, not shutdown a socket.

As far as I know, the other side does not know if you did a shutdown or close because in both situations a FIN is sent. And a FIN is expected back (and if not, no problem because it will timeout in FIN_WAIT2).

Since allowHalfOpen: false is the default setting, maybe then you can close the socket, otherwise shutdown?

I love node.js and I just want to contribute a potential problem which can be exploited :)

[bnoordhuis]

I discussed it with other committers. The consensus is that removing the shutdown() if allowHalfOpen is not set is the best way to go forward. The problem is that it breaks a fair number of tests - no doubt it'll affect real world code as well.

Ergo, don't expect it to land in a stable branch anytime soon - v0.10 at the soonest.

[indutny]

About, close() vs shutdown() after some investigation of linux code I can say that it's not nearly the same thing. close() will send RST frame in many cases.

  When an application closes a connection in such a way that it can
  no longer read any received data, the TCP SHOULD, per section
  4.2.2.13 of RFC 1122, send a RST if there is any unread received
  data, or if any new data is received. A TCP that fails to do so
  exhibits "Failure to RST on close with data pending".

[indutny]

So I propose following solution to this problem: joyent/libuv#498 . And we may add .forceClose() method to node.js which will us uv_tcp_reset_timed(...)