Handling of SO_REUSEADDR and friends

[njsmith]

Before I looked into it, my assumption was that we should default SO_REUSEADDR to enabled. But it turns out that things are more complicated than that.

The situation as I understand it:

On Unix:

• For clients, connect on an unbound socket will automatically pick a good port + interface, and part of picking a "good" one is that it knows who you're connecting to, so it can strategically re-use local ports. (It's okay to have two client connections use the same local port so long as the peers have different addresses.)
• For servers, bind will by default disallow re-use of ports that are in TIME_WAIT, which is generally considered over-fussy these days. So generally it's recommended to enable SO_REUSEADDR, which allows to bind to ports that are in TIME_WAIT but otherwise unused.
• For clients that call bind before connect, you probably don't want to use SO_REUSEADDR, because it makes it possible to get a port that ends up failing when you call connect (ref). Though really you're best off not calling bind at all, because connect can do a better job of binding than you can, because it has more information at hand. [Edit: on recent Linux there's also sock.setsockopt(IPPROTO_IP, IP_BIND_ADDRESS_NO_PORT, 1); sock.bind((host, 0)) which means "bind me to this host, but delay picking the port until I call connect.]

On Windows:

• For clients, the plain connect function acts similar to Unix, AFAIK. (The WSA-level functions like ConnectEx are different and require you to bind first, but ATM we aren't using those.)
• For servers, you have to enable SO_EXCLUSIVEADDRUSE or else any program with the same uid can hijack your port. (Yes! At least this is better than it used to be – in XP and earlier, they didn't even have the uid check.) This is also required to prevent weird problems like being allowed to bind to a wildcard address + port where there is already another program bound to that port on all the concrete addresses. However, the downside is that it also prevents re-using ports that are in TIME_WAIT.
• Never ever ever use SO_REUSEADDR, it's totally broken

(Reference for the delightful Windows behavior)

So one option would be to default-enable SO_REUSEADDR on Unix and SO_EXCLUSIVEADDRUSE on Windows. I'm a bit concerned about whether this will have a negative effect on clients, though – maybe we only want this to be the default for listening sockets? That's trickier. I guess we could set it in bind if not overridden? Or maybe we should keep it simple and say that it's default-enabled, and if you want to turn it off again then go for it.

Also, we should probably just not even expose SO_REUSEADDR on Windows, b/c it is a massive trap. Or even make trying to access it raise AttributeError: no really you don't want this, see <link>.

[njsmith]

Tentatively implemented, and closing in favor of #72