Mock network for testing

[njsmith]

@glyph gave a great talk at PyCon this year that involved using a virtual (= in memory, in python) networking layer to build a virtual server to test a real client.

As far as the virtual networking part goes, we have some of this, e.g. #107 has some pretty solid in-memory implementations of the stream abstraction. But it would be neat to virtualize more of networking, e.g. so in a test I can have tell my real server code to listen on some-server.example.org:12345 and tell my real client code to connect to that and they magically get an in-memory connection between them.

Fixing #159 would reduce the amount of monkeypatching needed to do this, but OTOH I guess monkeypatching the whole trio.socket module is probably the simplest and most direct way to do this anyway... or we could hook in at the socket layer (have it check a special flag before allocating a new socket) or at the high-level networking layer (open_tcp_stream checks a special flag and then returns a FakeSocketStream etc.). Fundamentally there's going to be some global state because no-one will put up with passing around the whole library interface as an argument everywhere, literally every async library has some kind of contextual/global state they use to solve this problem, and I can't think why it would matter a huge amount whether that's from twisted.internet import reactor vs asyncio._get_running_loop() vs trio.socket.socket(). So I'm leaning towards not worrying about monkeypatching. (The one practical issue I can think of is if someone is trying to use trio in two threads simultaneously, then this will cause some problems because the monkeypatch would be global, not thread-local. Maybe we can make it thread-local somehow? Or maybe we just don't care, because there really isn't any good reason to run your test suite multi-threaded in Python.)

Oh, or here's a horrible wonderful idea: embed the fake network into the regular network namespace, so like if you try to bind to 257.1.1.1 or example.trio-fake-tld then the regular functions notice and return faked results (we could even encode test parameters into the name, like getaddrinfo("example.ipv6.trio-fake-tld") returns fake ipv6 addresses...). Of course this would be a bit of a problem for code that wants to like, use the ipaddress library to parse getaddrinfo results. There are the reserved ip address ranges, but that gets dicey because they should give errors in normal use... In practice the solution might be to stick to mostly intercepting things at the hostname level (e.g. open_tcp_stream doesn't even need to resolve anything when it sees a fake hostname), though we do need to have some answer when the user asks for getpeername. I guess we could treat all addresses as regular until someone invokes this functionality with a hostname, at which point some ip addresses become magical.

BUT there would also still very much need to be a magic flag to make sure all this is opt-in at the run loop level, to make sure it could never be accidentally or maliciously invoked in real code, to avoid potential security bugs. At which point I suppose that magic flag could just make all hostnames/addresses magical. Oh well, I said it was a horrible (wonderful) idea :-). The bit about having hostnames determine host properties might still be a good idea.

There's also a big open question about how closely this API should mimic a real network. At the very least it would have to provide the interfaces to do things like set TCP_NODELAY (even as a no-op), for compatibility with code made to run on a real network. But there are also more subtle issues, like, should we simulate the large-but-finite buffers that real sockets have? Our existing in-memory stream implementations have either infinite buffering or zero buffering, both of which are often useful for testing, but neither of which is a great match to how networks actually work... and of course there are also all the usual questions about what's kind of API to provide for manipulating the virtual network within a test.

I suspect that this is a big enough problem and with enough domain-specific open questions that this should be a separate special-purpose library? Though I guess if we want to hook the regular functions without monkeypatching then there will need to be some core API for that.

Prerequisite: We'll need run- or task-local storage (#2) to store the state of the virtual network.

[njsmith]

Some notes:

I think the way to do this is to provide a hook for the socket and socketpair constructors, and for getaddrinfo and getnameinfo, and put the bulk of the code in a separate library. (The other constructors like from_stdlib_socket would remain unhooked. This is probably important for subprocess support, as well as being the Right Thing I think.)

The getaddrinfo hook should probably only be called after we check the numeric fast path and go through idna encoding, so it would always receive the hostname as a bytestring. (It can also receive the hostname as a numeric IP, iff the service is a symbolic name.)

A useful fake network probably needs to include features like:

• Special casing localhost and the numeric equivalents.

• Built in behaviors like IPs that work, that fail fast, that act as black holes. (There may be further subdivisions like, accept connections but then acts as a black hole, or accept connections but then error out when sending data? I guess error out on send is easily doable by putting up a fake server so maybe that's not needed at the network level.)

• Ability to make assertions about what operations happened, and when

• A sensible default set of mappings from hostnames to IPs, and from IPs/ports to behaviors.

• The ability to manipulate these mappings to set up canned configurations. (Possibly this should be a bit higher level than just "here's the desired output from getaddrinfo? getaddrinfo is pretty complicated when you get into all the different flags and protocol types.)

I'm not sure to what kind of behavior plugin API we want. Something like the memory_stream objects, with callbacks that fire to pump data? An active "switchboard" that gets notified of connection attempts etc and then decides what to do? Just a small set of canned behaviors + for actual interaction you can put up a fake server to talk to with whatever complex behavior you want? (This could even use an extended interface, e.g imagine a version accept that let you peek at the connection attempt and then decide what to do with it.)

[njsmith]

It would be neat to have a built in randomized/flakey-network mode; think like a 1 button hypothesis test suite for network protocol annoyances.

Possibly this might work like: run the code once to calibrate how much communication happens, and then rerun N times with random delays injected, or if more ambitious and requested, with network failures.

One might want to have ways to tune the kind and distribution of random delays (breaking up data by introducing a virtual delay after every byte is a good stress test, but might be too slow for larger tests). Some kind of hypothesis-like tuning towards "weird" patterns would also be good (e.g. adding tons of delays on one connection but not another, or on one direction but not another).

It would also be neat if there were a way to write down a pattern of injected faults, to extract repeatable regression test cases from a randomized run, and potentially to allow particular fault patterns or fault pattern generators to be written by hand.

[njsmith]

I added hooks!

#253 implements the core hooks needed to enable this. Specifically, I decided to allow hooking the main socket constructor (when fileno= is not given), the is-this-a-trio-socket logic, and getaddrinfo/getnameinfo.

Design notes on socket hooks

An interesting problem is what to do with the other socket constructors – socketpair, socket(fileno=...), fromfd, etc. I ultimately decided that:

• They don't seem to be necessary for our initial use case. socketpair can be replaced by memory_stream_pair if that's what you want, and all of these are mostly used when you want to do something clever with kernel-level socket objects, not in generic networking code.

• If you want to somehow use real sockets in the implementation of your custom socket class, then that becomes very difficult if all the real socket functions are intercepted.

If it turns out that being able to hook these "real socket" constructors is useful, then in the future we could add a wrap_real_socket method to SocketFactory, that gets called after the trio socket object is constructed and can wrap it or whatever you want. I guess this would only need to hook from_stdlib_socket since everything else funnels through there.

Design notes on hostname resolver hooks

It just lets you hook getaddrinfo and getnameinfo; see #254 for some discussion.

Currently, we optimize out the case getaddrinfo("127.0.0.1", 80) (skip calling the custom resolver), but we do call the custom resolver for getaddrinfo("localhost", 80) and for getaddrinfo("127.0.0.1", "http"). It might be nice to handle some more cases in the generic code, e.g. guarantee that they never see symbolic service names (because we resolve them first). I guess we could make sure they never see AI_NUMERICHOST or AI_NUMERICSERV too. But for now this just seemed too complicated -- to resolve the service name ourselves we'd need to use getservbyname in a thread, but it isn't thread-safe, and also it wants an argument "tcp" or "udp", even though these are supposed to always give the same results, and ... ugh, whatever, I decided to punt for now, and custom resolvers just have to be prepared to deal with this stuff as well as they're able. We can always tighten it up later (there's no backcompat break if we tighten things up so that the custom hooks see more regular input).

Next step

So the next step here is to make a package providing a sweet fake network simulator that hooks into trio.