feat: benchmark network traffic and database size

[brprice]

This uses the nixos-test framework to run each component (database, primer, client) in a different vm and generates traffic by replaying sessions. We measure the network traffic and database size.

[brprice]

This is pretty much done, except:
TODO:

• document format of fixture
• document how to extract fixture (I'll do these first two after feat: benchmark edit actions #863 is finished)
• rebase to remove "TODO" in commit msg and code (note: there is also one in code in a commit which does not have "TODO" in its commit message!)
• perhaps consider a more realistic fixture

One obvious extension is to support multiple fixtures. This should not be difficult, but I don't know enough about the nix setup to do this quickly. Perhaps leave this for future work?

@dhess: Could you look at the nix code with some suspicion? I think there are at least two bits that could probably be done better: using fromString a lot , and actually hooking up the benchmark into the flake (related to "generalise to have multiple fixtures", since due to my file layout I can't just use importFromDirectory). One higher-level question is where in the flake this should live -- currently I put it directly in benchmarks, gated by a x86_64-linux check, similar to the github-action benchmark.

[dhess]

I'll take a look, but FYI, I was able to refactor hacknix's NixOS tests so that I could use importFromDirectory by simply moving things around a bit, in case this helps: https://github.com/hackworthltd/hacknix/tree/main/tests

Note that there are subdirs full of test data in there, and so long as you're careful that the importFromDirectory dir only has .nix files with actual tests, it all works.

[brprice]

Thanks, I've managed to re-organise so I can use importFromDirectory -- probably best to hold of on looking at this until after I push that. EDIT: done

[dhess]

Let's move the new benchmark script to the same default.nix file as the others, but otherwise, this is really good stuff, and will be invaluable data. Thanks!

[dhess]

Here's a thought, though out of scope for this PR: how hard would it be to collect GHC RTS stats from these runs; e.g., number of GCs? Those might be really interesting, as these replays are as close as we can currently get to real-world loads.

[brprice]

Some future work is to add a fourth vm, a proxy. This would add compression, and bring this benchmark closer to the real-world. (Having both uncompressed and compressed stats would be useful, though perhaps they can be done in one benchmark, just by measuring either side of the proxy somehow (the current setup won't do that, as we only get tx/rx stats per interface; maybe setting up the proxy to have two interfaces: one for client, one for backend would work)). @dhess has indicated he'll look into this

[dhess]

As an aside, I believe this is considered to be bad practice by the nixpkgs maintainers, as you will sometimes want to use a different pkgs for tests etc. than the one from which your specific tooling comes from (jq, coreutils, etc.), but I'm not bothered, so this is fine for our needs.

[dhess]

The changes are fine, though I would prefer not to have fixup! commits in our history.

[brprice]

Here's a thought, though out of scope for this PR: how hard would it be to collect GHC RTS stats from these runs; e.g., number of GCs? Those might be really interesting, as these replays are as close as we can currently get to real-world loads.

Actually, I think it may be fairly straightforward. I'll have a look

[dhess]

Some future work is to add a fourth vm, a proxy. This would add compression, and bring this benchmark closer to the real-world. (Having both uncompressed and compressed stats would be useful, though perhaps they can be done in one benchmark, just by measuring either side of the proxy somehow (the current setup won't do that, as we only get tx/rx stats per interface; maybe setting up the proxy to have two interfaces: one for client, one for backend would work)). @dhess has indicated he'll look into this

See #889

[brprice]

Here's a thought, though out of scope for this PR: how hard would it be to collect GHC RTS stats from these runs; e.g., number of GCs? Those might be really interesting, as these replays are as close as we can currently get to real-world loads.

Actually, I think it may be fairly straightforward. I'll have a look

See #890