Enable vats to execute more independently of the kernel #334
Description
We currently inherit SwingSet's one-crank-at-a-time execution model, in which the kernel takes an item from its run queue and delivers it to the relevant vat, handling syscalls from that one running vat but waiting until the vat completes processing the delivery before proceeding to the next run queue item. We refer to the act of a vat processing a single message delivery to completion as a "crank", analogous to the more familiar concept of a "turn" but at the vat granularity rather than the JavaScript microtask granularity. At any given time, only one thing is happening: the kernel is starting or finishing a crank, the vat is running, or the kernel is processing a syscall; either the kernel or exactly one vat has agency. The key invariant this maintains is that each delivery to a vat is processed atomically, and the only way one vat can have a causal effect on another is by message passing, where, from a vat's point of view, message sends happen after a crank is finished and message receives happen before a crank starts.
This serialized execution strategy is favored by SwingSet because it is (relatively) easy to implement under the constraint of satisfying SwingSet's requirement for purely deterministic computation. However, we do not have that constraint. In principle at least, we should be able to gain a (hopefully) significant performance benefit by employing an alternative execution strategy that allows for greater run time parallelism while still preserving the essential atomicity of crank execution.
In brief:
Rather than accumulating deliveries in its run queue and mediating vats to execute these run queue entries one at a time, the kernel eagerly delivers messages and other events to their appropriate destination vats as soon as those deliveries become available to it. The vats individually queue those deliveries in their local vatstores. A vat processes its own message queue as rapidly as it is able. The vat itself still follows a one-crank-at-a-time pattern, but instead of handshaking with the kernel for each syscall, it accumulates these into a bundle that it gives to the kernel in a crank completion record at the end. In other words, from the kernel's point of view, when a vat executes a crank, all possible side effects of that crank are produced as a single atomic unit. The kernel processes these completion records in order, modifying its internal state or forwarding deliveries to other vats as appropriate. (Note that SwingSet could also employ this execution strategy, but it would require significant additional algorithmy and data structure magic to ensure that completion records were processed in the same order as the deliveries they are the completion records for; fortunately for us, this is complication that we need not bother with.)
For the sake of completeness, it's worth noting that the atomic completion record scheme itself is not a strict requirement: we could instead treat syscall events as a stream of items which are delivered to the kernel as they happen, while the kernel accumulates them incrementally in its own local storage and only acts upon them upon a signal that the crank is done. This might possibly lead to better memory utilization, though I suspect it would also be a little hairier to implement. I'm only mentioning this here so as to not lose track of the idea; I don't expect us to make this part of the implementation strategy of record unless circumstances end up requiring it.