Binary instrumentation of inline allocation functions and alloca

[stephenrkell]

This follows on from #11.

To handle alloca() in unmodified binaries, we need to be able to identify it where it appears in the output instruction stream. I believe this is doable, by a mixture of source- and binary-level analysis similar to what we currently do. A heuristic suffices for identifying the instruction that is actually doing the alloca (this is a sub of $rsp, on x86-64), cross-checked against the source code using debug info. Instead of inserting a wrapper, and indexing the alloca'd chunks as if they were malloc'd, we need to use a trampoline approach (like with #11) and keep a custom index (perhaps a bitmap hanging off the stack frame's bigalloc).

Handling inlineable allocation functions in general is hard. But it's rare for allocators to be inlined. We probably want to forbid this and warn if we see it happening, again by inspecting the DWARF. (A longer-term project is to support retroactive un-inlining....)

[stephenrkell]

Note the topic is binary instrumentation, not binary-level identification. Since we depend on source analysis for type info, we need to look at the source, which makes identification easy in the case of alloca. For other inlined callchain segments we might initially assume we can identify which callees are allocator functions. And to go further, note that we are proposing to identify allocation functions by some automatable criteria (#87).

(But just for fun, to identify alloca, we could say it means "has a frame pointer, and does at least one input-dependent sub from the stack pointer or does multiple constant-operand subs or does a single constant-operand sub but its frame was found to be holey by frametypes".)