allocscc takes very long when statically linking against big libraries

[mbyzhang]

allocscc takes very long (specifically alloctypes, allocsites, dwarftypes, extrasyms, etc.) when statically linking against big libraries (like openssl). This happens in libwebsocket configuration-time when the configuration script tries to detect a dozen of features in openssl (e.g. testing whether a function exists in openssl by compiling a small test C program that statically-links against openssl). As a result, configuring libwebsocket takes a couple of hours (which would normally take a minute or two).

It seems like on each feature test, the entire openssl library is redundantly processed by alloctypes, dwarftypes, etc. Is there a way to somehow cache the result to speed up?

[stephenrkell]

Thanks Peter. I agree. The main underlying bug is this one. I would like to address this bug (it is also what stops DWARF 5 from working) but it will take a chunk of consolidated time that I don't yet have on the horizon -- maybe this summer.

Without going that far, I can think of some improvements, roughly in increasing order of effort.

• preventing the configure tests from linking statically (but I guess that's an ad-hoc change to the libwebsocket build system)
• an environment variable ALLOCSCC_NO_METADATA or similar that we could use to skip metadata generation during configure tests (only)
• to combine dwarftypes, alloctypes, frametypes et al into a single tool, or rather to create a single tool that subsumes them all. It would create the root_die based on the main input file (the binary), then accept commands (maybe over stdin) for dispatch to the relevant function derived from each tool's main, passing in a reference to the already-constructed root_die. This means refactoring these main functions to support this style of invocation. The DWARF would only be read once, and since sticky_root_die would be used, each DIE would stay in memory once read.
• the same but somehow adding a long-lived cache of individual CUs' DIEs, so that when the same static library CUs are seen across multiple invocations, the relevant root_die state can be preserved. Since the offsets would differ, affecting the referential integrity of the DWARF, I think this is too much effort to be worth it. That said, I did consider representing DIE-to-DIE references completely differently at one point (a fixed-point version of Dwarf_Off; in order to accommodate insertion without invalidating existing references) and haven't ruled that out, so rolling this into that is not a silly idea -- but would take a lot of thinking through.

[stephenrkell]

Accepting commands over stdin is not necessary of course -- just command-line options requesting each distinct kind of metadata would do it. But that would assume we had replaced the existing invocation pattern, consisting of running multiple separate tools, driven by a Makefile. I like the simplicity of the latter, but creating a single, more Swiss-army-knifey tool could certainly be fine....

(... but I somehow fractionally prefer the server process idea. Perhaps it's because if we did want to preserve state across link jobs, like with the final option above, then I guess some kind of server-style process would be needed.)

[stephenrkell]

Another "Swiss-army-knifey" option would be to pull the Makefile.meta logic into the liballocs linker plugin and just rely on that for metadata builds from allocscc. Again I'm slightly reluctant to lose the simplicity of having the separate tools, but would not oppose it.

Peter, how about you do the ALLOCSCC_NO_METADATA thing for now and I work on the "server process" thing? Er, having said that, not sure exactly when I can make progress on that as I am struggling with teaching stuff just now.

[mbyzhang]

If we want to keep the independence of each tool while allowing the root_die to be shared, shall we introduce a driver that loads the root_die from the input file and passes the loaded root_die to the sub-tools as memfds or shared memory maps?

With a server-style process, we also have to worry about when to free the cache from the RAM. Do we have any ideas about it?

[stephenrkell]

I think sharing memory between subprocesses might be overkill. Here's a possible "simple as I can think of" design.

1. Each tool's main is refactored so that it can be built as normal or in library mode (say -DALLOCS_TOOL_AS_LIBRARY). The .o files built with the latter flag would get linked into a single mega-tool that is the server tool. Library mode chooses a unique name for main (and #ifdefs out the server dispatch logic below) but is otherwise the same.

2. Each tool's main (in non-library mode) also tests for an open fd 7 (or some equivalent... read on). If it's not open, it proceeds as normal. If it's open, it more-or-less just writes its own command line, say dwarftypes /path/to/my/binary, to that fd. Rather than hard-coding fd 7 we could get the path of a named pipe or socket from an env var, but that is more resource management than strictly necessary (someone would have to remove the pipe or socket later). I guess getting a filename from an env var is fine, because it can be a /proc/pid/fd/n filename if we want to minimise the resource management. (And to increase the robustness of this "magic fd" protocol we would ideally do a sanity-check handshake with the server, but am seeing that as an enhancement rather than a first-iteration issue.)

3. The server program has linked in all the tools' .o files built with -DALLOCS_TOOL_AS_LIBRARY and is a loop that reads tool commands on stdin, builds the root_die or picking up a cached version. It would be OK if it builds the root_die on first receiving a file name and then rejects any subsequent commands whose file name doesn't match.

4. Starting and stopping the server: we'd just do this in allocscompilerwrapper.py: start the server immediately before forking the Makefile.meta job, doing the latter with fd 7 pointing to the stdin of the server process. The spawned commands should all see the fd 7 and use the server. We can then kill it afterwards -- probably with a signal that it handles gracefully, setting a flag and then waiting until it hits the top of its main loop again before actually exiting... though if each tool waits for an "OK" completion response from the server, there's no need for that. Wrinkle: getting the OK would require a channel back to from server to client, which the above design does not yet have (and so would the handshaking mooted above).

Does that pass a sanity check?