allocsld.so and liballocs_preload.so should be the same binary

[stephenrkell]

We were covering this in #56 but it's really a separate issue.

In order to finish #98 and #97 and generally clean up the initialization of liballocs (needed indirectly by #123), we want there to no longer be separate DSOs performing the load-time (allocsld) and run-time (liballocs) functions. Instead, the loader will be a complete liballocs DSO whose memory image we later trick the host dynamic linker into "adopting" as the run-time memory image of the liballocs library.

One incidental obstacle is that allocsld has its own private no-TLS copy of meta-dso.o. I think we can fix this by symbol prefixing or similar.

[stephenrkell]

Making a self-contained loader within allocsld.os that can nevertheless be called by liballocs_preload.so is slightly harder than I expected. Not so difficult, but we have symbol collisions on the malloc hooks and a few other things.

More trickily, how do we ensure that the allocsld.os does not require a PLT? In order to keep the bootstrap relocations separate, this seems important... the linker will create a single PLT and we are opting for that not to be bootstrap-relocated, so it had better not be necessary for any of our internal calls. We could make all its defined symbols protected, perhaps? Then they will not be preemptible, so should not be called locally via PLT. And there should be no undefineds, if we link against a libc. Hmm, actually there are still a few.

$ readelf -Ws allocsld.os| grep UND
     0: 0000000000000000     0 NOTYPE  LOCAL  DEFAULT  UND 
   515: 0000000000000000     0 NOTYPE  LOCAL  DEFAULT  UND _GLOBAL_OFFSET_TABLE_
   536: 0000000000000000     0 NOTYPE  GLOBAL HIDDEN   UND _DYNAMIC
   561: 0000000000000000     0 NOTYPE  WEAK   HIDDEN   UND __fini_array_end
   596: 0000000000000000     0 NOTYPE  GLOBAL PROTECTED  UND instr_len_extended
   616: 0000000000000000     0 NOTYPE  GLOBAL DEFAULT  UND __ld_so_malloc_allocator
   656: 0000000000000000     0 NOTYPE  WEAK   HIDDEN   UND __fini_array_start
   664: 0000000000000000     0 NOTYPE  WEAK   HIDDEN   UND __init_array_end
   692: 0000000000000000     0 NOTYPE  GLOBAL HIDDEN   UND _begin
   716: 0000000000000000     0 NOTYPE  WEAK   HIDDEN   UND __init_array_start

It seems _GLOBAL_OFFSET_TABLE_ is not used by any relocation. In fact only _DYNAMIC and instr_len_extended are. Is it OK to pick these up from the surrounding DSO? I doubt it. Need to think more about this. The instr_len_extended comes from our Detours implementation, but maybe we should have another approach to hooking the ld.so allocator at this point?

[stephenrkell]

We could build all allocsld objects with -fvisibility=protected perhaps?

[stephenrkell]

I have been working on this. Just wanted to say so. :-)

The current blocker is how to do a relocatable link (of allocsld.os, which we wish to drop into liballocs_preload.so) that binds intra-section references, i.e. does not leave those calls/jumps as unbound relocation records. Otherwise we have a bootstrapping problem because the loader can't get as far as bootstrap relocation without already needing bootstrap relocation.