glibc 2.36 ld.so crash

[stephenrkell]

Startup of liballocs is broken for me after the upgrade to glibc 2.36. Somehow a null link map pointer makes its way into the dynamic linker's structures early during startup, and it chokes. I'm still tracking this down.

(Even after I fix this, see #77 -- we have problems because of libdwarf. For me, our old ibdwarf can't process the separate DWARF file from my Devuan libc6-dbg package... I could forward-port to latest libdwarf, which may have fixed this, or it may be time for the big libdw transition.)

[stephenrkell]

This turns out to be a glibc bug.

It seems necessary to have some detection/reporting of glibc bugs like this at liballocs startup, since buggy releases can stay in circulation a long time and we have a habit of finding the interesting corners. Doing a check/warning phase in allocsld seems like the right thing, although in liballocs's global init would also be early enough to work much of the time.

[stephenrkell]

If we detect the present glibc bug, is there a black-box workaround?

I have confirmed the same bug on glibc 2.35 and 2.40.

[stephenrkell]

One workaround might be to do some "reflective access" to the link map structure. We are in the fortunate position, from liballocs, of being able to assume that we have the debug info to enable this. But what?

Perhaps the right thing is to bisect this in glibc and then revert the breaking change. Then figure out a bodge fix that we can do at run time, reflectively if need be.

[stephenrkell]

This is more complicated. Even on the CI-passing glibc 2.27 I can reproduce the issue, using what I thought was the reproducer isolated from the crash. So, somehow in CI we are not triggering the crash even though the same glibc bug is present. What is different?

The crash arises in _dl_init When liballocs crashes, main_map->l_searchlist.r_nlist is 12 and the link map entries in main_map->l_initfini are for the following libs.

""
"/var/local/stephen/work/devel/liballocs.git/build/opt/liballocs_preload.so"
 "/var/local/stephen/work/devel/liballocs.git/tools/..//lib/liballocs_dummyweaks.so"
"/usr/lib/x86_64-linux-gnu/libunwind-x86_64.so.8"
"/usr/lib/x86_64-linux-gnu/libunwind.so.8"
"/lib/x86_64-linux-gnu/liblzma.so.5"
"/lib/x86_64-linux-gnu/libdl.so.2"
"/lib/x86_64-linux-gnu/libpthread.so.0"
"/lib/x86_64-linux-gnu/libc.so.6"
"/lib64/ld-linux-x86-64.so.2"
(then it's NULL)

... whereas in the good case main_map->l_searchlist.r_nlist is 10 and the link map entries in main_map->l_initfini are for the following libs.

""
"/root/project/build/debug/liballocs_preload.so"
"/root/project/tools/..//lib/liballocs_dummyweaks.so"
"/usr/lib/x86_64-linux-gnu/libunwind-x86_64.so.8"
"/usr/lib/x86_64-linux-gnu/libunwind.so.8"
"/lib/x86_64-linux-gnu/liblzma.so.5"
"/lib/x86_64-linux-gnu/libdl.so.2"
"/lib/x86_64-linux-gnu/libpthread.so.0"
"/lib/x86_64-linux-gnu/libc.so.6"
"/lib64/ld-linux-x86-64.so.2"
(then it's NULL)

i.e. exactly the same!

Is the difference whether add_to_global_resize is called? Or how many times it's called? In my crashing run of liballocs it's called twice. The function seems not to exist in glibc 2.27 however, but there is add_to_global.

[stephenrkell]

This is a glibc bug which we can pester the glibc people to fix, but that leaves a general issue: when a bad dynamic linker is in circulation, we need to work around its quirks.

I'm having some fun today trying to implement a binary patcher for malloc entry pathways, so that I can reflect on early allocations made within the ld.so, so I can write a workaround that patches up its internal data structures when liballocs is initialized (which is before this crash). But that's not really a generalisable solution. Instead we probably need allocsld to delegate to a 'known-good' dynamic linker which is built when liballocs is built. A serious liballocs developer (i.e. just me, for now) will take care to have a good ld.so on their system, e.g. by patching bugs like this one, so will avoid the long slow build of a glibc. But a typical, casual user/experimenter might not... that's fine but they should be prepared for the liballocs build process to go off and build a carefully selected/patched dynamic linker.

[stephenrkell]

Although this is still morally a glibc bug, it turns out that our use of DF_INITFIRST is regarded as an abuse by glibc, since only one object may use it and that's libpthread. The necessary approach if therefore to overhaul how we initialize ourselves, so as not to require being initfirst. This is discussed in #97... closing this now.