ERROR: "__memcat_p" [drivers/hwtracing/stm/stm_core.ko] undefined!

[nathanchance]

Split off from #514.

The config file in that issue produces the following error at link time:

ERROR: "__memcat_p" [drivers/hwtracing/stm/stm_core.ko] undefined!

A simpler/faster build is just x86_64_defconfig + CONFIG_STM=m.

This is really odd as __memcat_p is marked as EXPORT_SYMBOL_GPL in lib/memcat_p.c. Even changing it to EXPORT_SYMBOL doesn't resolve the build failure.

[nathanchance]

I narrowed this down to an ld.lld issue:

$ echo "CONFIG_STM=m" >> arch/x86/configs/x86_64_defconfig

No error:

$ make -j$(nproc) CC=clang O=out defconfig modules_prepare vmlinux modules

Error:

$ make -j$(nproc) CC=clang LD=ld.lld O=out defconfig modules_prepare vmlinux modules

I assume the problem here is that somehow __memcat_p isn't getting added to the __ksymtab section?

[ihalip]

I assume the problem here is that somehow __memcat_p isn't getting added to the __ksymtab section?

Looks like it, yes. The build steps create a linker script containing the exported symbols, including:

$ grep memcat lib/.lib-ksyms.o.lds 
EXTERN(__memcat_p)

But the behavior is different when ld.lld is invoked; it doesn't add any of the symbols in that script to the output file:

$ ld -m elf_x86_64  -z max-page-size=0x200000 -r -o lib/lib-ksyms.o -T lib/.lib-ksyms.o.lds lib/.lib_exports.o; objdump -x lib/lib-ksyms.o | grep memcat
0000000000000000         *UND*	0000000000000000 __memcat_p
$

$ ld.lld-9 -m elf_x86_64  -z max-page-size=0x200000 -r -o lib/lib-ksyms.o -T lib/.lib-ksyms.o.lds lib/.lib_exports.o; objdump -x lib/lib-ksyms.o | grep memcat
$

I tried a number of command-line params for ld.lld, with no success.

[ihalip]

Added review for lld: https://reviews.llvm.org/D63564

[nickdesaulniers]

great work @ihalip !

[nathanchance]

I can confirm that fix works, thanks!

[nickdesaulniers]

should this have a 32b or 64b x86 ARCH tag?

[nathanchance]

I did not test other architectures but I doubt this is an x86 specific issue.

[MaskRay]

I replied on https://reviews.llvm.org/D63564#1551417. My question is why the -r link needs EXTERN(__memcat_p)

[nathanchance]

@MaskRay I believe that comes from: https://github.com/torvalds/linux/blob/8c79f4cd441b27df6cadd11b70a50e06b3b3a2bf/scripts/Makefile.build#L438-L449

@masahir0y can also weigh if he has time.

[ihalip]

My question is why the -r link needs EXTERN(__memcat_p)

The trick is to create an "empty" object file (lib-ksyms.o) containing undefined references to all the symbols that need to be exported and are not built-in. For example, memcat_p is not used internally by the kernel but it is part of the kernel API (and it's used by the stm module), so it should be exported at all times.

The build process in the lib/ directory is as follows:

• generate lib.a, containing regular symbols
• generate lib-ksyms.o, containing undefined references to symbols that need to be exported
• generate built-in.a, containing built-in symbols and lib-ksyms.o
• link vmlinux against both lib.a and built-in.a

lib.a is added to the final linking step with --no-whole-archive. Because lib-ksyms.o contains undefined references which are found in lib.a, they are then added into the resulting binary.