Native debugging with symbols is broken on Linux/arm64

[echesakov]

It appears that #33716 causes the following issues on Linux/arm64 (it could be on other architectures as well)

This is what shows up when I run gdb

warning: the debug information found in "/home/robox/echesako/33749/artifacts/tests/coreclr/Linux.arm64.Checked/Tests/Core_Root/libclrjit.so.dbg" does not match "/home/robox/echesako/33749/artifacts/tests/coreclr/Linux.arm64.Checked/Tests/Core_Root/libclrjit.so" (CRC mismatch).

and this is output of bt command in lldb

(lldb) bt
* thread #12, name = 'corerun', stop reason = signal SIGTRAP
  * frame #0: 0x0000ffffb3644c0c libclrjit.so`DBG_DebugBreak
    frame #1: 0x0000ffffb35e6120 libclrjit.so`DebugBreak + 708
    frame #2: 0x0000ffffb33bdab8 libclrjit.so`assertAbort + 268
    frame #3: 0x0000ffffb339e334 libclrjit.so`Compiler::compInitOptions(JitFlags*) + 2064
    frame #4: 0x0000ffffb33a311c libclrjit.so`Compiler::compCompileHelper(CORINFO_MODULE_STRUCT_*, ICorJitInfo*, CORINFO_METHOD_INFO*, void**, unsigned int*, JitFlags*) + 240
    frame #5: 0x0000ffffb33a2580 libclrjit.so`Compiler::compCompile(CORINFO_METHOD_STRUCT_*, CORINFO_MODULE_STRUCT_*, ICorJitInfo*, CORINFO_METHOD_INFO*, void**, unsigned int*, JitFlags*) + 1824
    frame #6: 0x0000ffffb33a44f0 libclrjit.so`jitNativeCode(CORINFO_METHOD_STRUCT_*, CORINFO_MODULE_STRUCT_*, ICorJitInfo*, CORINFO_METHOD_INFO*, void**, unsigned int*, JitFlags*, void*) + 436
    frame #7: 0x0000ffffb33ae384 libclrjit.so`CILJit::compileMethod(ICorJitInfo*, CORINFO_METHOD_INFO*, unsigned int, unsigned char**, unsigned int*) + 292

As you can see there is no information about the source files.

If I manually disable stripping the symbols by commenting out the following lines

runtime/eng/native/functions.cmake

Lines 282 to 290 in db23750

	add_custom_command(
	TARGET ${targetName}
	POST_BUILD
	VERBATIM
	COMMAND ${CMAKE_OBJCOPY} --only-keep-debug ${strip_source_file} ${strip_destination_file}
	COMMAND ${CMAKE_OBJCOPY} --strip-debug ${strip_source_file}
	COMMAND ${CMAKE_OBJCOPY} --add-gnu-debuglink=${strip_destination_file} ${strip_source_file}
	COMMENT Stripping symbols from ${strip_source_file} into file ${strip_destination_file}
	)

everything goes back to normal

(lldb) bt
* thread #12, name = 'corerun', stop reason = signal SIGTRAP
  * frame #0: 0x0000ffffb3644c0c libclrjit.so`DBG_DebugBreak at debugbreak.S:8
    frame #1: 0x0000ffffb35e6120 libclrjit.so`::DebugBreak() at debug.cpp:405:9
    frame #2: 0x0000ffffb33bdab8 libclrjit.so`::assertAbort(why=0x0000ffffb3671ca1, file=0x0000ffffb3671880, line=2910) at error.cpp:295:9
    frame #3: 0x0000ffffb339e334 libclrjit.so`Compiler::compInitOptions(this=0x0000aaaaab3f0e28, jitFlags=0x0000fffee7ff99a0) at compiler.cpp:2910:9
    frame #4: 0x0000ffffb33a311c libclrjit.so`Compiler::compCompileHelper(this=0x0000aaaaab3f0e28, classPtr=0x0000ffff453ae3a8, compHnd=0x0000fffee7ff9cd8, methodInfo=0x0000fffee7ff9b90, methodCodePtr=0x0000fffee7ff9988, methodCodeSize=0x0000fffee7ff9b5c, compileFlags=0x0000fffee7ff99a0) at compiler.cpp:5785:5
    frame #5: 0x0000ffffb33a2580 libclrjit.so`Compiler::compCompile(CORINFO_METHOD_STRUCT_*, CORINFO_MODULE_STRUCT_*, ICorJitInfo*, CORINFO_METHOD_INFO*, void**, unsigned int*, JitFlags*) at compiler.cpp:5392:28
    frame #6: 0x0000ffffb33a2564 libclrjit.so`Compiler::compCompile(this=0x0000aaaaab3f0e28, methodHnd=<unavailable>, classPtr=0x0000ffff453ae3a8, compHnd=0x0000fffee7ff9cd8, methodInfo=0x0000fffee7ff9b90, methodCodePtr=0x0000fffee7ff9988, methodCodeSize=0x0000fffee7ff9b5c, compileFlags=0x0000fffee7ff99a0) at compiler.cpp:5411
    frame #7: 0x0000ffffb33a44f0 libclrjit.so`jitNativeCode(CORINFO_METHOD_STRUCT_*, CORINFO_MODULE_STRUCT_*, ICorJitInfo*, CORINFO_METHOD_INFO*, void**, unsigned int*, JitFlags*, void*) at compiler.cpp:6693:32
    frame #8: 0x0000ffffb33a4444 libclrjit.so`jitNativeCode(CORINFO_METHOD_STRUCT_*, CORINFO_MODULE_STRUCT_*, ICorJitInfo*, CORINFO_METHOD_INFO*, void**, unsigned int*, JitFlags*, void*) at compiler.cpp:6718
    frame #9: 0x0000ffffb33a4440 libclrjit.so`jitNativeCode(methodHnd=0x0000ffff454d4080, classPtr=0x0000ffff453ae3a8, compHnd=0x0000fffee7ff9cd8, methodInfo=0x0000fffee7ff9b90, methodCodePtr=0x0000fffee7ff9988, methodCodeSize=0x0000fffee7ff9b5c, compileFlags=0x0000fffee7ff99a0, inlineInfoPtr=0x0000000000000000) at compiler.cpp:6720
    frame #10: 0x0000ffffb33ae384 libclrjit.so`CILJit::compileMethod(this=<unavailable>, compHnd=0x0000fffee7ff9cd8, methodInfo=0x0000fffee7ff9b90, flags=<unavailable>, entryAddress=0x0000fffee7ff9b60, nativeSizeOfCode=0x0000fffee7ff9b5c) at ee_il_dll.cpp:304:14

cc @janvorli @am11

[am11]

@echesakovMSFT, does it work in Release, and only breaks in Checked and Debug, or only in Checked? If that is the case, then it sounds like a configuration-specific overwriting issue, which can be fixed.
Also, are you compiling it on device or using -cross?

[janvorli]

I can confirm the same issue happens on x64 with debug build. And the .dbg files definitely come from the same build as the executables. lldb just silently fails to load the symbols, dbg displays a warning that the CRC doesn't match.

[janvorli]

Interesting, if I comment out run the 2nd and 3rd strip commans

   COMMAND ${CMAKE_OBJCOPY} --strip-debug ${strip_source_file} 
   COMMAND ${CMAKE_OBJCOPY} --add-gnu-debuglink=${strip_destination_file} ${strip_source_file} 

and then run the strip commands manually in the target artifacts folder after the build, the crc32 matches and debugger can load it.

[am11]

@janvorli, interesting. I think if CRC check is satisfied in Release and only affected in Checked and Debug, we can conditionally add these two lines under:

if(NOT uppercase CMAKE_BUILD_TYPE MATCHES "^(DEBUG|CHECKED)$")

or perhaps investigate the root casue and see if we can fix that instead?

[janvorli]

Even more interesting, the .dbg file ended up being 10x larger when I ran the commands manually. And looking at the build log it seems we might be executing the whole custom command twice. I can see doubled lines like

$<TARGET_FILE:clrjit>.dbg
$<TARGET_FILE:clrjit>.dbg

[janvorli]

Yes, it is the case. I've added a message into the stripsymbols function and I can see that it is invoked twice for each target.

[janvorli]

And here we have the culprit:

runtime/src/coreclr/src/dlls/mscoree/coreclr/CMakeLists.txt

Lines 216 to 217 in 5a1d115

	install_clr(TARGETS coreclr)
	install_clr(TARGETS coreclr DESTINATION sharedFramework)

It used to be

install_clr(TARGETS coreclr)
install_clr(TARGETS coreclr DESTINATION sharedFramework SKIP_STRIP)

So the 2nd stripping was prevented before.

[janvorli]

I am going to fix that