CoreCLR native build on Linux / Linux_musl arm64 / arm causes clang crashes

[trylek]

OS: Linux / Linux_musl
Architecture: arm64, arm

Example run:
https://dev.azure.com/dnceng-public/public/_build/results?buildId=237422&view=logs&j=fd9dce65-fed6-5f8b-f099-1d407f24e80f&t=73b092a7-fc6e-5522-101b-81c9e52247b8

Going over the recent runs it seems to be pretty recent - first repro I've found is from yesterday afternoon:
https://dev.azure.com/dnceng-public/public/_build/results?buildId=237112&view=results

Diagnostics: clang crashes when compiling src/native/external/libunwind/src/elf32.c or elf64.c based on targeting architecture. I don't see any recent changes to their source code so I tend to suspect either a clang version update or some scripting update. The problem may not be fully deterministic - the above example run is the one where this also happens on Linux arm, in the remaining runs I saw it was only reproing on Linux / Linux_musl arm64.

PLEASE submit a bug report to https://bugs.llvm.org/ and include the crash backtrace, preprocessed source, and associated run script.
Stack dump:
0.	Program arguments: /usr/local/bin/clang-12 --target=arm-linux-gnueabihf --gcc-toolchain=/crossrootfs/arm/usr --sysroot=/crossrootfs/arm -DBUILDENV_DEBUG=1 -DCLR_ARM_FPU_CAPABILITY=0x7 -DDEBUG -DDISABLE_CONTRACTS -DHAVE_CONFIG_H=1 -DHAVE___THREAD=0 -DHOST_ARM -DHOST_UNIX -DPACKAGE_BUGREPORT=\"\" -DPACKAGE_STRING=\"\" -DTARGET_32BIT -DTARGET_ARM -DTARGET_LINUX -DTARGET_UNIX -DURTBLDENV_FRIENDLY=Debug -DUSE_STL -D_DBG -D_DEBUG -D_FILE_OFFSET_BITS=64 -D_GNU_SOURCE -Darm_search_unwind_table=UNW_OBJ(arm_search_unwind_table) -Dasm=__asm__ -I/__w/1/s/artifacts/obj/external/libunwind -I/__w/1/s/src/native/external/libunwind_extras -I/__w/1/s/src/native -I/__w/1/s/src/coreclr/pal/inc -I/__w/1/s/src/coreclr/pal/src -I/__w/1/s/src/coreclr/pal/../inc -I/__w/1/s/src/native/external/libunwind/include -I/__w/1/s/src/native/external/libunwind/include/tdep -I/__w/1/s/artifacts/obj/external/libunwind/include -I/__w/1/s/artifacts/obj/external/libunwind/include/tdep -I/__w/1/s/src/native/external/libunwind/include/tdep-arm -I/__w/1/s/src/native/external/libunwind/src -g -fPIC -mthumb -mfpu=vfpv3 -O0 -g -Wall -Wno-null-conversion -fno-omit-frame-pointer -fms-extensions -fwrapv -fstack-protector-strong -Werror -Wno-unused-variable -Wno-unused-value -Wno-unused-function -Wno-tautological-compare -Wno-unknown-pragmas -Wimplicit-fallthrough -Wno-unused-but-set-variable -ffp-contract=off -Wno-unknown-warning-option -ferror-limit=4096 -Wno-unused-private-field -Wno-constant-logical-operand -Wno-pragma-pack -Wno-incompatible-ms-struct -Wno-reserved-identifier -Wno-unsafe-buffer-usage -Wno-single-bit-bitfield-constant-conversion -Wno-cast-function-type-strict -fsigned-char -fvisibility=hidden -ffunction-sections -march=armv7-a -fexceptions -Wno-format -Wno-format-security -Wno-implicit-fallthrough -Wno-header-guard -Wno-absolute-value -Wno-inline-asm -Wno-implicit-function-declaration -Wno-implicit-int-conversion -std=gnu11 -MD -MT /__w/1/s/artifacts/obj/external/libunwind/CMakeFiles/libunwind.dir/__w/1/s/src/native/external/libunwind/src/elf32.c.o -MF CMakeFiles/libunwind.dir/__w/1/s/src/native/external/libunwind/src/elf32.c.o.d -o CMakeFiles/libunwind.dir/__w/1/s/src/native/external/libunwind/src/elf32.c.o -c /__w/1/s/src/native/external/libunwind/src/elf32.c
1.	 parser at end of file
2.	Code generation
[  8%] Building CXX object md/enc/CMakeFiles/mdruntimerw_wks_obj.dir/cmake_pch.hxx.pch
 #0 0x000056228ba9805e llvm::sys::PrintStackTrace(llvm::raw_ostream&, int) (/usr/local/bin/clang-12+0x213f05e)
 #1 0x000056228ba95e54 llvm::sys::RunSignalHandlers() (/usr/local/bin/clang-12+0x213ce54)
 #2 0x000056228ba0c438 CrashRecoverySignalHandler(int) (/usr/local/bin/clang-12+0x20b3438)
 #3 0x00007f29380e9e30 __restore_rt (/lib/libc.so.6+0x42e30)
 #4 0x000056228c53e310 llvm::DIE::getUnitDie() const (/usr/local/bin/clang-12+0x2be5310)
 #5 0x000056228c54869c llvm::DwarfDebug::finishEntityDefinitions() (/usr/local/bin/clang-12+0x2bef69c)
 #6 0x000056228c561a52 llvm::DwarfDebug::finalizeModuleInfo() (/usr/local/bin/clang-12+0x2c08a52)
 #7 0x000056228c564a68 llvm::DwarfDebug::endModule() (/usr/local/bin/clang-12+0x2c0ba68)
 #8 0x000056228c535aa9 llvm::AsmPrinter::doFinalization(llvm::Module&) (/usr/local/bin/clang-12+0x2bdcaa9)
 #9 0x000056228b3ef55d llvm::FPPassManager::doFinalization(llvm::Module&) (.localalias) (/usr/local/bin/clang-12+0x1a9655d)
#10 0x000056228b3fb020 llvm::legacy::PassManagerImpl::run(llvm::Module&) (/usr/local/bin/clang-12+0x1aa2020)
#11 0x000056228bd2bf2c (anonymous namespace)::EmitAssemblyHelper::EmitAssembly(clang::BackendAction, std::unique_ptr >) (.constprop.0) (/usr/local/bin/clang-12+0x23d2f2c)
#12 0x000056228bd2dc7c clang::EmitBackendOutput(clang::DiagnosticsEngine&, clang::HeaderSearchOptions const&, clang::CodeGenOptions const&, clang::TargetOptions const&, clang::LangOptions const&, llvm::DataLayout const&, llvm::Module*, clang::BackendAction, std::unique_ptr >) (/usr/local/bin/clang-12+0x23d4c7c)
#13 0x000056228c96670c clang::BackendConsumer::HandleTranslationUnit(clang::ASTContext&) (/usr/local/bin/clang-12+0x300d70c)
#14 0x000056228d430b39 clang::ParseAST(clang::Sema&, bool, bool) (/usr/local/bin/clang-12+0x3ad7b39)
#15 0x000056228c33b269 clang::FrontendAction::Execute() (/usr/local/bin/clang-12+0x29e2269)
#16 0x000056228c2deccb clang::CompilerInstance::ExecuteAction(clang::FrontendAction&) (/usr/local/bin/clang-12+0x2985ccb)
#17 0x000056228c3e9c58 clang::ExecuteCompilerInvocation(clang::CompilerInstance*) (/usr/local/bin/clang-12+0x2a90c58)
#18 0x000056228a50788f cc1_main(llvm::ArrayRef, char const*, void*) (/usr/local/bin/clang-12+0xbae88f)
#19 0x000056228a5052a0 ExecuteCC1Tool(llvm::SmallVectorImpl&) (/usr/local/bin/clang-12+0xbac2a0)
#20 0x000056228c18d5b5 void llvm::function_ref::callback_fn >, std::__cxx11::basic_string, std::allocator >*, bool*) const::'lambda'()>(long) (/usr/local/bin/clang-12+0x28345b5)
#21 0x000056228ba0c533 llvm::CrashRecoveryContext::RunSafely(llvm::function_ref) (/usr/local/bin/clang-12+0x20b3533)
#22 0x000056228c18e6df clang::driver::CC1Command::Execute(llvm::ArrayRef >, std::__cxx11::basic_string, std::allocator >*, bool*) const (.part.0) (/usr/local/bin/clang-12+0x28356df)
#23 0x000056228c164ad7 clang::driver::Compilation::ExecuteCommand(clang::driver::Command const&, clang::driver::Command const*&) const (/usr/local/bin/clang-12+0x280bad7)
#24 0x000056228c165519 clang::driver::Compilation::ExecuteJobs(clang::driver::JobList const&, llvm::SmallVectorImpl >&) const (/usr/local/bin/clang-12+0x280c519)
#25 0x000056228c172f29 clang::driver::Driver::ExecuteCompilation(clang::driver::Compilation&, llvm::SmallVectorImpl >&) (/usr/local/bin/clang-12+0x2819f29)
#26 0x000056228a47ecb2 main (/usr/local/bin/clang-12+0xb25cb2)
#27 0x00007f29380d557d __libc_start_call_main (/lib/libc.so.6+0x2e57d)
#28 0x00007f29380d5630 __libc_start_main_impl (/lib/libc.so.6+0x2e630)
#29 0x000056228a504c85 _start (/usr/local/bin/clang-12+0xbabc85)
clang-12: error: clang frontend command failed with exit code 139 (use -v to see invocation)
clang version 12.0.1
Target: arm-unknown-linux-gnueabihf
Thread model: posix
InstalledDir: /usr/local/bin
[  8%] Building CXX object md/compiler/CMakeFiles/mdcompiler_wks_obj.dir/assemblymd.cpp.o
clang-12: note: diagnostic msg: 
********************

PLEASE ATTACH THE FOLLOWING FILES TO THE BUG REPORT:
Preprocessed source(s) and associated run script(s) are located at:
clang-12: note: diagnostic msg: /tmp/elf32-2e3faa.c
clang-12: note: diagnostic msg: /tmp/elf32-2e3faa.sh
clang-12: note: diagnostic msg: 

********************

I must admit I don't yet fully understand what the "parser at end of file" refers to, whether it's the compiler command line or the source file.

/cc @dotnet/runtime-infrastructure @janvorli

Tagging subscribers to this area: @hoyosjs
See info in area-owners.md if you want to be subscribed.

Issue Details

---

OS: Linux / Linux_musl
Architecture: arm64, arm

Example run:
https://dev.azure.com/dnceng-public/public/_build/results?buildId=237422&view=logs&j=fd9dce65-fed6-5f8b-f099-1d407f24e80f&t=73b092a7-fc6e-5522-101b-81c9e52247b8

Going over the recent runs it seems to be pretty recent - first repro I've found is from yesterday afternoon:
https://dev.azure.com/dnceng-public/public/_build/results?buildId=237112&view=results

Diagnostics: clang crashes when compiling src/native/external/libunwind/src/elf32.c or elf64.c based on targeting architecture. I don't see any recent changes to their source code so I tend to suspect either a clang version update or some scripting update. The problem may not be fully deterministic - the above example run is the one where this also happens on Linux arm, in the remaining runs I saw it was only reproing on Linux / Linux_musl arm64.

PLEASE submit a bug report to https://bugs.llvm.org/ and include the crash backtrace, preprocessed source, and associated run script.
Stack dump:
0.	Program arguments: /usr/local/bin/clang-12 --target=arm-linux-gnueabihf --gcc-toolchain=/crossrootfs/arm/usr --sysroot=/crossrootfs/arm -DBUILDENV_DEBUG=1 -DCLR_ARM_FPU_CAPABILITY=0x7 -DDEBUG -DDISABLE_CONTRACTS -DHAVE_CONFIG_H=1 -DHAVE___THREAD=0 -DHOST_ARM -DHOST_UNIX -DPACKAGE_BUGREPORT=\"\" -DPACKAGE_STRING=\"\" -DTARGET_32BIT -DTARGET_ARM -DTARGET_LINUX -DTARGET_UNIX -DURTBLDENV_FRIENDLY=Debug -DUSE_STL -D_DBG -D_DEBUG -D_FILE_OFFSET_BITS=64 -D_GNU_SOURCE -Darm_search_unwind_table=UNW_OBJ(arm_search_unwind_table) -Dasm=__asm__ -I/__w/1/s/artifacts/obj/external/libunwind -I/__w/1/s/src/native/external/libunwind_extras -I/__w/1/s/src/native -I/__w/1/s/src/coreclr/pal/inc -I/__w/1/s/src/coreclr/pal/src -I/__w/1/s/src/coreclr/pal/../inc -I/__w/1/s/src/native/external/libunwind/include -I/__w/1/s/src/native/external/libunwind/include/tdep -I/__w/1/s/artifacts/obj/external/libunwind/include -I/__w/1/s/artifacts/obj/external/libunwind/include/tdep -I/__w/1/s/src/native/external/libunwind/include/tdep-arm -I/__w/1/s/src/native/external/libunwind/src -g -fPIC -mthumb -mfpu=vfpv3 -O0 -g -Wall -Wno-null-conversion -fno-omit-frame-pointer -fms-extensions -fwrapv -fstack-protector-strong -Werror -Wno-unused-variable -Wno-unused-value -Wno-unused-function -Wno-tautological-compare -Wno-unknown-pragmas -Wimplicit-fallthrough -Wno-unused-but-set-variable -ffp-contract=off -Wno-unknown-warning-option -ferror-limit=4096 -Wno-unused-private-field -Wno-constant-logical-operand -Wno-pragma-pack -Wno-incompatible-ms-struct -Wno-reserved-identifier -Wno-unsafe-buffer-usage -Wno-single-bit-bitfield-constant-conversion -Wno-cast-function-type-strict -fsigned-char -fvisibility=hidden -ffunction-sections -march=armv7-a -fexceptions -Wno-format -Wno-format-security -Wno-implicit-fallthrough -Wno-header-guard -Wno-absolute-value -Wno-inline-asm -Wno-implicit-function-declaration -Wno-implicit-int-conversion -std=gnu11 -MD -MT /__w/1/s/artifacts/obj/external/libunwind/CMakeFiles/libunwind.dir/__w/1/s/src/native/external/libunwind/src/elf32.c.o -MF CMakeFiles/libunwind.dir/__w/1/s/src/native/external/libunwind/src/elf32.c.o.d -o CMakeFiles/libunwind.dir/__w/1/s/src/native/external/libunwind/src/elf32.c.o -c /__w/1/s/src/native/external/libunwind/src/elf32.c
1.	 parser at end of file
2.	Code generation
[  8%] Building CXX object md/enc/CMakeFiles/mdruntimerw_wks_obj.dir/cmake_pch.hxx.pch
 #0 0x000056228ba9805e llvm::sys::PrintStackTrace(llvm::raw_ostream&, int) (/usr/local/bin/clang-12+0x213f05e)
 #1 0x000056228ba95e54 llvm::sys::RunSignalHandlers() (/usr/local/bin/clang-12+0x213ce54)
 #2 0x000056228ba0c438 CrashRecoverySignalHandler(int) (/usr/local/bin/clang-12+0x20b3438)
 #3 0x00007f29380e9e30 __restore_rt (/lib/libc.so.6+0x42e30)
 #4 0x000056228c53e310 llvm::DIE::getUnitDie() const (/usr/local/bin/clang-12+0x2be5310)
 #5 0x000056228c54869c llvm::DwarfDebug::finishEntityDefinitions() (/usr/local/bin/clang-12+0x2bef69c)
 #6 0x000056228c561a52 llvm::DwarfDebug::finalizeModuleInfo() (/usr/local/bin/clang-12+0x2c08a52)
 #7 0x000056228c564a68 llvm::DwarfDebug::endModule() (/usr/local/bin/clang-12+0x2c0ba68)
 #8 0x000056228c535aa9 llvm::AsmPrinter::doFinalization(llvm::Module&) (/usr/local/bin/clang-12+0x2bdcaa9)
 #9 0x000056228b3ef55d llvm::FPPassManager::doFinalization(llvm::Module&) (.localalias) (/usr/local/bin/clang-12+0x1a9655d)
#10 0x000056228b3fb020 llvm::legacy::PassManagerImpl::run(llvm::Module&) (/usr/local/bin/clang-12+0x1aa2020)
#11 0x000056228bd2bf2c (anonymous namespace)::EmitAssemblyHelper::EmitAssembly(clang::BackendAction, std::unique_ptr >) (.constprop.0) (/usr/local/bin/clang-12+0x23d2f2c)
#12 0x000056228bd2dc7c clang::EmitBackendOutput(clang::DiagnosticsEngine&, clang::HeaderSearchOptions const&, clang::CodeGenOptions const&, clang::TargetOptions const&, clang::LangOptions const&, llvm::DataLayout const&, llvm::Module*, clang::BackendAction, std::unique_ptr >) (/usr/local/bin/clang-12+0x23d4c7c)
#13 0x000056228c96670c clang::BackendConsumer::HandleTranslationUnit(clang::ASTContext&) (/usr/local/bin/clang-12+0x300d70c)
#14 0x000056228d430b39 clang::ParseAST(clang::Sema&, bool, bool) (/usr/local/bin/clang-12+0x3ad7b39)
#15 0x000056228c33b269 clang::FrontendAction::Execute() (/usr/local/bin/clang-12+0x29e2269)
#16 0x000056228c2deccb clang::CompilerInstance::ExecuteAction(clang::FrontendAction&) (/usr/local/bin/clang-12+0x2985ccb)
#17 0x000056228c3e9c58 clang::ExecuteCompilerInvocation(clang::CompilerInstance*) (/usr/local/bin/clang-12+0x2a90c58)
#18 0x000056228a50788f cc1_main(llvm::ArrayRef, char const*, void*) (/usr/local/bin/clang-12+0xbae88f)
#19 0x000056228a5052a0 ExecuteCC1Tool(llvm::SmallVectorImpl&) (/usr/local/bin/clang-12+0xbac2a0)
#20 0x000056228c18d5b5 void llvm::function_ref::callback_fn >, std::__cxx11::basic_string, std::allocator >*, bool*) const::'lambda'()>(long) (/usr/local/bin/clang-12+0x28345b5)
#21 0x000056228ba0c533 llvm::CrashRecoveryContext::RunSafely(llvm::function_ref) (/usr/local/bin/clang-12+0x20b3533)
#22 0x000056228c18e6df clang::driver::CC1Command::Execute(llvm::ArrayRef >, std::__cxx11::basic_string, std::allocator >*, bool*) const (.part.0) (/usr/local/bin/clang-12+0x28356df)
#23 0x000056228c164ad7 clang::driver::Compilation::ExecuteCommand(clang::driver::Command const&, clang::driver::Command const*&) const (/usr/local/bin/clang-12+0x280bad7)
#24 0x000056228c165519 clang::driver::Compilation::ExecuteJobs(clang::driver::JobList const&, llvm::SmallVectorImpl >&) const (/usr/local/bin/clang-12+0x280c519)
#25 0x000056228c172f29 clang::driver::Driver::ExecuteCompilation(clang::driver::Compilation&, llvm::SmallVectorImpl >&) (/usr/local/bin/clang-12+0x2819f29)
#26 0x000056228a47ecb2 main (/usr/local/bin/clang-12+0xb25cb2)
#27 0x00007f29380d557d __libc_start_call_main (/lib/libc.so.6+0x2e57d)
#28 0x00007f29380d5630 __libc_start_main_impl (/lib/libc.so.6+0x2e630)
#29 0x000056228a504c85 _start (/usr/local/bin/clang-12+0xbabc85)
clang-12: error: clang frontend command failed with exit code 139 (use -v to see invocation)
clang version 12.0.1
Target: arm-unknown-linux-gnueabihf
Thread model: posix
InstalledDir: /usr/local/bin
[  8%] Building CXX object md/compiler/CMakeFiles/mdcompiler_wks_obj.dir/assemblymd.cpp.o
clang-12: note: diagnostic msg: 
********************

PLEASE ATTACH THE FOLLOWING FILES TO THE BUG REPORT:
Preprocessed source(s) and associated run script(s) are located at:
clang-12: note: diagnostic msg: /tmp/elf32-2e3faa.c
clang-12: note: diagnostic msg: /tmp/elf32-2e3faa.sh
clang-12: note: diagnostic msg: 

********************

I must admit I don't yet fully understand what the "parser at end of file" refers to, whether it's the compiler command line or the source file.

/cc @dotnet/runtime-infrastructure @janvorli

Author:	trylek
Assignees:	-
Labels:	area-Infrastructure-coreclr, blocking-outerloop
Milestone:	-

[sbomer]

This looks like #84503. I was seeing it in linux arm64 mono build, but hadn't seen it in other builds. dotnet/dotnet-buildtools-prereqs-docker#842 seemed to fix it for me. But the images in dotnet-buildtools-prereqs-docker have already been updated to llvm 16, which we are not yet ready to use. I think we'll want to roll back the images to llvm 12 to fix this ASAP.

[janvorli]

@sbomer I am a bit confused - the failure logged above was caused by clang 12, why do we need to revert the upgrade to clang 16?

[sbomer]

Runtime is currently building with a set of mariner images that have clang 12, built with gcc. There is no published set of images that has clang 12, built with clang - because the llvm 16 upgrade went in before the build-with-clang fix. So I'm reverting the upgrade temporarily to get us a new set of images that have clang-built clang 12.

[janvorli]

Ah, got it, thank you for the explanation.

[trylek]

Frankly speaking I'm not getting why it doesn't fail the same way in the runtime (PR / CI) pipeline, e.g. in this Egor's PR run started about half an hour ago:

https://dev.azure.com/dnceng-public/public/_build/results?buildId=238687&view=results

[sbomer]

I'm not sure either - I was able to reproduce it consistently when I saw the failure building mono arm64. But it might be very sensitive to the inputs.

[sbomer]

I believe this is fixed with #84851. Please reopen if you hit this again.