[AMD64 Unix] Some CoreCLR tests fail with ELT hooks enabled (PR #7719)

[sergign60]

The following CoreCLR tests fail on Checked (-DFEATURE_GDBJIT=TRUE) build with the following environment

COMPlus_JitELTHookEnabled=1
COMPlus_TailcallStress=1
COMPlus_GCStress=0xC

with the PR dotnet/coreclr#7719
Some of the them (e.g. KeepAlive) fail because of the issue https://github.com/dotnet/coreclr/issues/8921

The tests are

./JIT/Regression/CLR-x86-JIT/V1.2-M01/b07900
./JIT/Regression/VS-ia64-JIT/M00/b112982
./JIT/Regression/VS-ia64-JIT/V1.2-M02/b26496
./JIT/Regression/VS-ia64-JIT/V2.0-Beta2/b302558
./JIT/Regression/VS-ia64-JIT/V2.0-Beta2/b309539
./JIT/jit64/opt/inl/caninline_do
./JIT/jit64/opt/inl/caninline_r
./Loader/classloader/explicitlayout/objrefandnonobjrefoverlap/case11
./Loader/classloader/explicitlayout/objrefandnonobjrefoverlap/case12
./Loader/classloader/explicitlayout/objrefandnonobjrefoverlap/case14
./Loader/classloader/explicitlayout/objrefandnonobjrefoverlap/case15
./Loader/classloader/explicitlayout/objrefandnonobjrefoverlap/case1
./Loader/classloader/explicitlayout/objrefandnonobjrefoverlap/case2
./Loader/classloader/explicitlayout/objrefandnonobjrefoverlap/case3
./Loader/classloader/explicitlayout/objrefandnonobjrefoverlap/case4
./Loader/classloader/explicitlayout/objrefandnonobjrefoverlap/case5
./Loader/classloader/explicitlayout/objrefandnonobjrefoverlap/case6
./Loader/classloader/explicitlayout/objrefandnonobjrefoverlap/case7
./Loader/classloader/explicitlayout/objrefandnonobjrefoverlap/case8
./Loader/classloader/explicitlayout/objrefandnonobjrefoverlap/case9
./JIT/jit64/regress/ddb/87766/ddb87766
./JIT/Generics/Exceptions/general_struct_instance01
./GC/API/GC/GetGeneration
./JIT/Methodical/casts/SEH/_il_dbgcastclass_catch
./JIT/Methodical/tailcall/_il_dbggcval
./JIT/Methodical/casts/SEH/_il_relcastclass_catch
./JIT/Methodical/casts/SEH/_il_relfilter
./GC/API/GC/KeepAlive
./JIT/Regression/CLR-x86-JIT/v2.1/DDB/B168384/LdfldaHack
./Interop/MarshalAPI/Miscellaneous/MarshalClassTests
./JIT/Methodical/flowgraph/bug619534/moduleHandleCache
./Loader/classloader/MethodImpl/override_override1
./JIT/Methodical/casts/SEH/_relthrow
./JIT/Methodical/casts/SEH/_speed_dbgthrow
./JIT/superpmi/superpmicollect
./JIT/Methodical/switch/switch6
./JIT/Methodical/tailcall
./JIT/Methodical/cctor/misc/throw_cs_d
./JIT/Methodical/cctor/misc/throw_cs_ro
./JIT/Methodical/cctor/misc/throw_cs_r
./JIT/Methodical/eh/basics/trythrowexcept_d

The tests

./JIT/Generics/Exceptions/general_struct_instance01
./GC/API/GC/GetGeneration
./GC/API/GC/KeepAlive

give wrong test results, others - Segmentation fault

The test

./GC/API/GC/KeepAlive

fails with

COMPlus_JitELTHookEnabled=1
COMPlus_TailcallStress=1
unset COMPlus_GCStress

others - pass.

[sergign60]

@janvorli @sivarv @Dmitri-Botcharnikov @kvochko

[sergign60]

#7228

[sergign60]

dotnet/coreclr#7719

[gkhanna79]

CC @RussKeldorph

[gkhanna79]

CC @lt72 @noahfalk

[sergign60]

Now I get the another set of failed tests, but the reason is not this profiling callbacks implementation.

For example,

coreclr/Tests/JIT/Regression/VS-ia64-JIT/V1.2-M01/b10827/b10827

fails (segmentation fault) with

COMPlus_JitELTHookEnabled=1
COMPlus_GCStress=0xC
COMPlus_TailcallStress=1

both with the profiler's callback calls and when I commented their calls.

[noahfalk]

@dotnet/jit-contrib

You said it failed when you 'commented their calls', is that the same as having the test fail without any of your changes being present?

[sergign60]

@noahfalk
Yes, please look at issue dotnet/coreclr#10765

[lt72]

@mikem8361 FYI, Noah being OOF

[sergign60]

dotnet/coreclr#12603

[lt72]

This should now be fixed, please re-open if not.