Multiple JIT Stress test failures

[tannergooding]

While working on dotnet/coreclr#16957 I found there were multiple unrelated tests that were failing.

General Failure

• baseservices_threading.regressions_269336_objmonhelper_objmonhelper: https://ci.dot.net/job/dotnet_coreclr/job/master/job/jitstress/job/x86_checked_windows_nt_gcstress0x3_prtest/19/testReport/junit/baseservices_threading/_regressions_269336_objmonhelper_objmonhelper_/_regressions_269336_objmonhelper_objmonhelper_cmd/
• baseservices_threading.generics_threadstart_GThread02_GThread02: https://ci.dot.net/job/dotnet_coreclr/job/master/job/jitstress/job/x86_checked_windows_nt_jitstressregs8_prtest/24/testReport/junit/baseservices_threading/_generics_threadstart_GThread02_GThread02_/_generics_threadstart_GThread02_GThread02_cmd/
• baseservices_threading.waithandle_waitany_waitanyex5_waitanyex5: https://ci.dot.net/job/dotnet_coreclr/job/master/job/jitstress/job/x64_checked_windows_nt_gcstress0x3_prtest/15/testReport/junit/baseservices_threading/_waithandle_waitany_waitanyex5_waitanyex5_/_waithandle_waitany_waitanyex5_waitanyex5_cmd/
• CoreMangLib_cti.system_reflection_emit_DynMethodJumpStubTests_DynMethodJumpStubTests_DynMethodJumpStubTests: https://ci.dot.net/job/dotnet_coreclr/job/master/job/jitstress/job/x64_checked_windows_nt_gcstress0xc_prtest/47/testReport/CoreMangLib_cti/_system_reflection_emit_DynMethodJumpStubTests_DynMethodJumpStubTests_DynMethodJumpStubTests_/_system_reflection_emit_DynMethodJumpStubTests_DynMethodJumpStubTests_DynMethodJumpStubTests_cmd/
• JIT_Methodical.ELEMENT_TYPE_IU__il_dbgu_fld__il_dbgu_fld: https://ci.dot.net/job/dotnet_coreclr/job/master/job/jitstress/job/x64_checked_windows_nt_gcstress0xc_minopts_heapverify1_prtest/10/testReport/JIT_Methodical/_ELEMENT_TYPE_IU__il_dbgu_fld__il_dbgu_fld_/_ELEMENT_TYPE_IU__il_dbgu_fld__il_dbgu_fld_cmd/
• JIT_Methodical.ELEMENT_TYPE_IU__il_relu_fld__il_relu_fld: https://ci.dot.net/job/dotnet_coreclr/job/master/job/jitstress/job/x86_checked_windows_nt_gcstress0xc_minopts_heapverify1_prtest/17/testReport/junit/JIT_Methodical/_ELEMENT_TYPE_IU__il_relu_fld__il_relu_fld_/_ELEMENT_TYPE_IU__il_relu_fld__il_relu_fld_cmd/
• JIT_Performance.CodeQuality_BenchmarksGame_k_nucleotide_k_nucleotide_9_k_nucleotide_9: https://ci.dot.net/job/dotnet_coreclr/job/master/job/jitstress/job/x64_checked_windows_nt_gcstress0xc_zapdisable_heapverify1_prtest/8/testReport/junit/JIT_Performance/_CodeQuality_BenchmarksGame_k_nucleotide_k_nucleotide_9_k_nucleotide_9_/_CodeQuality_BenchmarksGame_k_nucleotide_k_nucleotide_9_k_nucleotide_9_cmd/
• Loader_classloader.TypeInitialization_CircularCctors_CircularCctorFourThreads_CircularCctorFourThreads: https://ci.dot.net/job/dotnet_coreclr/job/master/job/jitstress/job/x64_checked_windows_nt_gcstress0xc_prtest/47/testReport/Loader_classloader/_TypeInitialization_CircularCctors_CircularCctorFourThreads_CircularCctorFourThreads_/_TypeInitialization_CircularCctors_CircularCctorFourThreads_CircularCctorFourThreads_cmd/
• Loader_classloader.TypeInitialization_CircularCctors_CircularCctorThreeThreads02_CircularCctorThreeThreads02: https://ci.dot.net/job/dotnet_coreclr/job/master/job/jitstress/job/x64_checked_windows_nt_gcstress0xc_zapdisable_heapverify1_prtest/8/testReport/junit/Loader_classloader/_TypeInitialization_CircularCctors_CircularCctorThreeThreads02_CircularCctorThreeThreads02_/_TypeInitialization_CircularCctors_CircularCctorThreeThreads02_CircularCctorThreeThreads02_cmd/

Jobs with many failures

• https://ci.dot.net/job/dotnet_coreclr/job/master/job/jitstress/job/x86_checked_windows_nt_gcstress0xc_zapdisable_jitstress2_prtest/8/#showFailuresLink
• https://ci.dot.net/job/dotnet_coreclr/job/master/job/jitstress/job/x86_checked_windows_nt_gcstress0xc_jitstress2_prtest/12/#showFailuresLink
• https://ci.dot.net/job/dotnet_coreclr/job/master/job/jitstress/job/x86_checked_windows_nt_gcstress0xc_jitstress1_prtest/12/#showFailuresLink
• https://ci.dot.net/job/dotnet_coreclr/job/master/job/jitstress/job/x86_checked_windows_nt_gcstress0xc_jitstress1/68/

baseservices_threading._mutex_misc_mutexctor1_mutexctor1

https://ci.dot.net/job/dotnet_coreclr/job/master/job/jitstress/job/x64_checked_windows_nt_gcstress0xc_minopts_heapverify1_prtest/10/testReport/baseservices_threading/_mutex_misc_mutexctor1_mutexctor1_/_mutex_misc_mutexctor1_mutexctor1_cmd/

	Assert failure(PID 9080 [0x00002378], Thread: 2304 [0x0900]): CONTRACT VIOLATION by Frame::Push at \"d:\\j\\workspace\\x64_checked_w---e8fbcb57\\src\\vm\\frames.cpp\" @ 394

	MODE_COOPERATIVE encountered while thread is in preemptive state.

	 CONTRACT in Frame::Push at \"d:\\j\\workspace\\x64_checked_w---e8fbcb57\\src\\vm\\frames.cpp\" @ 394
	 OVERRIDE_TYPE_LOAD_LEVEL_LIMIT in MethodDescCallSite::CallTargetWorker at \"d:\\j\\workspace\\x64_checked_w---e8fbcb57\\src\\vm\\callhelpers.cpp\" @ 371
	 CONTRACT in MethodDescCallSite::CallTargetWorker at \"d:\\j\\workspace\\x64_checked_w---e8fbcb57\\src\\vm\\callhelpers.cpp\" @ 361
	 CONTRACT in ThreadNative::KickOffThread_Worker at \"d:\\j\\workspace\\x64_checked_w---e8fbcb57\\src\\vm\\comsynchronizable.cpp\" @ 206
	 CONTRACT in ManagedThreadBase_DispatchInner at \"d:\\j\\workspace\\x64_checked_w---e8fbcb57\\src\\vm\\threads.cpp\" @ 8816
	 CONTRACT in ManagedThreadBase_FullTransitionWithAD at \"d:\\j\\workspace\\x64_checked_w---e8fbcb57\\src\\vm\\threads.cpp\" @ 9197
	 CONTRACT in ThreadNative::KickOffThread at \"d:\\j\\workspace\\x64_checked_w---e8fbcb57\\src\\vm\\comsynchronizable.cpp\" @ 305

	We can't find the violated contract. Look for an old-style non-holder-based contract.

	CORECLR! CONTRACT_ASSERT + 0x458 (0x00007ffe`f1db47f8)
	CORECLR! EEContract::DoChecks + 0x252 (0x00007ffe`f1ed1032)
	CORECLR! Frame::Push + 0xD4 (0x00007ffe`f216ecd4)
	CORECLR! LinkFrameAndThrow + 0x32 (0x00007ffe`f1ed0d92)
	CORECLR! NakedThrowHelper2 + 0x9 (0x00007ffe`f23981b9)
	CORECLR! NakedThrowHelper_RspAligned + 0x1E (0x00007ffe`f23981eb)
	CORECLR! NakedThrowHelper_FixRsp + 0x5 (0x00007ffe`f23981f5)
	SYSTEM.PRIVATE.CORELIB! <no symbol> + 0x0 (0x00007ffe`f1811e90)
	SYSTEM.PRIVATE.CORELIB! <no symbol> + 0x0 (0x00007ffe`f18f0ed4)
	<no module>! <no symbol> + 0x0 (0x00007ffe`92ce616f)
	 File: d:\\j\\workspace\\x64_checked_w---e8fbcb57\\src\\vm\\frames.cpp Line: 394
	 Image: D:\\j\\workspace\\x64_checked_w---e8fbcb57\\bin\\tests\\Windows_NT.x64.Checked\\Tests\\Core_Root\\CoreRun.exe

JIT_Methodical.refany__il_relarray3__il_relarray3

https://ci.dot.net/job/dotnet_coreclr/job/master/job/jitstress/job/x86_checked_windows_nt_gcstress0xc_minopts_heapverify1_prtest/17/testReport/junit/JIT_Methodical/_refany__il_relarray3__il_relarray3_/_refany__il_relarray3__il_relarray3_cmd/

	Assert failure(PID 5580 [0x000015cc], Thread: 7364 [0x1cc4]): !hasPartialArgInfo || isZero(argHigh) || (argHigh == CONSTRUCT_ptrArgTP(1, (argCnt-1)))

	CORECLR! scanArgRegTableI + 0x5F3 (0x7261e063)
	CORECLR! EECodeManager::EnumGcRefs + 0x322 (0x72617a62)
	CORECLR! GcStackCrawlCallBack + 0x169 (0x727eda29)
	CORECLR! Thread::MakeStackwalkerCallback + 0x45 (0x7246b345)
	CORECLR! Thread::StackWalkFramesEx + 0x177 (0x7246c9dc)
	CORECLR! Thread::StackWalkFrames + 0x15E (0x7246c7e2)
	CORECLR! ScanStackRoots + 0x18C (0x728f74da)
	CORECLR! GCToEEInterface::GcScanRoots + 0x10F (0x728f6adc)
	CORECLR! WKS::gc_heap::mark_phase + 0x1B1 (0x728c1c2e)
	CORECLR! WKS::gc_heap::gc1 + 0x133 (0x728bc800)
	 File: d:\\j\\workspace\\x86_checked_w---9da75386\\src\\vm\\eetwain.cpp Line: 2658
	 Image: D:\\j\\workspace\\x86_checked_w---9da75386\\bin\\tests\\Windows_NT.x86.Checked\\Tests\\Core_Root\\CoreRun.exe

JIT_Performance.CodeQuality_BenchmarksGame_regex_redux_regex_redux_5_regex_redux_5

https://ci.dot.net/job/dotnet_coreclr/job/master/job/jitstress/job/x86_checked_windows_nt_gcstress0xc_zapdisable_heapverify1_prtest/8/testReport/JIT_Performance/_CodeQuality_BenchmarksGame_regex_redux_regex_redux_5_regex_redux_5_/_CodeQuality_BenchmarksGame_regex_redux_regex_redux_5_regex_redux_5_cmd/

	Assert failure(PID 6868 [0x00001ad4], Thread: 760 [0x02f8]): Consistency check failed: hit privileged instruction!FAILED: !ExecutionManager::IsManagedCode(GetIP(pContext))
	 File: d:\\j\\workspace\\x86_checked_w---a898fbbe\\src\\vm\\excep.cpp Line: 6944
	 Image: D:\\j\\workspace\\x86_checked_w---a898fbbe\\bin\\tests\\Windows_NT.x86.Checked\\Tests\\Core_Root\\CoreRun.exe

JIT_HardwareIntrinsics.X86_Sse2_ConvertToDouble_r_ConvertToDouble_r

https://ci.dot.net/job/dotnet_coreclr/job/master/job/jitstress/job/x64_checked_windows_nt_jitstressregs4_prtest/31/testReport/junit/JIT_HardwareIntrinsics/_X86_Sse2_ConvertToDouble_r_ConvertToDouble_r_/_X86_Sse2_ConvertToDouble_r_ConvertToDouble_r_cmd/

	Assert failure(PID 16440 [0x00004038], Thread: 11024 [0x2b10]): Assertion failed '(unsigned)ins < _countof(insCodesRM)' in 'IntelHardwareIntrinsicTest.Program:Main(ref):int' (IL size 243)

	 File: d:\\j\\workspace\\x64_checked_w---db056dfb\\src\\jit\\emitxarch.cpp Line: 1249
	 Image: D:\\j\\workspace\\x64_checked_w---db056dfb\\bin\\tests\\Windows_NT.x64.Checked\\Tests\\Core_Root\\CoreRun.exe

[tannergooding]

These all need further investigating/resolution (and to possibly be split out into their own issues). Several of these issues were showing in multiple jobs.

[4creators]

@tannergooding

JIT_HardwareIntrinsics.X86_Sse2_ConvertToDouble_r_ConvertToDouble_r seems to be recent regression as test was always passing in the same job before.

[tannergooding]

@4creators, that one has been failing since https://ci.dot.net/job/dotnet_coreclr/job/master/job/jitstress/job/x64_checked_windows_nt_jitstressregs4/465/

It may have not been running before since the JIT Stress Reg jobs don't set EnableIncompleteISAClass.

@CarolEidt, @fiigii, we should ensure we run the full suite of JIT Stress jobs (Reg, GC, etc) on any PR which marks an ISA as complete.

[AndyAyersMS]

Think the ConvertToDouble case may need to use movsdsse2 to distinguish it from the normal movsd. Trying that now.

[AndyAyersMS]

Seems like that did the trick; the StressRegs=4 needs one reg, reg movsd while without stress regs the movsds are all reg/mem variants.

       C4410310F7           vmovsd   xmm14, xmm15

[AndyAyersMS]

I can't repro the k-nucleotide-9 failure in isolation (tried may be 50 runs). Seems like in CI we may be running a lot of tests concurrently and this test is multithreaded so it may experience quite different scheduling on a heavily loaded machine. So I'll see if I can simulate that sort of setup.

[AndyAyersMS]

Ok, I can repro it now, maybe once every 20 runs or so.

[AndyAyersMS]

Think the k-nucleotide-9 may be an issue with GCStress, I see a lot of c0000096 exceptions (priveledged instruction) before a final AV, RAX looks correct, and there is no heap corruption at the point of failure.

(1198.39b8): Unknown exception - code c0000096 (first chance)
(1198.3988): Unknown exception - code c0000096 (first chance)
(1198.3988): Unknown exception - code c0000096 (first chance)
(1198.39b8): Unknown exception - code c0000096 (first chance)
...
(1198.39b8): Unknown exception - code c0000096 (first chance)
(1198.3988): Access violation - code c0000005 (first chance)
First chance exceptions are reported before any exception handling.
This exception may be expected and handled.
System_Private_CoreLib!System.Collections.Generic.ArraySortHelper`1[[System.Int32, System.Private.CoreLib]].CreateArraySortHelper()+0x68:
00007ff9`55c0cc78 ff5028          call    qword ptr [rax+28h] ds:00007ff9`555fc920={System_Private_CoreLib!System.RuntimeType.get_TypeHandle() (00007ff9`55664790)}

;; memory at RAX
00007ff9`555fc8f8 00007ff955630840 System.Type.GetProperty(System.String, System.Reflection.BindingFlags)
00007ff9`555fc900 00007ff955665890 System_Private_CoreLib!System.Type.GetProperty(System.String, System.Reflection.BindingFlags, System.Reflection.Binder, System.Type, System.Type[], System.Reflection.ParameterModifier[])
00007ff9`555fc908 00007ff955665960 System_Private_CoreLib!System.RuntimeType.GetPropertyImpl(System.String, System.Reflection.BindingFlags, System.Reflection.Binder, System.Type, System.Type[], System.Reflection.ParameterModifier[])
00007ff9`555fc910 00007ff955630fe8 System.RuntimeType.GetProperties(System.Reflection.BindingFlags)
00007ff9`555fc918 00007ff955630d98 System.RuntimeType.GetDefaultMembers()
00007ff9`555fc920 00007ff955664790 System_Private_CoreLib!System.RuntimeType.get_TypeHandle()

Will see if I can catch one of these earlier exceptions...

[AndyAyersMS]

Instruction is hlt (0xF4). Looks like this is part of the gc stress infrastructure.

[AndyAyersMS]

Similar looking failure in regex-redux-5:

(276c.4d3c): Unknown exception - code c0000096 (first chance)
(276c.4e88): Access violation - code c0000005 (first chance)
First chance exceptions are reported before any exception handling.
This exception may be expected and handled.
System_Private_CoreLib!System.Reflection.Emit.DynamicILGenerator.GetTokenFor(System.Reflection.RuntimeFieldInfo, System.RuntimeType)+0x1b:
00007ff9`58d275cb ff5030          call    qword ptr [rax+30h] ds:00007ff9`5889ab20={System.Reflection.RtFieldInfo.get_FieldHandle() (00007ff9`58777618)}

Both these tests are heavily multi-threaded so would guess we are hitting some kind of race condition in gcstress here. @swaroop-sridhar does this look at all familiar?

Suggest we disable these from running in stress for now.

[AndyAyersMS]

JIT_Methodical.refany__il_relarray3__il_relarray3 fails deterministically for x86, gcstress 0xC, minopts. Have split this into its own issue dotnet/coreclr#17242.

[AndyAyersMS]

jit\HardwareIntrinsics\X86\Avx\ConvertToVector_ro fails deterministically for x86, gcstress, jitstress=2. Haves split this into its own issue dotnet/coreclr#17278.