AV in WKS::gc_heap::relocate_address

[jkotas]

Hit in Common.Tests on Windows x86 Checked in #34261:

ntdll!KiUserExceptionDispatcher+0xf [d:\rs1\minkernel\ntos\rtl\i386\userdisp.asm @ 597] 
coreclr!WKS::tree_search+0x6 [F:\workspace\_work\1\s\src\coreclr\src\gc\gc.cpp @ 24978] 
coreclr!WKS::gc_heap::relocate_address+0x5e [F:\workspace\_work\1\s\src\coreclr\src\gc\gc.cpp @ 25068] 
coreclr!WKS::GCHeap::Relocate+0x45 [F:\workspace\_work\1\s\src\coreclr\src\gc\gc.cpp @ 36864] 
coreclr!PromoteCarefully+0x83 [F:\workspace\_work\1\s\src\coreclr\src\vm\siginfo.cpp @ 4875] 
coreclr!GcEnumObject+0x4b [F:\workspace\_work\1\s\src\coreclr\src\vm\gcenv.ee.common.cpp @ 167] 
coreclr!EECodeManager::EnumGcRefs+0xc48 [F:\workspace\_work\1\s\src\coreclr\src\vm\eetwain.cpp @ 4702] 
coreclr!GcStackCrawlCallBack+0x152 [F:\workspace\_work\1\s\src\coreclr\src\vm\gcenv.ee.common.cpp @ 290] 
coreclr!Thread::MakeStackwalkerCallback+0x48 [F:\workspace\_work\1\s\src\coreclr\src\vm\stackwalk.cpp @ 841] 
coreclr!Thread::StackWalkFramesEx+0x193 [F:\workspace\_work\1\s\src\coreclr\src\vm\stackwalk.cpp @ 917] 
coreclr!Thread::StackWalkFrames+0x159 [F:\workspace\_work\1\s\src\coreclr\src\vm\stackwalk.cpp @ 1001] 
coreclr!ScanStackRoots+0x1e9 [F:\workspace\_work\1\s\src\coreclr\src\vm\gcenv.ee.cpp @ 146] 
coreclr!GCToEEInterface::GcScanRoots+0xe4 [F:\workspace\_work\1\s\src\coreclr\src\vm\gcenv.ee.cpp @ 175] 
coreclr!WKS::gc_heap::relocate_phase+0x35 [F:\workspace\_work\1\s\src\coreclr\src\gc\gc.cpp @ 25922] 
coreclr!WKS::gc_heap::plan_phase+0x1aa8 [F:\workspace\_work\1\s\src\coreclr\src\gc\gc.cpp @ 24246] 
coreclr!WKS::gc_heap::gc1+0x173 [F:\workspace\_work\1\s\src\coreclr\src\gc\gc.cpp @ 16782] 
coreclr!WKS::gc_heap::garbage_collect+0x3bd [F:\workspace\_work\1\s\src\coreclr\src\gc\gc.cpp @ 18414] 
coreclr!WKS::GCHeap::GarbageCollectGeneration+0x239 [F:\workspace\_work\1\s\src\coreclr\src\gc\gc.cpp @ 38169] 
coreclr!WKS::gc_heap::trigger_gc_for_alloc+0x40 [F:\workspace\_work\1\s\src\coreclr\src\gc\gc.cpp @ 14039] 
coreclr!WKS::gc_heap::try_allocate_more_space+0x1ba [F:\workspace\_work\1\s\src\coreclr\src\gc\gc.cpp @ 14157] 
coreclr!WKS::gc_heap::allocate_more_space+0x14 [F:\workspace\_work\1\s\src\coreclr\src\gc\gc.cpp @ 14587] 
coreclr!WKS::gc_heap::allocate+0x31 [F:\workspace\_work\1\s\src\coreclr\src\gc\gc.cpp @ 14618] 
coreclr!WKS::GCHeap::Alloc+0x199 [F:\workspace\_work\1\s\src\coreclr\src\gc\gc.cpp @ 37242] 
coreclr!Alloc+0x13e [F:\workspace\_work\1\s\src\coreclr\src\vm\gchelpers.cpp @ 231] 
coreclr!AllocateSzArray+0x2ea [F:\workspace\_work\1\s\src\coreclr\src\vm\gchelpers.cpp @ 486] 
coreclr!JIT_NewArr1+0x185 [F:\workspace\_work\1\s\src\coreclr\src\vm\jithelpers.cpp @ 2718] 
System_Private_CoreLib!System.Span`1[[System.Byte, System.Private.CoreLib]].ToArray()+0x9a6b2fa1 [/_/src/libraries/System.Private.CoreLib/src/System/Span.cs @ 477] 
Common_Tests!System.Net.Test.Common.VirtualNetworkStream.Write(Byte[], Int32, Int32)+0x67
Common_Tests!System.Net.Test.Common.VirtualNetworkStreamTest.VirtualNetworkStream_SingleThreadIntegrityTest_Ok()+0x137

[jkotas]

Crash while stackwalking Interop+Kernel32.GetFullPathNameW(Char ByRef, UInt32, Char ByRef, IntPtr)

[jkotas]

@fadimounir This looks like a duplicate of #31809

[fadimounir]

Thanks @jkotas, I will take a look. Are there crash dumps in the failures since I wasn't able to repro this locally?

[jkotas]

I have saved the dump and binaries at \jkotas9\drops\issue-34279 so that it won't get lost.

[fadimounir]

I took a look at the dump, and it looks like we're enumerating the gs cookie of the InlinedCallFrame as a gcref object. Still not sure why

[fadimounir]

@jkotas I've been able to trace the root cause of this issue. The problem is here: https://github.com/dotnet/runtime/blob/master/src/coreclr/src/vm/i386/PInvokeStubs.asm#L58

We're using the return address from the call to the JIT_PInvokeBegin helper, and not the return address of the actual PInvoke. The effect of this is that we report GC roots using incorrect GC data. I've been able to manually get a similar state under debugger, and saw that we were enumerating the gs cookie of the InlinedCallFrame. When I manually got this state under debugger however, the Promote() function was able to ignore this bad value, since it detected it wasn't a valid address in the GC heaps. In the crash dump though, things were not as lucky (maybe the gs cookie value we use in checked builds (0x12345678) fell between the low/high range of the gc heap?)

Anyways, I did test under debugger what the correct behavior would have been by manually patching the return address in the InlinedCallFrame to the correct one, and the GC refs enumeration seemed to be doing the correct thing (nothing was being enumerated from the pinvoke frame at that point).

I think the fix is to pass the correct return address value as a parameter to the JIT_PInvokeBegin helper here: https://github.com/dotnet/runtime/blob/master/src/coreclr/src/jit/lower.cpp#L3635

@dotnet/jit-contrib I need some help figuring out how to do that.

[fadimounir]

BTW, the bug exists on all architectures, all platforms, and not just on Windows-x86. It's just extremely rare that it would ever fail