Test failure: Interop\\COM\\NativeClients\\Events\\Events.cmd

[v-haren]

failed in job: runtime-coreclr gcstress0x3-gcstress0xc 20200531.1

Error message

Return code: 1
Raw output file: C:\h\w\A7A808D4\w\AD6E08F2\e\Interop\COM\Reports\Interop.COM\NativeClients\Events\Events.output.txt
Raw output:
BEGIN EXECUTION
Copying 'C:\h\w\A7A808D4\p\CoreShim.dll'...
 1 file(s) copied.
 "C:\h\w\A7A808D4\p\corerun.exe" Events.dll 
Searching for exe to launch in C:\h\w\A7A808D4\w\AD6E08F2\e\Interop\COM\NativeClients\Events...
Launching 'C:\h\w\A7A808D4\w\AD6E08F2\e\Interop\COM\NativeClients\Events\COMClientEvents.exe'...

Expected: 100
Actual: -1073741819
END EXECUTION - FAILED
FAILED
Test Harness Exitcode is : 1
To run the test:
> set CORE_ROOT=C:\h\w\A7A808D4\p
> C:\h\w\A7A808D4\w\AD6E08F2\e\Interop\COM\NativeClients\Events\Events.cmd
Expected: True
Actual: False


Stack trace
   at Interop_COM._NativeClients_Events_Events_._NativeClients_Events_Events_cmd() in F:\workspace\_work\1\s\artifacts\tests\coreclr\Windows_NT.x86.Checked\TestWrappers\Interop.COM\Interop.COM.XUnitWrapper.cs:line 916

[BruceForstall]

CoreCLR Windows_NT x86 Checked gcstress0xc @ Windows.10.Amd64.Open

[BruceForstall]

@jkoritzinsky @AaronRobinsonMSFT another interop gcstress failure

[AaronRobinsonMSFT]

I can't reproduce this locally at all. I have tried COMPlus_GCStress=0xC and COMPlus_GCStress=0x3 10 times each. I tried this on both DEBUG and CHECKED builds without any success. I am going to close this for now, but please reopen if it reproduces again.

There was a recent change in #37116 which fixed a long standing issue in x86 with GCStress. If this reoccurs in the future a deeper investigation will be warranted.

[BruceForstall]

@AaronRobinsonMSFT This failed again this weekend in the weekly GCstress run, e.g.:

https://dev.azure.com/dnceng/public/_build/results?buildId=687273&view=ms.vss-test-web.build-test-results-tab&runId=21339984&paneView=debug&resultId=108497

[AaronRobinsonMSFT]

I had to run this in a loop 3000 times... it finally triggered on 2984 which took about 2.5 hrs. This is not the test itself, but rather the runner. I have a full DMP of the issue.

0:000> !clrstack
OS Thread Id: 0x4da4 (0)
        Child SP               IP Call Site
000000E62897CE88 00007ffb62a4f97f [RedirectedThreadFrame: 000000e62897ce88] 
000000E62897D070 00007ffaacc32616 [InlinedCallFrame: 000000e62897d070] Interop+Kernel32.CreatePipe(Microsoft.Win32.SafeHandles.SafeFileHandle ByRef, Microsoft.Win32.SafeHandles.SafeFileHandle ByRef, SECURITY_ATTRIBUTES ByRef, Int32)
000000E62897D040 00007ffaacc32616 ILStubClass.IL_STUB_PInvoke(Microsoft.Win32.SafeHandles.SafeFileHandle ByRef, Microsoft.Win32.SafeHandles.SafeFileHandle ByRef, SECURITY_ATTRIBUTES ByRef, Int32)
000000E62897D1A0 00007ffaacc32389 System.Diagnostics.Process.CreatePipeWithSecurityAttributes(Microsoft.Win32.SafeHandles.SafeFileHandle ByRef, Microsoft.Win32.SafeHandles.SafeFileHandle ByRef, SECURITY_ATTRIBUTES ByRef, Int32) [D:\runtime\src\libraries\System.Diagnostics.Process\src\System\Diagnostics\Process.Windows.cs @ 828]
000000E62897D1F0 00007ffaacc32275 System.Diagnostics.Process.CreatePipe(Microsoft.Win32.SafeHandles.SafeFileHandle ByRef, Microsoft.Win32.SafeHandles.SafeFileHandle ByRef, Boolean) [D:\runtime\src\libraries\System.Diagnostics.Process\src\System\Diagnostics\Process.Windows.cs @ 866]
000000E62897D270 00007ffaacc30eac System.Diagnostics.Process.StartWithCreateProcess(System.Diagnostics.ProcessStartInfo) [D:\runtime\src\libraries\System.Diagnostics.Process\src\System\Diagnostics\Process.Windows.cs @ 513]
000000E62897D680 00007ffaacc2fd8e System.Diagnostics.Process.StartCore(System.Diagnostics.ProcessStartInfo)
000000E62897D6B0 00007ffaacc2c4d8 System.Diagnostics.Process.Start() [D:\runtime\src\libraries\System.Diagnostics.Process\src\System\Diagnostics\Process.cs @ 1228]
000000E62897D760 00007ffaacc2b7d2 System.Diagnostics.Process.Start(System.Diagnostics.ProcessStartInfo) [D:\runtime\src\libraries\System.Diagnostics.Process\src\System\Diagnostics\Process.cs @ 1277]
000000E62897D7B0 00007ffaacc19e5b Program.Main(System.String[]) [D:\runtime\src\coreclr\tests\src\Interop\common\ExeLauncherProgram.cs @ 65]

0:000> kb
 # RetAddr           : Args to Child                                                           : Call Site
00 00007ffb`0bfc1b4f : 00000000`00000002 00007ffb`0c96bfdc 000002b9`ed105fc8 00007ffb`0cb9d2d0 : KERNELBASE!RaiseFailFastException+0xaf [minkernel\kernelbase\xcpt.c @ 1133] 
01 (Inline Function) : --------`-------- --------`-------- --------`-------- --------`-------- : coreclr!TerminateOnAssert+0x17 [F:\workspace\_work\1\s\src\coreclr\src\utilcode\debug.cpp @ 190] 
02 00007ffb`0bfc12c0 : 00007ffb`0c983248 fefefefe`00001d46 00007ffb`0cb9a9c0 fefefefe`00000000 : coreclr!_DbgBreakCheck+0x483 [F:\workspace\_work\1\s\src\coreclr\src\utilcode\debug.cpp @ 428] 
03 00007ffb`0bfc14a2 : 00000000`00000000 00007ffb`0cb9a9c0 00000000`00000001 00000000`00000001 : coreclr!_DbgBreakCheckNoThrow+0x50 [F:\workspace\_work\1\s\src\coreclr\src\utilcode\debug.cpp @ 535] 
04 00007ffb`0bfc7d25 : 00007ffb`0c983248 000002b9`00001d46 000002b9`eebf3c48 000000e6`2897bbe8 : coreclr!DbgAssertDialog+0x18e [F:\workspace\_work\1\s\src\coreclr\src\utilcode\debug.cpp @ 696] 
05 00007ffb`0c020a86 : 000000e6`2897bbe8 000002b9`ed0fc6b8 000002b9`ed0c9e90 000002b9`ed0ee640 : coreclr!CHECK::Trigger+0x2c9 [F:\workspace\_work\1\s\src\coreclr\src\utilcode\check.cpp @ 140] 
06 00007ffb`0c020547 : 00000000`00000028 000002b9`eab2f890 00000000`00000001 000000e6`2897c110 : coreclr!CLRVectoredExceptionHandlerPhase3+0x2b2 [F:\workspace\_work\1\s\src\coreclr\src\vm\excep.cpp @ 7493] 
07 00007ffb`0c02048d : 00000000`00000028 00000000`00000028 000002b9`eaba1230 000002b9`eab2f890 : coreclr!CLRVectoredExceptionHandlerPhase2+0x97 [F:\workspace\_work\1\s\src\coreclr\src\vm\excep.cpp @ 7220] 
08 00007ffb`0c020d00 : 000002b9`eab2f890 00000000`00000000 000002b9`eaba1230 ffffffff`ffffffff : coreclr!CLRVectoredExceptionHandler+0x215 [F:\workspace\_work\1\s\src\coreclr\src\vm\excep.cpp @ 7186] 
09 00007ffb`655b85b6 : 00007ffb`0c020b60 000000e6`2897c100 000002b9`ec560880 00007ffb`65593550 : coreclr!CLRVectoredExceptionHandlerShim+0x1a0 [F:\workspace\_work\1\s\src\coreclr\src\vm\excep.cpp @ 7922] 
0a 00007ffb`655aa056 : 000000e6`2897c870 000000e6`2897c380 00007ffb`62971e00 00007ffb`0c7a2429 : ntdll!RtlpCallVectoredHandlers+0x196 [minkernel\ntdll\vectxcpt.c @ 204] 
0b (Inline Function) : --------`-------- --------`-------- --------`-------- --------`-------- : ntdll!RtlCallVectoredExceptionHandlers+0xe [minkernel\ntdll\vectxcpt.c @ 339] 
0c 00007ffb`655dfe3e : 00007ffb`0cb0cb00 00007ffb`65560299 00000000`00000000 000000e6`2897c448 : ntdll!RtlDispatchException+0x66 [minkernel\ntos\rtl\amd64\exdsptch.c @ 453] 
0d 00007ffb`0c6921e2 : 00000000`00000001 000000e6`2897cb39 00000000`00000000 000002b9`80150b80 : ntdll!KiUserExceptionDispatch+0x2e [minkernel\ntos\rtl\amd64\trampoln.asm @ 749] 
0e (Inline Function) : --------`-------- --------`-------- --------`-------- --------`-------- : coreclr!WKS::CObjectHeader::GetMethodTable+0xa [F:\workspace\_work\1\s\src\coreclr\src\gc\gc.cpp @ 3673] 
0f (Inline Function) : --------`-------- --------`-------- --------`-------- --------`-------- : coreclr!WKS::my_get_size+0xa [F:\workspace\_work\1\s\src\coreclr\src\gc\gc.cpp @ 9089] 
10 00007ffb`0c67c26f : 00000000`00000002 ffffffff`00000000 00007ffa`ad07bb60 00000000`00000000 : coreclr!WKS::gc_heap::verify_heap+0x316 [F:\workspace\_work\1\s\src\coreclr\src\gc\gc.cpp @ 34721] 
11 00007ffb`0c667a72 : 00000000`00002b01 00000000`00000000 00000000`00000000 00007ffb`0c683377 : coreclr!WKS::gc_heap::garbage_collect+0x273 [F:\workspace\_work\1\s\src\coreclr\src\gc\gc.cpp @ 17550] 
12 00007ffb`0c66a79a : 00000000`00000000 000000e6`ffffffff 00007ffb`00000008 000002b9`800d0320 : coreclr!WKS::GCHeap::GarbageCollectGeneration+0x286 [F:\workspace\_work\1\s\src\coreclr\src\gc\gc.cpp @ 37034] 
13 00007ffb`0c36b5e6 : 000002b9`eaba1230 000000e6`0000a5f4 00007ffa`acd053d0 000002b9`eaba1288 : coreclr!WKS::GCHeap::StressHeap+0x4ca [F:\workspace\_work\1\s\src\coreclr\src\gc\gc.cpp @ 35982] 
14 00007ffb`0c11a2ae : 000002b9`ed0634d0 000002b9`ed063400 00007ffa`ad124100 00007ffb`0c9a94f4 : coreclr!DoGcStress+0x362 [F:\workspace\_work\1\s\src\coreclr\src\vm\gccover.cpp @ 1815] 
15 00007ffb`0c314bd6 : 00000000`00000000 000000e6`2897d978 00000000`00000000 00000000`00000001 : coreclr!Thread::RedirectedHandledJITCase+0x1f6 [F:\workspace\_work\1\s\src\coreclr\src\vm\threadsuspend.cpp @ 2901] 
16 00007ffa`acc32616 : 00000000`00000000 000000e6`2897d978 00000000`00000000 00000000`00000004 : coreclr!RedirectedHandledJITCaseForGCStress_Stub+0x26 [F:\workspace\_work\1\s\src\coreclr\src\vm\amd64\RedirectedHandledJITCase.asm @ 104]
17 00007ffa`acc32389 : 000000e6`2897d240 000000e6`2897d558 000000e6`2897d248 00007ffb`00000000 : 0x00007ffa`acc32616 

@janvorli or @jkotas any suggestions on where to look are helpful. I will try the few things each of you have suggested in the past and see if anything pops.

[BruceForstall]

Looks like the object pointer on which coreclr!WKS::CObjectHeader::GetMethodTable() is being called is corrupt, since it appears the object pointer points to illegal memory. You need to figure out where that pointer came from, and why it is corrupt. Use the stress log and see if that has any clues.

[AaronRobinsonMSFT]

@BruceForstall Yeah I was starting down that path. I believe I have the object address, but it isn't in the stresslog. The object pointer I am looking at is a reasonable address but when dereferenced points to NULL - which is clearly the problem.

[AaronRobinsonMSFT]

The failure is happening on line 65 below. The test is a bit complicated because the actual test is a native EXE which is launched by a manged entry point that is run with corerun.exe. This convoluted path is needed because the test infrastructure didn't (doesn't?) support tests starting from a native context.

Based on the DMP there are two things going on:

1. The native process is starting to be launched.
2. There is a SafeProcessHandle in the finalizer queue. I have no idea why there would be a SafeProcessHandle in the queue, but it has an invalid handle value and is not marked as "closed" or "disposed" with a ref count of 1.

I can't find the object address in the stresslogs so unsure where it came from or perhaps is new/never moved? Given where the launcher is in the create process steps the native process hasn't been started yet. We are at the early stages of setting up pipes to communicate with the as yet unlaunched process.

runtime/src/coreclr/tests/src/Interop/common/ExeLauncherProgram.cs

Lines 65 to 75 in bbb5902

	using (Process p = Process.Start(startInfo))
	{
	p.OutputDataReceived += (_, args) => Console.WriteLine(args.Data);
	p.BeginOutputReadLine();
	p.ErrorDataReceived += (_, args) => Console.Error.WriteLine(args.Data);
	p.BeginErrorReadLine();
	p.WaitForExit();
	return p.ExitCode;
	}