Test failure: ComInterfaceGenerator.Tests

[BruceForstall]

Fails in runtime-coreclr libraries-jitstress-random

net9.0-windows-Release-x86-CoreCLR_checked-jitstress_random_1-Windows.10.Amd64.Open
net9.0-windows-Release-x86-CoreCLR_checked-jitstress_random_2-Windows.10.Amd64.Open
net9.0-linux-Release-arm-CoreCLR_checked-jitstress_random_2-(Ubuntu.1804.Arm32.Open)Ubuntu.2004.Armarch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm32v7
net9.0-linux-Release-arm-CoreCLR_checked-jitstress_random_1-(Ubuntu.1804.Arm32.Open)Ubuntu.2004.Armarch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm32v7

So, only 32-bit.

https://dev.azure.com/dnceng-public/public/_build/results?buildId=511155&view=ms.vss-test-web.build-test-results-tab

e.g.,

C:\h\w\A96D0928\w\9ED4086B\e>set DOTNET 
DOTNET_JitStress=ef
DOTNET_TieredCompilation=0

C:\h\w\A96D0928\w\9ED4086B\e>call RunTests.cmd --runtime-path C:\h\w\A96D0928\p 
========================= Begin custom configuration settings ==============================
set __TestArchitecture=x86
set __IsXUnitLogCheckerSupported=1
========================== End custom configuration settings ===============================
----- start Sat 12/30/2023  2:53:57.98 ===============  To repro directly: =====================================================
pushd C:\h\w\A96D0928\w\9ED4086B\e\
"C:\h\w\A96D0928\p\dotnet.exe" exec --runtimeconfig ComInterfaceGenerator.Tests.runtimeconfig.json --depsfile ComInterfaceGenerator.Tests.deps.json xunit.console.dll ComInterfaceGenerator.Tests.dll -xml testResults.xml -nologo -nocolor -notrait category=IgnoreForCI -notrait category=OuterLoop -notrait category=failing 
popd
===========================================================================================================

C:\h\w\A96D0928\w\9ED4086B\e>"C:\h\w\A96D0928\p\dotnet.exe" exec --runtimeconfig ComInterfaceGenerator.Tests.runtimeconfig.json --depsfile ComInterfaceGenerator.Tests.deps.json xunit.console.dll ComInterfaceGenerator.Tests.dll -xml testResults.xml -nologo -nocolor -notrait category=IgnoreForCI -notrait category=OuterLoop -notrait category=failing  
  Discovering: ComInterfaceGenerator.Tests (method display = ClassAndMethod, method display options = None)
  Discovered:  ComInterfaceGenerator.Tests (found 44 of 45 test cases)
  Starting:    ComInterfaceGenerator.Tests (parallel test collections = on, max threads = 4)
----- end Sat 12/30/2023  2:54:21.08 ----- exit code -1073741819 ----------------------------------------------------------
----- start ===============  XUnitLogChecker Output =====================================================

Tagging subscribers to this area: @JulieLeeMSFT, @jakobbotsch
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Fails in runtime-coreclr libraries-jitstress-random

net9.0-windows-Release-x86-CoreCLR_checked-jitstress_random_1-Windows.10.Amd64.Open
net9.0-windows-Release-x86-CoreCLR_checked-jitstress_random_2-Windows.10.Amd64.Open
net9.0-linux-Release-arm-CoreCLR_checked-jitstress_random_2-(Ubuntu.1804.Arm32.Open)Ubuntu.2004.Armarch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm32v7
net9.0-linux-Release-arm-CoreCLR_checked-jitstress_random_1-(Ubuntu.1804.Arm32.Open)Ubuntu.2004.Armarch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm32v7

So, only 32-bit.

https://dev.azure.com/dnceng-public/public/_build/results?buildId=511155&view=ms.vss-test-web.build-test-results-tab

e.g.,

C:\h\w\A96D0928\w\9ED4086B\e>set DOTNET 
DOTNET_JitStress=ef
DOTNET_TieredCompilation=0

C:\h\w\A96D0928\w\9ED4086B\e>call RunTests.cmd --runtime-path C:\h\w\A96D0928\p 
========================= Begin custom configuration settings ==============================
set __TestArchitecture=x86
set __IsXUnitLogCheckerSupported=1
========================== End custom configuration settings ===============================
----- start Sat 12/30/2023  2:53:57.98 ===============  To repro directly: =====================================================
pushd C:\h\w\A96D0928\w\9ED4086B\e\
"C:\h\w\A96D0928\p\dotnet.exe" exec --runtimeconfig ComInterfaceGenerator.Tests.runtimeconfig.json --depsfile ComInterfaceGenerator.Tests.deps.json xunit.console.dll ComInterfaceGenerator.Tests.dll -xml testResults.xml -nologo -nocolor -notrait category=IgnoreForCI -notrait category=OuterLoop -notrait category=failing 
popd
===========================================================================================================

C:\h\w\A96D0928\w\9ED4086B\e>"C:\h\w\A96D0928\p\dotnet.exe" exec --runtimeconfig ComInterfaceGenerator.Tests.runtimeconfig.json --depsfile ComInterfaceGenerator.Tests.deps.json xunit.console.dll ComInterfaceGenerator.Tests.dll -xml testResults.xml -nologo -nocolor -notrait category=IgnoreForCI -notrait category=OuterLoop -notrait category=failing  
  Discovering: ComInterfaceGenerator.Tests (method display = ClassAndMethod, method display options = None)
  Discovered:  ComInterfaceGenerator.Tests (found 44 of 45 test cases)
  Starting:    ComInterfaceGenerator.Tests (parallel test collections = on, max threads = 4)
----- end Sat 12/30/2023  2:54:21.08 ----- exit code -1073741819 ----------------------------------------------------------
----- start ===============  XUnitLogChecker Output =====================================================

Author:	BruceForstall
Assignees:	-
Labels:	JitStress, area-CodeGen-coreclr, blocking-clean-ci-optional
Milestone:	9.0.0

[BruceForstall]

I'm having trouble catching this in a debugger.

It fails on win-x86, test method ComInterfaceGenerator.Tests.PreserveSigTests.CallRcwFromGeneratedComInterface, with:

set DOTNET_JitStress=ef
set DOTNET_TieredCompilation=0

c:\gh\runtime\artifacts\bin\ComInterfaceGenerator.Tests\Release\net9.0>c:\gh\runtime\artifacts\bin\testhost\net9.0-windows-Release-x86\dotnet.exe exec --runtimeconfig ComInterfaceGenerator.Tests.runtimeconfig.json --depsfile ComInterfaceGenerator.Tests.deps.json xunit.console.dll ComInterfaceGenerator.Tests.dll -xml testResults.xml -notrait category=IgnoreForCI -notrait category=OuterLoop -notrait category=failing -parallel none -verbose -method "ComInterfaceGenerator.Tests.PreserveSigTests.CallRcwFromGeneratedComInterface"
Microsoft.DotNet.XUnitConsoleRunner v2.5.0 (32-bit .NET 9.0.0-dev)
Discovering: ComInterfaceGenerator.Tests (method display = ClassAndMethod, method display options = None)
Discovered:  ComInterfaceGenerator.Tests (found 1 of 45 test case)
Starting:    ComInterfaceGenerator.Tests (parallel test collections = off, max threads = 16)
ComInterfaceGenerator.Tests.PreserveSigTests.CallRcwFromGeneratedComInterface [STARTING]
c:\gh\runtime\artifacts\bin\ComInterfaceGenerator.Tests\Release\net9.0>echo errorlevel=-1073741819

(that's STATUS_ACCESS_VIOLATION)

Running under a debugger seems to lead to different behavior, and in fact, windbg disappears (crashes?).

[BruceForstall]

If I run under Visual Studio, I get:

Assert failure(PID 35868 [0x00008c1c], Thread: 11908 [0x2e84]): IsExecutingOnAltStack() || (curSP <= m_pFrame && m_pFrame < m_CacheStackBase) || m_pFrame == (Frame*) -1

CORECLR! Object::ValidateInner + 0xA3 (0x6a649e53)
CORECLR! Object::Validate + 0x98 (0x6a649d78)
CORECLR! OBJECTREF::OBJECTREF + 0x1F (0x6a6458d8)
CORECLR! `anonymous namespace'::TryGetOrCreateComInterfaceForObjectInternal + 0x4C5 (0x6a8c4988)
CORECLR! ComWrappers_TryGetOrCreateComInterfaceForObject + 0xF6 (0x6a8c6fe6)
SYSTEM.PRIVATE.CORELIB! <no symbol> + 0x0 (0x695ce415)
SYSTEM.PRIVATE.CORELIB! <no symbol> + 0x0 (0x695ce3c5)
SYSTEM.PRIVATE.CORELIB! <no symbol> + 0x0 (0x695ce352)
<no module>! <no symbol> + 0x0 (0x0e111abe)
<no module>! <no symbol> + 0x0 (0x0e1116d9)
    File: C:\gh\runtime\src\coreclr\vm\threads.h Line: 1654
    Image: c:\gh\runtime\artifacts\bin\testhost\net9.0-windows-Release-x86\dotnet.exe

Not sure why I hit this assert under a debugger, but not when run from the command-line.

And is this the same failure as from the command line?

[BruceForstall]

If I try running with corerun.exe I get:

c:\gh\runtime\artifacts\bin\ComInterfaceGenerator.Tests\Release\net9.0>c:\gh\runtime\artifacts\tests\coreclr\windows.x86.Checked\Tests\Core_Root\corerun.exe xunit.console.dll ComInterfaceGenerator.Tests.dll -xml testResults.xml -notrait category=IgnoreForCI -notrait category=OuterLoop -notrait category=failing -parallel none -verbose -method "ComInterfaceGenerator.Tests.PreserveSigTests.CallRcwFromGeneratedComInterface"
Microsoft.DotNet.XUnitConsoleRunner v2.5.0 (32-bit .NET 9.0.0-dev)
  Discovering: ComInterfaceGenerator.Tests (method display = ClassAndMethod, method display options = None)
  Discovered:  ComInterfaceGenerator.Tests (found 1 of 45 test case)
  Starting:    ComInterfaceGenerator.Tests (parallel test collections = off, max threads = 16)
    ComInterfaceGenerator.Tests.PreserveSigTests.CallRcwFromGeneratedComInterface [STARTING]
You must install or update .NET to run this application.

App: c:\gh\runtime\artifacts\tests\coreclr\windows.x86.Checked\Tests\Core_Root\corerun.exe
Architecture: x86
Framework: 'Microsoft.NETCore.App', version '9.0.0' (x86)
.NET location: C:\Program Files (x86)\dotnet\

The following frameworks were found:
  6.0.25 at [C:\Program Files (x86)\dotnet\shared\Microsoft.NETCore.App]
  7.0.14 at [C:\Program Files (x86)\dotnet\shared\Microsoft.NETCore.App]
  8.0.0 at [C:\Program Files (x86)\dotnet\shared\Microsoft.NETCore.App]

Learn more:
https://aka.ms/dotnet/app-launch-failed

To install missing framework, download:
https://aka.ms/dotnet-core-applaunch?framework=Microsoft.NETCore.App&framework_version=9.0.0&arch=x86&rid=win-x86&os=win10

c:\gh\runtime\artifacts\bin\ComInterfaceGenerator.Tests\Release\net9.0>echo errorlevel=-1073740791

(That's STATUS_STACK_BUFFER_OVERRUN)

[BruceForstall]

I verified that the test passes locally on x64 with the same JitStress number. (Since I don't know which JitStress modes are being enabled, and don't know the failing method(s), the JitStress set might be different for x64 versus x86.)

[BruceForstall]

@jkoritzinsky suggested that perhaps the DAC was crashing (and bringing down windbg).

When I move the DAC files (

mscordaccore.dll
mscordaccore.pdb
mscordaccore_amd64_amd64_42.42.42.42424.dll
mscordaccore_amd64_amd64_42.42.42.42424.pdb

) out of the host (c:\gh\runtime\artifacts\bin\testhost\net9.0-windows-Release-x64\shared\Microsoft.NETCore.App\9.0.0) and run the test, I get:

c:\gh\runtime\artifacts\bin\ComInterfaceGenerator.Tests\Release\net9.0>c:\gh\runtime\artifacts\bin\testhost\net9.0-windows-Release-x86\dotnet.exe exec --runtimeconfig ComInterfaceGenerator.Tests.runtimeconfig.json --depsfile ComInterfaceGenerator.Tests.deps.json xunit.console.dll ComInterfaceGenerator.Tests.dll -xml testResults.xml -notrait category=IgnoreForCI -notrait category=OuterLoop -notrait category=failing -parallel none -verbose -method "ComInterfaceGenerator.Tests.PreserveSigTests.CallRcwFromGeneratedComInterface"
Microsoft.DotNet.XUnitConsoleRunner v2.5.0 (32-bit .NET 9.0.0-dev)
  Discovering: ComInterfaceGenerator.Tests (method display = ClassAndMethod, method display options = None)
  Discovered:  ComInterfaceGenerator.Tests (found 1 of 45 test case)
  Starting:    ComInterfaceGenerator.Tests (parallel test collections = off, max threads = 16)
    ComInterfaceGenerator.Tests.PreserveSigTests.CallRcwFromGeneratedComInterface [STARTING]
    ComInterfaceGenerator.Tests.PreserveSigTests.CallRcwFromGeneratedComInterface [FAIL]
      System.BadImageFormatException : An attempt was made to load a program with an incorrect format. (0x8007000B (COR_E_BADIMAGEFORMAT))
      Stack Trace:
           at ComInterfaceGenerator.Tests.PreserveSigTests.NewNativeObject()
        C:\gh\runtime\src\libraries\System.Runtime.InteropServices\tests\ComInterfaceGenerator.Tests\PreserveSigTests.cs(20,0): at ComInterfaceGenerator.Tests.PreserveSigTests.CallRcwFromGeneratedComInterface()
           at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)
        C:\gh\runtime\src\coreclr\System.Private.CoreLib\src\System\Reflection\MethodBaseInvoker.CoreCLR.cs(36,0): at System.Reflection.MethodBaseInvoker.InterpretedInvoke_Method(Object obj, IntPtr* args)
        C:\gh\runtime\src\libraries\System.Private.CoreLib\src\System\Reflection\MethodBaseInvoker.cs(57,0): at System.Reflection.MethodBaseInvoker.InvokeWithNoArgs(Object obj, BindingFlags invokeAttr)
    ComInterfaceGenerator.Tests.PreserveSigTests.CallRcwFromGeneratedComInterface [FINISHED] Time: 0.1379286s
  Finished:    ComInterfaceGenerator.Tests
=== TEST EXECUTION SUMMARY ===
   ComInterfaceGenerator.Tests  Total: 1, Errors: 0, Failed: 1, Skipped: 0, Time: 4.175s

[BruceForstall]

However, without the DAC files, the test fails as above even without JitStress, without TieredCompilation=0, and in fact with DOTNET_JITMinOpts=1

So does the test require the DAC to function?

[jkoritzinsky]

Is it possible that you've accidentally mixed x86 and x64 assets in the build? Since these tests have a native component, you need to build them for 32-bit as well (specify the architecture for building the tests, not just the product build)

[BruceForstall]

Is it possible that you've accidentally mixed x86 and x64 assets in the build?

Yes, very likely (after I tried repro'ing x64). I'm rebuilding to double-check this.

(Unfortunately, the generated RunTests.cmd now have lines like:

set __TestArchitecture=x86
set __IsXUnitLogCheckerSupported=1

which is super annoying for local (non-CI) usage.)

[BruceForstall]

Is it possible that you've accidentally mixed x86 and x64 assets in the build? Since these tests have a native component, you need to build them for 32-bit as well (specify the architecture for building the tests, not just the product build)

It looks like microsoft.interop.tests.nativeexports.dll is one example of such a native asset?

I notice that this lives in the directory (along with all the other assets): artifacts\bin\ComInterfaceGenerator.Tests\Release\net9.0. This directory path has no architecture-specific component. That means that if I build tests for x86, then arm64, the arm64 native components will overwrite the x86 native components in place. This is bad! (I actually do build everything, including libs tests, for three different architectures). It seems like these need to be built to architecture-specific subdirectories. E.g., I see for some other tests artifacts\bin\System.Data.Common.Tests\Release\net9.0\runtimes\win-x64\native.