Triage tests regressing by building managed tests on OSX

[sdmaclea]

The following test started regressing (currently in the `dev/infrastructure branch) as a result of the #36253 change to start building all managed test assemblies on OSX.

They will be disabled (in the `dev/infrastructure branch) as part of #36351

<!-- Fixed by #36482
-->
<ExcludeList Include="$(XunitTestBinBase)/readytorun/tests/mainv**/*"/>
<ExcludeList Include="$(XunitTestBinBase)/Interop/COM/ComWrappers/GlobalInstance/GlobalInstanceMarshallingTests/*"/>
<ExcludeList Include="$(XunitTestBinBase)/Interop/COM/ComWrappers/GlobalInstance/GlobalInstanceTrackerSupportTests/*"/>
<ExcludeList Include="$(XunitTestBinBase)/Interop/COM/NativeClients/DefaultInterfaces/*"/>
<ExcludeList Include="$(XunitTestBinBase)/Interop/COM/NativeClients/Dispatch/*"/>
<ExcludeList Include="$(XunitTestBinBase)/Interop/COM/NativeClients/Events/*"/>
<ExcludeList Include="$(XunitTestBinBase)/Interop/COM/NativeClients/Licensing/*"/>
<ExcludeList Include="$(XunitTestBinBase)/Interop/COM/NativeClients/Primitives/*"/>
<ExcludeList Include="$(XunitTestBinBase)/Interop/COM/NETClients/ConsumeNETServer/ConsumeNETServer/*"/>
<!-- Regressions apparently caused by building all tests on OSX/Linux
        Oddly these pass when the helix payload is downloaded and the test is run locally
-->
<ExcludeList Include="$(XunitTestBinBase)/Interop/IJW/CopyConstructorMarshaler/CopyConstructorMarshaler/*"/>
<ExcludeList Include="$(XunitTestBinBase)/Interop/IJW/FixupCallsHostWhenLoaded/*"/>
<ExcludeList Include="$(XunitTestBinBase)/Interop/IJW/LoadIjwFromModuleHandle/LoadIjwFromModuleHandle/*"/>
<ExcludeList Include="$(XunitTestBinBase)/Interop/IJW/ManagedCallingNative/ManagedCallingNative/*"/>
<ExcludeList Include="$(XunitTestBinBase)/Interop/IJW/NativeCallingManaged/NativeCallingManaged/*"/>
<ExcludeList Include="$(XunitTestBinBase)/Interop/IJW/NativeVarargs/NativeVarargsTest/*"/>

Could someone take a look?
/cc @jkoritzinsky @sbomer @AaronRobinsonMSFT

[AaronRobinsonMSFT]

@sdmaclea All of the tests under Interop/ require native assets that are only built on Windows.

[sdmaclea]

So the native assets are supposed to be built on the target during the run-test-job.
Are these not hooked up properly so that they are being built during build-tests skipnative?

[AaronRobinsonMSFT]

They should be hooked up to build via the build-tests, but won't get built with build-tests skipnative. The native assets would build with build-tests skipmanaged. Some of those assets come from the same place as CoreRun, others come from the native project output path.

[sdmaclea]

So that shouldn't be the issue. AFAIK, the native assets are being built the same.

I am taking a look at Interop/IJW/NativeCallingManaged/NativeCallingManaged/* now

[sdmaclea]

So this is what I see in the payload for the test as sent to helix.

 ls -la
total 5188
drwxr-xr-x 1 stmaclea 1049089       0 May 13 20:50 ./
drwxr-xr-x 1 stmaclea 1049089       0 May 13 20:50 ../
-rwxr-xr-x 1 stmaclea 1049089   36864 May 13 20:50 IjwNativeCallingManagedDll.dll*
-rw-r--r-- 1 stmaclea 1049089     964 May 13 20:50 IjwNativeCallingManagedDll.exp
-rw-r--r-- 1 stmaclea 1049089    2062 May 13 20:50 IjwNativeCallingManagedDll.lib
-rw-r--r-- 1 stmaclea 1049089  536576 May 13 20:50 IjwNativeCallingManagedDll.pdb
-rwxr-xr-x 1 stmaclea 1049089  375808 May 13 20:50 mscoree.dll*
-rw-r--r-- 1 stmaclea 1049089    1167 May 13 20:50 mscoree.exp
-rw-r--r-- 1 stmaclea 1049089    2188 May 13 20:50 mscoree.lib
-rw-r--r-- 1 stmaclea 1049089 4272128 May 13 20:50 mscoree.pdb
-rw-r--r-- 1 stmaclea 1049089   10456 May 13 20:50 NativeCallingManaged.cmd
-rwxr-xr-x 1 stmaclea 1049089    6144 May 13 20:50 NativeCallingManaged.dll*
-rwxr-xr-x 1 stmaclea 1049089    8856 May 13 20:50 NativeCallingManaged.sh*
-rwxr-xr-x 1 stmaclea 1049089   35840 May 13 20:50 TestLibrary.dll*

[sdmaclea]

   Interop\IJW\NativeCallingManaged\NativeCallingManaged\NativeCallingManaged.cmd [FAIL]
      Unhandled exception. System.IO.FileNotFoundException: Could not load file or assembly 'IjwNativeCallingManagedDll, Culture=neutral, PublicKeyToken=null'. The system cannot find the file specified.
      File name: 'IjwNativeCallingManagedDll, Culture=neutral, PublicKeyToken=null'
         at System.Reflection.RuntimeAssembly.InternalLoad(ObjectHandleOnStack assemblyName, ObjectHandleOnStack requestingAssembly, StackCrawlMarkHandle stackMark, Boolean throwOnFileNotFound, ObjectHandleOnStack assemblyLoadContext, ObjectHandleOnStack retAssembly)
         at System.Reflection.RuntimeAssembly.InternalLoad(AssemblyName assemblyName, RuntimeAssembly requestingAssembly, StackCrawlMark& stackMark, Boolean throwOnFileNotFound, AssemblyLoadContext assemblyLoadContext)
         at System.Reflection.Assembly.Load(String assemblyString)
         at IjwHelper.LoadIjwAssembly(String name) in /Users/runner/runners/2.168.2/work/1/s/src/coreclr/tests/src/Interop/IJW/IjwHelper.cs:line 20
         at NativeCallingManaged.NativeCallingManaged.Main(String[] args) in /Users/runner/runners/2.168.2/work/1/s/src/coreclr/tests/src/Interop/IJW/NativeCallingManaged/NativeCallingManaged.cs:line 23
      
      Return code:      1
      Raw output file:      C:\h\w\97700803\w\B4A8097E\e\Interop\IJW\Reports\Interop.IJW\NativeCallingManaged\NativeCallingManaged\NativeCallingManaged.output.txt
      Raw output:
      BEGIN EXECUTION
       "C:\h\w\97700803\p/corerun.exe" NativeCallingManaged.dll 
      Expected: 100
      Actual: -532462766
      END EXECUTION - FAILED
      FAILED
      Test Harness Exitcode is : 1
      To run the test:
      > set CORE_ROOT=C:\h\w\97700803\p
      > C:\h\w\97700803\w\B4A8097E\e\Interop\IJW\NativeCallingManaged\NativeCallingManaged\NativeCallingManaged.cmd
      Expected: True
      Actual:   False
      Stack Trace:
        F:\workspace\_work\1\s\artifacts\tests\coreclr\Windows_NT.x64.Checked\TestWrappers\Interop.IJW\Interop.IJW.XUnitWrapper.cs(528,0): at Interop_IJW._NativeCallingManaged_NativeCallingManaged_NativeCallingManaged_._NativeCallingManaged_NativeCallingManaged_NativeCallingManaged_cmd()
      Output:
        Unhandled exception. System.IO.FileNotFoundException: Could not load file or assembly 'IjwNativeCallingManagedDll, Culture=neutral, PublicKeyToken=null'. The system cannot find the file specified.
        File name: 'IjwNativeCallingManagedDll, Culture=neutral, PublicKeyToken=null'
           at System.Reflection.RuntimeAssembly.InternalLoad(ObjectHandleOnStack assemblyName, ObjectHandleOnStack requestingAssembly, StackCrawlMarkHandle stackMark, Boolean throwOnFileNotFound, ObjectHandleOnStack assemblyLoadContext, ObjectHandleOnStack retAssembly)
           at System.Reflection.RuntimeAssembly.InternalLoad(AssemblyName assemblyName, RuntimeAssembly requestingAssembly, StackCrawlMark& stackMark, Boolean throwOnFileNotFound, AssemblyLoadContext assemblyLoadContext)
           at System.Reflection.Assembly.Load(String assemblyString)
           at IjwHelper.LoadIjwAssembly(String name) in /Users/runner/runners/2.168.2/work/1/s/src/coreclr/tests/src/Interop/IJW/IjwHelper.cs:line 20
           at NativeCallingManaged.NativeCallingManaged.Main(String[] args) in /Users/runner/runners/2.168.2/work/1/s/src/coreclr/tests/src/Interop/IJW/NativeCallingManaged/NativeCallingManaged.cs:line 23
        
        Return code:      1
        Raw output file:      C:\h\w\97700803\w\B4A8097E\e\Interop\IJW\Reports\Interop.IJW\NativeCallingManaged\NativeCallingManaged\NativeCallingManaged.output.txt
        Raw output:
        BEGIN EXECUTION
         "C:\h\w\97700803\p/corerun.exe" NativeCallingManaged.dll 
        Expected: 100
        Actual: -532462766
        END EXECUTION - FAILED
        FAILED
        Test Harness Exitcode is : 1
        To run the test:
        > set CORE_ROOT=C:\h\w\97700803\p
        > C:\h\w\97700803\w\B4A8097E\e\Interop\IJW\NativeCallingManaged\NativeCallingManaged\NativeCallingManaged.cmd

[sdmaclea]

It seems to be failing while trying to load the IjwNativeCallingManagedDll.dll assembly, which seems to be present.

[AaronRobinsonMSFT]

Where can I get the binaries from that run?

[sdmaclea]

CORE_ROOT

https://helixde107v0xdeko0k025g8.blob.core.windows.net/helix-job-81984b1d-673f-4c55-8991-8fee22a7d826bcd25907f2545e69d/48dfb514-0e41-4780-bbe6-606df2bab23b.zip?sv=2019-02-02&se=2020-06-02T17%3A12%3A59Z&sr=c&sp=rl&sig=YfBghT8jBGjEygX9h%2Bgoh9AWNYi%2FBTYYH7Nr2IwS6rM%3D

Interop tests

https://helixde107v0xdeko0k025g8.blob.core.windows.net/helix-job-81984b1d-673f-4c55-8991-8fee22a7d826bcd25907f2545e69d/5272378f-401b-42a1-a798-dc01320d7a46.zip?sv=2019-02-02&se=2020-06-02T17%3A12%3A59Z&sr=c&sp=rl&sig=YfBghT8jBGjEygX9h%2Bgoh9AWNYi%2FBTYYH7Nr2IwS6rM%3D

[sdmaclea]

Aaron and I worked on this a bit longer.

The tests downloaded from Helix work on my local machine. Aaron had similar experience with the tests on his machine with his own CORE_ROOT.

We did see that the cmd files do not match what we told MSBUILD to create. The directory separator has been replaced everywhere \ -> /. It is not clear why this is happening.

It is not clear why this works running locally, but not when run with helix.

/cc @trylek @jashook

Thanks @AaronRobinsonMSFT

[sdmaclea]

The typical usage of the directory separator is

REM Local CoreShim requested - see MSBuild property 'CLRTestScriptLocalCoreShim'
ECHO Copying '%CORE_ROOT%\CoreShim.dll'...
COPY /y %CORE_ROOT%\CoreShim.dll .

But since we are generating in MSBUILD, my working guess is that % is a MSBuild special character and needs to be escaped. Similar to the complaint in

src/runtest.proj:    <!-- NOTE! semicolons must be escaped with %3B boooo -->

    <!-- Write the file -->
    <WriteLinesToFile
      File="$(OutputPath)\$(MSBuildProjectName).cmd"
      Lines="$(_CLRTestExecutionScriptText)"
      Overwrite="true" />

Needs to become

    <!-- Write the file -->
    <WriteLinesToFile
      File="$(OutputPath)\$(MSBuildProjectName).cmd"
      Lines="$([MSBuild]::Escape($(_CLRTestExecutionScriptText)))"
      Overwrite="true" />

See https://docs.microsoft.com/en-us/visualstudio/msbuild/how-to-escape-special-characters-in-msbuild?view=vs-2019

I'll try this in a separate PR.

Edit

I am thinking $([MSBuild]::Escape($(_CLRTestExecutionScriptText))) won't work. We need some content treated specially.

The correct way to go seems to properly escape the special characters.

The % characters need to be escaped to %25. https://docs.microsoft.com/en-us/visualstudio/msbuild/msbuild-special-characters?view=vs-2019