PlatformSpecific tests for AnyOS assemblies running incorrectly

[stephentoub]

We're seeing tests being run for platforms they shouldn't be run for.

Specifically, it appears that the xunit command-line arguments like "-notrait category=nonwindowstests" are not being output when running the tests for platform-agnostic assemblies. But some such assemblies do have platform-specific tests, like System.Globalization. As a result, running those tests results in failures, because, for example, tests not intended for Windows are being run on Windows.

For example, when I do msbuild /t:rebuildandtest for System.Globalization.Extensions (which has a Windows and a Unix build), I correctly get this command-line:

RunTestsForProject:
  c:\Users\stoub\Source\repos\corefx\bin/tests/Windows_NT.AnyCPU.Debug/System.Globalization.Extensions.Tests/dnxcore50/CoreRun.exe xunit.console.netcore.exe System.Globalization.Extensions.Tests.dll  -xml testResults.xml -notrait category=nonwindowstests -notrait Benchmark=true  -notrait category=OuterLoop -notrait  category=failing

but when I do the same for System.Globalization, which is a facade for types in mscorlib that are platform-specific and which has platform-specific tests, I see this:

RunTestsForProject:
  c:\Users\stoub\Source\repos\corefx\bin/tests/AnyOS.AnyCPU.Debug/System.Globalization.Tests/dnxcore50/CoreRun.exe xunit.console.netcore.exe System.Globalization.Tests.dll  -xml testResults.xml -notrait Benchmark=true  -notrait category=OuterLoop -notrait category=failing

(no "-notrait nonwindowstests") and if I look at the tests getting run (with -showProgress), I see this test:

   TextInfoTests.EnUsPosixIsNotATurkishCasingLocale [STARTING]
   TextInfoTests.EnUsPosixIsNotATurkishCasingLocale [FINISHED] Time: 0.0031722s

which as shown here:
https://github.com/dotnet/corefx/blob/d55bead1604399a0e0ad17a2569a566340e0cba0/src/System.Globalization/tests/TextInfo.cs#L154
is attributed as being for Unix-only.

cc: @ericstj, @weshaggard, @dagood, @ellismg, @tarekgh

[stephentoub]

As an example of this causing failures:

cd src\System.IO.Compression.ZipFile\tests
msbuild /t:rebuildandtest /p:Outerloop=true

and several tests fail because it's running tests on Windows meant only to be run on Unix.

This is the cause of some of our outerloop Windows failures right now in corefx.
cc: @joshfree, @Priya91

[Priya91]

Will take a look.