Enable dotnet test in dotnet/runtime

[ViktorHofer]

Investigate how much work it is to switch away from the deprecated xunit.console runner to dotnet vstest. This is just about the runner, the test infra backend will still be xunit.

UAP will continue using Microsoft.DotNet.XUnitRunnerUap.
UAPAOT is problematic. We currently use xunit.console for it which doesn't work (https://github.com/dotnet/corefx/issues/30423). It may be our best option to add a ifdef configuration in dotnet/arcade#1627 which works fine with uapaot.

[clairernovotny]

xunit for devices supports UWP with .NET Native FYI. The Xamarin Essentials folks have done work to help automate and track that and I welcome improvements there too.

[ViktorHofer]

@onovotny From the readme it seems that you are injecting the MainPage.xaml into the test unit project. Would it be possible to change the test unit project to a UWP console app instead of injecting the MainPage.xaml so that we don't invoke a GUI?

Currently for UWP we use https://github.com/dotnet/arcade/tree/master/src/Microsoft.DotNet.XUnitRunnerUap which consists of a runner and and a launcher. The runner is a UWP console app and xunit runner which just runs the tests that are discover able and the launcher bundles the runner and the UWP test project together as we can't do Assembly.Load in UWP.

The injecting strategy is interesting and something I haven't thought of before. If xunit for devices could inject everything so that a UWP console app would be the result that could eliminate our custom runner for UWP which would be awesome.

[clairernovotny]

@ViktorHofer a UWP console app is in the plan for xUnit 3.0 and will be supported by "regular xUnit" @bradwilson. xUnit 3 plans to run all tests this way, essentially creating the exe that runs the tests. Whether it's .NET Framework, .NET Core, or UWP, is just a matter of what you target.

[clairernovotny]

A better question is if vstest supports UWP console apps? Right now, their test adapter for UWP is pretty specific and they provide their own GUI entry point.

If you use the latest xUnit for Devices templates (via the VSIX in the marketplace), it supports both VSTest and running standalone via F5. Either way though has GUI popup. However, I believe VSTest takes care of registering/deploying/running the test app so you don't need that C++ launcher code. I'm not completely sure though...

[ViktorHofer]

Thanks for the info. Does that mean with xunit 3 we will have two ways of testing UWP or will the UWP support be removed from xunit for devices?

Regarding the VSTest support I will open an issue. It would be great if we could get rid of all the bundling, registering, deploying and the whole custom runner. That said I don't expect that to happen anytime soon...

[clairernovotny]

You have two ways today and you'll have three ways in xUnit 3:

1. Create a new UWP Test project using the template and switch MSTest to xUnit (and use the xUnit visual studio adapter)
2. Use xUnit for Devices (supports both VS and standalone execution). IMO, no reason to use option 1.
3. [New for 3.0]: xUnit UWP console app. This should work with option 1 & 2 but TBD.

[ViktorHofer]

Understood. Thanks!

[ViktorHofer]

As you said option 3 is still tbd but can you guess if it'll be possible to just have a "normal" csproj sdk proejct and not a UWP template? So that it will inject all the UWP stuff into it, i.e. manifest, legacy csproj, etc.

[bradwilson]

Details are TBD but almost surely we will be telling people to make console apps for their target platform, not class libraries.

[ViktorHofer]

uapaot isn't a concern here anymore. uap will continue to use our custom runner.

[ViktorHofer]

The dotnet test and dotnet vstest commands will probably be merged into a single dotnet test command. Additionally I found out that xunit isn't happy with not generating deps.json files for the test assemblies.

[ViktorHofer]

I got dotnet vstest working locally with corefx. A few hurdles I hit:

• xunit.runner.visualstudio doesn't allow passing in runsettings from the CLI (-- ...)
• a direct dotnet test/vstest invocation (without overriding the VSTest target) won't work, unless a) dotnet is passed in as a full path, pointing to the testhost dotnet.exe and that dotnet is fully featured (has a sdk folder, etc. - not just bare bones) or b) https://github.com/dotnet/core-setup/issues/4809 is implemented.
• the testhost dotnet is just a rip-off, it doesn't allow executing anything besides the exec task. We might want to fully binplace/hardlink the sdk nevertheless for VS Test Explorer support.
• Warnings of tests with duplicate ids, need to investigate.

[ViktorHofer]

@BruceForstall @sandreenko @hoyosjs do you know if switching to dotnet vstest would cause issues on the coreclr side?

[BruceForstall]

@BruceForstall @sandreenko @hoyosjs do you know if switching to dotnet vstest would cause issues on the coreclr side?

As long as it supports response files for test exclusion, and doesn't have the bugs we fixed in our fork (dotnet/arcade#1613), it seems like it would be ok. Of course, it would be worth testing that hypothesis.