dotnet test - Output test results

[Greg15153]

Currently there is no way to output the test results from dotnet test. Our CI requires a file in order to determine if a job has failed or not plus it allows us to gather data from our builds.

Would be useful for dotnet test to accept a output path for test results.

[blackdwarf]

@Greg15153 can you provide some more details? Wouldn't dotnet test > output.txt work? Or are the details that it prints not enough?

/cc @livarcocc

[Greg15153]

Outputting the result of the command and parsing out what I need isn't too big of a deal but it is a little lacking. Previously before switching over to dotnet core we would use VSTest.Console and had the option to output TRX files.

To be able to know what tests were run and how long in an XML output would be great. Found this example of xunit results.

<?xml version="1.0" encoding="UTF-8"?>
<testsuite name="nosetests" tests="1" errors="1" failures="0" skip="0">
    <testcase classname="path_to_test_suite.TestSomething"
              name="test_it" time="0">
        <error type="exceptions.TypeError" message="oops, wrong type">
        Traceback (most recent call last):
        ...
        TypeError: oops, wrong type
        </error>
    </testcase>
</testsuite>

[brthor]

You can pass -xml /some/path/out.xml

We use this in our Jenkins CI to determine what tests failed in a build.

[Greg15153]

Thats perfect. Wish dotnet test --help showed me that :)

[livarcocc]

It does not because that's a capability of the runner dotnet-test-xunit and not of dotnet test. Maybe we should do something to show the capabilities of your runner as well, when you do dotnet-test --help. I have created https://github.com/dotnet/cli/issues/3143 to consider this.

[bradwilson]

We updated dotnet-test-xunit so that you can do dotnet test -help. I know this is not ideal, but we're stuck behind the command line parsing above us.

[tghcastro]

@brthor which Jenkins Plugin do you use? I need to run "donet test" and publish jenkins report
Thanks

[tommed]

This doesn't seem to work now msbuild is used for testing? Or have I missed something?

$ dotnet test -xml results.xml MyProject.csproj
MSBUILD : error MSB1001: Unknown switch.
Switch: -xml

I'm guessing we'd now need to use a value for dotnet test --logger but the only value I can see that would be valid is trx?

[bradwilson]

@tommed You should not expect to be able to use runner-specific features with dotnet test any more.

In the world of project.json (that is DNX, and .NET Core SDK prior to Preview 3), dotnet test was just a thin shim to a console application written by the unit test framework author. The code for dotnet-test-xunit was extremely similar to the code for xunit.console.exe.

In the world of MSBuild (that is, .NET Core Preview 3 and later), dotnet test is just a thin shim to something very much like the Visual Studio test runner executable; in fact, the API that test framework authors use to plug into dotnet test is actually the same API that has powered the Visual Studio GUI and command line runners (for desktop and devices) since VS 2012. In this world, the test framework author has very little control over the experiences that surround the runner; they simply provide a way to discover and execute tests, with very little influence beyond that; definitely, we have no impact over any reporting systems.

That means if you're using dotnet test in a CI scenario, you will need to find tools prepared to consume and report against Visual Studio's .trx files.

[TheRealPiotrP]

It will also be possible to create custom loggers that output in another format like xunit v2 xml.

[plioi]

@piotrpMSFT But report file format is just one example of how unusable preview3's dotnet test is, now. As is, "dotnet test" supports "any test framework as long as that test framework doesn't have features".

[blackdwarf]

@plioi what would be your suggestion?

[plioi]

Passing the command line arguments through, like before. In the earlier previews, everything after the special positional "--" argument was intended for the test framework to receive. Test frameworks were therefore allowed to have features, were allowed to skip throwing resources at output file formats the user wasn't asking for, etc.

My understanding is that preview 3 does command line integration through the interfaces used to integrate with Test Explorer. That appears to be quite limiting. If those interfaces can't change to take on a string[] input, then dotnet test is only going to be usable for Hello World demos.

Even if the preview 3 approach changes to allow args to pass through those interfaces, this whole approach seems needlessly costly at runtime. I'd expect every test framework team to have to give up and provide their own dotnet-test-framework-name-actual.exe while recommending users not use "dotnet test".

[plioi]

Better yet, IDE mode test running and CLI mode test running are simply different things, and ought to be separate. Allowing CLI runs to be lean and featureful wouldn't have to cripple IDE running, and IDE running wouldn't have to cripple CLI running.