MTPv2: clean output in GitHub Actions?

[bart-vmware]

When using MTPv2 in GitHub Actions, the output contains a lot of garbage that makes it difficult to read the log. For example, a single assembly occurs multiple times:

Compare that with VSTest, which just logs a clear single line:

Is there a way to produce a clean output in the log on GitHub Actions? Such as per assembly/framework:

• Number of tests passed/failed/skipped
• Each failed test with its name and stack trace

[Youssef1313]

Can you try passing --no-progress?

@nohwnd I strongly think that when the terminal isn't capable of updating the progress in-place, we should really disable progress by default. I have been adding --no-progress almost everywhere in CI and I think this should be the default.

[nohwnd]

The progress is reported every 3 seconds for every dll that is currently running. When the UI is not ansi capable this is the only way to report. I am not opposed to disabling the output in non-interactive scenarios. But then you also don't get any sense of what was progress.

[bart-vmware]

Using --no-progress is more readable, but it doesn't show the number of passed/failed/skipped tests per assembly.

[Youssef1313]

I am not opposed to disabling the output in non-interactive scenarios

Good. I think that's best. Please when you do so, make sure to fix that both on SDK and MTP (so we have consistency between running executable directly and running via dotnet test, and include it in release/10.0.2xx of SDK if possible)

But then you also don't get any sense of what was progress.

Maybe we can go with --no-progress <optional-bool> and if user wants progress they specify --no-progress false (double negation which is annoying). Alternatively, we can do --force-progress (and disallow specifying both --force-progress and --no-progress).

Using --no-progress is more readable, but it doesn't show the number of passed/failed/skipped tests per assembly.

@nohwnd When running on solution, do we show summary for each assembly? or only whole test run summary? Can we do some improvement here?

[nohwnd]

Yeah enabling progress is awkward. Imho there should be --ansi and --progress parameter that takes auto, true, false (and optionally 1,0, on, off like terminal logger parameter).

#5081 this is already requested here.

Currently none of the assembly summaries show the final number of tests that passed / failed / skipped / retried, it shows just final result of the assembly.

Adding counts to them should be easy, as we have already both the data and the way to render them.

However to me this seems a bit confusing, and we should probably color the outputs only to highlight with color to relevant number (like we do with the results in summary) = any failed, only red is highlighted, all passed only green is highlighted, all skipped only yellow is highlighted.

[Evangelink]

I love all your points here @nohwnd so +1000% for me!

[nohwnd]

@Copilot can you bring my design to reality? All the methods are already there. Change the coloring to be only highlighting the important status, as called out here:

color the outputs only to highlight with color to relevant number (like we do with the results in summary) = any failed, only red is highlighted, all passed only green is highlighted, all skipped only yellow is highlighted.

[nohwnd]

actually, wrong repo. the reporter here, does not report on assemblies since there is just 1.