`dotnet test` reports error even when exit code 8 is ignored

[Lexy2]

Describe the bug

Version used

Microsoft.Testing.Platform 2.1.0
net10.0
NUnit 4
NUnit3TestAdapter 6.1.0

Steps To Reproduce

Add test packages to a project but don't add any tests to it.
Run dotnet test --ignore-exit-code 8

Repro attached.

mtp-bug.zip

-->

Expected behavior

Command succeeds, exit code is 0.
Exit code 8 is respected at the dotnet test level.

Actual behavior

Command fails, exit code is 1.
Exit code 8 is respected only if you run the MTP executable itself.

Additional context

Workaround - dotnet test --ignore-exit-code 1;8

[Youssef1313]

This is by design. There is no exit code 8 involved in this situation.

The problem is that dotnet test command is considering the helper.csproj to be a test project, which is actually just a hello world console app that returns with exit code 0. This won't handshake correctly with Test command and thus fail.

The reason helper.csproj is considered as a test project is:

• It sets EnableNUnitRunner to true
• It references the NUnit adapter and Microsoft.NET.Test.Sdk.

The solution for this is to make two changes:

Helper.csproj should look as follows (note that false)

<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <OutputType>Exe</OutputType>
    <TargetFramework>net10.0</TargetFramework>
    <ImplicitUsings>enable</ImplicitUsings>
    <Nullable>enable</Nullable>
    <EnableNUnitRunner>false</EnableNUnitRunner>
  </PropertyGroup>

</Project>

And second, Directory.Build.props should look as follows:

<Project>
  <PropertyGroup>
    <EnableNUnitRunner>true</EnableNUnitRunner>
  </PropertyGroup>
  <ItemGroup>
    <PackageReference Include="Microsoft.NET.Test.Sdk" Version="17.14.0" Condition="'$(EnableNUnitRunner)'=='true'" />
    <PackageReference Include="NUnit" Version="4.3.2" />
    <PackageReference Include="NUnit.Analyzers" Version="4.7.0" />
    <PackageReference Include="NUnit3TestAdapter" Version="6.1.0" Condition="'$(EnableNUnitRunner)'=='true'" />
  </ItemGroup>
</Project>

Note that the EnableNUnitRunner conditions will work correctly and will consider when the property is set in csproj because of MSBuild order of evaluation. It evaluates first the properties, and then the items. So properties in Directory.Build.props are evaluated first, then properties in csproj, then items in Directory.Build.props, then items in csproj.

What is missing here is a better error message that explicitly states that handshake failed, which I'm surprised that it's not there. So that needs to be looked at.

[Youssef1313]

This is where we detect that there is a handshake failure.

https://github.com/dotnet/sdk/blob/57a4a8a7967da5450bb07173aa5472a9896e7e28/src/Cli/dotnet/Commands/Test/MTP/TestApplicationHandler.cs#L280

We print "zero tests" message in this case here:

https://github.com/dotnet/sdk/blob/57a4a8a7967da5450bb07173aa5472a9896e7e28/src/Cli/dotnet/Commands/Test/MTP/Terminal/TerminalTestReporter.cs#L775

The message should clearly indicate that it's handshake failure. There are also two callers of handshake failures:

• When negotiation of protocol version fails (i.e, no common supported protocol versions between test app and SDK)
• When handshake isn't received at all.

These two cases should have two different messages.

[Lexy2]

I'm afraid that this won't work this way.

First, if <EnableNUnitRunner>false</EnableNUnitRunner>, dotnet test will consider the helper project a VSTest project and refuse to run in a mixed environment.
Second, this helper project references test projects in my real case, so test packages are implicitly referenced in the helper project.
Third, Directory.Build.props files are evaluated before project files as far as I'm aware (I may be wrong, and your scenario of properties-then-items may be right), and thus <EnableNUnitRunner>true</EnableNUnitRunner> will override anything that would be set in the project file.

Can I manually add a TestHostHandshake to the helper app so that it doesn't cause dotnet test to fail in this case?

[Youssef1313]

If you made the exact changes suggested in #7557 (comment), it should work, and the helper project won't be seen as a VSTest project. This is because VSTest package (Microsoft.NET.Test.Sdk) and the NUnit adapter are both excluded.

Second, this helper project references test projects in my real case, so test packages are implicitly referenced in the helper project.

IIRC this wasn't reflected in the given repro so I haven't looked deeply in what would happen in that situation. But I think it might still work fine. The props/targets of both NUnit adapter and VSTest are defined for build and not buildTransitive, so they are not imported when the package is brought transitively. So I think my suggestion above should still work. If it doesn't, please upload a repro.

[Lexy2]

Both true, I haven't tested the exact changes and it wasn't reflected in the repro. Let me try instead of arguing and theoricising.

[Lexy2]

I've tried to do it the way you said - unfortunately, still the same.

mtp-bug-2.zip

[Youssef1313]

I'm running dotnet test on your repro and it works correctly.

Running tests from test\bin\Debug\net10.0\mtp-bug.dll (net10.0|x64)
test\bin\Debug\net10.0\mtp-bug.dll (net10.0|x64) passed (664ms)

Test run summary: Passed!
  total: 1
  failed: 0
  succeeded: 1
  skipped: 0
  duration: 925ms

[Lexy2]

This is my output if I run dotnet test on the solution:

helper\bin\Debug\net10.0\helper.dll (net10.0) Zero tests ran
Exit code: 0
  Standard output: Hello, World!

Running tests from test\bin\Debug\net10.0\mtp-bug.dll (net10.0|x64)
test\bin\Debug\net10.0\mtp-bug.dll (net10.0|x64) passed (322ms)

Test run summary: Failed!
  error: 1

  total: 1
  failed: 0
  succeeded: 1
  skipped: 0
  duration: 1s 778ms
Test run completed with non-success exit code: 1 (see: https://aka.ms/testingplatform/exitcodes)

This is my actual problem: in the real solution Tests folder contains many projects, and some of them are helper apps for tests. As such, I would rather not run every test project manually, but use dotnet test against the solution.

[Youssef1313]

Would you be able to run with dotnet test -bl and send a binlog? Before publicly attaching a binlog, please read http://aka.ms/binlog. Specifically:

⚠ NOTE: some build environments make secrets available using environment variables. Before sharing a binary log, make sure it does not expose API tokens or other important secrets.

There are also instructions in the link above for how to share a binlog anonymously with us.

[Lexy2]

There you go:
https://developercommunity.visualstudio.com/t/MTP-transitive-references-try-to-overwri/11060582?ref=native&refTime=1773650339551&refUserId=e717fc5f-7dce-638f-9671-a8eb77abb050