Document testconfig.json

[johncrim]

Summary

The testconfig.json configuration page is effectively useless, aside from confirming the existence of the config file, and that it is JSON.

How about some documentation about what settings can be specified in that file? Something equivalent to

• https://learn.microsoft.com/en-us/dotnet/core/testing/unit-testing-mstest-configure
• https://xunit.net/docs/config-testconfig-json

For example, I'd like to replace my *.runsettings.xml file with testconfig.json, but I can't figure out what the options are or how it should work.

[Youssef1313]

Thanks for reporting this @johncrim.

I do agree that the documentation can be improved here. There are not much testconfig.json options for the core platform though, and most are for MSTest, for which the documentation belongs to https://learn.microsoft.com/dotnet/core/testing/unit-testing-mstest-configure. I do think that:

1. We should certainly document the options relevant to the core platform in the page you mentioned.
2. We should add links to framework-specific testconfig.json documentation (e.g, MSTest, TUnit, and xUnit - will need to check if NUnit or Expecto is making use of any).

Meanwhile, we can still help if you have specific questions about the migration, if there is anything specific in your runsettings that you don't know yet how to do in Microsoft.Testing.Platform. It's useful also to know which test framework do you use.

[johncrim]

Thank you for the reply @Youssef1313 . We primarily use Xunit (recently migrated to xunit 3, which is why I discovered the missing docs); but we also use MSTest for playwright tests.

The first question I think the docs should provide is which of the MTP command-line options can be configured in the testconfig. For example I guessed that the results directory should be configurable (and I'd assume the path would be relative to the location of the testconfig.json file, and the name would be camelCased), but based on a little bit of testing (and no source code spelunking) I determined that it's not supported (so arg only works on the command-line). I tried this, but aside from the xunit docs I really didn't know if any of the other options are correct or not:

{ // testconfig.json
  "platformOptions": {
    "resultsDirectory": "../../TestResults"
  },
  "xUnit": {
    "longRunningTestSeconds": 20,
    "parallelizeTestCollections": false,
    "maxParallelThreads": -1,
    "showLiveOutput": true
  },
  "crashDump": {
    "type": "Mini"
  }
}

As another example, here is our old runsettings XML - ideally, I would like to be able to translate it to an equivalent testconfig.json:

<!-- ci.runsettings -->
<?xml version="1.0" encoding="utf-8"?>
<RunSettings>

  <RunConfiguration>
    <!-- Use 0 (the default) for maximum process-level parallelization. -->
    <MaxCpuCount>0</MaxCpuCount>
    <ResultsDirectory>../../Test-Results</ResultsDirectory>
  </RunConfiguration>

  <LoggerRunSettings>
    <Loggers>
      <!-- <Logger friendlyName="console" enabled="True">
        <Configuration>
          <Verbosity>quiet</Verbosity>
        </Configuration>
      </Logger> -->
      <Logger friendlyName="trx" enabled="True" />
      <Logger friendlyName="blame" enabled="True" />
      <!-- The main advantage of .trx files is that they can be opened in VisualStudio test runner UI.
        The xunit logger includes xunit Traits in the output (the vstest/trx unit doesn't), and the xunit XML
        format is generally clearer. Uncomment this for xunit XML output.
      <Logger friendlyName="xunit" enabled="True" /> -->
    </Loggers>
  </LoggerRunSettings>

  <!-- https://xunit.net/docs/runsettings#ParallelizeTestCollections -->
  <xUnit>
    <DiagnosticMessages>true</DiagnosticMessages>
    <LongRunningTestSeconds>20</LongRunningTestSeconds>

    <ParallelizeTestCollections>true</ParallelizeTestCollections>
    <ShowLiveOutput>true</ShowLiveOutput>
  </xUnit>

  <DataCollectionRunSettings>
    <DataCollectors>
      <!-- Enables blame -->
      <DataCollector friendlyName="blame" enabled="True">
        <Configuration>
          <!-- Enables crash dump, with dump type "Full" or "Mini". -->
          <CollectDump DumpType="Mini" />
          <!-- Enables hang dump or testhost and its child processes
          when a test hangs for more than 10 minutes.
          Dump type "Full", "Mini" or "None" (just kill the processes). -->
          <CollectDumpOnTestSessionHang TestTimeout="1min" HangDumpType="Full" />
        </Configuration>
      </DataCollector>
    </DataCollectors>
  </DataCollectionRunSettings>

</RunSettings>

I did find that MSTest supports .runsettings files, but only a subset of the options work - fortunately it reports which settings aren't supported. I'm pretty sure I can't use .runsettings with the xunit MTP runner.

[alexTr3]

awful release.. added Microsoft.Testing.Platform.MSBuild reference to my project and it does not even rename the testconfig.json!

I tried both in a core library (test) and in the executable test project ... and the testconfig.json is never renamed and stay unused..

if you have the ... to uselessly remake a framework from scratch and break everything at least do it correctly..

can you tell me why the need to even rename the testconfig.json to [appname].testconfig.json??

we dont want to use 100000 configuration files and configuration point.. just ONE! stop this madness

it should be a full move to MTP with FULL backward compatibility with VSTEST..

not 1000 pages of documentation to do the migration and when we try to follow the very bad documentation, the actual framework is riddled with bugs and unfinished business..

[alexTr3]

ok so I understand why it's not working, you need to have a copy of the testconfig.json in every executable test project. you cannot have it in a core library project..

just awful..

[Youssef1313]

Thanks for the feedback @alexTr3. Please make sure to follow Code of Conduct.

To answer your questions:

added Microsoft.Testing.Platform.MSBuild reference to my project and it does not even rename the testconfig.json

If your test app is MTP already, this should work (and actually you don't need to explicitly add the package reference as the major test frameworks bring it for you for free).

If you have a repro project that shows this doesn't work, please open a new bug report.

it should be a full move to MTP with FULL backward compatibility with VSTEST..

This unfortunately is not possible. We cannot keep full backward compatibility with VSTest. Otherwise, it would have just been a minor release of VSTest.

not 1000 pages of documentation to do the migration and when we try to follow the very bad documentation

We would love to hear specific feedback about documentation to improve it. We are actively looking into documentation improvements, and your feedback will definitely help us further.

the actual framework is riddled with bugs and unfinished business..

Please be more specific about bugs you see, and open new bug reports with repro project or details.

[Youssef1313]

ok so I understand why it's not working, you need to have the testconfig.json in ALL executable test project. you cannot have it in a core library project..

testconfig.json only works on MTP test projects. So it's expected that it will have no effect on a library project. If you want to have a single centralized testconfig.json, you can, but then you will need to pass it via --config-file command-line option.
In contexts where MSBuild is available (e.g, dotnet test on project/solution, or dotnet run), you can automatically add --config-file via TestingPlatformCommandLineArguments MSBuild property. You can add it in a centralized Directory.Build.props in repository root.

[alexTr3]

My advice.. make an example solution of each framework version (vstest vs mtp etc..) for each test platform (xunit) etc.. and each configuration..(testconfig, testrunner) etc.. so one can easily see the difference between all these and migrate correctly..

but stop with your limitation.. like the need to have the testconfig file in each project and it's not working when it's in a shared core project.. this is awful design..

[alexTr3]

I understand that you can add the testingplatformcommandlinearguments but .. you should support having the testconfig.json file in a core library and core library should be supported like in any other c# projects..

the problem with your TestingPlatformCommandLineArguments is that it will override specific testconfig for specific test projects.. and also I'm using that testconfig at the root for when I'm running the test in the pipeline..

also if I add the TestingPlatformCommandLineArguments is it used when I run tests in visual studio IDE??

[alexTr3]

so I repeat if I use the the TestingPlatformCommandLineArguments in Directory.Build.props will it be used when I run tests in visual studio IDE??

what config file is used when I run test in visual studio IDE??

Also how can I limit number of thread in total... I want to share max parallels threads in the same test project first..

(ex. if I say i want 10 thread) it would try to use all thread on the first test projects until it need only 5 threads to finish the rest of its test and the test framework would start moving these thread to test another test project etc...

now it's all over the place.. having 50 test projects running in parallel with no control.. I want to focus the maxparallelthreads on test project one after the other..

so if I have 32 max parallelthread.. the first testproject would finish fast... until the next ect.. i dont want 32 test projects/exe starting in parallel.. that's crazy stupid, more so with "test containers" that are hard to share because of your awesome ideas to make test projects independent executable that cannot share static data between each other..

my test are already made to run in parallel in each projects.. I dont see why you cannot focus the threads on projects run sequentially.. it would not solve the problem of not being able to share static state between test (for test containers) but at least the test containers would not have to be created uselessly in parallel take way too much ressource timing out the tests etc..

[Youssef1313]

My advice.. make an example solution of each framework version (vstest vs mtp etc..) for each test platform (xunit) etc.. and each configuration..(testconfig, testrunner) etc.. so one can easily see the difference between all these and migrate correctly..

Sure. We can add samples for those combinations. Tracking in dotnet/samples#7079.

so I repeat if I use the the TestingPlatformCommandLineArguments in Directory.Build.props will it be used when I run tests in visual studio IDE??

That's a question for @AbhitejJohn, @peterwald, and @drognanar. I would say it's a bug if it's not respected. Running the tests should follow most of the logic for running simple console apps. This includes respecting RunCommand, RunArguments, and RunWorkingDirectory.

For parallelization, I'm not completely following with your ask. Do you want to disable running different test projects in parallel? That's already achievable both in Visual Studio and dotnet test. For Visual Studio, there is this option:

For dotnet test, there is --max-parallel-test-modules. However, disabling parallelization can really slow down test run/discovery very significantly.

For any further questions/issues, please open separate issues/discussions so that each can be focused on one topic.

[alexTr3]

I want to run test in parallel so that dropdown is useless to me..

I have 32 threads, I want these threads to be used on the first test project and when there is free thread you move them to the second test project etc.. this should be the default behavior

you could use 2-3 of these threads for test discovery in non running projects when you test the first ones..... whatever..

--max-parallel-test-modules seem to be a way to kinda do what i want too but is it working in MTP? And why would it slow down anything?? I have 32 thread I have 32 thread.. it should not be slower in any way shape or form! it should actually be faster..

since while you focus on running the first projects, you can do test discovery or optimization of other tests projects in the background until it's their turn to run..