Microsoft.Testing.Platform for xUnit

[bradwilson]

The approach xUnit.net v3 is taking for supporting Microsoft.Testing.Platform (since it already produces executables) is to conditionally use the "create TestApplication.CreateBuilderAsync and build it" approach when it understands that it's time to run in that way, vs. letting it run with our entry point.

Our current strategy is to:

• Link all v3 test projects against Microsoft.Testing.Platform and Microsoft.Testing.Platform.MSBuild
• Set a NuGet package variable via .props file inside the xunit.v3.core NuGet package to indicate that a v3 test project is in place
• Ship a (not linked by default) DLL (xunit.v3.runner.testingplatform) inside our xunit.v3.core NuGet package that has all the Microsoft.Testing.Platform logic
• Detect from within xunit.runner.visualstudio when it's got a v3 test project, and override the entry point to conditionally dispatch to the code inside xunit.v3.runner.testingplatform when it realizes that we're being asked to run inside of dotnet test

There are two branches right now that perform this work in concert with one another:

https://github.com/xunit/xunit/tree/microsoft-testing-platform
https://github.com/xunit/visualstudio.xunit/tree/microsoft-testing-platform

My question is about the documentation & support behind the hook. Right now, through a combination of decompilation of Microsoft.Testing.Extensions.VSTestBridge (which we are not using) and inspecting the logs from dotnet test when using that, it appears that dotnet test is utilizing a command line option that looks like --internal-msbuild-node <argument>.

Based on this comment, this is currently an undocumented and unsupported way to get dotnet test to work:

This is the reason why the option starts with --internal-, options that start with it are hidden internal.

So my question is:

When will the interaction between Microsoft.Testing.Platform test frameworks and dotnet test become documented and supported? If this isn't the way we should be doing it, then what is the officially supported way to enable dotnet test for Microsoft.Testing.Platform test framework implementations that are not based on Microsoft.Testing.Extensions.VSTestBridge?

[bradwilson]

@MarcoRossignoli @drognanar

[MarcoRossignoli]

My question is about the documentation & support behind the hook. Right now, through a combination of decompilation of Microsoft.Testing.Extensions.VSTestBridge (which we are not using) and inspecting the logs from dotnet test when using that, it appears that dotnet test is utilizing a command line option that looks like --internal-msbuild-node .

Let's start to describe what's inside the Microsoft.Testing.Platform.MSBuild: It's not a mandatory package, it's used to:

• ship a target that integrates with the current dotnet test infrastructure that handle through a predefined VSTest target the run for single tfm or multi-tfm, the setup or the target is here https://github.com/microsoft/testfx/blob/main/src/Platform/Microsoft.Testing.Platform.MSBuild/buildMultiTargeting/Microsoft.Testing.Platform.MSBuild.targets#L148-L260

In this case we didn't do any update to the sdk we simply rely on the current support. The task in this case will run the application as-is connecting through a pipe with a custom internal protocol. It means that we're skipping the VSTest infra.

NB. We're working on a new version for dotnet test to skip completely MSBuild and start directly the tests module in server mode(for now with pipe binary protocol, not jsonRpc). This will solve the problem of unclear and confusing UX and giving us a new model for execution like dotnet test --testsmodule /**/*Tests.dll to support scenario where users build on one machine move to another one and run tests. Let's say we're working to avoid the needs to run through MSBuild because it imposes some UX and runtime limitation and make complex the evolutions (i.e. pre and post actions, post assets elaboration etc...).

• Generates the new configuration file. The testing platform supports a new configuration format, a json file for now (but we could plug supports for more format like toml). Similar to what the runtime is doing we take the user configuration file defined at project level and we transform it close to the startup module. Here the definition https://github.com/microsoft/testfx/blob/main/src/Platform/Microsoft.Testing.Platform.MSBuild/buildMultiTargeting/Microsoft.Testing.Platform.MSBuild.targets#L30

• If $(GenerateTestingPlatformEntryPoint) = true (default) we inject the entry point. Initially the idea was to avoid it and propose the Program.cs in clear to the users. After some feedback we decided to make it hidden to have a transparent porting from old projects.

The target is defined here https://github.com/microsoft/testfx/blob/main/src/Platform/Microsoft.Testing.Platform.MSBuild/buildMultiTargeting/Microsoft.Testing.Platform.MSBuild.targets#L71

The testing framework supports extension and as you know they're linked at build-time. So this task takes also a specific Items collection and generates a call to an extension provided method passing the application builder and the entry point args. So to "plug" the "compile time" registration (vs manual registration where user can manually write the code inside the Main) an extension MUST provide a prop with the registration method. The task will emit the code in the autogenerated main. The sample schema for the Items is for instance the build-time registration of the Trx extension https://github.com/microsoft/testfx/blob/main/src/Platform/Microsoft.Testing.Extensions.TrxReport/buildMultiTargeting/Microsoft.Testing.Extensions.TrxReport.props

Thanks to the TestingPlatformBuilderHook we know how to emit the call, we use the TypeFullName provided by the extension and we "append" a method on it (.AddExtensions(builder, args);).

== Why you see internal-msbuild-node <argument> as args?

Because the Microsoft.Testing.Platform.MSBuild it's nothing special than an extension relying on the built-in extensibility point and so it's injecting an IDataConsumer https://github.com/microsoft/testfx/blob/main/src/Platform/Microsoft.Testing.Platform.MSBuild/TestPlatformExtensions/MSBuildExtensions.cs using the static registration mechanism https://github.com/microsoft/testfx/blob/main/src/Platform/Microsoft.Testing.Platform.MSBuild/buildMultiTargeting/Microsoft.Testing.Platform.MSBuild.props. That pipe is used to transfer the information needed by the InvokeTestingPlatformTask using a custom consumer https://github.com/microsoft/testfx/blob/main/src/Platform/Microsoft.Testing.Platform.MSBuild/TestPlatformExtensions/MSBuildConsumer.cs#L65.

So Microsoft.Testing.Platform.MSBuild is not a mandatory package to run tests it's shipping some support to help users to write less code generating some code. This package has got nothing to do with the Microsoft.Testing.Extensions.VSTestBridge. The Microsoft.Testing.Extensions.VSTestBridge is trasparent and is plugged under the test framework adapter interface https://github.com/microsoft/testfx/blob/main/docs/testingplatform/itestframework.md. This is used to have 0 update for old VSTest adapters. The future goal would be remove it and avoid deps on VSTest infrastructure. Clearly the jurney won't be quick, the new testing platform has got a complete different extensibility and runtime model so extensions like collectors, loggers are no more supported, and for that we need to fill the gap with new extensions like we did here https://learn.microsoft.com/en-us/dotnet/core/testing/unit-testing-platform-extensions.

From versioning perspective Microsoft.Testing.Extensions.VSTestBridge and Microsoft.Testing.Platform.MSBuild depend on the single core platform with 0 dependency Microsoft.Testing.Platform. This means that at built-time nuget will resolve the version, giving us (if we play good with semver) pretty stable and deterministic/testable platform+extensions configuration, a pain point of VSTest.

Microsoft.Testing.Platform.MSBuild is a transitive dep so the targets will flow up. The issue I see with the custom main with the if is that we need to take the control of program.cs if you want to give to xUnit access to the extensions. We emit the full program.cs with the registration using the MSBuild Items above.
This is what we codegen:

//------------------------------------------------------------------------------
// <auto-generated>
//     This code was generated by Microsoft.Testing.Platform.MSBuild
// </auto-generated>
//------------------------------------------------------------------------------

[global::System.Diagnostics.CodeAnalysis.ExcludeFromCodeCoverage]
internal sealed class TestingPlatformEntryPoint
{
    public static async global::System.Threading.Tasks.Task<int> Main(string[] args)
    {
        global::Microsoft.Testing.Platform.Builder.ITestApplicationBuilder builder = await global::Microsoft.Testing.Platform.Builder.TestApplication.CreateBuilderAsync(args);
        Microsoft.Testing.Platform.MSBuild.TestingPlatformBuilderHook.AddExtensions(builder, args);
        Microsoft.Testing.Extensions.Telemetry.TestingPlatformBuilderHook.AddExtensions(builder, args);
        Microsoft.VisualStudio.TestTools.UnitTesting.TestingPlatformBuilderHook.AddExtensions(builder, args);
        Microsoft.Testing.Extensions.CodeCoverage.TestingPlatformBuilderHook.AddExtensions(builder, args);
        Microsoft.Testing.Extensions.TrxReport.TestingPlatformBuilderHook.AddExtensions(builder, args);
        using (global::Microsoft.Testing.Platform.Builder.ITestApplication app = await builder.BuildAsync())
        {
            return await app.RunAsync();
        }
    }
}

Collecting from MSBuild

So I think that if you want to plug to the new testing platform you have to opt-out your one and give us the control on the starting point or extensions won't work.

Another possible issue I see is that the new testing platform support custom command line arguments so if you want your users use your v3 options at command line you should provide the extension for it. Here the documentation https://github.com/microsoft/testfx/blob/main/docs/testingplatform/icommandlineoptionsprovider.md with a sample here https://github.com/microsoft/testfx/blob/main/docs/testingplatform/Source/TestingPlatformExplorer/TestingFramework/TestingFramework.CommandLineOptions.cs

[MarcoRossignoli]

If this isn't the way we should be doing it, then what is the officially supported way to enable dotnet test for Microsoft.Testing.Platform test framework implementations that are not based on Microsoft.Testing.Extensions.VSTestBridge?

Finally if you plug to the Microsoft.Testing.Platform you will have built-in support to all the "protocol" model we will ship for dotnet test and VS/VS Code transparently. The server mode is developed, maintained and evolved inside the core Microsoft.Testing.Platform.

You could have an msbuild property like <UseTestingPlatform> and if true you ref Microsoft.Testing.Platform and Microsoft.Testing.Platform.MSBuild same version and opting out your codegen.
In this way users can opt-in the new testing platform for what you've already implemented the framework and we take control of the entry-point and extensions(if users will add some). In that way you will have everything out of the box.

Another suggestion maybe for next loop is to have all this inside an MSBuild sdk. It helps a lot with clean opt-in/out without giving to users a way to break the config https://learn.microsoft.com/en-us/dotnet/core/testing/unit-testing-mstest-sdk source here https://github.com/microsoft/testfx/tree/main/src/Package/MSTest.Sdk

[bradwilson]

So Microsoft.Testing.Platform.MSBuild is not a mandatory package to run tests

When users are asking us to support Microsoft.Testing.Platform (primarily because they've been told there are performance benefits), I don't think this statement is true, because they will expect dotnet test to run with Microsoft.Testing.Platform if we claim to support it.

Another way to phrase this is: if any unit test framework (ignore xUnit.net v3 the moment) only supported Microsoft.Testing.Platform and took no other dependencies, how would users be able to see that benefit? There would be no support in dotnet test and no support in Test Explorer. The only way they'd "see" it is if they tried to directly run the test project itself.

If that's the intended goal, to be able to run directly run a project and have it be faster than dotnet test, we already meet that goal with our v3 design, without needing Microsoft.Testing.Platform.

I personally believe that from a user perspective they're going to expect that we use Microsoft.Testing.Platform for dotnet test and Test Explorer in order to consider this feature complete. There would be zero benefit for us shipping an "only Microsoft.Testing.Platform" feature today, because all we'd end up doing is adding your layer of indirection around a discovery and execution system that already exists, and by definition is faster when we don't involve you. (That assumes that we let you control the entry point.) Do you agree?

This is used to have 0 update for old VSTest adapters. The future goal would be remove it and avoid deps on VSTest infrastructure.

We're ready now. 😄

Our current split plan right now with xunit.runner.visualstudio is that we'll continue to use TPv2 for v1 and v2 unit tests, and Microsoft.Testing.Platform for v3 unit tests. I have isolated the logic to "turn on" Microsoft.Testing.Platform for v3 tests into this package, but there's no reason for that to be true today. My original prototype just enabled it straight away for all v3 test projects without needing that separate package. I'm honestly torn on which is better, but for sure because of the way NuGet dependencies work in my current plan, all v3 test projects will end up getting linked against Microsoft.Testing.Platform whether they end up bringing in xunit.runner.visualstudio or not, simply because there's no other way for me to conditionally add that reference based on whether the user has a v3 test project. NuGet's package dependency logic is "always on" or "always off": your dependencies are in your .nuspec, no way around that. And we don't want to split v1/v2 users off to one package and v3 users off to a different package. Besides, there appears to be zero downside to our users having that dependency when it isn't used, precisely because you've designed it for zero upstream dependencies.

The issue I see with the custom main with the if is that we need to take the control of program.cs if you want to give to xUnit access to the extensions. We emit the full program.cs with the registration using the MSBuild Items above.

You've botched the dependency tree, though. Microsoft.Testing.Platform.MSBuild requires references, based on that auto-generated code, that you haven't brought along with you. You're assuming they'll come from users who are referencing Microsoft.Testing.Extensions.VSTestBridge. Right now you probably assume that those two packages always live together, but that's not true for me. That's the reason that I don't have any of the extension registrations in my code that you do, other than the one from Microsoft.Testing.Platform.MSBuild, because I haven't taken any of those other dependencies: https://github.com/xunit/xunit/blob/cccc2b35f58bf2503b63b5ab380d20f511006621/src/xunit.v3.runner.testingplatform/TestPlatformTestFramework.cs#L227-L232

We can't unilaterally give you control of our entry point, though. That's out of the question, because our users need to be able to run tests outside of dotnet test and Test Explorer. Not only do we have the interactive scenario (where a user just runs the test project) where we need strong control over the UX, there's also the scenario where our multi-assembly runners (like xunit.runner.visualstudio and xunit.v3.runner.console) need to be able to launch and control v3 unit test projects, and that is simply not possible if we let you own the command line. I looked at your command line infrastructure and it's not sufficient for the kinds of parsing and validation we do. We also need strong control over StdOut when we're running in what I called "automated" mode, which is how v3 unit test projects are controlled by our multi-assembly runners. The only output permitted on StdOut is one-per-line JSON serialization of our runner message model.

So I think that if you want to plug to the new testing platform you have to opt-out your one and give us the control on the starting point or extensions won't work.

The two questions I have here aren't around the core of what we're doing right now in that code snippet above, but:

1. How stable is this extension list over time? Would it be possible for us to discover that list at build time without using your injected entry point, like maybe you inject a function at a well known namespace/class/method that could be called by us at the right time to do extension registration? That's basically 90% of your auto-gen'd code, but in a way that we could consume; the other 10% is understanding what command line switches tell us we need to delegate control to you for dotnet test and Test Explorer to function (like --server).
2. How many dependencies do you force into us as a result?

You speak above about the zero-dependency Microsoft.Testing.Platform, but those extensions aren't in Microsoft.Testing.Platform. They're spread among a bunch of other NuGet packages, which do have real dependency impacts directly on our end users. The top level dependency list from Microsoft.Testing.Extensions.VSTestBridge 1.3.2 is:

• Microsoft.ApplicationInsights (>= 2.22.0)
• Microsoft.Testing.Extensions.Telemetry (>= 1.3.2)
• Microsoft.Testing.Extensions.TrxReport.Abstractions (>= 1.3.2)
• Microsoft.Testing.Platform (>= 1.3.2)
• Microsoft.TestPlatform.ObjectModel (>= 17.10.0)

Digging into those, you can see more upstream dependencies of things that unit test users very well may be using, and therefore have potential places to conflict when different versions are needed that end up not being compatible.

Bottom Line

Is the "we must own your entry point" a 100% requirement that you are immovable from? If so, then it sounds like we will never support Microsoft.Testing.Platform, because it is a 100% requirement on our side that we own our entry point.

We'd prefer a model where we work with you to find a way so that you can evolve Microsoft.Testing.Platform while we still own our own command line UX. An ideal solution would be for us to be able to just update NuGet package versions and it "just works", but if that means occasionally updating our entry point to accommodate new extensions and the evolution of your command line requirements, I'm fine with that. We can move forward with that strategy, assuming that things get documented appropriately. Right now having to dig everything out of decompiled binaries and log file isn't really a tenable solution. I understand that that's because you're still in an experimental place (at least as it pertains to dotnet test; I expect we're going to have a similar conversation in my other issue related to Test Explorer). But right now, I know that you would like for us to support Microsoft.Testing.Platform, and our users would like the presumed performance benefits when using dotnet test and Test Explorer. It just seems like it's too early for us to be able to do that officially, because we are unable to use the VSTestBridge (where it seems like the heavy investment went in order to get things bootstrapped went).

If the entry point ownership is 100% no compromise, then please say that plainly so that I can close this issue and point users here when they ask why we won't support Microsoft.Testing.Platform.

[MarcoRossignoli]

I personally believe that from a user perspective they're going to expect that we use Microsoft.Testing.Platform for dotnet test and Test Explorer in order to consider this feature complete. There would be zero benefit for us shipping an "only Microsoft.Testing.Platform" feature today, because all we'd end up doing is adding your layer of indirection around a discovery and execution system that already exists, and by definition is faster when we don't involve you. (That assumes that we let you control the entry point.) Do you agree?

We're towards that goal but we added already some benefits that are currently used.

First step was have the new platform with all the needed extensibility point.
This already solved some problem for some team, they're running in CI with dotnet run (with AzureCLI@2 for instance) or manually with scripts.
Others are using F5 in VS/VS Code working in a focus way without the TestExplorer (they setup the filter in the launcher and they debug in that way) for tests that are hard to run or failing inside VSTest host.
We shipped some interesting extension like Retry that now is portable in every OS/.NET version, this is another already used scenario, where the user friendliness of dotnet test is less important than the feature.
Plus allowing to run aot or with custom configuration like in fan out or without the needs to have the sdk on the machine are anyway appreciated additions.
We're close to have Test Explorer using directly the testing application in server mode (we're dogfooding it and we're adding the last needed feature to have the parity with VSTest).

So the Microsoft.Testing.Platform.MSBuild currently is giving the same performance as dotnet run but with the known gesture. And we're working to drop it in favor of the new dotnet test version that will talk directly with the test application like Test Explorer without the needs of the MSBuild infra, this should solve the old confusing UX output. This needs some time.

We're ready now. 😄

I saw our branch, happy to see that the documentation and the sample helped to quickly integrate. On our side we need a bit more time to "adapt" to the rest of the environment and tooling.

NuGet's package dependency logic is "always on" or "always off": your dependencies are in your .nuspec, no way around that.

We solved that problem with the MSBuild SDK there you can conditionally opt-in out set of different packages. We switch completely the implementation of the engine when we go AOT. Preserving the UseVSTest for users that wants to opt-in old set of packages.

You've botched the dependency tree, though. Microsoft.Testing.Platform.MSBuild requires references, based on that auto-generated code, that you haven't brought along with you. You're assuming they'll come from users who are referencing Microsoft.Testing.Extensions.VSTestBridge

What you mean? Microsoft.Testing.Platform.MSBuild depends only on the core plat to insert the extension needed for dotnet test (IDataConsumer) https://www.nuget.org/packages/Microsoft.Testing.Platform.MSBuild#dependencies-body-tab it doesn't need Microsoft.Testing.Extensions.VSTestBridge. It generates the entry point and the configuration but you can opt-out it.

I looked at your command line infrastructure and it's not sufficient for the kinds of parsing and validation we do.

What's missing?

Not only do we have the interactive scenario (where a user just runs the test project) where we need strong control over the UX

We ship by default with our console UX to report the result and the errors but we don't capture the output in any way, if you do Console.WriteLine it will end in console. If you need the complete control on the UX(we call it OutputDevice) we can open this extension (closed because we were waiting to understand if can be interesting for someone) https://github.com/microsoft/testfx/blob/main/src/Platform/Microsoft.Testing.Platform/OutputDevice/IPlatformOutputDevice.cs and you can substitute completely the UX. The new testing platform can run headless. @nohwnd is working to make the built-in one better at the moment #3292.

How stable is this extension list over time? Would it be possible for us to discover that list at build time without using your injected entry point, like maybe you inject a function at a well known namespace/class/method that could be called by us at the right time to do extension registration? That's basically 90% of your auto-gen'd code, but in a way that we could consume; the other 10% is understanding what command line switches tell us we need to delegate control to you for dotnet test and Test Explorer to function (like --server).

This is not super clear as question, at the moment we're doing what you said, we use MSBuild prop Items that we codegen at build-time as I've explained above(https://github.com/microsoft/testfx/blob/main/src/Platform/Microsoft.Testing.Platform.MSBuild/buildMultiTargeting/Microsoft.Testing.Platform.MSBuild.props). Extension will be shipped by us and by users and if they provide the correct Item inside the prop they will register themselves without the need of any change inside the program.cs. Otherwise users can disable the codegen and do it manually with the classic extension method.

How many dependencies do you force into us as a result?

For a test adapter the only dependency is Microsoft.Testing.Platform. Microsoft.Testing.Platform.MSBuild is needed today to inject the entry point with the custom extension registration that you can do by your own or providing a clear Program.cs and to plug dotnet test using MSBuild that you can do by your own using the current infrastructure. When TextExplorer and the new dotnet test will be completed the usage of Microsoft.Testing.Platform.MSBuild can be opted-out and used only to insert the entry point and transform the configuration file.
Anyway these 2 extension are pretty light Microsoft.Testing.Platform takes no deps and Microsoft.Testing.Platform.MSBuild takes Microsoft.Testing.Platform. So we should not pollute or interfere in any way.

Microsoft.Testing.Extensions.Telemetry is used for telemetry, useful to understand what we can deprecate, here all the info collected https://learn.microsoft.com/en-us/dotnet/core/testing/unit-testing-platform-telemetry, but optional.
Microsoft.Testing.Extensions.TrxReport.Abstractions: it's part of the Trx generator extension. The new testing platform is shipped with a new capability system that allows extensions to talk and provide information to other extension. This lib export the object model for some custom property to push in the message bus to make the trx complete (trx is based on class/method names and the default list of properties shipped with the core lib doesn't containt specific extension information). So it's optional, good to have to produce a good trx, the deps is without any deps but again the core one. Here the "properties" exposed by the trx extension https://github.com/microsoft/testfx/blob/main/src/Platform/Microsoft.Testing.Extensions.TrxReport.Abstractions/TrxReportProperties.cs

After users can add extensions and extensions can take dependencies our of our control.

You speak above about the zero-dependency Microsoft.Testing.Platform, but those extensions aren't in Microsoft.Testing.Platform. They're spread among a bunch of other NuGet packages, which do have real dependency impacts directly on our end users. The top level dependency list from Microsoft.Testing.Extensions.VSTestBridge 1.3.2 is:

You don't need Microsoft.Testing.Extensions.VSTestBridge if you drop the old VSTest support. For what I see you didn't use it in xunit v3 so you should not see it in the list of deps. We use it in MSTest because unfortunately we're not yet ready to drop the object model of VSTest that's down the code and we don't have separation between engine objects and VSTest objects.

Is the "we must own your entry point" a 100% requirement that you are immovable from? If so, then it sounds like we will never support Microsoft.Testing.Platform, because it is a 100% requirement on our side that we own our entry point.

If you copy our way to at compile time register the dependency you can take it. It's explained above, if you codegen the registration by your own it's fine we don't need the entry point. You need to flow the options to our builder for server mode (for dotnet test and TestExplorer)

But right now, I know that you would like for us to support Microsoft.Testing.Platform, and our users would like the presumed performance benefits when using dotnet test and Test Explorer. It just seems like it's too early for us to be able to do that officially, because we are unable to use the VSTestBridge (where it seems like the heavy investment went in order to get things bootstrapped went).

Our goal is to drop the VSTestBridge so you're one step ahead. If you want to wait till we will have complete the new dotnet test and the TestExplorer to complete the integration I think it's understandable. We need to understand how to handle the command line issue. If you try a --help you'll see some options and extension are also registering custom options. I think that if you want to take the entry is doable for the compile-time extension registration so that problem I think it's solved(here how we're doing and the Items spec https://github.com/microsoft/testfx/blob/main/src/Platform/Microsoft.Testing.Platform.MSBuild/Tasks/TestingPlatformEntryPointTask.cs#L45 and we can help there too given that we spec the MSBuild Items), we provide the server mode for dotnet test and TestExplorer and we can fill the gap in case something is missing so also that part should be fine, the only tricky part is the command line.

[bradwilson]

What you mean? Microsoft.Testing.Platform.MSBuild depends only on the core plat to insert the extension needed for dotnet test (IDataConsumer) https://www.nuget.org/packages/Microsoft.Testing.Platform.MSBuild#dependencies-body-tab it doesn't need Microsoft.Testing.Extensions.VSTestBridge. It generates the entry point and the configuration but you can opt-out it.

Your claim was that you auto-generate this entry point (which, BTW, I cannot duplicate):

[global::System.Diagnostics.CodeAnalysis.ExcludeFromCodeCoverage]
internal sealed class TestingPlatformEntryPoint
{
    public static async global::System.Threading.Tasks.Task<int> Main(string[] args)
    {
        global::Microsoft.Testing.Platform.Builder.ITestApplicationBuilder builder = await global::Microsoft.Testing.Platform.Builder.TestApplication.CreateBuilderAsync(args);
        Microsoft.Testing.Platform.MSBuild.TestingPlatformBuilderHook.AddExtensions(builder, args);
        Microsoft.Testing.Extensions.Telemetry.TestingPlatformBuilderHook.AddExtensions(builder, args);
        Microsoft.VisualStudio.TestTools.UnitTesting.TestingPlatformBuilderHook.AddExtensions(builder, args);
        Microsoft.Testing.Extensions.CodeCoverage.TestingPlatformBuilderHook.AddExtensions(builder, args);
        Microsoft.Testing.Extensions.TrxReport.TestingPlatformBuilderHook.AddExtensions(builder, args);
        using (global::Microsoft.Testing.Platform.Builder.ITestApplication app = await builder.BuildAsync())
        {
            return await app.RunAsync();
        }
    }
}

My counterpoint was that several of these lines would not compile without additional references; specifically:

        Microsoft.Testing.Extensions.Telemetry.TestingPlatformBuilderHook.AddExtensions(builder, args);
        Microsoft.VisualStudio.TestTools.UnitTesting.TestingPlatformBuilderHook.AddExtensions(builder, args);
        Microsoft.Testing.Extensions.CodeCoverage.TestingPlatformBuilderHook.AddExtensions(builder, args);
        Microsoft.Testing.Extensions.TrxReport.TestingPlatformBuilderHook.AddExtensions(builder, args);

Given that I tried to repro this, and saw only this as your auto-generated entry point, I'm confused by what you claimed previously:

Note that this is exactly what we're doing as well:

(The extra Register call is us registering ourselves as a test framework.)

So I'm not sure what/how we go from what I'm seeing today to what you claimed above. Is your claim above an idealized goal for the future? Is the auto-generation dependent on the user adding references to NuGet packages that include register-able things (for lack of a better term)? Are you sniffing the TestingPlatformBuilderHook item groups from the NuGet packages to decide how to generate the entry point? If so, that's something I could do as well, and that would always keep us in sync. Users could add the packages they want and we'll make sure they get registered. Is that how it works today?

What's missing?

• How can I make the -? output look like this? Also note: there are two command line options that aren't based on dashes: you can specify :<seed> to set the randomization seed and you can specify a path to the configuration file. This was done to match the command line option in xunit.v3.runner.console which allows you to specify <assembly>:<seed> [<configfile>] [<assembly>:<seed> [<configfile>]...].

  

• Is there a hook that exists for me, before you ask to discover or execute, that I could inspect those command line options and set up things like the reporter?

• How would I validate something complex like the -maxThreads option?

• Can I use single-dashes for long names, which has historically been how our console runner works? Or will I be forced to change all my command line options that my users have a decade of muscle memory (and build scripts)?

We ship by default with our console UX to report the result and the errors but we don't capture the output in any way, if you do Console.WriteLine it will end in console.

I'll take your word for it if you say that you don't ever output anything to the console. My experience so far has been with dotnet test, but those headers and the VSTest style output may be a result of running via dotnet test rather than running directly.

You need to flow the options to our builder for server mode (for dotnet test and TestExplorer)

Does this mean I only have to look for you to pass --server, at which point I just hand you all the args and go? That's effectively what I'm doing today (except that I'm also looking for --internal-msbuild-node, but if we remove Microsoft.Testing.Platform.MSBuild and the temporary dotnet test overloads, then I could stop looking for that).

If you want to wait till we will have complete the new dotnet test and the TestExplorer to complete the integration I think it's understandable.

If there's value in me providing something in the meantime that people could opt into that would work, then I'd be happy to do that (as I presume it would also help you all do testing and validation of the proposed usage).

Today that looks like me including Microsoft.Testing.Platform.MSBuild and recognizing --internal-msbuild-node to offer that transitional dotnet test support, which I'm okay with if that's what people want.

Also today, as #3479 illustrates, I have no idea whether I'm being used via TPLv2 or Microsoft.Testing.Platform inside of Visual Studio. I do know the tests are being run, but there is no visible way to know which path they're taking (short of giving myself environment debugger hacks to watch and see which path things take), but I'm happy to have that discussion over in that issue, if it's appropriate.

I feel like, based on this conversation, the appropriate path is probably:

• Remove our dependency on Microsoft.Testing.Platform.MSBuild and accept that dotnet test for now will still be done via TPLv2
• Enable Microsoft.Testing.Platform in v3 test projects, always, and remove the necessity of xunit.runner.visualstudio (which will remain how you enable TPLv2)
• Trust that when I have both Microsoft.Testing.Platform and TPLv2 support (because almost all users will still want to reference xunit.runner.visualstudio, at least until such time that realistically zero users will need the support of a TPLv2 runner), you all are picking up that we support M.T.P and using us that way.

Given the non-intrusive nature of this (basically, I'm just sniffing for --server and users bring along one dependency they won't care about), I have no problem doing this now and not waiting. I just want to get clarification on my question about how your auto-generation is triggered today, and make sure I can do something equivalent on my end.

[bradwilson]

I do see one problem right now with my current thought:

<TestingPlatformBuilderHook Include="98058041-B5B6-4A75-9834-58E6DF796A22" Condition=" '$(GenerateTestingPlatformEntryPoint)' == 'True' " >

We obviously have to set GenerateTestingPlatformEntryPoint to false so you won't generate the entry point, which means I can't sniff to TestingPlatformBuilderHook entries to do the dynamic registration.

Is there any reason this should actually be conditional? What's the harm in all of your extensions just always having TestingPlatformBuilderHook MSBuild items? It's just a couple strings. Unless you make this stuff always on, nobody else will be able to generate an entry point but you.

(That said, I could make it work today, because removing Microsoft.Testing.Platform.MSBuild turns off your entry point generation anyway, but it does sound like eventually you're going to move that somewhere else and then we'll be stuck.)

[bradwilson]

I have updated the microsoft-testing-platform branch in the core framework, and I'm throwing the one from xunit.runner.visualstudio away, since it's not necessary in the current proposed model.

If you want to take a look at the changeset and see if this aligns with what you'd expect: xunit/xunit@0073f0b

This is a project that just references xunit.v3 (albeit one built from my branch):

It references the Microsoft.Testing.Extensions.TrxReport NuGet package, so this is what the auto-generated entry point looks like:

// <auto-generated> This file has been auto generated by xunit.v3.core. </auto-generated>

[global::System.Diagnostics.CodeAnalysis.ExcludeFromCodeCoverage]
public class XunitAutoGeneratedEntryPoint
{
    public static int Main(string[] args)
    {
        if (args.Length > 0 && args[0] == "--server")
            return global::Xunit.Runner.InProc.SystemConsole.TestingPlatform.TestPlatformTestFramework.RunAsync(args, typeof(Microsoft.Testing.Extensions.TrxReport.TestingPlatformBuilderHook)).GetAwaiter().GetResult();
        else
            return global::Xunit.Runner.InProc.SystemConsole.ConsoleRunner.Run(args).GetAwaiter().GetResult();
    }
}

I followed the pattern of the code you linked to that ensures that the TrxReport extension is always registered last.

[bradwilson]

I followed the pattern of the code you linked to that ensures that the TrxReport extension is always registered last.

I feel like maybe an optional priority might be better than hard coding this logic.

[MarcoRossignoli]

So I'm not sure what/how we go from what I'm seeing today to what you claimed above. Is your claim above an idealized goal for the future? Is the auto-generation dependent on the user adding references to NuGet packages that include register-able things (for lack of a better term)? Are you sniffing the TestingPlatformBuilderHook item groups from the NuGet packages to decide how to generate the entry point? If so, that's something I could do as well, and that would always keep us in sync. Users could add the packages they want and we'll make sure they get registered. Is that how it works today?

Yes we're "sniffing the TestingPlatformBuilderHook" so the lines you see inside the codegen depends on the extension packages added.

I feel like maybe an optional priority might be better than hard coding this logic.

I agree, we discussed and for first version we decided to embed it. But it's on the table.

How can I make the -? output look like this? Also note: there are two command line options that aren't based on dashes: you can specify : to set the randomization seed and you can specify a path to the configuration file. This was done to match the command line option in xunit.v3.runner.console which allows you to specify : [] [: []...].

I think that is not fundamental have 100% same UX for the help. Under the new(in-progress) "dotnet test --help" for instance we aggregate all the options and we notify the users if some tests module has got different options (usually brought by extensions) because at runtime for that specific test module with the missing extension if will fail for invalid option. I mean the UX will be for sure different.

So I think that you can map all your options under the testing plat and for the option without a "-some" add a new one i.e. --config ... --seed ... only when is running in testing platform mode. Despite your UX looks really well organized I think that at least for the start have a plain list where users can read and maybe look at the documentation if something is not clear is a good starting point. Usually once users understand the option they won't change it for some time. I mean the important thing I see is that users can in some way opt-in/out all the xunit feature.

PS. It's not so bad idea give an extensibility point where test framework author can write the custom help, I'll open a ticket for it. cc: @Evangelink

Is there a hook that exists for me, before you ask to discover or execute, that I could inspect those command line options and set up things like the reporter?

In your ITestFramework implementation you can get thought the IServiceProvider the ICommandLineOptions and query the command line and apply the options. For instance https://github.com/microsoft/testfx/blob/main/docs/testingplatform/Source/TestingPlatformExplorer/TestingFramework/TestingFramework.cs#L47

How would I validate something complex like the -maxThreads option?

The ICommandLineOptionsProvider allows you to validate single options and global configuration at the end for instance https://github.com/microsoft/testfx/blob/main/docs/testingplatform/Source/TestingPlatformExplorer/TestingFramework/TestingFramework.CommandLineOptions.cs

Can I use single-dashes for long names, which has historically been how our console runner works? Or will I be forced to change all my command line options that my users have a decade of muscle memory (and build scripts)?

At the moment we support -- and - also if we advice for the -- in the documentation and in the help.

I'll take your word for it if you say that you don't ever output anything to the console. My experience so far has been with dotnet test, but those headers and the VSTest style output may be a result of running via dotnet test rather than running directly.

We give a way to personalize the banner implementing the IBannerMessageOwnerCapability and at the moment we ship our built-in output console. But we can if you need add an option that will allow you to have headless runtime and you can write your own UX. If it's a requirement for you open a ticket for it.

Does this mean I only have to look for you to pass --server, at which point I just hand you all the args and go? That's effectively what I'm doing today (except that I'm also looking for --internal-msbuild-node, but if we remove Microsoft.Testing.Platform.MSBuild and the temporary dotnet test overloads, then I could stop looking for that).

Yes, in--server modes (jsonRpc for VS and pipe binary for dotnet test) we expect to invoke the tests module passing --server --client-port xx --client-host xxxx (VS create the socket at the moment) or --server --dotnet-test-pipe xxxx (but you don't have to take care of it we could change in future at the moment the options for server mode are hidden and used only by tools) to decide which protocol we want to use...so if you do an index of inside the args[] and you find this parameter and it's not one of yours you could "forward" to the testing platform workflow. And anyway if this is not enough we can add an hidden option (we support hidden options) like --internal-run-testingplatform and you can check that one or something specific.

Today that looks like me including Microsoft.Testing.Platform.MSBuild and recognizing --internal-msbuild-node to offer that transitional dotnet test support, which I'm okay with if that's what people want.

True current dotnet integration that rely on the current(old) MSBuild infra can be recognized using --internal-msbuild-node . NB --internal- prefix is reserved.

Also today, as #3479 illustrates, I have no idea whether I'm being used via TPLv2 or Microsoft.Testing.Platform inside of Visual Studio. I do know the tests are being run, but there is no visible way to know which path they're taking (short of giving myself environment debugger hacks to watch and see which path things take), but I'm happy to have that discussion over in that issue, if it's appropriate.

VS decide if start using the server mode (start process passing args) looking at the ProjectCapability named TestingPlatformServer. We set it inside the Microsoft.Testing.Platform.MSBuild at the moment https://github.com/microsoft/testfx/blob/main/src/Platform/Microsoft.Testing.Platform.MSBuild/buildMultiTargeting/Microsoft.Testing.Platform.MSBuild.targets#L11-L16 if that is found and the preview feature(not yet public) is enabled the TestExplorer will use the jsonRpc protocol and will skip VSTest infra.
Again, if you accept for now to link Microsoft.Testing.Platform.MSBuild and Microsoft.Testing.Platform that are deps free we can "incrementally" provide the glue and when we will be there we can drop the Microsoft.Testing.Platform.MSBuild. The reason for the split of the two lib is because we want to be able to "always" in every situation run tests, to have always the plan b. You can link only Microsoft.Testing.Platform and we can run the test, the services provided by msbuild can be done manually(entry point, config) and in some scenario dotnet test is not available at all so your can anyway run your tests.

Remove our dependency on Microsoft.Testing.Platform.MSBuild and accept that dotnet test for now will still be done via TPLv2
Enable Microsoft.Testing.Platform in v3 test projects, always, and remove the necessity of xunit.runner.visualstudio (which will remain how you enable TPLv2)
Trust that when I have both Microsoft.Testing.Platform and TPLv2 support (because almost all users will still want to reference xunit.runner.visualstudio, at least until such time that realistically zero users will need the support of a TPLv2 runner), you all are picking up that we support M.T.P and using us that way.

I think you could keep both deps Microsoft.Testing.Platform.MSBuild and Microsoft.Testing.Platform for v3, when you're running with Tpv2 we're using reflection/scan files to load the adapter and you're "entering" in your code ignoring the 2 dll (Microsoft.Testing.Platform.MSBuild.dll and Microsoft.Testing.Platform), isn't it? We ship both libs aligned always so if they're harmful(if not 2 small file inside the bin) update them to take new stuff should be easy upgrade. Users can opt-in/out the integration with dotnet test using the TestingPlatformDotnetTestSupport.
But up to you if you want to start without the current dotnet test support. The good part is that the msbuild package allows you also to plug VS thanks to the TestingPlatformServer. Let's say that we could think to have the same setup we have in MSTest but you don't have the VSTestBridge and the others deps that we ship by-default (extensions like codecoverage, trx and telemetry).

Is there any reason this should actually be conditional? What's the harm in all of your extensions just always having TestingPlatformBuilderHook MSBuild items? It's just a couple strings. Unless you make this stuff always on, nobody else will be able to generate an entry point but you.

The reason is to not pollute MSBuild with unused items. We didn't expect the needs to provide the same items outside our codege. But it's fine for me remove the conditional and let you sniff.

If you want to take a look at the changeset and see if this aligns with what you'd expect: xunit/xunit@0073f0b

I'll check and let you know soon. Thanks for the effort.