Improvement needed to run tests with live CoreCLR

[BruceForstall]

There is an option to build corefx against live coreclr bits, passed to build.cmd as -- /p:CoreCLROverridePath=<path>. This option has been used in a CI testing scenario in the dotnet/coreclr repo, which (1) builds coreclr, (2) clones corefx and builds it using that option. This very frequently leads to failures in these CI jobs due to breaking changes. To make these tests more reliable, the CI job is changing (with dotnet/coreclr#16263) to build corefx as-is, without the /p:CoreCLROverridePath argument, and then overwrite the generated runtime files (which were copied from downloaded packages) with built coreclr files. The idea is that there will be fewer failures with this method, for several reasons: (1) there should never be a build break since corefx is building as it does in the CI, (2) test breaks due to breaking changes might not occur, or should be minor, as new apis are added or changed in coreclr.

Ideally, corefx could provide an argument like /p:CoreCLROverridePath to do what our run-corefx-tests.py script is now going to do: that is, copy coreclr binaries to the generated runtime directory AFTER the corefx build, and NOT build against them. This would encapsulate the information about where this runtime directory lives in the right place -- the corefx scripts.

[jkotas]

We will also need to:

• Be able to run subset of tests (e.g. we may not want to run networking tests in CoreCLR because of they take time, are flaky, and do not provide interesting extra coverage for the runtime)
• Suppress tests via exclusion file that lives outside of CoreFX repo
• Be able to download the prebuilt test binaries, instead of building them each time. This is nice-to-have when the corefx legs in CoreCLR repo are optional, but I think it is must-have if we were to turn them on by default.

@A-And work in dotnet/corert#5329 is related to this.

[weshaggard]

Can you help me understand the issue with CoreCLROverridePath? If it is just that the build fails I think that is by design because if you cannot build against it then you also cannot expect to be able to run against it. Tracking down runtime failures that should have been caught by the compiling of the corefx tests seems like a waist of everyone's time.

[BruceForstall]

In the absence of breaking changes, CoreCLROverridePath is fine.

The problem is that we have many, many breaking changes. Most of them seem to originate on the CoreCLR side, and these often seem to cause the corefx build using CoreCLROverridePath to fail. This breaks our corefx-in-the-coreclr repo CI testing.

However, if we build corefx against whatever coreclr packages it has been validated against in its CI, then the corefx build will succeed. If we then run against our just-built coreclr, that includes some kind of breaking changes, it's likely some tests will fail. However, the most common case is that very few tests will fail. Ideally, with a test exclusion mechanism that is independent of the corefx repo (namely, a "file" or something that is checked into the coreclr repo), we can easily disable those (few) tests.

This allows our corefx-testing-in-the-coreclr repo to work more more reliably, and be more useful more of the time.

@jkotas has much more experience with actual cases of breaking changes to be able to explain this better than me.

[jkotas]

@BruceForstall description of the problem is accurate. The problem is mostly caused by the not-yet-released APIs that folks are iterating on a lot. We have seen it a lot with Span APIs or hardware intrinsic APIs recently.

However, I do not think it is a good idea to be modifying the CoreFX build system for CoreFX test runs in CoreCLR CI. I think it would be better to download the CoreFX test binaries for CoreCLR CI (ie the contract would be zip file or package layout, not build output layout), and have custom script that runs the tests in CoreCLR CI.

[weshaggard]

However, I do not think it is a good idea to be modifying the CoreFX build system for CoreFX test runs in CoreCLR CI. I think it would be better to download the CoreFX test binaries for CoreCLR CI (ie the contract would be zip file or package layout, not build output layout), and have custom script that runs the tests in CoreCLR CI.

I prefer that approach as well. Have you guys looked at using the test drops we publish as part of our corefx official builds that we run in helix?

[BruceForstall]

Have you guys looked at using the test drops we publish as part of our corefx official builds that we run in helix?

I briefly asked @A-And about this (he had some recent experience), but it was quite complicated, and didn't solve the immediate problem I had.

Note that we need to be able to run on all platforms. Windows ARM32, for example, is quite limited, so we already have a (hacky) script (tests\run-corefx-tests.bat) that just walks the tree looking for RunTests.cmd files to invoke.

[jkotas]

it was quite complicated

What is the quite complicated part?

[BruceForstall]

I'm going to close this, due to dotnet/coreclr#18365 and subsequent changes.