[XHarness] Reduce the number of test applications for the BCL tests. #5768

mandel-macaque · 2019-03-14T13:03:22Z

This commit improves the state of the BCL testing in the following ways:

Improve the device tests running. Less apps, faster results.
Reduces the chances of getting issue
https://github.com/xamarin/maccore/issues/581 when running the tests
since we have less apps to run.

We now support the ignore files per assembly name to simplify the
tracking of the ignored tests.

mandel-macaque · 2019-03-14T13:03:44Z

This moves to have just 4 apps for ALL the BCL tests. Once landed, I'll try to move all BCL tests to 2 single Mac OS X apps, one for NUnit, other for xUnit.

spouliot

Make sure this runs correct on device for 32bits, i.e. both iOS and watchOS
I'm concerned that either/both the native linker (24bits) limits and watchOS timeouts could break this

Maybe it's possible to have this grouping optional ?

monojenkins · 2019-03-14T13:36:57Z

Build failure
❌ Build was aborted

rolfbjarne · 2019-03-15T10:34:52Z

build

rolfbjarne

Maybe it's possible to have this grouping optional ?

One possibility is to add both options to the list of tests when running locally (in server mode). This would mean the non-grouped versions won't even exist on the bots.

rolfbjarne · 2019-03-15T10:38:36Z

tools/bcl-test-importer/BCLTestImporter/BCLTestProjectGenerator.cs

+									"monotouch_System.Security_xunit-test.dll",
+									"monotouch_System.Runtime.Serialization_xunit-test.dll",
+									"monotouch_System.Runtime.CompilerServices.Unsafe_xunit-test.dll", }),
+			(name:"BCL tests group 4", assemblies: new[] { "monotouch_System.Core_xunit-test.dll",


A couple of really minor naming points:

Maybe name the XUnit and Unit tests differently: BCL tests group X (NUnit) and BCL tests group X (XUnit).

I'd intuitively expect the most basic/important assemblies to use a lower number: that the mscorlib/System tests would be in group 1, and more peripheral assemblies in group >1.

monojenkins · 2019-03-15T11:08:21Z

Build failure
❌ Build was aborted

rolfbjarne · 2019-03-15T11:10:58Z

build

monojenkins · 2019-03-15T11:44:20Z

Build failure
❌ Build was aborted

monojenkins · 2019-03-15T12:35:17Z

Build failure
❌ Build was aborted

This commit improves the state of the BCL testing in the following ways: 1. Improve the device tets running. Less apps, faster results. 2. Reduces the chances of getting issue https://github.com/xamarin/maccore/issues/581 when running the tests since we have less apps to run. We now support the ignore files per assembly name to simplify the tracking of the ignored tests.

mandel-macaque · 2019-03-15T13:32:48Z

@rolfbjarne got a nice idea on how to do it so that locally they are not grouped (better for the developer to test and debug) and group on jenkins). You might or not like it ;)

akoeplinger · 2019-03-15T14:07:13Z

@mandel-macaque it'd be nice if this was an option, I'd like to use the grouped version locally too.

rolfbjarne · 2019-03-15T14:32:22Z

build

monojenkins · 2019-03-15T16:35:17Z

Build success
✅ Build succeeded
✅ API Diff (from stable)
✅ API Diff (from PR only) (no change)
✅ Generator Diff (no change)
✅ Test run succeeded

mandel-macaque · 2019-03-15T19:11:17Z

@spouliot @akoeplinger after the last push, it is an option. You can make xharness think you are on Jenkins setting the env var https://github.com/xamarin/xamarin-macios/blob/master/tests/xharness/Harness.cs#L692 else, if that env var does not have the right value, you will get one app per assembly as we used.

I have not look much into the group naming, so ideas are welcome (simple string replacement).

monojenkins · 2019-03-15T19:15:26Z

Build failure
❌ Build was aborted

✅ Build succeeded
✅ API Diff (from stable)
✅ API Diff (from PR only) (no change)
✅ Generator Diff (no change)

akoeplinger · 2019-03-15T20:06:24Z

@mandel-macaque that's pretty non-discoverable and reusing the Jenkins variable probably has other implications? why not add a separate option?

monojenkins · 2019-03-16T03:18:06Z

Build failure
❌ Build was aborted

mandel-macaque · 2019-03-18T11:40:03Z

@akoeplinger code/var reuse, but add a new one in the last commit: 343efd7#diff-8f44b7ce671e35190b27341716ad7e8cR699

monojenkins · 2019-03-18T17:10:57Z

Build failure
✅ Build succeeded
✅ API Diff (from stable)
✅ API Diff (from PR only) (no change)
✅ Generator Diff (no change)
🔥 Test run failed 🔥

Test results

5 tests failed, 0 tests skipped, 181 tests passed.

Failed tests

mscorlib/Mac/Modern: Failed (Test run failed.)
[NUnit] Mono Mac OS X BCL tests group 1/Mac/Full: TimedOut (Execution timed out after 1200 seconds.)
[xUnit] Mono Mac OS X BCL tests group 2/Mac/Full: TimedOut (Execution timed out after 1200 seconds.)
[NUnit] Mono Mac OS X BCL tests group 1/Mac/Modern: TimedOut (Execution timed out after 1200 seconds.)
[xUnit] Mono Mac OS X BCL tests group 2/Mac/Modern: Failed (Test run crashed (exit code: 255).)

…ot contain ignored assemblies.

mandel-macaque · 2019-03-18T23:20:53Z

Looks like the combination of tests assemblies on mac os x makes the tests block. Will provide a sample to @akoeplinger

Current split works. Also, if an assembly is ignore in a platform, it will not be included in the grouped app.

monojenkins · 2019-03-19T03:37:24Z

Build failure
✅ Build succeeded
✅ API Diff (from stable)
✅ API Diff (from PR only) (no change)
✅ Generator Diff (no change)
🔥 Test run failed 🔥

Test results

2 tests failed, 0 tests skipped, 190 tests passed.

Failed tests

[NUnit] Mono Mac OS X BCL tests group 1/Mac/Full: TimedOut (Execution timed out after 1200 seconds.)
[NUnit] Mono Mac OS X BCL tests group 1/Mac/Modern: TimedOut (Execution timed out after 1200 seconds.)

monojenkins · 2019-03-19T15:20:21Z

Build success
✅ Build succeeded
✅ API Diff (from stable)
✅ API Diff (from PR only) (no change)
✅ Generator Diff (no change)
✅ Test run succeeded

mandel-macaque · 2019-03-19T17:13:34Z

As per las commit, iOS/tvOS and MacOS apps are grouped, watchOS are not due to small issues with the deployment to the device and their execution.

monojenkins · 2019-03-19T21:28:15Z

Build failure
✅ Build succeeded
✅ API Diff (from stable)
✅ API Diff (from PR only) (no change)
✅ Generator Diff (no change)
🔥 Test run failed 🔥

Test results

1 tests failed, 0 tests skipped, 219 tests passed.

Failed tests

mscorlib/watchOS - simulator/Debug: Failed

mandel-macaque · 2019-03-20T11:07:44Z

The failing issues is known: https://github.com/xamarin/maccore/issues/619

…otnet#5768) This commit improves the state of the BCL testing in the following ways: 1. Improve the device tets running. Less apps, faster results. 2. WatchOS apps are left as they were to ensure that we do not have deplouyment/run issues. We now support the ignore files per assembly name to simplify the tracking of the ignored tests. All

…5768) This commit improves the state of the BCL testing in the following ways: 1. Improve the device tets running. Less apps, faster results. 2. WatchOS apps are left as they were to ensure that we do not have deplouyment/run issues. We now support the ignore files per assembly name to simplify the tracking of the ignored tests. All

…5980) * [tests] Don't run BitcodeNotSupported tests on watchOS. (#5744) Ref: mono/mono#13007 * [tests] Check all architectures when verifying public symbols. (#5745) This also means updating the whitelisted symbols. * [XHarness] Don't run BitcodeNotSupported BCL tests on watchOS. (#5750) * [xharness] Remove bitcode from tvOS apps. (#5781) We don't need the bitcode for testing, it makes test apps bigger (so big that the Apple TV might refuse to install them). * [XHarness] Reduce the number of test applications for the BCL tests. (#5768) This commit improves the state of the BCL testing in the following ways: 1. Improve the device tets running. Less apps, faster results. 2. WatchOS apps are left as they were to ensure that we do not have deplouyment/run issues. We now support the ignore files per assembly name to simplify the tracking of the ignored tests. All * [XHarness] Ensure we do not crash if the framework node is missing. (#5792) * [xharness] Fix BuildOnly logic (#5813) #4884 introduced the logic of only building certain `RunTestTask`. This was meant to disable iOS Extensions as part of a fix to xamarin/maccore#1008. However this didn't quite work and iOS extensions were still running (and failing). The reason being that `BuildOnly` was set to a `RunDeviceTask` that's added to a list which is then given to `CreateTestVariations` which creates new instances of `RunDeviceTask`. We now propagate `BuildOnly` to the new variation instance. * [XHarness] Fix IntermediateOutputPath getting inserted in the wrong place. (#5841) * Ignore tests for devices (#5807) * [XHarness] Ignore some tests that fail on device. Added the ignore, which can be later removed on the new mono 2019-02 since the issues do not happen there. Fixes: xamarin/maccore#1495 * [XHarness] Ignore corlib tets failing on devices. Added the required ignores to get the devices green and workaround mono issue mono/mono#13641 * [XHarness] Ignore a failing test on devices. (#5848) * [XHarness] Update ignore for some failing tests. (#5887) The mono bump added some new tests that fail on device + release configurations. This is a known issue by mono and they'll fix it in the next version but will not be backported to 2018-10. * [XHarness] Ignore monotouch_Microsoft.CSharp_xunit-test.dll until issue 5868 is fixed. (#5869) If not ignored, we will have an issue building the test app on device resulting on a failure. * [d16-1][xharness] Skip fsharp iOS tests until #5886 is merged This can be reverted if #5886 makes it in. * [XHarness] Fix issues building the today extensions for the BCL tests apps. (#5854) * [d16-1][xharness] Ignore failing device tests because mono backports are impossible * [d16-1][xharness] Ignore correct interp tests and more BCL group 1 * [d16-1][xharness] Ignore mscorlib interpreter (debug) test * [tests] Adjust SetupBlockPerfTest to expect at least a 4x speedup instead of 6x We hit this: ``` Expected: greater than 6 But was: 4.9317220348827826d ``` So adjusting to 4 to avoid those random failures. See xamarin/maccore#649 for more details.

mandel-macaque added the run-bcl-tests Run the BCL tests label Mar 14, 2019

VincentDondain approved these changes Mar 14, 2019

View reviewed changes

spouliot reviewed Mar 14, 2019

View reviewed changes

dalexsoto approved these changes Mar 14, 2019

View reviewed changes

rolfbjarne reviewed Mar 15, 2019

View reviewed changes

Group the bcl test applications only on jenkins.

1bcdda4

mandel-macaque force-pushed the group-bcl-tests branch from 65197d7 to 1bcdda4 Compare March 15, 2019 19:06

mandel-macaque added 2 commits March 18, 2019 12:32

Merge branch 'master' into group-bcl-tests

73755de

Use grouped app if on Jenkins OR a special env var is set.

343efd7

Split the mac apps so that the run. Ensure that the grouped apps do n…

bae3160

…ot contain ignored assemblies.

mandel-macaque added 2 commits March 19, 2019 12:41

Move blocking test out of first app.

884aff5

Merge branch 'master' into group-bcl-tests

adce47f

Set all watchos bcl apps not to be grouped due to size and run issues.

63c3cd9

mandel-macaque merged commit 1615b28 into dotnet:master Mar 20, 2019

mandel-macaque deleted the group-bcl-tests branch March 20, 2019 11:10

[XHarness] Reduce the number of test applications for the BCL tests. #5768

[XHarness] Reduce the number of test applications for the BCL tests. #5768

Uh oh!

Conversation

mandel-macaque commented Mar 14, 2019 • edited by rolfbjarne Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mandel-macaque commented Mar 14, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

spouliot left a comment

Choose a reason for hiding this comment

Uh oh!

monojenkins commented Mar 14, 2019

Uh oh!

rolfbjarne commented Mar 15, 2019

Uh oh!

rolfbjarne left a comment

Choose a reason for hiding this comment

Uh oh!

rolfbjarne Mar 15, 2019

Choose a reason for hiding this comment

Uh oh!

monojenkins commented Mar 15, 2019

Uh oh!

rolfbjarne commented Mar 15, 2019

Uh oh!

monojenkins commented Mar 15, 2019

Uh oh!

monojenkins commented Mar 15, 2019

Uh oh!

mandel-macaque commented Mar 15, 2019

Uh oh!

akoeplinger commented Mar 15, 2019

Uh oh!

rolfbjarne commented Mar 15, 2019

Uh oh!

monojenkins commented Mar 15, 2019

Uh oh!

mandel-macaque commented Mar 15, 2019

Uh oh!

monojenkins commented Mar 15, 2019

Uh oh!

akoeplinger commented Mar 15, 2019

Uh oh!

monojenkins commented Mar 16, 2019

Uh oh!

mandel-macaque commented Mar 18, 2019

Uh oh!

monojenkins commented Mar 18, 2019

Test results

Failed tests

Uh oh!

mandel-macaque commented Mar 18, 2019

Uh oh!

monojenkins commented Mar 19, 2019

Test results

Failed tests

Uh oh!

monojenkins commented Mar 19, 2019

Uh oh!

mandel-macaque commented Mar 19, 2019

Uh oh!

monojenkins commented Mar 19, 2019

Test results

Failed tests

Uh oh!

mandel-macaque commented Mar 20, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

mandel-macaque commented Mar 14, 2019 •

edited by rolfbjarne

Loading

mandel-macaque commented Mar 14, 2019 •

edited

Loading