Ignore tests for devices #5807
Merged
Ignore tests for devices #5807
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Added the ignore, which can be later removed on the new mono 2019-02 since the issues do not happen there. Fixes: https://github.com/xamarin/maccore/issues/1495
Added the required ignores to get the devices green and workaround mono issue mono/mono#13641
rolfbjarne
approved these changes
Mar 25, 2019
VincentDondain
approved these changes
Mar 25, 2019
Collaborator
|
Build failure Test results1 tests failed, 0 tests skipped, 218 tests passed.Failed tests
|
dalexsoto
approved these changes
Mar 26, 2019
Contributor
Author
|
Failure is a known issue: https://github.com/xamarin/maccore/issues/581 |
VincentDondain
pushed a commit
to VincentDondain/xamarin-macios
that referenced
this pull request
Apr 24, 2019
* [XHarness] Ignore some tests that fail on device. Added the ignore, which can be later removed on the new mono 2019-02 since the issues do not happen there. Fixes: xamarin/maccore#1495 * [XHarness] Ignore corlib tets failing on devices. Added the required ignores to get the devices green and workaround mono issue mono/mono#13641
VincentDondain
added a commit
that referenced
this pull request
May 9, 2019
…5980) * [tests] Don't run BitcodeNotSupported tests on watchOS. (#5744) Ref: mono/mono#13007 * [tests] Check all architectures when verifying public symbols. (#5745) This also means updating the whitelisted symbols. * [XHarness] Don't run BitcodeNotSupported BCL tests on watchOS. (#5750) * [xharness] Remove bitcode from tvOS apps. (#5781) We don't need the bitcode for testing, it makes test apps bigger (so big that the Apple TV might refuse to install them). * [XHarness] Reduce the number of test applications for the BCL tests. (#5768) This commit improves the state of the BCL testing in the following ways: 1. Improve the device tets running. Less apps, faster results. 2. WatchOS apps are left as they were to ensure that we do not have deplouyment/run issues. We now support the ignore files per assembly name to simplify the tracking of the ignored tests. All * [XHarness] Ensure we do not crash if the framework node is missing. (#5792) * [xharness] Fix BuildOnly logic (#5813) #4884 introduced the logic of only building certain `RunTestTask`. This was meant to disable iOS Extensions as part of a fix to xamarin/maccore#1008. However this didn't quite work and iOS extensions were still running (and failing). The reason being that `BuildOnly` was set to a `RunDeviceTask` that's added to a list which is then given to `CreateTestVariations` which creates new instances of `RunDeviceTask`. We now propagate `BuildOnly` to the new variation instance. * [XHarness] Fix IntermediateOutputPath getting inserted in the wrong place. (#5841) * Ignore tests for devices (#5807) * [XHarness] Ignore some tests that fail on device. Added the ignore, which can be later removed on the new mono 2019-02 since the issues do not happen there. Fixes: xamarin/maccore#1495 * [XHarness] Ignore corlib tets failing on devices. Added the required ignores to get the devices green and workaround mono issue mono/mono#13641 * [XHarness] Ignore a failing test on devices. (#5848) * [XHarness] Update ignore for some failing tests. (#5887) The mono bump added some new tests that fail on device + release configurations. This is a known issue by mono and they'll fix it in the next version but will not be backported to 2018-10. * [XHarness] Ignore monotouch_Microsoft.CSharp_xunit-test.dll until issue 5868 is fixed. (#5869) If not ignored, we will have an issue building the test app on device resulting on a failure. * [d16-1][xharness] Skip fsharp iOS tests until #5886 is merged This can be reverted if #5886 makes it in. * [XHarness] Fix issues building the today extensions for the BCL tests apps. (#5854) * [d16-1][xharness] Ignore failing device tests because mono backports are impossible * [d16-1][xharness] Ignore correct interp tests and more BCL group 1 * [d16-1][xharness] Ignore mscorlib interpreter (debug) test * [tests] Adjust SetupBlockPerfTest to expect at least a 4x speedup instead of 6x We hit this: ``` Expected: greater than 6 But was: 4.9317220348827826d ``` So adjusting to 4 to avoid those random failures. See xamarin/maccore#649 for more details.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There are a number of tests that fail on device. We have an issue for the corlib tests (mono/mono#13641 ) and we do not have one for the icrosoft.CSharp_xunit-test.dll because it does not happen on 2019-02 yet we have issue
mono/mono#13639 that is related with that dll.