[WIP][DDFun Pipeline] Test test test #8171
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Collaborator
|
❌ Build was (probably) aborted 🔥 Jenkins job (on internal Jenkins) failed in stage(s) 'Running XM tests on '10.10'' 🔥 : hudson.AbortException: Xamarin.Mac tests on macOS 10.10 failed (xammac_tests) ✅ Build succeeded ✅ API Diff (from stable) |
Collaborator
|
Build success |
Collaborator
|
❌ Build was (probably) aborted 🔥 Jenkins job (on internal Jenkins) failed in stage(s) 'Running XM tests on '10.10', Running XM tests on '10.12', Running XM tests on '10.11'' 🔥 : hudson.AbortException: Xamarin.Mac tests on macOS 10.10 failed (xammac_tests) ✅ Build succeeded ✅ API Diff (from stable) |
Collaborator
|
Build success |
Collaborator
|
❌ Build was (probably) aborted 🔥 Jenkins job (on internal Jenkins) failed in stage(s) 'Running XM tests on '10.10', Running XM tests on '10.11'' 🔥 : hudson.AbortException: Xamarin.Mac tests on macOS 10.10 failed (xammac_tests) ✅ Build succeeded ✅ API Diff (from stable) |
Collaborator
|
Build success |
Collaborator
|
❌ Build was (probably) aborted 🔥 Jenkins job (on internal Jenkins) failed in stage(s) 'Running XM tests on '10.10'' 🔥 : hudson.AbortException: Xamarin.Mac tests on macOS 10.10 failed (xammac_tests) ✅ Build succeeded ✅ API Diff (from stable) |
Collaborator
|
Build success |
Collaborator
|
❌ Build was (probably) aborted 🔥 Jenkins job (on internal Jenkins) failed in stage(s) 'Running XM tests on '10.10'' 🔥 : hudson.AbortException: Xamarin.Mac tests on macOS 10.10 failed (xammac_tests) ✅ Build succeeded ✅ API Diff (from stable) |
Collaborator
|
Build success |
Collaborator
|
❌ Build was (probably) aborted 🔥 Jenkins job (on internal Jenkins) failed in stage(s) 'Running XM tests on '10.10'' 🔥 : hudson.AbortException: Xamarin.Mac tests on macOS 10.10 failed (xammac_tests) ✅ Build succeeded ✅ API Diff (from stable) |
Collaborator
|
Build success |
Collaborator
|
❌ Build was (probably) aborted 🔥 Jenkins job (on internal Jenkins) failed in stage(s) 'Running XM tests on '10.10'' 🔥 : hudson.AbortException: Xamarin.Mac tests on macOS 10.10 failed (xammac_tests) ✅ Build succeeded ✅ API Diff (from stable) |
Collaborator
|
Build success |
Collaborator
|
Build success |
Collaborator
|
❌ Build was (probably) aborted 🔥 Jenkins job (on internal Jenkins) failed in stage(s) 'Checkout' 🔥 : hudson.AbortException: There is already a newer build in progress (#14) |
Collaborator
|
❌ Build was (probably) aborted 🔥 Jenkins job (on internal Jenkins) failed in stage(s) 'Build' 🔥 : hudson.AbortException: script returned exit code 2 🔥 Build failed 🔥 |
whitneyschmidt
added a commit
that referenced
this pull request
May 15, 2020
commit 1a6c4bf Author: Whitney Schmidt <whschm@microsoft.com> Date: Thu May 14 15:41:56 2020 -0400 fix syntax... commit 246b5c5 Author: Whitney Schmidt <whschm@microsoft.com> Date: Thu May 14 15:35:02 2020 -0400 fix provisioning status is always set to failure... commit 603930d Author: Whitney Schmidt <whschm@microsoft.com> Date: Thu May 14 15:32:21 2020 -0400 make context for per-test failure _actually_ unique commit b5975f9 Author: Whitney Schmidt <whschm@microsoft.com> Date: Thu May 14 15:14:06 2020 -0400 set TestState to failure before we try to run tests, add partial provisioning improvements commit 08eea78 Author: Whitney Schmidt <whschm@microsoft.com> Date: Thu May 14 11:53:15 2020 -0400 re-add back tests, devices commit 7ec48a8 Author: Whitney Schmidt <whschm@microsoft.com> Date: Wed May 13 17:33:25 2020 -0400 clean up publish_failure + add try/catch commit 7169485 Author: Whitney Schmidt <whschm@microsoft.com> Date: Wed May 13 17:12:25 2020 -0400 clean up + add try/catch to publish_results commit a01d5ff Author: Whitney Schmidt <whschm@microsoft.com> Date: Tue May 12 19:02:38 2020 -0400 add bolded text, clean up a bit commit cad4470 Author: Whitney Schmidt <whschm@microsoft.com> Date: Tue May 12 18:15:52 2020 -0400 fix syntax commit b460224 Author: Whitney Schmidt <whschm@microsoft.com> Date: Tue May 12 18:06:12 2020 -0400 handle error in run-tests correctly commit e665eb6 Author: Whitney Schmidt <whschm@microsoft.com> Date: Tue May 12 17:14:36 2020 -0400 try out not passing -e to script, ssome cleanup commit 855a9f4 Author: Whitney Schmidt <whschm@microsoft.com> Date: Tue May 12 16:56:51 2020 -0400 swap description with context since context is unique... commit 4207bee Author: Whitney Schmidt <whschm@microsoft.com> Date: Tue May 12 16:35:37 2020 -0400 clean up some stuff, update status with VSTS prefix + add real description and context to aggregate status commit 0512e59 Author: Whitney Schmidt <whschm@microsoft.com> Date: Tue May 12 16:33:08 2020 -0400 temporary testing change to always run publish_failure commit 86613a8 Author: Whitney Schmidt <whschm@microsoft.com> Date: Tue May 12 16:14:21 2020 -0400 give a better fake exit code commit bbf6ad1 Author: Whitney Schmidt <whschm@microsoft.com> Date: Tue May 12 15:52:27 2020 -0400 test out w/ test failures commit 1568e7f Author: Whitney Schmidt <whschm@microsoft.com> Date: Tue May 12 15:35:34 2020 -0400 add succeededorfailed(), update TestState to failed when no test summary file is found commit 6a29feb Author: Whitney Schmidt <whschm@microsoft.com> Date: Tue May 12 14:27:08 2020 -0400 fix condition on tests failure commit be94e20 Author: Whitney Schmidt <whschm@microsoft.com> Date: Mon May 11 18:49:23 2020 -0400 test with device tests passing commit 235c764 Author: Whitney Schmidt <whschm@microsoft.com> Date: Mon May 11 17:19:30 2020 -0400 fix syntax error commit 4b21d92 Author: Whitney Schmidt <whschm@microsoft.com> Date: Mon May 11 17:13:37 2020 -0400 clean up publish_failure, add vso var to device-tests to check for test failure commit 150dfb8 Author: Whitney Schmidt <whschm@microsoft.com> Date: Mon May 11 16:42:13 2020 -0400 tweak formatting commit 1c33516 Author: Whitney Schmidt <whschm@microsoft.com> Date: Mon May 11 16:03:11 2020 -0400 fix file.fullname commit 1156b5f Author: Whitney Schmidt <whschm@microsoft.com> Date: Mon May 11 15:16:39 2020 -0400 fix length? commit 8556e2f Author: Whitney Schmidt <whschm@microsoft.com> Date: Mon May 11 14:36:54 2020 -0400 grab fullname then substring commit 81e3cf7 Author: Whitney Schmidt <whschm@microsoft.com> Date: Mon May 11 13:55:16 2020 -0400 try adding device + test info to aggregate results commit 5abd35f Author: Whitney Schmidt <whschm@microsoft.com> Date: Mon May 11 13:22:43 2020 -0400 fix file path?? commit 2ad64eb Author: Whitney Schmidt <whschm@microsoft.com> Date: Fri May 8 20:45:13 2020 -0400 try out xamarin-macios/tests commit cb2bae4 Author: Whitney Schmidt <whschm@microsoft.com> Date: Fri May 8 20:35:20 2020 -0400 try accessing xamarin-macios rather than upload location because there seems to be a delay with access to the upload commit c43f8d4 Author: Whitney Schmidt <whschm@microsoft.com> Date: Fri May 8 20:20:19 2020 -0400 fix file path for per-test message again... commit 11b82c0 Author: Whitney Schmidt <whschm@microsoft.com> Date: Fri May 8 20:12:41 2020 -0400 actually fix file path for per-device tests? commit ebde63f Author: Whitney Schmidt <whschm@microsoft.com> Date: Fri May 8 20:02:34 2020 -0400 try fixing per-device pwsh env var for summaries commit e8c3541 Author: Whitney Schmidt <whschm@microsoft.com> Date: Fri May 8 19:54:17 2020 -0400 try out more debugging messages for per-test failure commit 281f709 Author: Whitney Schmidt <whschm@microsoft.com> Date: Fri May 8 19:38:31 2020 -0400 fix leftover pwsh commands causing failure commit b642e4a Author: Whitney Schmidt <whschm@microsoft.com> Date: Fri May 8 19:19:32 2020 -0400 remote aggregate test summary bash task, try out publishing correctly formatted test results for per-test failure messages commit 951dbd0 Author: Whitney Schmidt <whschm@microsoft.com> Date: Wed May 6 15:49:42 2020 -0400 try removing recursive call by adding more wildcards to dir path commit 9363b88 Author: Whitney Schmidt <whschm@microsoft.com> Date: Wed May 6 15:24:47 2020 -0400 recursive call to get-childitem since we have Summaries/blah-testname/testsummary.md commit cca7f2d Author: Whitney Schmidt <whschm@microsoft.com> Date: Wed May 6 14:56:05 2020 -0400 actually fix dir path?? commit 03f8548 Author: Whitney Schmidt <whschm@microsoft.com> Date: Wed May 6 13:22:06 2020 -0400 fix env var for dir path, add debugging print commit ae0bdbc Author: Whitney Schmidt <whschm@microsoft.com> Date: Wed May 6 12:54:34 2020 -0400 try out double quotes around commit 4d52b29 Author: Whitney Schmidt <whschm@microsoft.com> Date: Tue May 5 19:00:55 2020 -0400 fix path... commit 7e6bb67 Author: Whitney Schmidt <whschm@microsoft.com> Date: Tue May 5 18:41:38 2020 -0400 fix file path for summaries... commit 4da25bb Author: Whitney Schmidt <whschm@microsoft.com> Date: Tue May 5 18:20:29 2020 -0400 fix file path for testsummaries commit 942eb1a Author: Whitney Schmidt <whschm@microsoft.com> Date: Tue May 5 18:08:18 2020 -0400 try out with all testsummary aggregation logic inside pwsh script commit 591c5f1 Author: Whitney Schmidt <whschm@microsoft.com> Date: Mon May 4 11:56:44 2020 -0400 try out quotation marks commit 257d61e Author: Whitney Schmidt <whschm@microsoft.com> Date: Fri May 1 17:47:45 2020 -0400 remove quotation marks from json body text commit c5c63b8 Author: Whitney Schmidt <whschm@microsoft.com> Date: Fri May 1 17:32:45 2020 -0400 try converting to json text before adding as text body to post commit 548a9db Author: Whitney Schmidt <whschm@microsoft.com> Date: Fri May 1 17:05:29 2020 -0400 try making not .md file commit 99f8ec3 Author: Whitney Schmidt <whschm@microsoft.com> Date: Fri May 1 16:22:52 2020 -0400 try out formatting python commit 8bc52cb Author: Whitney Schmidt <whschm@microsoft.com> Date: Fri May 1 16:05:17 2020 -0400 first try at preserving formatting in .md commit a7b1623 Author: Whitney Schmidt <whschm@microsoft.com> Date: Fri May 1 14:50:31 2020 -0400 fix extra spaces in bash script commit 19a8421 Author: Whitney Schmidt <whschm@microsoft.com> Date: Fri May 1 14:30:29 2020 -0400 fix bash for loop commit e1b5006 Author: Whitney Schmidt <whschm@microsoft.com> Date: Fri May 1 14:15:48 2020 -0400 try fixing bash syntax error commit 2d38eeb Author: Whitney Schmidt <whschm@microsoft.com> Date: Thu Apr 30 19:39:08 2020 -0400 fix syntax error for prefix commit 3c83f30 Author: Whitney Schmidt <whschm@microsoft.com> Date: Thu Apr 30 18:08:24 2020 -0400 add string manipulation commit 6bfec0f Author: Whitney Schmidt <whschm@microsoft.com> Date: Thu Apr 30 16:51:45 2020 -0400 wildcard testsummary in publish-results commit b6a57a6 Author: Whitney Schmidt <whschm@microsoft.com> Date: Thu Apr 30 16:30:13 2020 -0400 create + grab real test summary from correct location commit 210ff21 Author: Whitney Schmidt <whschm@microsoft.com> Date: Wed Apr 29 17:35:53 2020 -0400 add test summary file name to aggregate commit ead00b7 Author: Whitney Schmidt <whschm@microsoft.com> Date: Tue Apr 28 16:09:53 2020 -0400 fix dir path for aggregate commit fb4e08b Author: Whitney Schmidt <whschm@microsoft.com> Date: Tue Apr 28 15:09:56 2020 -0400 debug testsummary contents commit cc2ca82 Author: Whitney Schmidt <whschm@microsoft.com> Date: Tue Apr 28 14:43:06 2020 -0400 debug summary location commit 629d944 Author: Whitney Schmidt <whschm@microsoft.com> Date: Tue Apr 28 14:13:53 2020 -0400 fix aggregate step commit 75a18d7 Author: Whitney Schmidt <whschm@microsoft.com> Date: Tue Apr 28 13:50:57 2020 -0400 fix again... commit ad36208 Author: Whitney Schmidt <whschm@microsoft.com> Date: Tue Apr 28 13:49:08 2020 -0400 fix succeeededorfailed conditions commit d949474 Author: Whitney Schmidt <whschm@microsoft.com> Date: Tue Apr 28 13:45:57 2020 -0400 adjust final commit message for aggregate test file commit ee19b9c Author: Whitney Schmidt <whschm@microsoft.com> Date: Tue Apr 28 13:35:10 2020 -0400 remove provisioning success condition, first try (wrong) at publishing test summary in final commit message commit b46df75 Author: Whitney Schmidt <whschm@microsoft.com> Date: Mon Apr 27 18:39:47 2020 -0400 add step for test summary if provisioning fails, fix summary path in publish_failure commit c2b3aac Author: Whitney Schmidt <whschm@microsoft.com> Date: Mon Apr 27 17:48:13 2020 -0400 skip test execution, instead immediately exit w/ failure code commit 55a52a5 Author: Whitney Schmidt <whschm@microsoft.com> Date: Mon Apr 27 16:42:14 2020 -0400 fix *indentation* that was causing pwsh scripts not to run on failure commit 2dfad31 Author: Whitney Schmidt <whschm@microsoft.com> Date: Mon Apr 27 15:54:25 2020 -0400 change condition for yaml so that everything runs regardless of success/failure commit e73c194 Author: Whitney Schmidt <whschm@microsoft.com> Date: Mon Apr 27 15:17:25 2020 -0400 first try at adding test summary to per-test failure comment, moved when test summaries get published to vsts as artifacts commit 07b6432 Author: Whitney Schmidt <whschm@microsoft.com> Date: Fri Apr 24 17:53:33 2020 -0400 fix accessing testsummary.md commit 5b29a27 Author: Whitney Schmidt <whschm@microsoft.com> Date: Fri Apr 24 15:08:24 2020 -0400 re-add debugging code to print out env vars commit 80b8418 Author: Whitney Schmidt <whschm@microsoft.com> Date: Fri Apr 24 15:00:08 2020 -0400 update with CORRECT env var commit c2885a2 Author: Whitney Schmidt <whschm@microsoft.com> Date: Thu Apr 23 17:55:43 2020 -0400 remove set-location from publish_results commit 64f5c82 Author: Whitney Schmidt <whschm@microsoft.com> Date: Thu Apr 23 17:09:23 2020 -0400 Revert "change device pool" This reverts commit 52d51fb. commit ef6164d Author: Whitney Schmidt <whschm@microsoft.com> Date: Thu Apr 23 17:09:03 2020 -0400 Revert "change capabilities for xamarin-ios device pool" This reverts commit 469b57c. commit d2732d6 Author: Whitney Schmidt <whschm@microsoft.com> Date: Thu Apr 23 13:54:15 2020 -0400 remove buggy pwsh line from publish_results, uncomment real tests in run-tests.sh commit 469b57c Author: Whitney Schmidt <whschm@microsoft.com> Date: Wed Apr 22 20:09:57 2020 -0400 change capabilities for xamarin-ios device pool commit 52d51fb Author: Whitney Schmidt <whschm@microsoft.com> Date: Wed Apr 22 20:02:14 2020 -0400 change device pool commit 57bae65 Author: Whitney Schmidt <whschm@microsoft.com> Date: Wed Apr 22 19:37:54 2020 -0400 try out grabbing test summary files location commit d5fc0de Author: Whitney Schmidt <51677938+whitneyschmidt@users.noreply.github.com> Date: Wed Apr 8 14:14:08 2020 -0400 Delete publish-results-jenkins.ps1 commit 3c99706 Author: Whitney Schmidt <51677938+whitneyschmidt@users.noreply.github.com> Date: Wed Apr 8 14:10:29 2020 -0400 Delete foo.sh commit c529e93 Author: Whitney Schmidt <51677938+whitneyschmidt@users.noreply.github.com> Date: Wed Apr 8 14:01:44 2020 -0400 [WIP][DDFun Pipeline] Test test test (#8171) * comment out most of templates/device-tests.yml * fix yaml formatting * remove condition from publish path * try modifying path to TestSummary.md * add .md * try out merge pwd * try out searching one dir up for summary .mds, cat merged summary * fix dir path * narrow down .md files grabbed * try out publishing + downloading merged file * try out some pwsh * fix yaml for testing pwsh * try out messy pwsh * try pointing at right script location * try out fuller path for pwsh script * try out fuller path for pwsh script * fix hash literal * fix pwsh bug in * try out Bearer * change pwsh to PUT, try and print to check validity * try out new evaluation of token * try print auth * try print everything * fix variables? * remove jobs from matrix * change some env var stuff * more env * try out new target_url * try GitHub.Token * add testing for Gh token * try pwd for ps1 * try changing location to pwsh path * try out ps1 from jenkins dir * print out env vars * fix target_url? * fix for string concat * fix GH url? * fix GH token? * unquote Authorization * add user agent? * Authorization -> AUthentication * try adding quotes to json * print out bash json, remove hash from pwsh json payload * fix underscore url * change put to post * try out full restmethod syntax * manually enter gh token * remove extra code that changes location * re-add real token * try using @params * tools/devops/templates/publish-results.yml * add back env vars * remove some env variables * env vars are actually needed... * try adding context? * uncomment a bunch of stuff in device-tests.yml * move comment? * try out dir path * ls some more * try out fix for run-tests.sh * try setting variable for gh_status * replace vsts device tests with dummy test * add own sh script * try setting job vars in script * try out setting var in a bash task * try out setting global var directly * fix failure * add GH status for failures, set tests to always fail: * add publish_failure.ps1 * fix dir path for script * skip iOS device test stage * re-add stages keyword * remove broken var stuff * comment out dependencie on iOS * try adding some more env vars to pwsh script * add env vars to json payload for status, run more than one test suite * switch from tvos to ios to avoid agent issues * fix broken stage * fix broken syntax * update dependencies to ios * whoops * print out commit comment json payload * fix error in publish_failure * try switching to tvos to see whether queue is shorter * whoops, try again for tvos * initial try at adding commit message in publish_failure.ps1 * fix json payload error * fix formatting and emojis * fix broken escape characters in json payload * tweak context + add AGGREGATE to status for final status, try out getting status * use api.github.com * fix combined status url * try setting status by querying GH for combined status * fix the way that we get json object status * try out accessing response like pwsh object * Get-Location in pwsh scripts * uncomment real test runs, try accessing testsummary.md for aggregate result comment * revert stuff commit deb53eb Merge: 9d650c3 2b733ae Author: Manuel de la Pena <mandel@microsoft.com> Date: Wed Feb 26 10:08:01 2020 -0500 Merge branch 'master' into matrix-testing commit 9d650c3 Merge: 60627cc 09c4c80 Author: Manuel de la Pena <mandel@microsoft.com> Date: Fri Feb 21 14:04:32 2020 -0500 Merge branch 'master' into matrix-testing commit 60627cc Merge: 047aed6 c35389f Author: Manuel de la Pena <mandel@microsoft.com> Date: Fri Feb 21 14:03:07 2020 -0500 Merge branch 'matrix-testing' of github.com:xamarin/xamarin-macios into matrix-testing commit c35389f Merge: 6c78d46 9fd1c2d Author: Manuel de la Pena <mandel@microsoft.com> Date: Fri Feb 21 06:03:31 2020 -0500 Merge branch 'add-start-time' into matrix-testing commit 6c78d46 Merge: d250012 c4978a9 Author: Manuel de la Pena <mandel@microsoft.com> Date: Fri Feb 21 06:03:23 2020 -0500 Merge branch 'single-variation' into matrix-testing commit 9fd1c2d Author: Manuel de la Pena <mandel@microsoft.com> Date: Fri Feb 21 05:55:55 2020 -0500 [Harness] Add start-time to work around a bug in the publishing tool. The publishing tool is a little fragile. In run https://devdiv.visualstudio.com/DevDiv/_build/results?buildId=3490479&view=logs&j=67d14776-f827-5fe4-2625-2db4b5987fd1&t=fa262eec-9d97-5ba4-b4cc-a9292beecd8f I noticed that valid test runs with a failing test (launch issues) were not being uploaded. I found out that the reason is a flaw in the logic on the parser of the publishing tool. The tool assumes, that if there is no start-time, there are no test results (do remember that NUnitV3 is schemaless we don't know exactly what attrs are compulsory). The culprint is line: https://dev.azure.com/mseng/AzureDevOps/_git/AzureDevOps?path=%2FTa%2FTasks%2FPublishTestResults%2FParser%2FNUnitResultParser.cs&version=GBmaster&line=473&lineEnd=473&lineStartColumn=63&lineEndColumn=64&lineStyle=plain Basically: ```csharp if (testRunNode?.Attributes?["start-time"] != null) { // import test data } // do nothing interesting since there is no data ``` This commit fixes it by setting the start time as the current one, we dont care since it is a failure xml result. commit c4978a9 Author: Manuel de la Pena <mandel@microsoft.com> Date: Fri Feb 21 05:28:00 2020 -0500 [Harness] Do not add the variation two times to failure messages. AppName should only be the app name since we are passing the varation as a parameter too. Else we end up with $"{appname} {variation} {variation}". commit d250012 Merge: 8e3363c 13b402f Author: Manuel de la Pena <mandel@microsoft.com> Date: Wed Feb 19 22:51:32 2020 -0500 Merge branch 'name-device-installation' into matrix-testing commit 13b402f Author: Manuel de la Pena <mandel@microsoft.com> Date: Wed Feb 19 22:48:52 2020 -0500 [Harness] Provide device name in those errors where is device specific. In order to simplify the monitoring job add the device name to the following failures: * Installation * Launch * Tcp Connection All the above are most of the time due to a misconfigured device. The device name is useful information for the monitoring person to be able to reach IT and address the issue. commit 8e3363c Merge: 87ee051 0fb6d89 Author: Manuel de la Pena <mandel@microsoft.com> Date: Wed Feb 19 22:38:31 2020 -0500 Merge branch 'add-missing-node' into matrix-testing commit 0fb6d89 Author: Manuel de la Pena <mandel@microsoft.com> Date: Wed Feb 19 22:17:13 2020 -0500 [Harness] Close message element correctly in NUnit3. Xml should be: ``` <failure> <message>Foo</message> <stack-trace>Bar</stack-trace> </failure> ``` But we generate: ``` <failure> <message>Foo <stack-trace>Bar</stack-trace> </message> </failure> ``` Makes the parsing of the failures impossible. commit 87ee051 Merge: 36149a8 44bd2c8 Author: Manuel de la Pena <mandel@microsoft.com> Date: Wed Feb 19 21:11:19 2020 -0500 Merge branch 'master' into matrix-testing commit 047aed6 Author: Manuel de la Pena <mandel@microsoft.com> Date: Wed Feb 19 09:18:07 2020 -0500 Remove CWL, everything worked. commit 36149a8 Author: Manuel de la Pena <mandel@microsoft.com> Date: Wed Feb 19 00:16:42 2020 -0500 Poor mans debugger. commit 09c0d56 Author: Manuel de la Pena <mandel@microsoft.com> Date: Wed Feb 19 00:14:41 2020 -0500 Revert "Do not autoconfig xharness, we 'know' better." This reverts commit 804f242. commit 318f597 Author: Manuel de la Pena <mandel@microsoft.com> Date: Wed Feb 19 00:05:34 2020 -0500 Also add harness logs. commit 804f242 Author: Manuel de la Pena <mandel@microsoft.com> Date: Wed Feb 19 00:05:20 2020 -0500 Do not autoconfig xharness, we 'know' better. commit 101a980 Author: Manuel de la Pena <mandel@microsoft.com> Date: Tue Feb 18 09:15:18 2020 -0500 [VSTS] Correct order of params for make. (#7927) commit cb84dfd Author: Manuel de la Pena <mandel@microsoft.com> Date: Tue Feb 18 15:29:35 2020 -0500 Fix logic for test selection using labels. commit 2ef35c8 Merge: e63a00c c3d847d Author: Manuel de la Pena <mandel@microsoft.com> Date: Mon Feb 17 23:38:33 2020 -0500 Merge branch 'create-crash-xml-tcp' into matrix-testing commit c3d847d Author: Manuel de la Pena <mandel@microsoft.com> Date: Mon Feb 17 23:32:35 2020 -0500 [Harness] Create an xml crash file when we do not have a result file. On devices that cannot reach the host via TCP we do not have a log, this means that in the if statement needs to have a case for it. The main problem is that when the device cannot connect to the host, we do not get a log OR a crash reason from the crash logs. It makes sense not to have a crash reason, because the app did not crash. In these sitations, we have to create a xml crash report (since we really do not know if we can parse the file) that will tell vsts that there was an issue. Adding the main log will let the monitoring person see the results of the test run. commit e63a00c Merge: 1495d64 4937a5d Author: Manuel de la Pena <mandel@microsoft.com> Date: Mon Feb 17 18:44:20 2020 -0500 Merge branch 'master' into matrix-testing commit 1495d64 Author: Manuel de la Pena <mandel@microsoft.com> Date: Sat Feb 15 01:15:43 2020 -0500 [Devops] Move to a 'matrix' pipeline to improve the device tests feedback Our bots spend most of the time compiling and deploying rather than executing tests. This means that a bot with an iOS device is most of the time not really running tests, but deploying them. With the old pipeline the average runtime to get feedback for all the device tests is around 6 to 7 hours. In order to improve the speed in which we get results the new pipeline spawns 5 different testing jobs and a last 6th job that is a fan in to report the completion to the monitoring person. The tests have been divided in the following way: 1. Xamarin tests, wich are all those tests that are not bcl tests or monotouch. 2. Monotouch tests, since it is a large set of tests. 3. All xUnit based BCL tests. 4. All NUnit based BCL tests. 5. mscorlib tests (all 3 splitted assemblies). In average the jobs individually take the following times: 1. Xamarin tests: 1h 30m 2. Monotouch tests: 45/50m 3. xUnit based BCL: 1h 30m 4. NUnit based BCL: 45 m 5. mscorlib 1h 30m In a perfect world with infinite bots, we can get device test results in under 2 hours, but in reality we might be getting around the 3 hours, which is half of the time atm. In order to make things easier for the future different templates in yml have been created under tools/devops/templates - tools/devops/templates/device-tests.yml Basic template that contains all the required steps needed to run the tests on device. This template takes two env vars. One which will contains the extra labels to pass to xharness and the context to be used to set the device tests status. - tools/devops/templates/job-matrix.yml Main template that allows to create pipelines with all the 5 jobs targeting different devices. The template takes a numer of parameters to set the targetd device and the capabilities of the agents. - tools/devops/templates/publish-results.yml Small helper template for posting messages. Once the 5 jobs are completed, a 6th job is kicked that does not need an iOS/tvOS device (we have more agents without capabilities). The jobs grabs the artifacts from the previous jos, unzipts them and performs the publication of the test results to vsts. Once completed it reports the result in a status to github. This job will execute no matter what were the results of the previous jobs (notice the always() condition). commit 0f53f01 Author: Manuel de la Pena <mandel@microsoft.com> Date: Sat Feb 15 00:43:40 2020 -0500 [Harness] Use fullname for test-case in the xunit xslt. After reading the code of the parser used in VSTS I found out that name is not used, but fullname. That attribute is the one used in their xml parser to set the 'testCaseResultData.AutomatedTestName'.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.