Some iOS tests are failing to launch test app

[safern]

I just saw a couple of failures on a PR on iOS, with:

[16:59:08] dbug: 16:59:08.5015570 Xamarin.Hosting: Simulator watchdogs are already disabled for 'iOS 13.5 (17F61) - iPhone X'.
[16:59:08] dbug: 16:59:08.5075460 Xamarin.Hosting: Device 'iOS 13.5 (17F61) - iPhone X' is already booted and the simulator is running.
[16:59:08] dbug: 16:59:08.5182020 Xamarin.Hosting: No need to boot (already booted): iPhone X (iOS 13.5) - created by XHarness
[16:59:08] dbug: 16:59:08.5218370 Xamarin.Hosting: Installing /tmp/helix/working/9F630914/w/A3C008F7/e/Microsoft.Extensions.Configuration.Functional.Tests.app with Bundle Identifier net.dot.Microsoft.Extensions.Configuration.Functional.Tests on 'iOS 13.5 (17F61) - iPhone X'...
[17:28:44] dbug: 17:28:44.9031320 Killing process 14971 as it was cancelled
[17:28:44] dbug: 17:28:44.9088890 Process exited with 137
[17:28:44] dbug: 17:28:44.9095210 Test run failed
[17:28:44] dbug: 17:28:44.9175530 Could not find pid in mtouch output.
[17:28:44] dbug: 17:28:44.9224100 Test run started
[17:28:46] dbug: 17:28:46.0798530 Test run failed to launch
[17:28:46] dbug: 17:28:46.0844090 No crash reports, waiting a second to see if the crash report service just didn't complete in time (4)
[17:28:47] dbug: 17:28:47.2295520 No crash reports, waiting a second to see if the crash report service just didn't complete in time (3)
[17:28:48] dbug: 17:28:48.3149390 No crash reports, waiting a second to see if the crash report service just didn't complete in time (2)
[17:28:49] dbug: 17:28:49.4627860 No crash reports, waiting a second to see if the crash report service just didn't complete in time (1)
[17:28:50] dbug: 17:28:50.5966390 No crash reports, waiting a second to see if the crash report service just didn't complete in time (0)
[17:28:51] fail: Failed to launch the application:
                 Launch failure
                 
                 Check logs for more information.
XHarness exit code: 83
+ exit_code=83

#45172
https://dev.azure.com/dnceng/public/_build/results?buildId=898906&view=results
https://helixre107v0xdeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-pull-45172-merge-f3086f428107446aa7/Microsoft.Extensions.Configuration.Functional.Tests/console.c96502f5.log?sv=2019-07-07&se=2020-12-15T00%3A55%3A49Z&sr=c&sp=rl&sig=i1uWKdwS6iGhcUW0XoWG5uWfxxQF%2FwXqK6V6beKh6YM%3D

cc: @premun @dotnet/runtime-infrastructure

[Dotnet-GitSync-Bot]

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

Tagging subscribers to this area: @ViktorHofer
See info in area-owners.md if you want to be subscribed.

Issue Details

---

I just saw a couple of failures on a PR on iOS, with:

[16:59:08] dbug: 16:59:08.5015570 Xamarin.Hosting: Simulator watchdogs are already disabled for 'iOS 13.5 (17F61) - iPhone X'.
[16:59:08] dbug: 16:59:08.5075460 Xamarin.Hosting: Device 'iOS 13.5 (17F61) - iPhone X' is already booted and the simulator is running.
[16:59:08] dbug: 16:59:08.5182020 Xamarin.Hosting: No need to boot (already booted): iPhone X (iOS 13.5) - created by XHarness
[16:59:08] dbug: 16:59:08.5218370 Xamarin.Hosting: Installing /tmp/helix/working/9F630914/w/A3C008F7/e/Microsoft.Extensions.Configuration.Functional.Tests.app with Bundle Identifier net.dot.Microsoft.Extensions.Configuration.Functional.Tests on 'iOS 13.5 (17F61) - iPhone X'...
[17:28:44] dbug: 17:28:44.9031320 Killing process 14971 as it was cancelled
[17:28:44] dbug: 17:28:44.9088890 Process exited with 137
[17:28:44] dbug: 17:28:44.9095210 Test run failed
[17:28:44] dbug: 17:28:44.9175530 Could not find pid in mtouch output.
[17:28:44] dbug: 17:28:44.9224100 Test run started
[17:28:46] dbug: 17:28:46.0798530 Test run failed to launch
[17:28:46] dbug: 17:28:46.0844090 No crash reports, waiting a second to see if the crash report service just didn't complete in time (4)
[17:28:47] dbug: 17:28:47.2295520 No crash reports, waiting a second to see if the crash report service just didn't complete in time (3)
[17:28:48] dbug: 17:28:48.3149390 No crash reports, waiting a second to see if the crash report service just didn't complete in time (2)
[17:28:49] dbug: 17:28:49.4627860 No crash reports, waiting a second to see if the crash report service just didn't complete in time (1)
[17:28:50] dbug: 17:28:50.5966390 No crash reports, waiting a second to see if the crash report service just didn't complete in time (0)
[17:28:51] fail: Failed to launch the application:
                 Launch failure
                 
                 Check logs for more information.
XHarness exit code: 83
+ exit_code=83

#45172
https://dev.azure.com/dnceng/public/_build/results?buildId=898906&view=results
https://helixre107v0xdeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-pull-45172-merge-f3086f428107446aa7/Microsoft.Extensions.Configuration.Functional.Tests/console.c96502f5.log?sv=2019-07-07&se=2020-12-15T00%3A55%3A49Z&sr=c&sp=rl&sig=i1uWKdwS6iGhcUW0XoWG5uWfxxQF%2FwXqK6V6beKh6YM%3D

cc: @premun @dotnet/runtime-infrastructure

Author:	safern
Assignees:	-
Labels:	area-Infrastructure, blocking-clean-ci, os-ios, untriaged
Milestone:	-

[premun]

Having a look. This happened before when the MacOS sometimes slows down and I was waiting for this to prepare a final fix.

Usually it's tied to a specific machine, so I will take them offline to unblock runtime and prepare the fix.

[premun]

@safern I took these machines offline:

• dci-mac-build-133
• dci-mac-build-136
• dci-mac-build-146

The rest looked healthy from a quick Kusto search.
I will add work item retry + machine reboot for these cases and that should resolve this for good.

[radical]

Another instance of this, IIUC, on #45839:

https://dnceng.visualstudio.com/public/_build/results?buildId=914235&view=logs&j=8580ecfb-912a-5dbd-35ce-e64d0d51ddbf&t=ebd4788e-80c8-5104-29f3-a1792da138c8

console: https://helixre107v0xdeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-pull-45839-merge-13b48d1686ba4343b7/Microsoft.Bcl.AsyncInterfaces.Tests/console.fa2f6cc5.log?sv=2019-07-07&se=2020-12-29T16%3A46%3A22Z&sr=c&sp=rl&sig=tn4cgwt0brU45SKlzBlHEbsrbDNIhpZNsqrQkidt7Ss%3D

This is for Microsoft.Bcl.AsyncInterfaces.Tests. Also, I see an extra message before the process gets killed (2nd line):

[09:16:32] dbug: 09:16:32.4000240 Xamarin.Hosting: Installed 'net.dot.Microsoft.Bcl.AsyncInterfaces.Tests' from /tmp/helix/working/A1B108B5/w/B28E0A07/e/Microsoft.Bcl.AsyncInterfaces.Tests.app
[09:16:39] dbug: 09:16:39.9175400 Xamarin.Hosting: Can't process potentially bad load commands, because the app is signed.
[09:16:52] dbug: 09:16:52.6508610 Killing process 943 as it was cancelled

[radical]

And two more library test runs fail with the same issue, on the PR #45839 from the last comment:
System.Data.Odbc.Tests, and System.Runtime.Serialization.Json.ReflectionOnly.Tests.

[steveisok]

dci-mac-build-133, dci-mac-build-145, and dci-mac-build-068.local are the 3 machines that have any badexits in the last 24 hours.

[premun]

We have a fix in Arcade that was supposed to retry the test on another machine in this case and reboot the machine.
However, the fix seems to have failed:

+ /usr/local/bin/python3 -c 'from helix.workitemutil import request_infra_retry; request_infra_retry('\''Retrying because iOS Simulator application install hung'\'')'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'helix'
+ /usr/local/bin/python3 -c 'from helix.workitemutil import request_reboot; request_reboot('\''Rebooting because iOS Simulator application install hung '\'')'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'helix'

I disabled the machines and will have a look at the retry