[6.0] AppleTV CI failures: Failure to launch simulator OR failure to authenticate request

[carlossanlop]

• Example 6.0 backport PR: [release/6.0] [mono][debugger] Fix Debugger.Break() behavior when running on release mode #80615
• Queue: Build tvOSSimulator x64 Release AllSubsets_Mono
• Job results: https://dev.azure.com/dnceng-public/public/_build/results?buildId=136740&view=logs&j=e4e455c5-b032-5051-d2c5-a64982c9ac45&t=0263a457-10d3-5b56-3414-171b09292e09

The tests in tvOS fail with warnings due to one of two possible problems, and can affect almost any test:

A)
Log file: https://helixre107v0xdeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-pull-80615-merge-be0a3fea1fd0435e85/System.Globalization.Calendars.Tests/1/console.47352c66.log?helixlogtype=result

Could not launch the simulator application (expand)

[11:28:03] dbug: Xamarin.Hosting: Simulator watchdogs are already disabled for 'tvOS 15.0 (19J344) - Apple TV'.
[11:28:03] dbug: Xamarin.Hosting: Launching simulator application 'com.apple.iphonesimulator'
[11:28:03] dbug: error HE0018: Could not launch the simulator application: The application “Simulator” could not be launched because a miscellaneous error occurred.
[11:28:03] dbug: 
[11:28:03] dbug: at Xamarin.Hosting.SimulatorApplication.LaunchAsync (Xamarin.Hosting.Bindings.SimDevice device) [0x0020d] in /Users/builder/azdo/_work/1/s/maccore/tools/mlaunch/Xamarin.Hosting/Xamarin.Hosting/SimulatorApplication.cs:203
[11:28:03] dbug: at Xamarin.Hosting.Services.LaunchSimulatorAsync (Xamarin.Hosting.Bindings.SimDevice device) [0x000a7] in /Users/builder/azdo/_work/1/s/maccore/tools/mlaunch/Xamarin.Hosting/Xamarin.Hosting/Services.cs:1390
[11:28:03] dbug: at Xamarin.Launcher.Driver.LaunchSimulatorAsync () [0x00144] in /Users/builder/azdo/_work/1/s/maccore/tools/mlaunch/Xamarin.Hosting/Xamarin.Launcher/Main.cs:781
[11:28:03] dbug: at Xamarin.Utils.NSRunLoopExtensions.RunUntilTaskCompletion[T] (Foundation.NSRunLoop this, System.Threading.Tasks.Task`1[TResult] task) [0x00082] in /Users/builder/azdo/_work/1/s/maccore/tools/mlaunch/Xamarin.Hosting/Xamarin.Utils/Extensions.cs:35
[11:28:03] dbug: at Xamarin.Launcher.Driver.Main2 (System.String[] args) [0x00b43] in /Users/builder/azdo/_work/1/s/maccore/tools/mlaunch/Xamarin.Hosting/Xamarin.Launcher/Main.cs:459
[11:28:03] dbug: at Xamarin.Launcher.Driver.Main (System.String[] args) [0x0006d] in /Users/builder/azdo/_work/1/s/maccore/tools/mlaunch/Xamarin.Hosting/Xamarin.Launcher/Main.cs:151
[11:28:03] dbug: Process mlaunch exited with 1
[11:28:03] dbug: Test run failed
[11:28:03] dbug: Could not find pid in mtouch output.
[11:28:03] dbug: Test run crashed before it started (no log file produced)
[11:28:03] dbug: No crash reports, waiting 30 seconds for the crash report service...
[11:28:34] dbug: Test execution started
[11:28:36] fail: Application run failed:
                 System.NullReferenceException: Object reference not set to an instance of an object.
                    at Microsoft.DotNet.XHarness.Apple.ErrorKnowledgeBase.TryFindErrors(IFileBackedLog log, Dictionary`2 errorMap, Nullable`1& failureMessage) in /_/src/Microsoft.DotNet.XHarness.Apple/ErrorKnowledgeBase.cs:line 80
                    at Microsoft.DotNet.XHarness.Apple.ErrorKnowledgeBase.IsKnownTestIssue(IFileBackedLog runLog, Nullable`1& knownFailureMessage) in /_/src/Microsoft.DotNet.XHarness.Apple/ErrorKnowledgeBase.cs:line 59
                    at Microsoft.DotNet.XHarness.Apple.TestOrchestrator.<ParseResult>g__LogProblem|10_0(String message, <>c__DisplayClass10_0& ) in /_/src/Microsoft.DotNet.XHarness.Apple/Orchestration/TestOrchestrator.cs:line 260
                    at Microsoft.DotNet.XHarness.Apple.TestOrchestrator.ParseResult(TestExecutingResult testResult, String resultMessage) in /_/src/Microsoft.DotNet.XHarness.Apple/Orchestration/TestOrchestrator.cs:line 294
                    at Microsoft.DotNet.XHarness.Apple.TestOrchestrator.ExecuteApp(AppBundleInformation appBundleInfo, TestTargetOs target, IDevice device, IDevice companionDevice, TimeSpan timeout, TimeSpan launchTimeout, CommunicationChannel communicationChannel, XmlResultJargon xmlResultJargon, IEnumerable`1 singleMethodFilters, IEnumerable`1 classMethodFilters, IReadOnlyCollection`1 environmentalVariables, IEnumerable`1 passthroughArguments, Boolean signalAppEnd, CancellationToken cancellationToken) in /_/src/Microsoft.DotNet.XHarness.Apple/Orchestration/TestOrchestrator.cs:line 195
                    at Microsoft.DotNet.XHarness.Apple.BaseOrchestrator.OrchestrateRun(TestTargetOs target, String deviceName, Boolean includeWirelessDevices, Boolean resetSimulator, Boolean enableLldb, AppBundleInformation appBundleInfo, Func`2 executeMacCatalystApp, Func`4 executeApp, CancellationToken cancellationToken) in /_/src/Microsoft.DotNet.XHarness.Apple/Orchestration/BaseOrchestrator.cs:line 184
                 
[11:28:36] info: Uninstalling the application 'net.dot.System.Globalization.Calendars.Tests' from 'Apple TV'
[11:28:36] dbug: Running /Applications/Xcode_13.1.app/Contents/Developer/usr/bin/simctl uninstall 5C312D2E-4E67-483D-ACF7-36580EC84FC2 net.dot.System.Globalization.Calendars.Tests
[11:28:39] dbug: Process simctl exited with 0
[11:28:39] info: Application 'net.dot.System.Globalization.Calendars.Tests' was uninstalled successfully
XHarness exit code: 83 (APP_LAUNCH_FAILURE)
Removing empty log files:
/tmp/helix/working/BACE09A0/w/A87E08F7/uploads/net.dot.System.Globalization.Calendars.Tests.log
ls: /tmp/helix/working/BACE09A0/w/A87E08F7/uploads/xunit-*.xml: No such file or directory
+ exit_code=83
+ '[' -f /tmp/helix/working/BACE09A0/w/A87E08F7/e/.retry ']'
+ '[' -f /tmp/helix/working/BACE09A0/w/A87E08F7/e/.reboot ']'
+ exit 83
+ export _commandExitCode=83
+ _commandExitCode=83
+ /usr/local/bin/python3 -u /tmp/helix/working/BACE09A0/w/A87E08F7/u/xharness-event-processor.py
Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.
RequestId:d4af8ecd-801e-004f-406c-27ab70000000
Time:2023-01-13T16:28:43.3664817Z
ErrorCode:AuthenticationFailed

B)
Log file: https://helixre107v0xdeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-pull-80615-merge-be0a3fea1fd0435e85/tvOS.Simulator.Interpreter.Test/1/console.a264eebb.log?helixlogtype=result

Server failed to authenticate the request (expand)

Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.
RequestId:2e70b597-d01e-006d-746b-276e6f000000
Time:2023-01-13T16:27:54.9362060Z
ErrorCode:AuthenticationFailed

{
  "ErrorMessage": "Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.",
  "BuildRetry": false,
  "ErrorPattern": "",
  "ExcludeConsoleLog": false
}

Report

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
0	0	0

Tagging subscribers to 'os-tvos': @steveisok, @akoeplinger
See info in area-owners.md if you want to be subscribed.

Issue Details

---

• Example 6.0 backport PR: [release/6.0] [mono][debugger] Fix Debugger.Break() behavior when running on release mode #80615
• Queue: Build tvOSSimulator x64 Release AllSubsets_Mono
• Job results: https://dev.azure.com/dnceng-public/public/_build/results?buildId=136740&view=logs&j=e4e455c5-b032-5051-d2c5-a64982c9ac45&t=0263a457-10d3-5b56-3414-171b09292e09

The tests in tvOS fail with warnings due to one of two possible problems, and can affect almost any test:

A)
Log file: https://helixre107v0xdeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-pull-80615-merge-be0a3fea1fd0435e85/System.Globalization.Calendars.Tests/1/console.47352c66.log?helixlogtype=result

Could not launch the simulator application (expand)

[11:28:03] dbug: Xamarin.Hosting: Simulator watchdogs are already disabled for 'tvOS 15.0 (19J344) - Apple TV'.
[11:28:03] dbug: Xamarin.Hosting: Launching simulator application 'com.apple.iphonesimulator'
[11:28:03] dbug: error HE0018: Could not launch the simulator application: The application “Simulator” could not be launched because a miscellaneous error occurred.
[11:28:03] dbug: 
[11:28:03] dbug: at Xamarin.Hosting.SimulatorApplication.LaunchAsync (Xamarin.Hosting.Bindings.SimDevice device) [0x0020d] in /Users/builder/azdo/_work/1/s/maccore/tools/mlaunch/Xamarin.Hosting/Xamarin.Hosting/SimulatorApplication.cs:203
[11:28:03] dbug: at Xamarin.Hosting.Services.LaunchSimulatorAsync (Xamarin.Hosting.Bindings.SimDevice device) [0x000a7] in /Users/builder/azdo/_work/1/s/maccore/tools/mlaunch/Xamarin.Hosting/Xamarin.Hosting/Services.cs:1390
[11:28:03] dbug: at Xamarin.Launcher.Driver.LaunchSimulatorAsync () [0x00144] in /Users/builder/azdo/_work/1/s/maccore/tools/mlaunch/Xamarin.Hosting/Xamarin.Launcher/Main.cs:781
[11:28:03] dbug: at Xamarin.Utils.NSRunLoopExtensions.RunUntilTaskCompletion[T] (Foundation.NSRunLoop this, System.Threading.Tasks.Task`1[TResult] task) [0x00082] in /Users/builder/azdo/_work/1/s/maccore/tools/mlaunch/Xamarin.Hosting/Xamarin.Utils/Extensions.cs:35
[11:28:03] dbug: at Xamarin.Launcher.Driver.Main2 (System.String[] args) [0x00b43] in /Users/builder/azdo/_work/1/s/maccore/tools/mlaunch/Xamarin.Hosting/Xamarin.Launcher/Main.cs:459
[11:28:03] dbug: at Xamarin.Launcher.Driver.Main (System.String[] args) [0x0006d] in /Users/builder/azdo/_work/1/s/maccore/tools/mlaunch/Xamarin.Hosting/Xamarin.Launcher/Main.cs:151
[11:28:03] dbug: Process mlaunch exited with 1
[11:28:03] dbug: Test run failed
[11:28:03] dbug: Could not find pid in mtouch output.
[11:28:03] dbug: Test run crashed before it started (no log file produced)
[11:28:03] dbug: No crash reports, waiting 30 seconds for the crash report service...
[11:28:34] dbug: Test execution started
[11:28:36] fail: Application run failed:
                 System.NullReferenceException: Object reference not set to an instance of an object.
                    at Microsoft.DotNet.XHarness.Apple.ErrorKnowledgeBase.TryFindErrors(IFileBackedLog log, Dictionary`2 errorMap, Nullable`1& failureMessage) in /_/src/Microsoft.DotNet.XHarness.Apple/ErrorKnowledgeBase.cs:line 80
                    at Microsoft.DotNet.XHarness.Apple.ErrorKnowledgeBase.IsKnownTestIssue(IFileBackedLog runLog, Nullable`1& knownFailureMessage) in /_/src/Microsoft.DotNet.XHarness.Apple/ErrorKnowledgeBase.cs:line 59
                    at Microsoft.DotNet.XHarness.Apple.TestOrchestrator.<ParseResult>g__LogProblem|10_0(String message, <>c__DisplayClass10_0& ) in /_/src/Microsoft.DotNet.XHarness.Apple/Orchestration/TestOrchestrator.cs:line 260
                    at Microsoft.DotNet.XHarness.Apple.TestOrchestrator.ParseResult(TestExecutingResult testResult, String resultMessage) in /_/src/Microsoft.DotNet.XHarness.Apple/Orchestration/TestOrchestrator.cs:line 294
                    at Microsoft.DotNet.XHarness.Apple.TestOrchestrator.ExecuteApp(AppBundleInformation appBundleInfo, TestTargetOs target, IDevice device, IDevice companionDevice, TimeSpan timeout, TimeSpan launchTimeout, CommunicationChannel communicationChannel, XmlResultJargon xmlResultJargon, IEnumerable`1 singleMethodFilters, IEnumerable`1 classMethodFilters, IReadOnlyCollection`1 environmentalVariables, IEnumerable`1 passthroughArguments, Boolean signalAppEnd, CancellationToken cancellationToken) in /_/src/Microsoft.DotNet.XHarness.Apple/Orchestration/TestOrchestrator.cs:line 195
                    at Microsoft.DotNet.XHarness.Apple.BaseOrchestrator.OrchestrateRun(TestTargetOs target, String deviceName, Boolean includeWirelessDevices, Boolean resetSimulator, Boolean enableLldb, AppBundleInformation appBundleInfo, Func`2 executeMacCatalystApp, Func`4 executeApp, CancellationToken cancellationToken) in /_/src/Microsoft.DotNet.XHarness.Apple/Orchestration/BaseOrchestrator.cs:line 184
                 
[11:28:36] info: Uninstalling the application 'net.dot.System.Globalization.Calendars.Tests' from 'Apple TV'
[11:28:36] dbug: Running /Applications/Xcode_13.1.app/Contents/Developer/usr/bin/simctl uninstall 5C312D2E-4E67-483D-ACF7-36580EC84FC2 net.dot.System.Globalization.Calendars.Tests
[11:28:39] dbug: Process simctl exited with 0
[11:28:39] info: Application 'net.dot.System.Globalization.Calendars.Tests' was uninstalled successfully
XHarness exit code: 83 (APP_LAUNCH_FAILURE)
Removing empty log files:
/tmp/helix/working/BACE09A0/w/A87E08F7/uploads/net.dot.System.Globalization.Calendars.Tests.log
ls: /tmp/helix/working/BACE09A0/w/A87E08F7/uploads/xunit-*.xml: No such file or directory
+ exit_code=83
+ '[' -f /tmp/helix/working/BACE09A0/w/A87E08F7/e/.retry ']'
+ '[' -f /tmp/helix/working/BACE09A0/w/A87E08F7/e/.reboot ']'
+ exit 83
+ export _commandExitCode=83
+ _commandExitCode=83
+ /usr/local/bin/python3 -u /tmp/helix/working/BACE09A0/w/A87E08F7/u/xharness-event-processor.py
Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.
RequestId:d4af8ecd-801e-004f-406c-27ab70000000
Time:2023-01-13T16:28:43.3664817Z
ErrorCode:AuthenticationFailed

B)
Log file: https://helixre107v0xdeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-pull-80615-merge-be0a3fea1fd0435e85/tvOS.Simulator.Interpreter.Test/1/console.a264eebb.log?helixlogtype=result

Server failed to authenticate the request (expand)

Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.
RequestId:2e70b597-d01e-006d-746b-276e6f000000
Time:2023-01-13T16:27:54.9362060Z
ErrorCode:AuthenticationFailed

Author:	carlossanlop
Assignees:	-
Labels:	area-VM-meta-mono, os-tvos, test-failure
Milestone:	-

[premun]

@akoeplinger this looks like there is some old version of XHarness in the branch:

[08:27:54] fail: Unknown arguments: --launch-timeout 00:30:00
XHarness exit code: 3 (INVALID_ARGUMENTS)

I'd say it should be safe to bump the version. I did update the XHarness Helix SDK some time ago in that branch but if need be, we could do another round. Though I think we should be good on that side.

The other error (the NRE) would get resolved by this hopefully too.

[akoeplinger]

@premun yeah let's do that

[carlossanlop]

From the same job, I noticed the other platforms (the Apple ones, not sure about Android) are having some very similar failure messages, but I'm unsure if they have the same root cause as this. Can you please check?

[premun]

I think the reason is that the runtime scripts that run this were backported at some point but the XHarness version wasn't bumped. @akoeplinger is this something your team could do? I am very unsure on how to do 6.0 branch changes in runtime

[premun]

This should get better with #85725

[SamMonoRT]

@akoeplinger @premun - can we close this issue?