Enable macOS-arm64 libraries tests on runtime-staging rolling builds

[sdmaclea]

I believe the Apple Silicon Libraries tests are now stable enough to turn on rolling libraries tests.

My hack in #49316 has allowed me to run all libraries tests in CI on Apple Silicon. The recent runs with the disables from #49400 are mostly clean. There was one spurious network timeout in the last four runs. See https://dev.azure.com/dnceng/public/_build?definitionId=678&_a=summary runs 20210309.4 - 20210309.7

[ghost]

Tagging subscribers to this area: @safern, @ViktorHofer, @Anipik
See info in area-owners.md if you want to be subscribed.

Issue Details

---

I believe the Apple Silicon Libraries tests are now stable enough to turn on rolling libraries tests.

My hack in #49316 has allowed me to run all libraries tests in CI on Apple Silicon. The recent runs with the disables from #49400 are mostly clean. There was one spurious network timeout in the last four runs. See https://dev.azure.com/dnceng/public/_build?definitionId=678&_a=summary runs 20210309.4 - 20210309.7

Author:	sdmaclea
Assignees:	sdmaclea
Labels:	arch-arm64, area-Infrastructure-libraries, os-mac-os-x-big-sur
Milestone:	6.0.0

[sdmaclea]

I triggered this fullMatrix run https://dev.azure.com/dnceng/public/_build/results?buildId=1031819

[sdmaclea]

Hmmm... the net6.0-OSX-Release-arm64-CoreCLR_checked-OSX.1100.ARM64.Open leg failed....

What is weird is it failed on several tests which were disabled by #49400. For instance I am sure the cryptography tests were disabled.

Wonder if somehow IgnoreForCI and ActiveIssues are being ignored for this leg??

[sdmaclea]

I guess I'll switch to using Mono's disable method (as mentioned by @safern #49400 (comment))

[sdmaclea]

So using the disable like @safern doesn't work either. The issue is that we are using the x64 libraries tests builds on arm64 so the tests can't be disabled at build time.

These worked in libraries outerloop because it doesn't use dependsOnTestArchitecture: x64. It uses a build-job instead of a run-tests job.

I'll move to a [assembly:ActiveIssue()] approach.

[safern]

😢 sounds good.

[ViktorHofer]

The recent runs with the disables from #49400 are mostly clean. There was one spurious network timeout in the last four runs.

Before adding them to rolling builds in the main runtime pipeline and potentially regressing the pass rate, should we add them to the runtime-staging pipeline for rolling builds first?

[sdmaclea]

add them to the runtime-staging pipeline for rolling builds first?

@ViktorHofer Once I get the patch for getting these to run properly in runtime.yml. I was thinking I would make 10-100 runs on my M1 to make sure I disabled all/most intermittent tests. That is what I had done with the runtime tests and it seemed to work well.

Running locally I was seeing a lot of weird error messages which seemed to imply failure, but didn't result in a failing test. I'll probably open one or more issues just to ask questions.

If you think it would be much better, I can move to runtime staging, it just will require a lot of duplicate/redundant builds.

[ViktorHofer]

Either approach sounds good to me. I trust you on this. If we would enable the configuration in every PR I would probably prefer the runtime-staging pipeline but for rolling builds (which we usually have about 10 per day) it should be fine.

[sdmaclea]

I have run several hundred runs, disabling failing tests and test suites. I have disabled lots of tests. There are still a lot of spurious failures. There were 12 failures in the most recent 69 test runs. That still almost 20% failure rate. The macOS-arm64 runtime is too unstable for the runtime pipeline.

I'll move this to the runtime-staging pipeline as suggested and give up trying to stabilize runs by disabling tests.

[ViktorHofer]

That still almost 20% failure rate. The macOS-arm64 runtime is too unstable for the runtime pipeline.

Thanks a lot Steve for the transparency and following the process. This helps us with getting our pass rate up again.

[sdmaclea]

I triggered a full matrix run here https://dev.azure.com/dnceng/public/_build/results?buildId=1042491.

[sdmaclea]

The full matrix build completed w/o a Apple Silicon failure.

The main build failed however, because I had undone some changes and missed a line. This build should now pass.

[sdmaclea]

The macOS-arm64 runtime is too unstable for the runtime pipeline.

This helps us with getting our pass rate up again.

To be clearer, I meant that it was too unstable to run the libraries test suite. I did look at the runtime pipeline osx-arm64 failures. They seemed to be all related to a single test which @directhex disabled for me earlier today.

[sdmaclea]

All checks have passed.

@ViktorHofer @safern Can I get a review. I think thought this is was ready to merge.

I guess I have to deal with merge conflicts

[ViktorHofer]

LGTM. cc @danmoseley for the amount of disabled tests and test projects on macOS arm64.

[danmoseley]

LGTM with the one crucial fix above

[danmoseley]

(I didn't review the yaml)

[safern]

LGTM