Several numeric operator tests failed on macOS in CI

[stephentoub]

https://ci3.dot.net/job/dotnet_corefx/job/master/job/osx-TGroup_netcoreapp+CGroup_Debug+AGroup_x64+TestOuter_false_prtest/2743/consoleText

System.Numerics.Tests.ToStringTest.RunRegionSpecificStandardFormatToStringTests [FAIL]
        Unexpected Exception:Xunit.Sdk.TrueException: Assert.True() Failure
        Expected: True
        Actual:   False
           at Xunit.Assert.True(Nullable`1 condition, String userMessage)
           at System.Numerics.Tests.ToStringTest.VerifyToString(String test, String format, IFormatProvider provider, Boolean expectError, String expectedResult) in /Users/dotnet-bot/j/workspace/dotnet_corefx/master/osx-TGroup_netcoreapp+CGroup_Debug+AGroup_x64+TestOuter_false_prtest/src/System.Runtime.Numerics/tests/BigInteger/BigIntegerToStringTests.cs:line 1439
        Expected: True
        Actual:   False
        Stack Trace:
           /Users/dotnet-bot/j/workspace/dotnet_corefx/master/osx-TGroup_netcoreapp+CGroup_Debug+AGroup_x64+TestOuter_false_prtest/src/System.Runtime.Numerics/tests/BigInteger/BigIntegerToStringTests.cs(1448,0): at System.Numerics.Tests.ToStringTest.VerifyToString(String test, String format, IFormatProvider provider, Boolean expectError, String expectedResult)
           /Users/dotnet-bot/j/workspace/dotnet_corefx/master/osx-TGroup_netcoreapp+CGroup_Debug+AGroup_x64+TestOuter_false_prtest/src/System.Runtime.Numerics/tests/BigInteger/BigIntegerToStringTests.cs(1397,0): at System.Numerics.Tests.ToStringTest.VerifyToString(String test, String format, String expectedResult)
           /Users/dotnet-bot/j/workspace/dotnet_corefx/master/osx-TGroup_netcoreapp+CGroup_Debug+AGroup_x64+TestOuter_false_prtest/src/System.Runtime.Numerics/tests/BigInteger/BigIntegerToStringTests.cs(478,0): at System.Numerics.Tests.ToStringTest.RunStandardFormatToStringTests(Random random, String format, String negativeSign, Int32 precision, StringFormatter formatter)
           /Users/dotnet-bot/j/workspace/dotnet_corefx/master/osx-TGroup_netcoreapp+CGroup_Debug+AGroup_x64+TestOuter_false_prtest/src/System.Runtime.Numerics/tests/BigInteger/BigIntegerToStringTests.cs(253,0): at System.Numerics.Tests.ToStringTest.RunRegionSpecificStandardFormatToStringTests()
     System.Numerics.Tests.op_orTest.RunOrTests [FAIL]
        Assert.Equal() Failure
                                         ↓ (pos 240)
        Expected: ···6007339264329025061429377265939311317315953000824212942620456···
        Actual:   ···6007339264329025061465590802459251221408169200421330713897122···
                                         ↑ (pos 240)
        Stack Trace:
           /Users/dotnet-bot/j/workspace/dotnet_corefx/master/osx-TGroup_netcoreapp+CGroup_Debug+AGroup_x64+TestOuter_false_prtest/src/System.Runtime.Numerics/tests/BigInteger/op_or.cs(126,0): at System.Numerics.Tests.op_orTest.VerifyOrString(String opstring)
           /Users/dotnet-bot/j/workspace/dotnet_corefx/master/osx-TGroup_netcoreapp+CGroup_Debug+AGroup_x64+TestOuter_false_prtest/src/System.Runtime.Numerics/tests/BigInteger/op_or.cs(25,0): at System.Numerics.Tests.op_orTest.RunOrTests()
     System.Numerics.Tests.op_rightshiftTest.RunRightShiftTests [FAIL]
        Assert.Equal() Failure
                                         ↓ (pos 2632)
        Expected: ···5000711332234539142747417137995476834885943827938409825669015···
        Actual:   ···5000711332234539142785686200374855839256444202561770627110816···
                                         ↑ (pos 2632)
        Stack Trace:
           /Users/dotnet-bot/j/workspace/dotnet_corefx/master/osx-TGroup_netcoreapp+CGroup_Debug+AGroup_x64+TestOuter_false_prtest/src/System.Runtime.Numerics/tests/BigInteger/op_rightshift.cs(167,0): at System.Numerics.Tests.op_rightshiftTest.VerifyRightShiftString(String opstring)
           /Users/dotnet-bot/j/workspace/dotnet_corefx/master/osx-TGroup_netcoreapp+CGroup_Debug+AGroup_x64+TestOuter_false_prtest/src/System.Runtime.Numerics/tests/BigInteger/op_rightshift.cs(118,0): at System.Numerics.Tests.op_rightshiftTest.RunRightShiftTests()

[mellinoe]

My guess is that this is a codegen issue. @CarolEidt Any recent changes that might have caused this that you are aware of?

[CarolEidt]

Nothing obvious occurs to me, though of course one suspects something optimization-related. Do you have an idea when these were last known to work with the latest coreclr/jit?

[mellinoe]

@CarolEidt This showed up when we updated from the August 25th package to the August 28th package. So it would have to be a change taken in that timeframe.

[CarolEidt]

@mellinoe There were a number of checkins, none of which are an obvious culprit. I suspect that it will require a binary search to track down the culprit. I see from dotnet/corefx#23563 that there were other failures as well, though I'm a bit confused because @stephentoub opened this 2 days ago, but you say the failures showed up with the August 28th package which is today.

[mellinoe]

Good point, I guess I'm not sure when the issue popped up. @stephentoub Any clarification?

[stephentoub]

The failures I saw were associated with dotnet/corefx@7a4a948

[mellinoe]

Thanks, that looks like yesterday's build. So that would narrow down the commit range a little bit.

[CarolEidt]

@mellinoe @stephentoub can you help me figure out what coreclr commit range that corresponds to?

[danmoseley]

@CarolEidt here is the quickest way I have found to get the CoreCLR commit range from a CoreFX update job like dotnet/corefx#23563 in this case.

The issues were noted just after Update CoreClr, CoreFx, CoreSetup, External to preview2-25627-01,.... (Because the bot force pushes, we can't see earlier CI results in the PR so I'm going by the comments.) That gives us preview2-25627-01 as the endpoint.

To get the start point, I looked at the left of the diff on the PR and see <CoreClrPackageVersion>2.1.0-preview2-25625-02</CoreClrPackageVersion> is what is currently in CoreFX.

To get the CoreCLR commit hashes from the version range, just add them to myget URLs like this:

https://dotnet.myget.org/feed/dotnet-core/package/nuget/Microsoft.NETCore.Runtime.CoreCLR/2.1.0-preview2-25625-02
https://dotnet.myget.org/feed/dotnet-core/package/nuget/Microsoft.NETCore.Runtime.CoreCLR/2.1.0-preview2-25627-01

The hashes are in the description, eg The .NET Core runtime, called CoreCLR, and the base library, called System.Private.CoreLib. It includes the garbage collector, JIT compiler, base .NET data types and many low-level classes. 1596d06b6180a58782cf831b435dbe0046b5be9c

This gives me this range e9e3a69..1596d06 which gives:

*   18ee973ffc Merge pull request dotnet/corefx#13579 from CarolEidt/RemoveDupLowerBlockStore
|\
| * 22ca4e0c4a PR Feedback
| * 388660ba7f Arm Containment Cleanup
* c539df9eda Fix generic context passed to resolved constrained method calls (#13577)
* 06fc72a620 Do not devirtualize indirect calls (#13561)
*   a7ffdeca6f Merge pull request dotnet/runtime#19176 from pgavlin/NoExpandIndex
|\
| * 71b9152b70 Address PR feedback.
| * cdec3b5ed0 Use a smaller expansion of `GT_INDEX` in MinOpts.
* 62a8d935e7 [RyuJIT/armel] Fix ifdef block in TreeNodeInfoInit

Assuming I did this correctly the next question is how to verify hypotheses. I did not repro the Windows failure when I ran it locally, but this could be tried out. I also kicked off a CoreCLR job (dotnet/coreclr#13679) and if the failures repro there then you can throw up CoreCLR PR's with experiments.

[jkotas]

dotnet/coreclr#13245 is the bad change. I am reliably reproducing the failure with the change, and not reproducing it without the change, with COMPlus_GCStress=1. Reverting in dotnet/coreclr#13682

[jkotas]

cc @pgavlin

[pgavlin]

@jkotas what was the exact scenario under which you were reproducing this? I am able to reproduce this without GCStress on OS X. Were you doing something different?

[jkotas]

It does reproduce without GCStress too on both Windows and Unix, but it was not reliable repro for me.