Assert failure(PID 5984 [0x00001760], Thread: 1888 [0x0760]): CONTRACT VIOLATION by CallCountingManager::OnCallCountThresholdReached within calls to KERNELBASE! EnumCalendarInfoExEx

[danmoseley]

Globalization related tests are periodically failing with output like this

Assert failure(PID 5984 [0x00001760], Thread: 1888 [0x0760]): CONTRACT VIOLATION by CallCountingManager::OnCallCountThresholdReached at "F:\workspace\_work\1\s\src\coreclr\vm\callcounting.cpp" @ 725

MODE_COOPERATIVE encountered while thread is in preemptive state.

                        CONTRACT in CallCountingManager::OnCallCountThresholdReached at "F:\workspace\_work\1\s\src\coreclr\vm\callcounting.cpp" @ 725
                        CONTRACT in DispatchCallSimple at "F:\workspace\_work\1\s\src\coreclr\vm\callhelpers.cpp" @ 185
                        CONTRACT in ThreadNative::KickOffThread_Worker at "F:\workspace\_work\1\s\src\coreclr\vm\comsynchronizable.cpp" @ 148
                        CONTRACT in ManagedThreadBase_DispatchInner at "F:\workspace\_work\1\s\src\coreclr\vm\threads.cpp" @ 7293
                        CONTRACT in ManagedThreadBase_FullTransition at "F:\workspace\_work\1\s\src\coreclr\vm\threads.cpp" @ 7542
                        CONTRACT in ThreadNative::KickOffThread at "F:\workspace\_work\1\s\src\coreclr\vm\comsynchronizable.cpp" @ 196

We can't find the violated contract. Look for an old-style non-holder-based contract.


CORECLR! <no symbol> + 0x0 (0x00007ff8`11a93c82)
CORECLR! GetCLRRuntimeHost + 0x259212 (0x00007ff8`11de2f32)
CORECLR! <no symbol> + 0x0 (0x00007ff8`11af7229)
CORECLR! GetCLRRuntimeHost + 0x5CD7D5 (0x00007ff8`121574f5)
KERNELBASE! RegLoadMUIStringA + 0x17B37 (0x00007ff8`3c5b6437)
KERNELBASE! EnumCalendarInfoExEx + 0x5A (0x00007ff8`3c57fcda)
SYSTEM.PRIVATE.CORELIB! <no symbol> + 0x0 (0x00007ff8`114ea4dc)
SYSTEM.PRIVATE.CORELIB! <no symbol> + 0x0 (0x00007ff8`115e2528)
SYSTEM.PRIVATE.CORELIB! <no symbol> + 0x0 (0x00007ff8`115e22ca)
SYSTEM.PRIVATE.CORELIB! <no symbol> + 0x0 (0x00007ff8`115e1e85)
    File: F:\workspace\_work\1\s\src\coreclr\vm\callcounting.cpp Line: 725
    Image: C:\h\w\A2F3094A\p\dotnet.exe

----- end Thu 12/31/2020  8:15:58.95 ----- exit code -1073740286 ----------------------------------------------------------

It is happening ~2x a day over the last 10 days and only on windows.10.amd64.open.rt ### OS: Distro=WindowsProductType=8 WindowsInstallationType=Server Description=Microsoft Windows 10.0.14393 Version=Microsoft Windows NT 10.0.14393.0 Arch=X64 Rid=win10-x64

It occurs in these test jobs

System.Globalization.Tests
System.Globalization.Nls.Tests
System.Runtime.Tests

the stack always contains

KERNELBASE! Internal_EnumCalendarInfo + 0x225 (0x00007ff8`b074ff15)
KERNELBASE! EnumCalendarInfoExEx + 0x5A (0x00007ff8`b074fcda)

or

KERNELBASE! RegLoadMUIStringA + 0x17B37 (0x00007ff8`3c5b6437)
KERNELBASE! EnumCalendarInfoExEx + 0x5A (0x00007ff8`3c57fcda)

[Dotnet-GitSync-Bot]

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

[danmoseley]

https://engsrvprod.kusto.windows.net/engineeringdata?query=H4sIAAAAAAAEAIWRUU%2fCMBDH30n4Dpc9QTK2UhZniPNBogYfTFQMz4WeWWG7krYLYPzwFkE2gsQ%2btEl7ze%2f3v5tqsxw7LC20W3H8BescDcKDImVzlJBlIIVDp0rscMZZr897%2fRRYMuR8mLCIDxLmV7fdOvt6C6TXnV6fyS4IkvXDzeHB359CjUKSxfZZlAjWCePsWrkcgret9YZR0IBcKP2r9n6j3EhL3GXxNukgTRi%2fvmqSXyqs8AxbzSpyVRSANhcqtN3sSQutCJaKZKaI0MCTnlnQtDvHsla5M4Lm%2bU4kMPhh4xyFtHEpvLIJ9kIroxc4dzUwPAYIYbJd%2bb0ZPjy2NYSRJqsLfDcq3INDmB6m25Q46V2W%2fXbs1Yf1U44maJ39yfzfPB4LPROF%2bhROafL67dY3GylF%2bE0CAAA%3d&web=0

WorkItems
| where Finished > now(-10d) and Finished < now(-1d)
| where ExitCode == -1073740286
| join kind=inner Jobs on JobId
| where Branch == "refs/heads/master"
| where FriendlyName =="System.Runtime.Tests" or FriendlyName startswith "System.Globalization"

[danmoseley]

First instance was 2020-12-15 08:03:15.8700000 - previously it didn't appear at all. Given how regular the runs are on this queue, whatever caused it likely was a change made between about 12-10 through 12-15

[danmoseley]

Setting interop because it seems like we shouldn't be able to cause this with a managed code bug? The relevant signatures did not change in that period.

[danmoseley]

#45625 was about the right time and mentions UnmanagedCallersOnly. However that was reverted a couple days later, whereas these failures continue:

Finished	ExitCode
2020-12-15 08:03:15.8700000	-1073740286 (first one)
2020-12-15 08:17:15.7180000	-1073740286
2020-12-17 08:06:36.9600000	-1073740286
2020-12-17 08:20:15.3760000	-1073740286
2020-12-19 08:14:09.1880000	-1073740286
2020-12-19 08:36:50.8180000	-1073740286
2020-12-20 08:05:30.0100000	-1073740286
2020-12-20 08:28:17.1140000	-1073740286
2020-12-22 08:22:54.2930000	-1073740286
2020-12-24 08:00:07.2940000	-1073740286
2020-12-24 08:21:41.5050000	-1073740286
2020-12-26 08:21:48.7070000	-1073740286
2020-12-26 08:35:58.2240000	-1073740286
2020-12-27 07:35:32.4310000	-1073740286
2020-12-27 07:43:52.6100000	-1073740286
2020-12-29 07:49:39.7490000	-1073740286

[danmoseley]

@jkoritzinsky

[jkoritzinsky]

I'll take a look.

[jkoritzinsky]

I've been able to triage it. These failures are happening since the only place we block Tiered Compilation for UnmanagedCallersOnly is when we load the UnmanagedCallersOnly method from a R2R image. We don't block it when we JIT the method. #45625 removed R2R support for UnmanagedCallersOnly in crossgen1, so that's why the failures started happening now.

The issue is that UnmanagedCallersOnly methods are entered and exited in preemptive mode, but the CodeVersioning stubs expect the GC to be in co-op mode. That's what's causing the assert. I think the best solution for now is to disabled Tiered Compilation for UnmanagedCallersOnly methods in all scenarios.

@AaronRobinsonMSFT @jkotas does that sound good to you? Any recommendations on the least perf-impacting way to do this?

[AaronRobinsonMSFT]

I think the best solution for now is to disabled Tiered Compilation for UnmanagedCallersOnly methods in all scenarios.

I agree with this. I assume that these methods will simply be JITed for tier1 instead of tier0?

but the CodeVersioning stubs expect the GC to be in co-op mode.

Do we know how baked in this assumption is in the implementation?

/cc @kouvel

[jkotas]

@jkotas does that sound good to you?

Sounds good to me.

[kouvel]

I assume that these methods will simply be JITed for tier1 instead of tier0?

They would be generated with optimizations but without the tier1 flag, which is probably similar at the moment.

Do we know how baked in this assumption is in the implementation?

Call counting is currently done at the boundary of the call, when it reaches the call count threshold for promotion it currently uses coop mode to synchronize with deleting call counting stubs, that ensures that a call counting stub would not be deleted while managed code is running through it, and that it's safe to read from it to associate it with the info necessary to set up the method for promotion.

[jkotas]

Do we know how baked in this assumption is in the implementation?

The call counting reclamation strategy uses GC mode for synchronization, so it is pretty baked in.

runtime/src/coreclr/vm/callcounting.cpp

Line 733 in 69e114c

// Get the code version from the call counting stub/info in cooperative GC mode to synchronize with deletion. The stub/info