tracing/eventsource/eventsourcetrace/eventsourcetrace/eventsourcetrace.sh failure

[mikedn]

In Ubuntu16.04 arm64 Cross Checked Innerloop Build and Test this test failed with the following error:

Assert failure(PID 32988 [0x000080dc], Thread: 32988 [0x80dc]): Consistency check failed: Crst Level violation: Can't take level 18 lock CrstPendingTypeLoadEntry because you already holding level 17 lock CrstEventPipe

in https://ci.dot.net/job/dotnet_coreclr/job/master/job/arm64_cross_checked_ubuntu16.04_innerloop_flow_prtest/545/ (triggered in dotnet/coreclr#20045).

I think I've already seen some issue(s) about tracing but I can't seem to find anything that involves crst level violation so opened a new issue.

[mikedn]

The entire test output in case the build gets deleted:

        
        Assert failure(PID 32988 [0x000080dc], Thread: 32988 [0x80dc]): Consistency check failed: Crst Level violation: Can't take level 18 lock CrstPendingTypeLoadEntry because you already holding level 17 lock CrstEventPipe
        FAILED: false
            File: /mnt/j/workspace/dotnet_coreclr/master/arm64_cross_checked_ubuntu16.04_innerloop_prtest/src/vm/crst.cpp Line: 761
            Image: /home/robox/j/workspace/dotnet_coreclr/master/arm64_cross_checked_ubuntu16.04_innerloop_tst_prtest/bin/tests/Linux.arm64.Checked/Tests/Core_Root/corerun
        
        /home/robox/j/workspace/dotnet_coreclr/master/arm64_cross_checked_ubuntu16.04_innerloop_tst_prtest/bin/tests/Linux.arm64.Checked/tracing/eventsource/eventsourcetrace/eventsourcetrace/eventsourcetrace.sh: line 251: 32988 Aborted                 (core dumped) $_DebuggerFullPath "$CORE_ROOT/corerun" $ExePath $CLRTestExecutionArguments
        /n/nReturn code:      1/nRaw output file:      /home/robox/j/workspace/dotnet_coreclr/master/arm64_cross_checked_ubuntu16.04_innerloop_tst_prtest/bin/tests/Linux.arm64.Checked/tracing/eventsource/Reports/tracing.eventsource/eventsourcetrace/eventsourcetrace/eventsourcetrace.output.txt/nRaw output:/nBEGIN EXECUTION
        /home/robox/j/workspace/dotnet_coreclr/master/arm64_cross_checked_ubuntu16.04_innerloop_tst_prtest/bin/tests/Linux.arm64.Checked/Tests/Core_Root/corerun eventsourcetrace.exe
        	Start: Enable tracing.
        	End: Enable tracing.
        
        	Start: Messaging.
        	End: Messaging.
        
        	Start: Disable tracing.
        Expected: 100
        Actual: 134
        END EXECUTION - FAILED
        Test Harness Exitcode is : 1
        /nTo run the test:/n> set CORE_ROOT=/home/robox/j/workspace/dotnet_coreclr/master/arm64_cross_checked_ubuntu16.04_innerloop_tst_prtest/bin/tests/Linux.arm64.Checked/Tests/Core_Root/n> /home/robox/j/workspace/dotnet_coreclr/master/arm64_cross_checked_ubuntu16.04_innerloop_tst_prtest/bin/tests/Linux.arm64.Checked/tracing/eventsource/eventsourcetrace/eventsourcetrace/eventsourcetrace.sh/n
        Expected: True
        Actual:   False
        Stack Trace:
          /mnt/j/workspace/dotnet_coreclr/master/arm64_cross_checked_ubuntu16.04_innerloop_prtest/bin/tests/Linux.arm64.Checked/TestWrappers/tracing.eventsource/tracing.eventsource.XUnitWrapper.cs(109,0): at tracing_eventsource._eventsourcetrace_eventsourcetrace_eventsourcetrace_._eventsourcetrace_eventsourcetrace_eventsourcetrace_sh()

[RussKeldorph]

@brianrob

[brianrob]

This is not something that I've seen before. I suspect some code path changed that is now affecting the lock order.

Adding @noahfalk and @jorive who own this area now.

[noahfalk]

Thanks for the report @mikedn ! Let us know if there is anything pressing here, otherwise it might be a while before we can get to this. We just inherited the area from Brian and there has been quite an influx new issues that we aren't yet ramped up to handle quickly.

[mikedn]

Seems like a rare issue, at least in comparation to other issues that hit CI rather often. I've yet to see this again.

[jkotas]

Hit by: dotnet/coreclr#20878

[jkotas]

Hit again by dotnet/coreclr#21071 (https://ci.dot.net/job/dotnet_coreclr/job/master/job/arm64_cross_checked_ubuntu16.04_innerloop_flow_prtest/1787/)

[tommcdon]

cc @jorive

[tommcdon]

@sywhang has a WIP that should address this
Related to https://github.com/dotnet/coreclr/issues/23412

[sywhang]

dotnet/coreclr#24101 should have addressed this issue. I opened a PR to re-enable this test: dotnet/coreclr#24365.

[sywhang]

dotnet/coreclr#24365 re-enabled this test. Please feel free to open this issue again if another failure is observed.