System.Diagnostics.Tests.PerformanceCounterTests failed in CI

[eerhardt]

Frequency:

Day	Run	Failures	Notes
4/1-7/22
6/30	PR #68648	10 failures	Win11 - Mono
6/28	PR #71397	10 failures	Win10 - x86
6/15	Official run	10 failures	Win11
6/13	PR #70403	10 failures	Win10 - x86
5/7	PR #68993	10 failures	Win11
4/6	PR #67557	10 failures	Win11
4/2	Official run	10 failures	Win10

let failedTests = (methodName : string, includePR : bool, messageSubstr: string, includePassedOnRerun : bool) {
cluster('engsrvprod.kusto.windows.net').database('engineeringdata').AzureDevOpsTests
    | where TestName contains methodName
    | where includePassedOnRerun or (Outcome == 'Failed')
    | extend startOfTestName = indexof_regex(TestName, @"[^.]+$")
    | extend Method = substring(TestName, startOfTestName)
    | extend Type = substring(TestName, 0, startOfTestName - 1)
    | project-away startOfTestName
    | where Message contains messageSubstr
    | distinct JobId, WorkItemId, Message, StackTrace, Method, Type, Arguments, Outcome
    | join kind=inner (cluster('engsrvprod.kusto.windows.net').database('engineeringdata').Jobs
        | where ((Branch == 'refs/heads/main') or (Branch == 'refs/heads/master') or (includePR and (Source startswith "pr/")))
        | where Type startswith "test/functional/cli/"
            and not(Properties contains "runtime-staging")
        | summarize arg_max(Finished, Properties, Type, Branch, Source, Started, QueueName) by JobId
        | project-rename JobType = Type) on JobId
    | extend PropertiesJson = parse_json(Properties)
    | extend OS = replace_regex(tostring(PropertiesJson.operatingSystem), @'\((.*)\).*|([^\(].*)', @'\1\2')
    | extend Runtime = iif(PropertiesJson.runtimeFlavor == "mono", "Mono", iif(PropertiesJson.DefinitionName contains "coreclr", "CoreCLR", ""))
    | extend TargetBranch = extractjson("$.['System.PullRequest.TargetBranch']", Properties)
    | extend Architecture = PropertiesJson.architecture
    | extend Scenario = iif(isempty(PropertiesJson.scenario), "--", PropertiesJson.scenario)
    | project-away PropertiesJson
};
failedTests('System.Diagnostics.Tests.InstanceDataTests', true, 'Category', true);

Test System.Diagnostics.Tests.PerformanceCounterTests.PerformanceCounter_BeginInitEndInit_ProcessorCounter has failed. Check the Test tab or this console log: https://helix.dot.net/api/2019-06-17/jobs/7fb481c8-b204-4809-9ac6-1f7db90f5e5d/workitems/System.Diagnostics.PerformanceCounter.Tests/console

Console log: 'System.Diagnostics.PerformanceCounter.Tests' from job 7fb481c8-b204-4809-9ac6-1f7db90f5e5d workitem a0e2d365-2975-4dd7-b36e-cc56d3df3997 (windows.7.amd64.open.rt) executed on machine a002OCP

C:\h\w\AA5A0982\w\ADD20987\e>taskkill.exe /f /im corerun.exe 
ERROR: The process "corerun.exe" not found.

C:\h\w\AA5A0982\w\ADD20987\e>call RunTests.cmd --runtime-path C:\h\w\AA5A0982\p 
----- start Wed 10/27/2021  3:46:34.80 ===============  To repro directly: ===================================================== 
pushd C:\h\w\AA5A0982\w\ADD20987\e\
"C:\h\w\AA5A0982\p\dotnet.exe" exec --runtimeconfig System.Diagnostics.PerformanceCounter.Tests.runtimeconfig.json --depsfile System.Diagnostics.PerformanceCounter.Tests.deps.json xunit.console.dll System.Diagnostics.PerformanceCounter.Tests.dll -xml testResults.xml -nologo -nocolor -notrait category=IgnoreForCI -notrait category=OuterLoop -notrait category=failing 
popd
===========================================================================================================

C:\h\w\AA5A0982\w\ADD20987\e>"C:\h\w\AA5A0982\p\dotnet.exe" exec --runtimeconfig System.Diagnostics.PerformanceCounter.Tests.runtimeconfig.json --depsfile System.Diagnostics.PerformanceCounter.Tests.deps.json xunit.console.dll System.Diagnostics.PerformanceCounter.Tests.dll -xml testResults.xml -nologo -nocolor -notrait category=IgnoreForCI -notrait category=OuterLoop -notrait category=failing  
  Discovering: System.Diagnostics.PerformanceCounter.Tests (method display = ClassAndMethod, method display options = None)
  Discovered:  System.Diagnostics.PerformanceCounter.Tests (found 91 test cases)
  Starting:    System.Diagnostics.PerformanceCounter.Tests (parallel test collections = on, max threads = 2)
    System.Diagnostics.Tests.PerformanceCounterTests.PerformanceCounter_BeginInitEndInit_ProcessorCounter [FAIL]
      System.InvalidOperationException : Category does not exist.
      Stack Trace:
        /_/src/libraries/System.Diagnostics.PerformanceCounter/src/System/Diagnostics/PerformanceCounterLib.cs(373,0): at System.Diagnostics.PerformanceCounterLib.CounterExists(String machine, String category, String counter)
        /_/src/libraries/System.Diagnostics.PerformanceCounter/src/System/Diagnostics/PerformanceCounter.cs(479,0): at System.Diagnostics.PerformanceCounter.InitializeImpl()
        /_/src/libraries/System.Diagnostics.PerformanceCounter/src/System/Diagnostics/PerformanceCounter.cs(453,0): at System.Diagnostics.PerformanceCounter.Initialize()
        /_/src/libraries/System.Diagnostics.PerformanceCounter/src/System/Diagnostics/PerformanceCounter.cs(78,0): at System.Diagnostics.PerformanceCounter..ctor(String categoryName, String counterName, String instanceName, String machineName)
        /_/src/libraries/System.Diagnostics.PerformanceCounter/tests/PerformanceCounterTests.cs(192,0): at System.Diagnostics.Tests.PerformanceCounterTests.PerformanceCounter_BeginInitEndInit_ProcessorCounter()

Info: Successfully installed performance counters in C:\h\w\AA5A0982\t\dxbpy5oa.jfn\provider.man    System.Diagnostics.Tests.PerformanceDataTests.PerformanceCounter_PerformanceData [SKIP]
      Condition(s) not met: "IsRunnableEnvironment"

Info: Successfully uninstalled the performance counters from the counter definition XML file C:\h\w\AA5A0982\t\dxbpy5oa.jfn\provider.man.  Finished:    System.Diagnostics.PerformanceCounter.Tests
=== TEST EXECUTION SUMMARY ===
   System.Diagnostics.PerformanceCounter.Tests  Total: 94, Errors: 0, Failed: 1, Skipped: 1, Time: 66.084s
----- end Wed 10/27/2021  3:47:42.10 ----- exit code 1 ----------------------------------------------------------
2021-10-27 03:47:42,569: INFO: 2516: run(48): main: Beginning reading of test results.
2021-10-27 03:47:42,569: INFO: 2516: __init__(48): get_log_files: Searching 'C:\h\w\AA5A0982\w\ADD20987\e\..' for log files
2021-10-27 03:47:42,569: INFO: 2516: __init__(66): construct_log_list: Generated log list: console.0927879d.log:
  https://helixre8s23ayyeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-pull-60900-merge-7fb481c8b20448099a/System.Diagnostics.PerformanceCounter.Tests/console.0927879d.log?sv=2019-07-07&se=2021-11-16T03%3A32%3A07Z&sr=c&sp=rl&sig=w7TmCMmLjLzI7c%2Bfh8x2jkOV9DWDX64m%2BgXE4z5eMCg%3D

2021-10-27 03:47:42,569: INFO: 2516: __init__(90): read_results: Searching 'C:\h\w\AA5A0982\w\ADD20987\e' for test results files
2021-10-27 03:47:42,569: INFO: 2516: __init__(96): read_results: Found results file C:\h\w\AA5A0982\w\ADD20987\e\testResults.xml with format xunit
2021-10-27 03:47:42,569: INFO: 2516: __init__(90): read_results: Searching 'C:\h\w\AA5A0982\w\ADD20987\uploads' for test results files
2021-10-27 03:47:42,569: INFO: 2516: packing_test_reporter(30): report_results: Packing 94 test reports to 'C:\h\w\AA5A0982\w\ADD20987\e\__test_report.json'
2021-10-27 03:47:42,569: INFO: 2516: packing_test_reporter(33): report_results: Packed 31823 bytes
Found log 'C:\h\w\AA5A0982\w\ADD20987\e\..\console.0927879d.log'
Uri 'https://helixre8s23ayyeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-pull-60900-merge-7fb481c8b20448099a/System.Diagnostics.PerformanceCounter.Tests/console.0927879d.log?sv=2019-07-07&se=2021-11-16T03%3A32%3A07Z&sr=c&sp=rl&sig=w7TmCMmLjLzI7c%2Bfh8x2jkOV9DWDX64m%2BgXE4z5eMCg%3D'
Did not find dumps, skipping dump docs generation.

Build	Kind	Start Time
1583845	Rolling	2022-01-02

[krwq]

I'm reusing this issue since I've seen similar issue in different random tests across PerformanceCounters.
List of tests I've seen failing similarly in the past 14 days:

• System.Diagnostics.Tests.PerformanceCounterCategoryTests.PerformanceCounterCategory_CategoryType_MultiInstance
• System.Diagnostics.Tests.PerformanceCounterCategoryTests.PerformanceCounterCategory_InstanceExists_Static
• System.Diagnostics.Tests.PerformanceCounterCategoryTests.PerformanceCounterCategory_ReadCategory
• System.Diagnostics.Tests.PerformanceCounterTests.PerformanceCounter_BeginInitEndInit_ProcessorCounter
• System.Diagnostics.Tests.PerformanceCounterTests.PerformanceCounter_CreateCounter_ProcessorCounter
• System.Diagnostics.Tests.PerformanceCounterTests.PerformanceCounter_Decrement_DecrementReadOnly
• System.Diagnostics.Tests.PerformanceCounterTests.PerformanceCounter_NextValue_ProcessorCounter

[krwq]

I'm unclear if Mono issue: #53261 is the same or not so leaving that one for now. The above are Windows only (happened on Win 10, Win 11, Win 7) all on amd64 (although unclear if it's specific to this arch)

[krwq]

Also related: #60403

[danmoseley]

Noting from offline discussion -- for some reason, we are getting an incomplete read from the registry key (which I think is dynamically populated?). Recommendation is that if a category isn’t present, we should sleep for a little while, call CloseSharedResources(), and try again. We would do this in tests only -- unless something has changed in Windows, it doesn't seem to be affecting real-world use unless we hear otherwise. If it does affect customers, we could potentially do it inside CounterExists(..) but could not "fix" GetCategories() since we have no way to tell whether it's returning everything.

[sbomer]

Hit again in ci for net7.0-windows-Release-x86-CoreCLR_release-Windows.7.Amd64.Open: build, log.

C:\h\w\C7BE09D6\w\A195091E\e>"C:\h\w\C7BE09D6\p\dotnet.exe" exec --runtimeconfig System.Diagnostics.PerformanceCounter.Tests.runtimeconfig.json --depsfile System.Diagnostics.PerformanceCounter.Tests.deps.json xunit.console.dll System.Diagnostics.PerformanceCounter.Tests.dll -xml testResults.xml -nologo -nocolor -notrait category=IgnoreForCI -notrait category=OuterLoop -notrait category=failing  
  Discovering: System.Diagnostics.PerformanceCounter.Tests (method display = ClassAndMethod, method display options = None)
  Discovered:  System.Diagnostics.PerformanceCounter.Tests (found 91 test cases)
  Starting:    System.Diagnostics.PerformanceCounter.Tests (parallel test collections = on, max threads = 2)
    System.Diagnostics.Tests.PerformanceCounterTests.PerformanceCounter_BeginInitEndInit_ProcessorCounter [FAIL]
      System.InvalidOperationException : Category 'Processor' does not exist.
      Stack Trace:
        /_/src/libraries/System.Diagnostics.PerformanceCounter/src/System/Diagnostics/PerformanceCounterLib.cs(374,0): at System.Diagnostics.PerformanceCounterLib.CounterExists(String machine, String category, String counter)
        /_/src/libraries/System.Diagnostics.PerformanceCounter/src/System/Diagnostics/PerformanceCounter.cs(479,0): at System.Diagnostics.PerformanceCounter.InitializeImpl()
        /_/src/libraries/System.Diagnostics.PerformanceCounter/src/System/Diagnostics/PerformanceCounter.cs(453,0): at System.Diagnostics.PerformanceCounter.Initialize()
        /_/src/libraries/System.Diagnostics.PerformanceCounter/src/System/Diagnostics/PerformanceCounter.cs(78,0): at System.Diagnostics.PerformanceCounter..ctor(String categoryName, String counterName, String instanceName, String machineName)
        /_/src/libraries/System.Diagnostics.PerformanceCounter/tests/PerformanceCounterTests.cs(194,0): at System.Diagnostics.Tests.PerformanceCounterTests.PerformanceCounter_BeginInitEndInit_ProcessorCounter()