Test failure: System.Diagnostics.Metrics.Tests.MetricsTests.PassingVariableTagsParametersTest

[VincentBu]

Failed in: runtime-coreclr libraries-jitstress2-jitstressregs 20260307.1

Failed tests:

net11.0-linux-Release-x64-jitstress2_jitstressregs2-AzureLinux.3.Amd64.Open
- System.Diagnostics.Metrics.Tests.MetricsTests.PassingVariableTagsParametersTest
net11.0-linux-Release-x64-jitstress2_jitstressregs0x80-AzureLinux.3.Amd64.Open
- System.Diagnostics.Metrics.Tests.MetricsTests.PassingVariableTagsParametersTest

Error message:

Microsoft.DotNet.RemoteExecutor.RemoteExecutionException : Remote process failed with an unhandled exception.

Stack trace:

Child exception:
  Xunit.Sdk.EqualException: Assert.Equal() Failure: Collections differ
           ↓ (pos 0)
Expected: [["K1"] = "V1", ["K2"] = "V2", ["K3"] = "V3", ["K4"] = "V4", ["K5"] = "V5", ···]
Actual:   [[null] = null, ["K2"] = "V2", ["K3"] = "V3", ["K4"] = "V4", ["K5"] = "V5", ···]
           ↑ (pos 0)
   at System.Diagnostics.Metrics.Tests.MetricsTests.<>c__DisplayClass32_0`1.<InstrumentPassingVariableTagsParametersTest>b__1(Instrument inst, T measurement, ReadOnlySpan`1 tags, Object state) in /_/src/libraries/System.Diagnostics.DiagnosticSource/tests/MetricsTests.cs:line 1965
   at System.Diagnostics.Metrics.Instrument`1.RecordMeasurement(T measurement, ReadOnlySpan`1 tags) in /_/src/libraries/System.Diagnostics.DiagnosticSource/src/System/Diagnostics/Metrics/Instrument.common.cs:line 99
   at System.Diagnostics.Metrics.Tests.MetricsTests.PublishCounterMeasurement[T](Counter`1 counter, T value, KeyValuePair`2[] tags) in /_/src/libraries/System.Diagnostics.DiagnosticSource/tests/MetricsTests.cs:line 1791
   at System.Diagnostics.Metrics.Tests.MetricsTests.<PassingVariableTagsParametersTest>b__10_3(Instrument`1 instrument, Int32 value, KeyValuePair`2[] tags) in /_/src/libraries/System.Diagnostics.DiagnosticSource/tests/MetricsTests.cs:line 751
   at System.Diagnostics.Metrics.Tests.MetricsTests.InstrumentPassingVariableTagsParametersTest[T](Instrument`1 instrument, Func`4 record) in /_/src/libraries/System.Diagnostics.DiagnosticSource/tests/MetricsTests.cs:line 2015
   at System.Diagnostics.Metrics.Tests.MetricsTests.<PassingVariableTagsParametersTest>b__10_0() in /_/src/libraries/System.Diagnostics.DiagnosticSource/tests/MetricsTests.cs:line 749
   at System.Reflection.MethodBaseInvoker.InterpretedInvoke_Method(Object obj, IntPtr* args) in /_/src/coreclr/System.Private.CoreLib/src/System/Reflection/MethodBaseInvoker.CoreCLR.cs:line 36
   at System.Reflection.MethodBaseInvoker.InvokeWithNoArgs(Object obj, BindingFlags invokeAttr) in /_/src/libraries/System.Private.CoreLib/src/System/Reflection/MethodBaseInvoker.cs:line 57

Child process:
  System.Diagnostics.DiagnosticSource.Tests, Version=11.0.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51 System.Diagnostics.Metrics.Tests.MetricsTests Void <PassingVariableTagsParametersTest>b__10_0()

[VincentBu]

Feel free to close it if it's a dup of #117520

[VincentBu]

Failed in: runtime-coreclr libraries-jitstress2-jitstressregs 20260314.1

Failed tests:

net11.0-linux-Release-x64-jitstress2_jitstressregs2-AzureLinux.3.Amd64.Open
- System.Diagnostics.Metrics.Tests.MetricsTests.PassingVariableTagsParametersTest
net11.0-linux-Release-x64-jitstress2_jitstressregs0x80-AzureLinux.3.Amd64.Open
- System.Diagnostics.Metrics.Tests.MetricsTests.PassingVariableTagsParametersTest

Error message:

Microsoft.DotNet.RemoteExecutor.RemoteExecutionException : Remote process failed with an unhandled exception.

Stack trace:

Child exception:
  Xunit.Sdk.EqualException: Assert.Equal() Failure: Collections differ
           ↓ (pos 0)
Expected: [["K1"] = "V1", ["K2"] = "V2", ["K3"] = "V3", ["K4"] = "V4", ["K5"] = "V5", ···]
Actual:   [[null] = null, ["K2"] = "V2", ["K3"] = "V3", ["K4"] = "V4", ["K5"] = "V5", ···]
           ↑ (pos 0)
   at System.Diagnostics.Metrics.Tests.MetricsTests.<>c__DisplayClass32_0`1.<InstrumentPassingVariableTagsParametersTest>b__1(Instrument inst, T measurement, ReadOnlySpan`1 tags, Object state) in /_/src/libraries/System.Diagnostics.DiagnosticSource/tests/MetricsTests.cs:line 1965
   at System.Diagnostics.Metrics.Instrument`1.RecordMeasurement(T measurement, ReadOnlySpan`1 tags) in /_/src/libraries/System.Diagnostics.DiagnosticSource/src/System/Diagnostics/Metrics/Instrument.common.cs:line 99
   at System.Diagnostics.Metrics.Tests.MetricsTests.PublishCounterMeasurement[T](Counter`1 counter, T value, KeyValuePair`2[] tags) in /_/src/libraries/System.Diagnostics.DiagnosticSource/tests/MetricsTests.cs:line 1791
   at System.Diagnostics.Metrics.Tests.MetricsTests.<PassingVariableTagsParametersTest>b__10_3(Instrument`1 instrument, Int32 value, KeyValuePair`2[] tags) in /_/src/libraries/System.Diagnostics.DiagnosticSource/tests/MetricsTests.cs:line 751
   at System.Diagnostics.Metrics.Tests.MetricsTests.InstrumentPassingVariableTagsParametersTest[T](Instrument`1 instrument, Func`4 record) in /_/src/libraries/System.Diagnostics.DiagnosticSource/tests/MetricsTests.cs:line 2015
   at System.Diagnostics.Metrics.Tests.MetricsTests.<PassingVariableTagsParametersTest>b__10_0() in /_/src/libraries/System.Diagnostics.DiagnosticSource/tests/MetricsTests.cs:line 749
   at System.Reflection.MethodBaseInvoker.InterpretedInvoke_Method(Object obj, IntPtr* args) in /_/src/coreclr/System.Private.CoreLib/src/System/Reflection/MethodBaseInvoker.CoreCLR.cs:line 36
   at System.Reflection.MethodBaseInvoker.InvokeWithNoArgs(Object obj, BindingFlags invokeAttr) in /_/src/libraries/System.Private.CoreLib/src/System/Reflection/MethodBaseInvoker.cs:line 57

Child process:
  System.Diagnostics.DiagnosticSource.Tests, Version=11.0.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51 System.Diagnostics.Metrics.Tests.MetricsTests Void <PassingVariableTagsParametersTest>b__10_0()

[dotnet-policy-service]

Tagging subscribers to this area: @JulieLeeMSFT, @jakobbotsch
See info in area-owners.md if you want to be subscribed.

[mangod9]

Here is some copilot analysis for this failure:

Crash Context

  ┌────────────┬───────────────────────────────────────────────────────────────────────────────────────────────────┐
  │ Field      │ Value                                                                                             │
  ├────────────┼───────────────────────────────────────────────────────────────────────────────────────────────────┤
  │ Build      │ runtime-coreclr libraries-jitstress2-jitstressregs 20260307.1                                     │
  │            │ (https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_buil                   │
  │            │ d/results?buildId=1324993)                                                                        │
  ├────────────┼───────────────────────────────────────────────────────────────────────────────────────────────────┤
  │ Helix Job  │ 3f3cc108-6550-42e6-80a5-5e70ed9c556a                                                              │
  │ (regs2)    │                                                                                                   │
  ├────────────┼───────────────────────────────────────────────────────────────────────────────────────────────────┤
  │ Helix Job  │ acfbbeb5-9555-4c5b-b5ea-892f475ef22b                                                              │
  │ (regs0x80) │                                                                                                   │
  ├────────────┼───────────────────────────────────────────────────────────────────────────────────────────────────┤
  │ Work Item  │ System.Diagnostics.DiagnosticSource.Tests                                                         │
  ├────────────┼───────────────────────────────────────────────────────────────────────────────────────────────────┤
  │ Test       │ MetricsTests.PassingVariableTagsParametersTest                                                    │
  ├────────────┼───────────────────────────────────────────────────────────────────────────────────────────────────┤
  │ Platform   │ Linux x64, AzureLinux 3.0, azurelinux.3.amd64.open                                                │
  ├────────────┼───────────────────────────────────────────────────────────────────────────────────────────────────┤
  │ Dump files │ Generated (blobs expired ~9 days)                                                                 │
  └────────────┴───────────────────────────────────────────────────────────────────────────────────────────────────┘

  -----------------------------------------------------------------------------------------------------------------

  Root Cause Analysis

  Call Chain

   PassingVariableTagsParametersTest()
     → RemoteExecutor.Invoke(...)
       → InstrumentPassingVariableTagsParametersTest<byte>(counter, record)
         → record(instrument, value, [K1,K2,K3,K4])         ← expectedTags.Length == 4
           → PublishCounterMeasurement(counter, value, tags)
             → counter.Add(value, tags[0], tags[1], tags[2], tags[3])
               → Counter<T>.Add(T, params ReadOnlySpan<KVP>)  ← NO 4-param overload!
                 → inline array built at call site: [{K1,V1}, {K2,V2}, {K3,V3}, {K4,V4}]
                   → RecordMeasurement(delta, span)
                     → listener callback: Assert.Equal(expected, tags.ToArray())  ← FAIL

  Why 4 tags fail but 1–3 don't

  Counter<T> has explicit overloads for 1, 2, and 3 tags:

   public void Add(T, KVP tag1, KVP tag2, KVP tag3) => RecordMeasurement(delta, tag1, tag2, tag3);

  These call into Instrument.netcore.cs which creates the collection expression [tag1, tag2, tag3] inside the callee
  where args are already in stable stack positions.

  For 4+ tags, there is no explicit overload. The call falls through to:

   public void Add(T delta, params ReadOnlySpan<KeyValuePair<string, object?>> tags)

  The C# compiler generates an inline array at the call site in PublishCounterMeasurement, loading four KeyValuePair
  structs out of a heap array and storing them into a stack-allocated inline array. Under JitStressRegs, the JIT
  misallocates registers for this sequence, zeroing one element.

  Corruption pattern confirms register allocator bug

  ┌──────────────────────────────────────┬────────────────────┬─────────────────────────────────┐
  │ JIT Stress Config                    │ Corrupted Position │ Corrupted Element               │
  ├──────────────────────────────────────┼────────────────────┼─────────────────────────────────┤
  │ JitStress=2 + JitStressRegs=2        │ pos 0              │ [K1]→V1 becomes [null]→null     │
  ├──────────────────────────────────────┼────────────────────┼─────────────────────────────────┤
  │ JitStress=2 + JitStressRegs=0x80     │ pos 1              │ [K2]→V2 becomes [null]→null     │
  └──────────────────────────────────────┴────────────────────┴─────────────────────────────────┘

  Different register mask bits (0x2 vs 0x80) force different register assignments, causing the miswrite to hit a
  different slot — a classic register allocator bug fingerprint. Both the Key (string ref) and Value (object ref)
  fields are nulled out, consistent with the entire KeyValuePair<string, object?> struct (16 bytes on x64) being
  written to the wrong offset.

  -----------------------------------------------------------------------------------------------------------------

  Stack Trace (Child Process — from console log)

   Unhandled exception. Xunit.Sdk.EqualException: Assert.Equal() Failure: Collections differ
                         ↓ (pos 1)
   Expected: [["K1"] = "V1", ["K2"] = "V2", ["K3"] = "V3", ["K4"] = "V4"]
   Actual:   [["K1"] = "V1", [null] = null, ["K3"] = "V3", ["K4"] = "V4"]
                         ↑ (pos 1)
     at MetricsTests.<InstrumentPassingVariableTagsParametersTest>b__1(...)  [MetricsTests.cs:1965]
     at Instrument<T>.RecordMeasurement(T, ReadOnlySpan<KVP> tags)           [Instrument.common.cs:99]
     at MetricsTests.PublishCounterMeasurement<T>(...)                       [MetricsTests.cs:1789]
     at MetricsTests.<PassingVariableTagsParametersTest>b__10_1(...)         [MetricsTests.cs:741]
     at MetricsTests.InstrumentPassingVariableTagsParametersTest<T>(...)     [MetricsTests.cs:1996]
     at RemoteExecutor.Program.Main(...)
   [createdump] Writing minidump with heap to coredump.3738.dmp  (78 MB)

  -----------------------------------------------------------------------------------------------------------------

  Next Steps

   - Owner: area-CodeGen-coreclr (JIT team) — this is not a test bug
   - Nature: Pre-existing JIT register allocator bug, exposed only under JitStress=2 + JitStressRegs={2,0x80} — not
  a regression in the test or metrics library
   - Repro key: The params ReadOnlySpan<> + 4-element inline array at call site + struct with GC-reference fields +
  limited register set. Could repro with a small standalone test: DOTNET_TieredCompilation=0 DOTNET_JitStress=2
  DOTNET_JitStressRegs=0x80
    // Call a method with params ReadOnlySpan<KeyValuePair<string,object?>> passing 4+ literal pairs
   - Fix direction: JIT bug in inline-array-at-call-site lowering for params ReadOnlySpan<T> when T contains GC
  reference fields and register count is restricted. The JIT owners should inspect codegen for
  PublishCounterMeasurement under these stress modes with COMPlus_JitDump.

  -----------------------------------------------------------------------------------------------------------------

  Confidence Level

  95/100 — the console log proves the corruption conclusively, the different-position corruption across two register
  masks is a textbook register allocator bug signature, and the code path difference (explicit overloads for ≤3 tags
  vs. params ReadOnlySpan for ≥4) precisely explains why only the 4+ case fails.

[VincentBu]

Failed in: runtime-coreclr libraries-jitstress2-jitstressregs 20260321.1

Failed tests:

net11.0-linux-Release-x64-jitstress2_jitstressregs2-AzureLinux.3.Amd64.Open
- System.Diagnostics.Metrics.Tests.MetricsTests.PassingVariableTagsParametersTest
net11.0-linux-Release-x64-jitstress2_jitstressregs0x80-AzureLinux.3.Amd64.Open
- System.Diagnostics.Metrics.Tests.MetricsTests.PassingVariableTagsParametersTest

Error message:

Microsoft.DotNet.RemoteExecutor.RemoteExecutionException : Remote process failed with an unhandled exception.

Stack trace:

Child exception:
  Xunit.Sdk.EqualException: Assert.Equal() Failure: Collections differ
           ↓ (pos 0)
Expected: [["K1"] = "V1", ["K2"] = "V2", ["K3"] = "V3", ["K4"] = "V4", ["K5"] = "V5", ···]
Actual:   [[null] = null, ["K2"] = "V2", ["K3"] = "V3", ["K4"] = "V4", ["K5"] = "V5", ···]
           ↑ (pos 0)
   at System.Diagnostics.Metrics.Tests.MetricsTests.<>c__DisplayClass32_0`1.<InstrumentPassingVariableTagsParametersTest>b__1(Instrument inst, T measurement, ReadOnlySpan`1 tags, Object state) in /_/src/libraries/System.Diagnostics.DiagnosticSource/tests/MetricsTests.cs:line 1965
   at System.Diagnostics.Metrics.Instrument`1.RecordMeasurement(T measurement, ReadOnlySpan`1 tags) in /_/src/libraries/System.Diagnostics.DiagnosticSource/src/System/Diagnostics/Metrics/Instrument.common.cs:line 99
   at System.Diagnostics.Metrics.Tests.MetricsTests.PublishCounterMeasurement[T](Counter`1 counter, T value, KeyValuePair`2[] tags) in /_/src/libraries/System.Diagnostics.DiagnosticSource/tests/MetricsTests.cs:line 1791
   at System.Diagnostics.Metrics.Tests.MetricsTests.<PassingVariableTagsParametersTest>b__10_3(Instrument`1 instrument, Int32 value, KeyValuePair`2[] tags) in /_/src/libraries/System.Diagnostics.DiagnosticSource/tests/MetricsTests.cs:line 751
   at System.Diagnostics.Metrics.Tests.MetricsTests.InstrumentPassingVariableTagsParametersTest[T](Instrument`1 instrument, Func`4 record) in /_/src/libraries/System.Diagnostics.DiagnosticSource/tests/MetricsTests.cs:line 2015
   at System.Diagnostics.Metrics.Tests.MetricsTests.<PassingVariableTagsParametersTest>b__10_0() in /_/src/libraries/System.Diagnostics.DiagnosticSource/tests/MetricsTests.cs:line 749
   at System.Reflection.MethodBaseInvoker.InterpretedInvoke_Method(Object obj, IntPtr* args) in /_/src/coreclr/System.Private.CoreLib/src/System/Reflection/MethodBaseInvoker.CoreCLR.cs:line 39
   at System.Reflection.MethodBaseInvoker.InvokeWithNoArgs(Object obj, BindingFlags invokeAttr) in /_/src/libraries/System.Private.CoreLib/src/System/Reflection/MethodBaseInvoker.cs:line 57

Child process:
  System.Diagnostics.DiagnosticSource.Tests, Version=11.0.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51 System.Diagnostics.Metrics.Tests.MetricsTests Void <PassingVariableTagsParametersTest>b__10_0()

[mangod9]

more analysis:

Root Cause

A register move ordering bug in genConsumeBlockOp() in src/coreclr/jit/codegenlinear.cpp.

For CpObjUnroll (GC-aware struct copy), the codegen requires rdi=destination and rsi=source. Under JitStressRegs,
the LSRA legitimately allocates the source address to rdi and the destination to rax. Two resolution moves are
needed:

 - rdi → rsi (move source to its required register)
 - rax → rdi (move destination to its required register)

genConsumeBlockOp always does destination first:

 mov  rdi, rax     ; dst → rdi  (CLOBBERS source in rdi!)
 mov  rsi, rdi     ; src → rsi  (copies destination, not source!)
 call ASSIGN_BYREF ; writes wrong data into inline array

The correct order is source first, then destination.

Why Linux-only

On Linux (System V ABI), ReadOnlySpan<T> is decomposed into two integer registers, creating higher register
pressure. On Windows, it's passed by hidden pointer. The extra pressure under JitStressRegs forces the LSRA to
assign the source address to rdi, triggering the conflict.

Fix

 void CodeGen::genConsumeBlockOp(GenTreeBlk* blkNode, regNumber dstReg, regNumber srcReg, regNumber sizeReg)
  {
      // We have to consume the registers, and perform any copies, in the actual execution order: dst, src, size.
      //
      // Note that the register allocator ensures that the registers ON THE NODES will not interfere
      // with one another if consumed (i.e. reloaded or moved to their ASSIGNED reg) in execution order.
      // Further, it ensures that they will not interfere with one another if they are then copied
      // to the REQUIRED register (if a fixed register requirement) in execution order.  This requires,
      // then, that we first consume all the operands, then do any necessary moves.
  
      GenTree* const dstAddr = blkNode->Addr();
  
      // First, consume all the sources in order, and verify that registers have been allocated appropriately,
      // based on the 'gtBlkOpKind'.
  
      // The destination is always in a register; 'genConsumeReg' asserts that.
      genConsumeReg(dstAddr);
      // The source may be a local or in a register; 'genConsumeBlockSrc' will check that.
      genConsumeBlockSrc(blkNode);
  
      // Next, perform any necessary moves.
 -    genCopyRegIfNeeded(dstAddr, dstReg);
 -    genSetBlockSrc(blkNode, srcReg);
 +    // We must be careful about the ordering: if the source address is currently in dstReg,
 +    // we must move the source to srcReg first before overwriting dstReg with the destination.
 +    // Otherwise, copying dst to dstReg would clobber the source address value.
 +    regNumber srcCurReg = REG_NA;
 +    if (srcReg != REG_NA && blkNode->OperIsCopyBlkOp())
 +    {
 +        GenTree* src = blkNode->Data();
 +        if (src->OperIs(GT_IND))
 +        {
 +            srcCurReg = src->AsOp()->gtOp1->GetRegNum();
 +        }
 +        else if (src->OperIs(GT_LCL_VAR, GT_LCL_FLD))
 +        {
 +            // Local source: genSetBlockSrc will emit a LEA, so no conflict possible.
 +        }
 +        else
 +        {
 +            srcCurReg = src->GetRegNum();
 +        }
 +    }
 +
 +    if (srcCurReg == dstReg && srcCurReg != srcReg)
 +    {
 +        // The source is in the register that the destination needs. Move source first.
 +        genSetBlockSrc(blkNode, srcReg);
 +        genCopyRegIfNeeded(dstAddr, dstReg);
 +    }
 +    else
 +    {
 +        genCopyRegIfNeeded(dstAddr, dstReg);
 +        genSetBlockSrc(blkNode, srcReg);
 +    }
      genSetBlockSize(blkNode, sizeReg);
  }

The original code unconditionally did dst→dstReg then src→srcReg. The fix checks if the source's current register
would be clobbered by the destination copy, and if so, reverses the order.

Verification

┌───────────────────────────────────────────┬───────────────────────────────────┬───────────┐
│ Configuration                             │ Before Fix                        │ After Fix │
├───────────────────────────────────────────┼───────────────────────────────────┼───────────┤
│ JitStress=2 JitStressRegs=2 (Linux x64)   │ ❌ FAIL (4 tag corruption errors) │ ✅ PASS   │
├───────────────────────────────────────────┼───────────────────────────────────┼───────────┤
│ JitStress=2 JitStressRegs=0x80            │ ❌ FAIL                           │ ✅ PASS   │
├───────────────────────────────────────────┼───────────────────────────────────┼───────────┤
│ JitStress=2 JitStressRegs=0x10            │ N/A                               │ ✅ PASS   │
├───────────────────────────────────────────┼───────────────────────────────────┼───────────┤
│ JitStress=2 JitStressRegs=0xff            │ N/A                               │ ✅ PASS   │
├───────────────────────────────────────────┼───────────────────────────────────┼───────────┤
│ No stress                                 │ ✅ PASS                           │ ✅ PASS   │
└───────────────────────────────────────────┴───────────────────────────────────┴───────────┘