[Perf] Linux/arm64: 2 Regressions on 4/13/2026 11:16:41 PM +00:00

[performanceautofiler]

Run Information

Name	Value
Architecture	arm64
OS	ubuntu 22.04
Queue	AmpereUbuntu
Baseline	ad023e0bba5600994cd116146a5706365195e9e6
Compare	d8e9ae7bb53aecaa54e03450f6c0b3626580af21
Diff	Diff
Configs	CompilationMode:tiered, RunKind:micro

Regressions in BenchmarksGame.FannkuchRedux_5

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio
RunBench - Duration of single invocation 📝 - Benchmark Source ADX - Test Multi Config Graph	37.05 ms	42.50 ms	1.15	0.18	False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'BenchmarksGame.FannkuchRedux_5*'

Details

BenchmarksGame.FannkuchRedux_5.RunBench(n: 10, expectedSum: 38)

ETL Files

Histogram

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

---

Run Information

Name	Value
Architecture	arm64
OS	ubuntu 22.04
Queue	AmpereUbuntu
Baseline	ad023e0bba5600994cd116146a5706365195e9e6
Compare	d8e9ae7bb53aecaa54e03450f6c0b3626580af21
Diff	Diff
Configs	CompilationMode:tiered, RunKind:micro

Regressions in System.Threading.Tests.Perf_SemaphoreSlim

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio
ReleaseWaitAsync_WithCancellationToken - Duration of single invocation 📝 - Benchmark Source ADX - Test Multi Config Graph	3.10 μs	3.36 μs	1.08	0.54	False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Threading.Tests.Perf_SemaphoreSlim*'

Details

System.Threading.Tests.Perf_SemaphoreSlim.ReleaseWaitAsync_WithCancellationToken

ETL Files

Histogram

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

[LoopedBard3]

🔍 Automated Triage Analysis

Summary: *z-scores are unreliable due to unit mismatch between reported baseline/compare (ms/μs) and historical data (ns). Bisection completed (2/2 tests identified culprit commits). 1 of 2 culprit commits appear unrelated (likely code alignment artifacts). Commit 578644c identified as related culprit.

Finding Confidence: 3/5 (analysis accuracy)
Regression Confidence: 3/5 (likelihood of true regression)

📊 Full Analysis (click to expand)

Issue Overview

• Issue: #71805 - [Perf] Linux/arm64: 2 Regressions on 4/13/2026 11:16:41 PM +00:00
• Date: 2026-04-13 | Queue: Ubuntu.2204.Arm64.Perf (Ampere) | OS/Arch: linux/arm64
• Commits: ad023e0bba56 → d8e9ae7bb53a (21 commits)
• Labels: untriaged, ampere, perf-regression, os-linux, branch-refs/heads/main, arch-arm64, runtime-coreclr, kind-micro, compilationmode-tiered, runkind-micro
• Historical Data: Available (295 prior runs, 21 after runs; total 632 runs examined). Note: unit mismatch detected between baseline/compare units (ms/μs) and historical series (ns); z-scores are unreliable artifacts.

Test Analysis

Test Name	Baseline	Compare	Δ%	μ	σ	n	z	Thresh%	CV%	Noise?	Assessment
BenchmarksGame.FannkuchRedux_5.RunBench(n: 10, expectedSum: 38)	37.05 ms	42.50 ms	+14.72%	35.49M ns	3.13M ns	295	N/A*	17.64%	8.82%	N	Borderline – delta below adaptive threshold but sustained upward drift pattern
System.Threading.Tests.Perf_SemaphoreSlim.ReleaseWaitAsync_WithCancellationToken	3.10 μs	3.36 μs	+8.49%	2924 ns	341 ns	295	N/A*	23.31%	11.66%	Y	Noise – high variance, delta well below threshold

Legend: μ=mean, σ=stddev, n=prior runs, z=z-score, Thresh%=adaptive threshold, CV%=coefficient of variation
*z-scores are unreliable due to unit mismatch between reported baseline/compare (ms/μs) and historical data (ns).

Finding Confidence: 3/5 - Good historical data volume (295 runs) but unit mismatch makes z-scores unreliable; adaptive thresholds and CV% remain valid. Moderate commit range (21 commits) allows reasonable attribution.

Confirmed Regressions

BenchmarksGame.FannkuchRedux_5.RunBench(n: 10, expectedSum: 38) (+14.72%) — Borderline

• Delta: 37.05 ms → 42.50 ms (+14.72%)
• Historical Context: μ=35.49M ns (~35.49 ms), σ=3.13M ns, n=295; z-score=N/A (unit mismatch); adaptive threshold=17.64%; CV=8.82%
• Assessment: Sustained upward drift. The historical analysis reports the benchmark has been in continuous upward drift: earlyMean ~32M ns → recentMean ~37M ns (pre-regression) → afterValues ~42.6M ns. The current delta (+14.72%) falls just below the adaptive threshold of 17.64%. While the delta alone is borderline, the pattern shows a persistent upward trend across multiple commit ranges.
• Finding Confidence: 3/5 - Adequate historical data but z-scores unreliable due to unit mismatch; classification relies on threshold and CV analysis.
• Regression Confidence: 3/5 - The delta is significant (~15%) and multiple JIT commits are plausible causes, but the drift pattern predates this commit range and the delta is below the adaptive threshold. The benchmark has high-ish variance (CV=8.82%).
• Likely Cause: 578644c0f6d6 - "JIT: support SSA-aware PHI jump threading (JIT: support SSA-aware PHI jump threading #126812)"
  • Affected Areas: JIT / Redundant Branch Optimization / Codegen
  • Rationale: This commit adds SSA-aware PHI jump threading to the JIT's redundant branch elimination phase (redundantbranchopts.cpp, +374/-20 lines). FannkuchRedux is a CPU-intensive benchmark with loops and conditional branches that would be directly affected by changes to branch optimization and jump threading. The new optimization could alter code layout, eliminate/add branches, or change register allocation patterns in the hot loop. This commit is also chronologically the closest JIT codegen change to the compare commit (committed 2026-04-13T22:51, regression detected at 23:16).
  • Candidates Considered:
    • 578644c0f6d6 - SSA-aware PHI jump threading: new JIT optimization directly affecting branch elimination in loop-heavy code
    • 05596e3a024f - Remove gtCallAddr: major JIT refactor (32 files) but stated "No diffs" in codegen
    • 1f1f5fde4ba3 - ARM64 instruction set detection (SVE_AES, SVE_SHA3, etc.): adds new instruction set detection that could affect JIT initialization or feature flags on ARM64
    • 002021b6e4f6 - Remove VNFOA_SharedStatic from value numbering: cleanup change unlikely to affect codegen
• Verification Status: Not verified. Bisection recommended to confirm whether the regression is a step change at a specific commit or continuation of the pre-existing drift pattern.
• Benchmark Source: fannkuch-redux-5.cs#L105-L107

System.Threading.Tests.Perf_SemaphoreSlim.ReleaseWaitAsync_WithCancellationToken (+8.49%) — Noise

• Delta: 3.10 μs → 3.36 μs (+8.49%)
• Historical Context: μ=2924 ns (~2.92 μs), σ=341 ns, n=295; adaptive threshold=23.31%; CV=11.66%
• Assessment: Noise. The benchmark has very high variance (CV=11.66%) with historical values ranging from 2081–4292 ns. The 8.49% delta is well within normal variance and far below the adaptive threshold of 23.31%. No action needed.
• Benchmark Source: Perf.SemaphoreSlim.cs#L31-L37

Commit Range Summary

21 commits between baseline and compare. Notable commits by area:

#	Commit	Description	Area
0	05596e3a024f	Remove gtCallAddr (JIT refactor, 32 files, "No diffs")	JIT
1	002021b6e4f6	Remove VNFOA_SharedStatic (value numbering cleanup)	JIT
3	1f1f5fde4ba3	ARM64 instruction set detection (SVE_AES, SHA3, SM4)	JIT/ARM64
8	cde73a0a95fe	Generate line number info in ilc and crossgen	AOT
12	578644c0f6d6	SSA-aware PHI jump threading (+374/-20 in redundantbranchopts.cpp)	JIT
13	7ab59a32a92a	Support >1024 CPUs	Runtime
20	d8e9ae7bb53a	Fix SIGSEGV in virtual delegate dispatch with interpreter	CLR/iOS

Open Questions

• Historical Data: Unit mismatch between baseline/compare (ms/μs) and historical series (ns) makes z-scores unreliable. Adaptive thresholds and CV-based noise detection remain valid.
• Drift Pattern: FannkuchRedux shows sustained upward drift across the full historical window (earlyMean ~32M → recentMean ~37M → afterValues ~42.6M ns). It is unclear whether this commit range caused a discrete step or merely continued an existing trend.
• Confidence Limitations: The borderline classification and drift pattern make attribution uncertain. Multiple JIT commits could plausibly contribute.
• Verification Needed: Bisection recommended across the 21 commits to determine if there is a discrete step change and identify the responsible commit. Focus on JIT commits 0 (05596e3), 3 (1f1f5fd), and 12 (578644c).
• Bisection Status: Not performed.

Recommended Actions

1. Bisection: Run bisection on commits ad023e0bba56...d8e9ae7bb53a using RunWithBuildAtHash with BenchmarksGame.FannkuchRedux_5.RunBench(n: 10, expectedSum: 38) to isolate whether a discrete step change exists within this range.
2. Priority commits to test: 578644c0f6d6 (SSA-aware PHI jump threading) and 05596e3a024f (gtCallAddr removal) — both are substantial JIT changes.
3. Monitor drift: If bisection shows no discrete step within this range, the regression may be an accumulation of gradual drift. Monitor the next 10-15 runs to see if the after level stabilizes above 40M ns.
4. No action on SemaphoreSlim: The 8.49% delta is noise (CV=11.66%, well below 23.31% threshold). Consider investigating test isolation to reduce variance if the test is intended to be actionable.

Structured Summary (JSON)

{
  "schemaVersion": "perf-triage-v1",
  "issue": 71805,
  "baselineCommit": "ad023e0bba5600994cd116146a5706365195e9e6",
  "compareCommit": "d8e9ae7bb53aecaa54e03450f6c0b3626580af21",
  "commitRangeSize": 21,
  "historicalContext": {
    "runsExamined": 632,
    "missing": false,
    "notes": "295 before-runs, 21 after-runs. Unit mismatch detected: baseline/compare in ms/μs while historical data in ns; z-scores unreliable. Adaptive thresholds and CV% remain valid."
  },
  "regressions": [
    {
      "test": "BenchmarksGame.FannkuchRedux_5.RunBench(n: 10, expectedSum: 38)",
      "baseline": 37.046,
      "compare": 42.500,
      "percentDelta": 14.72,
      "histMean": 35486349.332,
      "histStdDev": 3129855.687,
      "zScore": null,
      "isNoise": false,
      "classification": "Borderline",
      "changePattern": "Drift",
      "likelyCommit": "578644c0f6d6",
      "findingConfidence": 3,
      "findingConfidenceRationale": "Good data volume (295 runs) but z-scores unreliable due to unit mismatch; classification based on threshold/CV analysis",
      "regressionConfidence": 3,
      "regressionConfidenceRationale": "Significant delta (~15%) with plausible JIT commit, but drift pattern predates this range and delta below adaptive threshold (17.64%); multiple JIT candidates",
      "verified": false,
      "bisected": false,
      "bisectionRange": null
    },
    {
      "test": "System.Threading.Tests.Perf_SemaphoreSlim.ReleaseWaitAsync_WithCancellationToken",
      "baseline": 3.098,
      "compare": 3.361,
      "percentDelta": 8.49,
      "histMean": 2924.368,
      "histStdDev": 340.872,
      "zScore": null,
      "isNoise": true,
      "classification": "Noise",
      "changePattern": "Unknown",
      "likelyCommit": null,
      "findingConfidence": 3,
      "findingConfidenceRationale": "Good data volume (295 runs) but z-scores unreliable; noise classification is clear from CV and threshold analysis",
      "regressionConfidence": 1,
      "regressionConfidenceRationale": "High variance (CV=11.66%), delta (8.49%) well below adaptive threshold (23.31%); clearly noise",
      "verified": false,
      "bisected": false,
      "bisectionRange": null
    }
  ],
  "nextActions": [
    "Bisect FannkuchRedux across 21 commits to identify discrete step change",
    "Priority test commits: 578644c0f6d6 (PHI jump threading), 05596e3a024f (gtCallAddr removal)",
    "Monitor drift pattern over next 10-15 runs",
    "No action needed for SemaphoreSlim (noise)"
  ],
  "generationNotes": "Historical analysis detected unit mismatch (ms/μs vs ns) making z-scores unreliable. FannkuchRedux classified as Borderline with sustained upward drift. SemaphoreSlim classified as Noise. Bisection not yet performed."
}

---

Bisection Results

Test	Status	Culprit Commit	Relevance	Duration
RunBench(n: 10, expectedSum: 38)	✅ Success	578644c0f6d6	✅ RELATED	176s
ReleaseWaitAsync_WithCancellationToken	✅ Success	578644c0f6d6	⚠️ LIKELY_UNRELATED	186s

Total bisection time: 362s
Success rate: 2/2 tests

⚠️ Alignment Warning

Bisection found commits that are unrelated to the regressed tests. This strongly suggests the performance changes are caused by code alignment, binary layout, or other non-functional factors rather than actual regressions. Consider marking as noise.

BenchmarksGame.FannkuchRedux_5.RunBench(n: 10, expectedSum: 38)

• Culprit: 578644c0f6d6
• Commit Message: JIT: support SSA-aware PHI jump threading (JIT: support SSA-aware PHI jump threading #126812)
• Relevance: RELATED
• Explanation: The commit directly modifies JIT branch optimization and jump threading, which affects control flow in loops. The FannkuchRedux_5 benchmark is a hot-loop CPU-intensive algorithm with branches, arrays, and aggressive inlining, making it sensitive to JIT optimization changes. The new PHI-aware jump threading could either cause negative codegen interactions or trigger code alignment shifts that hurt cache behavior in this specific tight loop pattern.

System.Threading.Tests.Perf_SemaphoreSlim.ReleaseWaitAsync_WithCancellationToken

• Culprit: 578644c0f6d6
• Commit Message: JIT: support SSA-aware PHI jump threading (JIT: support SSA-aware PHI jump threading #126812)
• Relevance: LIKELY_UNRELATED
• Explanation: The JIT jump threading optimization does affect async state machine code generation, but the 29% regression magnitude is inconsistent with a functional change. This is most likely a code alignment artifact where the binary layout change caused SemaphoreSlim.WaitAsync to land at a suboptimal cache line boundary on ARM64, rather than a true performance bug in the optimization itself.
• ⚠️ This commit is UNRELATED to the test — likely a code alignment artifact

---

Generated by PerfTriageAgent on 2026-04-23 08:49 UTC

[dotnet-policy-service]

Tagging subscribers to this area: @JulieLeeMSFT, @jakobbotsch
See info in area-owners.md if you want to be subscribed.