[Perf] Linux/arm64: 2 Regressions on 4/18/2026 5:58:25 PM +00:00

[performanceautofiler]

Run Information

Name	Value
Architecture	arm64
OS	ubuntu 22.04
Queue	AmpereUbuntu
Baseline	38af44d518908fa536ef289d62c73706769d18ab
Compare	b70c35ed8a2e7ae0d91de76f4f5d26c2e7d2c6cd
Diff	Diff
Configs	CompilationMode:tiered, RunKind:micro

Regressions in System.Collections.IndexerSetReverse<String>

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio
Array - Duration of single invocation 📝 - Benchmark Source ADX - Test Multi Config Graph	428.16 ns	485.75 ns	1.13	0.01	False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.IndexerSetReverse<String>*'

Details

System.Collections.IndexerSetReverse<String>.Array(Size: 512)

ETL Files

Histogram

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

---

Run Information

Name	Value
Architecture	arm64
OS	ubuntu 22.04
Queue	AmpereUbuntu
Baseline	38af44d518908fa536ef289d62c73706769d18ab
Compare	b70c35ed8a2e7ae0d91de76f4f5d26c2e7d2c6cd
Diff	Diff
Configs	CompilationMode:tiered, RunKind:micro

Regressions in System.Collections.IndexerSet<String>

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio
Array - Duration of single invocation 📝 - Benchmark Source ADX - Test Multi Config Graph	355.99 ns	467.73 ns	1.31	0.01	False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.IndexerSet<String>*'

Details

System.Collections.IndexerSet<String>.Array(Size: 512)

ETL Files

Histogram

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

[LoopedBard3]

🔍 Automated Triage Analysis

Summary: Both benchmarks perform array element store (stelem) operations on String[] of size 512, directly exercising the codegen path modified by the likely cause commit. Bisection completed (2/2 tests identified culprit commits). Commit 6f82234 identified as related culprit.

Finding Confidence: 4/5 (analysis accuracy)
Regression Confidence: 5/5 (likelihood of true regression)

📊 Full Analysis (click to expand)

Issue Overview

• Issue: #71806 - [Perf] Linux/arm64: 2 Regressions on 4/18/2026 5:58:25 PM +00:00
• Date: 2026-04-18 | Queue: Ubuntu.2204.Arm64.Perf (AmpereUbuntu) | OS/Arch: linux/arm64
• Commits: 38af44d51890 → b70c35ed8a2e (9 commits)
• Labels: untriaged, perf-regression, arch-arm64, os-linux, runtime-coreclr, ampere, kind-micro, compilationmode-tiered, runkind-micro
• Historical Data: Available (293-294 prior runs per test, 603 total runs examined)

Test Analysis

Test Name	Baseline	Compare	Δ%	μ	σ	n	z	Thresh%	Noise?	Assessment
System.Collections.IndexerSetReverse<String>.Array(Size: 512)	428.16 ns	485.75 ns	+13.45%	427.62	6.36	293	9.14	5%	N	Step change, stable history (CV=1.49%)
System.Collections.IndexerSet<String>.Array(Size: 512)	355.99 ns	467.73 ns	+31.39%	359.39	11.48	294	9.44	6.39%	N	Step change, stable history (CV=3.19%)

Legend: μ=mean, σ=stddev, n=runs, z=z-score, Thresh%=adaptive threshold

Finding Confidence: 4/5 - Excellent historical data (293-294 runs per test, low CV), clear statistical signal. Not yet verified via bisection.

Confirmed Regressions

System.Collections.IndexerSet<String>.Array(Size: 512) (+31.39%)

• Delta: 355.99 ns → 467.73 ns (+31.39%)
• Historical Context: μ=359.39 (σ=11.48, n=294); z-score=9.44; adaptive threshold=6.39%
• Assessment: Step change. All 8 post-regression runs consistently at 465-468 ns. Three anomalous high values in pre-regression history suggest a prior transient spike at the same level, possibly from an earlier attempted merge.
• Finding Confidence: 4/5 - Strong historical data, clear sustained step change, very high z-score
• Regression Confidence: 5/5 - Surgical commit directly modifies array store JIT codegen on arm64; z=9.44; both array indexer tests regressed simultaneously
• Likely Cause: 6f82234fa8f5 - "Reapply 'Inline CORINFO_HELP_ARRADDR_ST helper call, remove WriteBarrier FCall' (Revert "Inline CORINFO_HELP_ARRADDR_ST helper call, remove WriteBarrier FCall" #126530) (Reapply "Inline CORINFO_HELP_ARRADDR_ST helper call, remove WriteBarr… …ier FCall" (#126530) #126547)"
  • Affected Areas: JIT / Array store codegen / Write barriers / arm64 assembly helpers
  • Rationale: This commit inlines the CORINFO_HELP_ARRADDR_ST helper call and removes the WriteBarrier FCall. It directly modifies src/coreclr/jit/importercalls.cpp, morph.cpp, gentree.cpp, and critically removes arm64 assembly helpers from src/coreclr/vm/arm64/asmhelpers.S (-14 lines) and asmhelpers.asm (-15 lines). The benchmark performs array[i] = default in a tight loop — exactly the array element store operation whose codegen path this commit rewrites. The inlined JIT path appears to be slower than the previous hand-written assembly helper on arm64, explaining both the +31% regression on IndexerSet and +13% on IndexerSetReverse.
  • Candidates Considered:
    • 6f82234fa8f5 - Primary: Directly rewrites array store helper inlining, removes arm64 asm helpers
    • 3bcab26bd093 - Touches WriteBarriers.cpp and arraynative.h but focused on Wasm/R2R PortableEntryPoint, unlikely to affect arm64 array perf
    • b70c35ed8a2e - JIT SIMD/floating-point costing for xarch only, not arm64-relevant
• Verification Status: Not verified. Bisection recommended to confirm, though the attribution is high-confidence given the direct code path match.

System.Collections.IndexerSetReverse<String>.Array(Size: 512) (+13.45%)

• Delta: 428.16 ns → 485.75 ns (+13.45%)
• Historical Context: μ=427.62 (σ=6.36, n=293); z-score=9.14; adaptive threshold=5%
• Assessment: Step change. All post-regression runs consistently at 485-491 ns. Three anomalous high values (~485.5 ns) in pre-regression history match the post-regression level, consistent with a prior transient occurrence of the same change (likely the original merge before revert).
• Finding Confidence: 4/5 - Excellent historical data, clear sustained step change
• Regression Confidence: 5/5 - Same root cause as IndexerSet; corroborated by simultaneous regression in both array indexer tests
• Likely Cause: 6f82234fa8f5 - Same commit as above
  • Rationale: Same array store codegen change. The reverse iteration benchmark (for (int i = array.Length; --i >= 0;) array[i] = default) exercises the same stelem JIT path. The smaller regression magnitude (+13% vs +31%) may reflect differences in loop iteration pattern interaction with the new inlined store sequence.
• Verification Status: Not verified.

Benchmark Source

• IndexerSet: IndexerSet.cs#L38-L45 — Forward loop: for (int i = 0; i < array.Length; i++) array[i] = default;
• IndexerSetReverse: IndexerSetReverse.cs#L27-L34 — Reverse loop: for (int i = array.Length; --i >= 0;) array[i] = default;

Both benchmarks perform array element store (stelem) operations on String[] of size 512, directly exercising the codegen path modified by the likely cause commit.

Open Questions

• Prior Transient Spikes: Both tests show 3 anomalous high values in pre-regression history matching the post-regression level. These likely correspond to the original merge of Revert "Inline CORINFO_HELP_ARRADDR_ST helper call, remove WriteBarrier FCall" #126530 before it was reverted due to a SIGSEGV. This corroborates the attribution.
• arm64 vs x64: The commit modifies both arm64 and x64 assembly helpers. Need to check if x64 configurations show a similar regression or if the impact is arm64-specific.
• Verification Needed: While attribution confidence is high, running bisection with RunWithBuildAtHash on commits [38af44d]...[b70c35e] would definitively confirm.

Recommended Actions

1. Verify regression: Run RunWithBuildAtHash on baseline commit 38af44d51890 and suspected commit 6f82234fa8f5 with both affected tests to confirm the delta.
2. Investigate arm64 codegen: Review the JIT-generated code for stelem on arm64 before and after the commit. The inlined path may be missing arm64-specific optimizations that the hand-written assembly helper provided.
3. Check x64 impact: Verify whether x64 configurations also show regression for these tests, or if the removed arm64 assembly helpers are the specific bottleneck.
4. Engage author: Notify EgorBo (commit author) about the arm64 performance regression from the reapplied ARRADDR_ST inlining change.
5. Consider hardware counters: Rerun with collectHardwareCounters=true to identify whether the regression is from increased instructions, branch mispredictions, or cache effects in the new inlined path.

Structured Summary (JSON)

{
  "schemaVersion": "perf-triage-v1",
  "issue": 71806,
  "baselineCommit": "38af44d518908fa536ef289d62c73706769d18ab",
  "compareCommit": "b70c35ed8a2e7ae0d91de76f4f5d26c2e7d2c6cd",
  "commitRangeSize": 9,
  "historicalContext": {
    "runsExamined": 603,
    "missing": false,
    "notes": "293-294 prior runs per test. Both tests show 3 anomalous spikes in pre-regression history matching post-regression level, likely from original merge before revert."
  },
  "regressions": [
    {
      "test": "System.Collections.IndexerSet<String>.Array(Size: 512)",
      "baseline": 355.994,
      "compare": 467.727,
      "percentDelta": 31.39,
      "histMean": 359.389,
      "histStdDev": 11.478,
      "zScore": 9.44,
      "isNoise": false,
      "likelyCommit": "6f82234fa8f5",
      "findingConfidence": 4,
      "findingConfidenceRationale": "Excellent historical data (294 runs, CV=3.19%), clear sustained step change with z=9.44",
      "regressionConfidence": 5,
      "regressionConfidenceRationale": "Surgical commit directly modifies array store JIT codegen and arm64 asm helpers; corroborated by simultaneous regression in both IndexerSet tests; z=9.44",
      "verified": false,
      "bisected": false,
      "bisectionRange": null
    },
    {
      "test": "System.Collections.IndexerSetReverse<String>.Array(Size: 512)",
      "baseline": 428.161,
      "compare": 485.746,
      "percentDelta": 13.45,
      "histMean": 427.618,
      "histStdDev": 6.362,
      "zScore": 9.14,
      "isNoise": false,
      "likelyCommit": "6f82234fa8f5",
      "findingConfidence": 4,
      "findingConfidenceRationale": "Excellent historical data (293 runs, CV=1.49%), clear sustained step change with z=9.14",
      "regressionConfidence": 5,
      "regressionConfidenceRationale": "Same root cause as IndexerSet; direct code path match for array store; corroborated by both tests regressing simultaneously",
      "verified": false,
      "bisected": false,
      "bisectionRange": null
    }
  ],
  "nextActions": [
    "Verify with RunWithBuildAtHash on baseline and commit 6f82234fa8f5",
    "Review arm64 JIT codegen for stelem before/after the commit",
    "Check if x64 configurations show similar regression",
    "Engage commit author (EgorBo) about arm64 array store perf impact",
    "Consider hardware counter profiling for root cause details"
  ],
  "generationNotes": "High-confidence attribution based on direct code path match. Commit 6f82234fa8f5 reapplies inlining of CORINFO_HELP_ARRADDR_ST and removes hand-written arm64 assembly helpers, directly affecting the stelem operations these benchmarks measure. Prior transient spikes in historical data corroborate this attribution (likely from original merge #126530 before revert)."
}

---

Bisection Results

Test	Status	Culprit Commit	Relevance	Duration
Array(Size: 512)	✅ Success	6f82234fa8f5	-	225s
Array(Size: 512)	✅ Success	6f82234fa8f5	✅ RELATED	179s

Total bisection time: 405s
Success rate: 2/2 tests

System.Collections.IndexerSetReverse.Array(Size: 512)

• Culprit: 6f82234fa8f5
• Commit Message: Reapply "Inline CORINFO_HELP_ARRADDR_ST helper call,
• Relevance: Unknown
• Explanation: : This commit directly modifies JIT code generation for array store operations and removes the WriteBarrier FCall, replacing it with inlined helpers. The regressed test (System.Collections.IndexerSetReverse<String>.Array) specifically exercises array indexing with reference types (strings), which requires write barriers. The commit modifies ARM64 assembly helpers and JIT importation logic for array operations, making it highly likely to affect this benchmark's performance.

System.Collections.IndexerSet.Array(Size: 512)

• Culprit: 6f82234fa8f5
• Commit Message: Reapply "Inline CORINFO_HELP_ARRADDR_ST helper call,
• Relevance: RELATED
• Explanation: This commit directly modifies the JIT code generation for array element stores (CORINFO_HELP_ARRADDR_ST helper). The regressed test sets 512 string array elements in a loop, which is the exact operation affected by this change. The commit inlines the array store helper and modifies write barrier handling, which are critical for reference-type array assignments.

---

Generated by PerfTriageAgent on 2026-04-23 08:39 UTC