Skip to content

Regressions in multiple benchmarks after char.Is* update #69727

@performanceautofiler

Description

@performanceautofiler

Run Information

Architecture x64
OS Windows 10.0.18362
Baseline 6e4eba8c98f4e05ece31a883dc23076a78f7c7b9
Compare 15dda0efe52c2aaa7a4e783f443eefb558529248
Diff Diff

Regressions in System.Numerics.Tests.Perf_VectorOf<SByte>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
SquareRootBenchmark - Duration of single invocation 67.72 ns 72.62 ns 1.07 0.02 False
MultiplyOperatorBenchmark - Duration of single invocation 51.64 ns 64.82 ns 1.26 0.03 False
DivisionOperatorBenchmark - Duration of single invocation 110.14 ns 115.99 ns 1.05 0.01 False
GetHashCodeBenchmark - Duration of single invocation 102.10 ns 114.58 ns 1.12 0.04 False

graph
graph
graph
graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Numerics.Tests.Perf_VectorOf&lt;SByte&gt;*'
Details

Payloads

Baseline
Compare

Histogram

System.Numerics.Tests.Perf_VectorOf<SByte>.SquareRootBenchmark


Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 72.616263426816 > 70.3165027262756.
IsChangePoint: Marked as a change because one of 3/17/2022 6:54:53 PM, 5/18/2022 10:04:15 AM, 5/24/2022 7:58:36 AM falls between 5/15/2022 6:57:22 PM and 5/24/2022 7:58:36 AM.
IsRegressionStdDev: Marked as regression because -45.0318918142146 (T) = (0 -72.8527869121108) / Math.Sqrt((0.10060373577538348 / (33)) + (0.29639641692658697 / (25))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (33) + (25) - 2, .025) and -0.08162207084295842 = (67.35512234447401 - 72.8527869121108) / 67.35512234447401 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;SByte&gt;.MultiplyOperatorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 64.81981764655713 > 53.46745717697251.
IsChangePoint: Marked as a change because one of 3/11/2022 8:42:43 PM, 3/29/2022 11:07:04 AM, 4/25/2022 11:05:49 AM, 5/11/2022 7:02:26 PM, 5/20/2022 10:18:01 PM, 5/24/2022 7:58:36 AM falls between 5/15/2022 6:57:22 PM and 5/24/2022 7:58:36 AM.
IsRegressionStdDev: Marked as regression because -28.529296478118287 (T) = (0 -65.17046109518371) / Math.Sqrt((7.8020485166881395 / (43)) + (0.030602190839587127 / (14))) is less than -2.0040447832881556 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (43) + (14) - 2, .025) and -0.23090626675282228 = (52.94510463993808 - 65.17046109518371) / 52.94510463993808 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;SByte&gt;.DivisionOperatorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 115.98556839828156 > 115.70109878652117.
IsChangePoint: Marked as a change because one of 3/17/2022 6:54:53 PM, 5/18/2022 10:04:15 AM, 5/24/2022 7:58:36 AM falls between 5/15/2022 6:57:22 PM and 5/24/2022 7:58:36 AM.
IsRegressionStdDev: Marked as regression because -155.80186125871447 (T) = (0 -116.382354674369) / Math.Sqrt((0.007821590082532347 / (31)) + (0.03303517114606407 / (25))) is less than -2.0048792881871513 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (25) - 2, .025) and -0.05608511730511969 = (110.2016804965014 - 116.382354674369) / 110.2016804965014 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;SByte&gt;.GetHashCodeBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 114.57668798889473 > 107.65639252344216.
IsChangePoint: Marked as a change because one of 3/17/2022 6:54:53 PM, 4/25/2022 11:05:49 AM, 5/20/2022 10:18:01 PM, 5/24/2022 7:58:36 AM falls between 5/15/2022 6:57:22 PM and 5/24/2022 7:58:36 AM.
IsRegressionStdDev: Marked as regression because -283.03839865316087 (T) = (0 -114.59128121623345) / Math.Sqrt((0.07882115090493173 / (44)) + (0.0012095622524388827 / (14))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (44) + (14) - 2, .025) and -0.11986198350643835 = (102.32625350619793 - 114.59128121623345) / 102.32625350619793 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

### Run Information
Architecture x64
OS Windows 10.0.18362
Baseline 6e4eba8c98f4e05ece31a883dc23076a78f7c7b9
Compare 15dda0efe52c2aaa7a4e783f443eefb558529248
Diff Diff

Regressions in System.Numerics.Tests.Perf_VectorOf<Byte>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
SquareRootBenchmark - Duration of single invocation 67.56 ns 72.62 ns 1.07 0.01 False
GetHashCodeBenchmark - Duration of single invocation 105.18 ns 114.93 ns 1.09 0.06 False
MultiplyOperatorBenchmark - Duration of single invocation 50.92 ns 65.31 ns 1.28 0.07 False

graph
graph
graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Numerics.Tests.Perf_VectorOf&lt;Byte&gt;*'
Details

Payloads

Baseline
Compare

Histogram

System.Numerics.Tests.Perf_VectorOf<Byte>.SquareRootBenchmark


Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 72.61622324189207 > 70.29034196284785.
IsChangePoint: Marked as a change because one of 3/17/2022 6:54:53 PM, 5/18/2022 10:04:15 AM, 5/24/2022 7:58:36 AM falls between 5/15/2022 6:57:22 PM and 5/24/2022 7:58:36 AM.
IsRegressionStdDev: Marked as regression because -43.44383568394873 (T) = (0 -73.07887024394701) / Math.Sqrt((0.4275740748536778 / (33)) + (0.10853637117538197 / (24))) is less than -2.0040447832881556 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (33) + (24) - 2, .025) and -0.08529938753935677 = (67.33521743676181 - 73.07887024394701) / 67.33521743676181 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Byte&gt;.GetHashCodeBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 114.93390136649963 > 107.37540770947281.
IsChangePoint: Marked as a change because one of 4/25/2022 11:05:49 AM, 5/20/2022 10:18:01 PM, 5/24/2022 7:58:36 AM falls between 5/15/2022 6:57:22 PM and 5/24/2022 7:58:36 AM.
IsRegressionStdDev: Marked as regression because -48.89414960870857 (T) = (0 -114.77622672101295) / Math.Sqrt((2.158761094640268 / (44)) + (0.028601638891270608 / (13))) is less than -2.0040447832881556 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (44) + (13) - 2, .025) and -0.10674676117540412 = (103.70595220817863 - 114.77622672101295) / 103.70595220817863 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Byte&gt;.MultiplyOperatorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 65.30981681633082 > 53.33145200677543.
IsChangePoint: Marked as a change because one of 3/17/2022 6:54:53 PM, 3/29/2022 11:07:04 AM, 4/25/2022 11:05:49 AM, 5/11/2022 7:02:26 PM, 5/18/2022 10:04:15 AM, 5/24/2022 7:58:36 AM falls between 5/15/2022 6:57:22 PM and 5/24/2022 7:58:36 AM.
IsRegressionStdDev: Marked as regression because -15.112172425806873 (T) = (0 -62.63261316978918) / Math.Sqrt((0.11165366968483725 / (32)) + (14.904766609836882 / (25))) is less than -2.0040447832881556 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (25) - 2, .025) and -0.22978084202205626 = (50.92989826285313 - 62.63261316978918) / 50.92989826285313 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS Windows 10.0.18362
Baseline 6e4eba8c98f4e05ece31a883dc23076a78f7c7b9
Compare 15dda0efe52c2aaa7a4e783f443eefb558529248
Diff Diff

Regressions in System.Tests.Perf_String

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Format_MultipleArgs - Duration of single invocation 244.70 ns 295.96 ns 1.21 0.06 False
Concat_CharEnumerable - Duration of single invocation 7.03 μs 8.46 μs 1.20 0.06 False

graph
graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Tests.Perf_String*'
Details

Payloads

Baseline
Compare

Histogram

System.Tests.Perf_String.Format_MultipleArgs


Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 295.96439696558804 > 258.5723078295418.
IsChangePoint: Marked as a change because one of 4/16/2022 5:36:25 PM, 5/18/2022 10:04:15 AM, 5/24/2022 7:58:36 AM falls between 5/15/2022 6:57:22 PM and 5/24/2022 7:58:36 AM.
IsRegressionStdDev: Marked as regression because -43.214416060613296 (T) = (0 -297.6240622174258) / Math.Sqrt((14.284845301254313 / (33)) + (23.17517385721064 / (24))) is less than -2.0040447832881556 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (33) + (24) - 2, .025) and -0.2073050297315947 = (246.5193591412378 - 297.6240622174258) / 246.5193591412378 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_String.Concat_CharEnumerable

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 8.459203054806828 > 7.38004250196541.
IsChangePoint: Marked as a change because one of 3/15/2022 11:00:34 PM, 3/23/2022 6:35:24 PM, 3/29/2022 11:07:04 AM, 5/5/2022 12:13:02 AM, 5/11/2022 7:02:26 PM, 5/20/2022 10:18:01 PM, 5/24/2022 7:58:36 AM falls between 5/15/2022 6:57:22 PM and 5/24/2022 7:58:36 AM.
IsRegressionStdDev: Marked as regression because -190.01315732087696 (T) = (0 -8472.773343347046) / Math.Sqrt((78.21027311391161 / (44)) + (730.5417624546898 / (13))) is less than -2.0040447832881556 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (44) + (13) - 2, .025) and -0.20591487712144863 = (7026.0127842288375 - 8472.773343347046) / 7026.0127842288375 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS Windows 10.0.18362
Baseline 6e4eba8c98f4e05ece31a883dc23076a78f7c7b9
Compare 0c6d412f86205043726e273127d289b65d5b6c7a
Diff Diff

Regressions in System.Globalization.Tests.StringEquality

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Compare_Same - Duration of single invocation 975.11 ns 1.61 μs 1.65 0.01 True

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Globalization.Tests.StringEquality*'
Details

Payloads

Baseline
Compare

Histogram

System.Globalization.Tests.StringEquality.Compare_Same(Count: 1024, Options: (en-US, OrdinalIgnoreCase))


Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.6128745088408643 > 1023.9368522813845.
IsChangePoint: Marked as a change because one of 5/18/2022 10:04:15 AM, 5/24/2022 7:58:36 AM falls between 5/15/2022 6:57:22 PM and 5/24/2022 7:58:36 AM.
IsRegressionStdDev: Marked as regression because -14.572122467414548 (T) = (0 -1455.1143152156847) / Math.Sqrt((0.005368131784794816 / (33)) + (26039.113951732077 / (24))) is less than -2.0040447832881556 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (33) + (24) - 2, .025) and -0.4922315216536375 = (975.1263755661582 - 1455.1143152156847) / 975.1263755661582 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Metadata

Metadata

Assignees

Labels

Type

No type

Projects

No projects

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions