-
Notifications
You must be signed in to change notification settings - Fork 5.3k
Closed
Labels
arch-x64area-CodeGen-coreclrCLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMICLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMIruntime-coreclrspecific to the CoreCLR runtimespecific to the CoreCLR runtimetenet-performancePerformance related issuePerformance related issuetenet-performance-benchmarksIssue from performance benchmarkIssue from performance benchmark
Milestone
Description
Run Information
| Architecture | x64 |
|---|---|
| OS | Windows 10.0.18362 |
| Baseline | 6e4eba8c98f4e05ece31a883dc23076a78f7c7b9 |
| Compare | 15dda0efe52c2aaa7a4e783f443eefb558529248 |
| Diff | Diff |
Regressions in System.Numerics.Tests.Perf_VectorOf<SByte>
| Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL |
|---|---|---|---|---|---|---|---|---|---|---|
| SquareRootBenchmark - Duration of single invocation | 67.72 ns | 72.62 ns | 1.07 | 0.02 | False | |||||
| MultiplyOperatorBenchmark - Duration of single invocation | 51.64 ns | 64.82 ns | 1.26 | 0.03 | False | |||||
| DivisionOperatorBenchmark - Duration of single invocation | 110.14 ns | 115.99 ns | 1.05 | 0.01 | False | |||||
| GetHashCodeBenchmark - Duration of single invocation | 102.10 ns | 114.58 ns | 1.12 | 0.04 | False |
Repro
git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Numerics.Tests.Perf_VectorOf<SByte>*'Details
Payloads
Histogram
System.Numerics.Tests.Perf_VectorOf<SByte>.SquareRootBenchmark
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 72.616263426816 > 70.3165027262756.
IsChangePoint: Marked as a change because one of 3/17/2022 6:54:53 PM, 5/18/2022 10:04:15 AM, 5/24/2022 7:58:36 AM falls between 5/15/2022 6:57:22 PM and 5/24/2022 7:58:36 AM.
IsRegressionStdDev: Marked as regression because -45.0318918142146 (T) = (0 -72.8527869121108) / Math.Sqrt((0.10060373577538348 / (33)) + (0.29639641692658697 / (25))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (33) + (25) - 2, .025) and -0.08162207084295842 = (67.35512234447401 - 72.8527869121108) / 67.35512234447401 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Numerics.Tests.Perf_VectorOf<SByte>.MultiplyOperatorBenchmark
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 64.81981764655713 > 53.46745717697251.
IsChangePoint: Marked as a change because one of 3/11/2022 8:42:43 PM, 3/29/2022 11:07:04 AM, 4/25/2022 11:05:49 AM, 5/11/2022 7:02:26 PM, 5/20/2022 10:18:01 PM, 5/24/2022 7:58:36 AM falls between 5/15/2022 6:57:22 PM and 5/24/2022 7:58:36 AM.
IsRegressionStdDev: Marked as regression because -28.529296478118287 (T) = (0 -65.17046109518371) / Math.Sqrt((7.8020485166881395 / (43)) + (0.030602190839587127 / (14))) is less than -2.0040447832881556 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (43) + (14) - 2, .025) and -0.23090626675282228 = (52.94510463993808 - 65.17046109518371) / 52.94510463993808 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Numerics.Tests.Perf_VectorOf<SByte>.DivisionOperatorBenchmark
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 115.98556839828156 > 115.70109878652117.
IsChangePoint: Marked as a change because one of 3/17/2022 6:54:53 PM, 5/18/2022 10:04:15 AM, 5/24/2022 7:58:36 AM falls between 5/15/2022 6:57:22 PM and 5/24/2022 7:58:36 AM.
IsRegressionStdDev: Marked as regression because -155.80186125871447 (T) = (0 -116.382354674369) / Math.Sqrt((0.007821590082532347 / (31)) + (0.03303517114606407 / (25))) is less than -2.0048792881871513 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (25) - 2, .025) and -0.05608511730511969 = (110.2016804965014 - 116.382354674369) / 110.2016804965014 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Numerics.Tests.Perf_VectorOf<SByte>.GetHashCodeBenchmark
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 114.57668798889473 > 107.65639252344216.
IsChangePoint: Marked as a change because one of 3/17/2022 6:54:53 PM, 4/25/2022 11:05:49 AM, 5/20/2022 10:18:01 PM, 5/24/2022 7:58:36 AM falls between 5/15/2022 6:57:22 PM and 5/24/2022 7:58:36 AM.
IsRegressionStdDev: Marked as regression because -283.03839865316087 (T) = (0 -114.59128121623345) / Math.Sqrt((0.07882115090493173 / (44)) + (0.0012095622524388827 / (14))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (44) + (14) - 2, .025) and -0.11986198350643835 = (102.32625350619793 - 114.59128121623345) / 102.32625350619793 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
Docs
Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
| Architecture | x64 |
|---|---|
| OS | Windows 10.0.18362 |
| Baseline | 6e4eba8c98f4e05ece31a883dc23076a78f7c7b9 |
| Compare | 15dda0efe52c2aaa7a4e783f443eefb558529248 |
| Diff | Diff |
Regressions in System.Numerics.Tests.Perf_VectorOf<Byte>
| Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL |
|---|---|---|---|---|---|---|---|---|---|---|
| SquareRootBenchmark - Duration of single invocation | 67.56 ns | 72.62 ns | 1.07 | 0.01 | False | |||||
| GetHashCodeBenchmark - Duration of single invocation | 105.18 ns | 114.93 ns | 1.09 | 0.06 | False | |||||
| MultiplyOperatorBenchmark - Duration of single invocation | 50.92 ns | 65.31 ns | 1.28 | 0.07 | False |
Repro
git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Numerics.Tests.Perf_VectorOf<Byte>*'Details
Payloads
Histogram
System.Numerics.Tests.Perf_VectorOf<Byte>.SquareRootBenchmark
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 72.61622324189207 > 70.29034196284785.
IsChangePoint: Marked as a change because one of 3/17/2022 6:54:53 PM, 5/18/2022 10:04:15 AM, 5/24/2022 7:58:36 AM falls between 5/15/2022 6:57:22 PM and 5/24/2022 7:58:36 AM.
IsRegressionStdDev: Marked as regression because -43.44383568394873 (T) = (0 -73.07887024394701) / Math.Sqrt((0.4275740748536778 / (33)) + (0.10853637117538197 / (24))) is less than -2.0040447832881556 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (33) + (24) - 2, .025) and -0.08529938753935677 = (67.33521743676181 - 73.07887024394701) / 67.33521743676181 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Numerics.Tests.Perf_VectorOf<Byte>.GetHashCodeBenchmark
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 114.93390136649963 > 107.37540770947281.
IsChangePoint: Marked as a change because one of 4/25/2022 11:05:49 AM, 5/20/2022 10:18:01 PM, 5/24/2022 7:58:36 AM falls between 5/15/2022 6:57:22 PM and 5/24/2022 7:58:36 AM.
IsRegressionStdDev: Marked as regression because -48.89414960870857 (T) = (0 -114.77622672101295) / Math.Sqrt((2.158761094640268 / (44)) + (0.028601638891270608 / (13))) is less than -2.0040447832881556 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (44) + (13) - 2, .025) and -0.10674676117540412 = (103.70595220817863 - 114.77622672101295) / 103.70595220817863 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Numerics.Tests.Perf_VectorOf<Byte>.MultiplyOperatorBenchmark
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 65.30981681633082 > 53.33145200677543.
IsChangePoint: Marked as a change because one of 3/17/2022 6:54:53 PM, 3/29/2022 11:07:04 AM, 4/25/2022 11:05:49 AM, 5/11/2022 7:02:26 PM, 5/18/2022 10:04:15 AM, 5/24/2022 7:58:36 AM falls between 5/15/2022 6:57:22 PM and 5/24/2022 7:58:36 AM.
IsRegressionStdDev: Marked as regression because -15.112172425806873 (T) = (0 -62.63261316978918) / Math.Sqrt((0.11165366968483725 / (32)) + (14.904766609836882 / (25))) is less than -2.0040447832881556 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (25) - 2, .025) and -0.22978084202205626 = (50.92989826285313 - 62.63261316978918) / 50.92989826285313 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
Docs
Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
Run Information
| Architecture | x64 |
|---|---|
| OS | Windows 10.0.18362 |
| Baseline | 6e4eba8c98f4e05ece31a883dc23076a78f7c7b9 |
| Compare | 15dda0efe52c2aaa7a4e783f443eefb558529248 |
| Diff | Diff |
Regressions in System.Tests.Perf_String
| Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL |
|---|---|---|---|---|---|---|---|---|---|---|
| Format_MultipleArgs - Duration of single invocation | 244.70 ns | 295.96 ns | 1.21 | 0.06 | False | |||||
| Concat_CharEnumerable - Duration of single invocation | 7.03 μs | 8.46 μs | 1.20 | 0.06 | False |
Repro
git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Tests.Perf_String*'Details
Payloads
Histogram
System.Tests.Perf_String.Format_MultipleArgs
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 295.96439696558804 > 258.5723078295418.
IsChangePoint: Marked as a change because one of 4/16/2022 5:36:25 PM, 5/18/2022 10:04:15 AM, 5/24/2022 7:58:36 AM falls between 5/15/2022 6:57:22 PM and 5/24/2022 7:58:36 AM.
IsRegressionStdDev: Marked as regression because -43.214416060613296 (T) = (0 -297.6240622174258) / Math.Sqrt((14.284845301254313 / (33)) + (23.17517385721064 / (24))) is less than -2.0040447832881556 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (33) + (24) - 2, .025) and -0.2073050297315947 = (246.5193591412378 - 297.6240622174258) / 246.5193591412378 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Tests.Perf_String.Concat_CharEnumerable
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 8.459203054806828 > 7.38004250196541.
IsChangePoint: Marked as a change because one of 3/15/2022 11:00:34 PM, 3/23/2022 6:35:24 PM, 3/29/2022 11:07:04 AM, 5/5/2022 12:13:02 AM, 5/11/2022 7:02:26 PM, 5/20/2022 10:18:01 PM, 5/24/2022 7:58:36 AM falls between 5/15/2022 6:57:22 PM and 5/24/2022 7:58:36 AM.
IsRegressionStdDev: Marked as regression because -190.01315732087696 (T) = (0 -8472.773343347046) / Math.Sqrt((78.21027311391161 / (44)) + (730.5417624546898 / (13))) is less than -2.0040447832881556 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (44) + (13) - 2, .025) and -0.20591487712144863 = (7026.0127842288375 - 8472.773343347046) / 7026.0127842288375 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
Docs
Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
Run Information
| Architecture | x64 |
|---|---|
| OS | Windows 10.0.18362 |
| Baseline | 6e4eba8c98f4e05ece31a883dc23076a78f7c7b9 |
| Compare | 0c6d412f86205043726e273127d289b65d5b6c7a |
| Diff | Diff |
Regressions in System.Globalization.Tests.StringEquality
| Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL |
|---|---|---|---|---|---|---|---|---|---|---|
| Compare_Same - Duration of single invocation | 975.11 ns | 1.61 μs | 1.65 | 0.01 | True |
Repro
git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Globalization.Tests.StringEquality*'Details
Payloads
Histogram
System.Globalization.Tests.StringEquality.Compare_Same(Count: 1024, Options: (en-US, OrdinalIgnoreCase))
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.6128745088408643 > 1023.9368522813845.
IsChangePoint: Marked as a change because one of 5/18/2022 10:04:15 AM, 5/24/2022 7:58:36 AM falls between 5/15/2022 6:57:22 PM and 5/24/2022 7:58:36 AM.
IsRegressionStdDev: Marked as regression because -14.572122467414548 (T) = (0 -1455.1143152156847) / Math.Sqrt((0.005368131784794816 / (33)) + (26039.113951732077 / (24))) is less than -2.0040447832881556 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (33) + (24) - 2, .025) and -0.4922315216536375 = (975.1263755661582 - 1455.1143152156847) / 975.1263755661582 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.
Docs
Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
Metadata
Metadata
Assignees
Labels
arch-x64area-CodeGen-coreclrCLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMICLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMIruntime-coreclrspecific to the CoreCLR runtimespecific to the CoreCLR runtimetenet-performancePerformance related issuePerformance related issuetenet-performance-benchmarksIssue from performance benchmarkIssue from performance benchmark
_1.png)
_2.png)
_3.png)
_4.png)
_1.png)
_2.png)
_3.png)


