[Perf] Linux/x64: 9 Regressions on 12/1/2022 11:48:13 PM #79337
Description
Run Information
| Architecture | x64 |
|---|---|
| OS | ubuntu 18.04 |
| Baseline | 776a87d9f0c9105c2729d1ab781d74bffa2a182b |
| Compare | 9d7ffb5b0f9c56376460a0a45b90cb3ca90302ea |
| Diff | Diff |
Regressions in System.Numerics.Tests.Perf_Matrix4x4
| Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL |
|---|---|---|---|---|---|---|---|---|---|---|
| CreatePerspectiveBenchmark - Duration of single invocation | 46.88 ns | 106.40 ns | 2.27 | 0.23 | False | |||||
| CreatePerspectiveOffCenterBenchmark - Duration of single invocation | 56.08 ns | 100.36 ns | 1.79 | 0.19 | False | |||||
| CreateScaleFromScalarWithCenterBenchmark - Duration of single invocation | 33.20 ns | 61.18 ns | 1.84 | 0.33 | False | |||||
| CreateScaleFromVectorWithCenterBenchmark - Duration of single invocation | 50.28 ns | 76.30 ns | 1.52 | 0.23 | False | |||||
| CreatePerspectiveFieldOfViewBenchmark - Duration of single invocation | 63.69 ns | 152.52 ns | 2.39 | 0.17 | False |
Repro
git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Numerics.Tests.Perf_Matrix4x4*'Details
Payloads
Histogram
System.Numerics.Tests.Perf_Matrix4x4.CreatePerspectiveBenchmark
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 106.40167473232303 > 50.089019744049914.
IsChangePoint: Marked as a change because one of 10/19/2022 7:27:15 PM, 12/1/2022 7:29:59 PM, 12/6/2022 5:44:36 AM falls between 11/27/2022 4:37:47 PM and 12/6/2022 5:44:36 AM.
IsRegressionStdDev: Marked as regression because -29.54742807413155 (T) = (0 -96.53403777573979) / Math.Sqrt((2.0842638368238435 / (31)) + (50.35869810254004 / (19))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (19) - 2, .025) and -1.0185334186441561 = (47.82384917886644 - 96.53403777573979) / 47.82384917886644 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Numerics.Tests.Perf_Matrix4x4.CreatePerspectiveOffCenterBenchmark
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 100.36403988735348 > 58.79696350489112.
IsChangePoint: Marked as a change because one of 10/17/2022 12:47:34 PM, 12/1/2022 7:29:59 PM, 12/6/2022 5:44:36 AM falls between 11/27/2022 4:37:47 PM and 12/6/2022 5:44:36 AM.
IsRegressionStdDev: Marked as regression because -62.96910756137589 (T) = (0 -101.91213463508836) / Math.Sqrt((3.1128228061074505 / (31)) + (8.126852446138692 / (19))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (19) - 2, .025) and -0.8149874229088963 = (56.15032553324964 - 101.91213463508836) / 56.15032553324964 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Numerics.Tests.Perf_Matrix4x4.CreateScaleFromScalarWithCenterBenchmark
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 61.175483380092 > 33.07993844635303.
IsChangePoint: Marked as a change because one of 10/11/2022 6:23:40 AM, 10/19/2022 7:27:15 PM, 12/5/2022 3:14:07 PM, 12/6/2022 5:44:36 AM falls between 11/27/2022 4:37:47 PM and 12/6/2022 5:44:36 AM.
IsRegressionStdDev: Marked as regression because -240.36780006315195 (T) = (0 -61.18321579634298) / Math.Sqrt((0.7338720353181785 / (48)) + (0.00011958052215698276 / (2))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (48) + (2) - 2, .025) and -0.9482623727783919 = (31.403991911567015 - 61.18321579634298) / 31.403991911567015 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Numerics.Tests.Perf_Matrix4x4.CreateScaleFromVectorWithCenterBenchmark
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 76.29638516883118 > 52.98744532824259.
IsChangePoint: Marked as a change because one of 10/11/2022 6:23:40 AM, 12/5/2022 3:14:07 PM, 12/6/2022 5:44:36 AM falls between 11/27/2022 4:37:47 PM and 12/6/2022 5:44:36 AM.
IsRegressionStdDev: Marked as regression because -20.23418898038416 (T) = (0 -75.14045784284106) / Math.Sqrt((5.187993144752031 / (48)) + (2.67233596594134 / (2))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (48) + (2) - 2, .025) and -0.47845555759071756 = (50.82361621020892 - 75.14045784284106) / 50.82361621020892 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Numerics.Tests.Perf_Matrix4x4.CreatePerspectiveFieldOfViewBenchmark
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 152.5239787610467 > 66.84254185778312.
IsChangePoint: Marked as a change because one of 10/17/2022 12:47:34 PM, 12/1/2022 7:29:59 PM, 12/6/2022 5:44:36 AM falls between 11/27/2022 4:37:47 PM and 12/6/2022 5:44:36 AM.
IsRegressionStdDev: Marked as regression because -50.33783026021656 (T) = (0 -143.0980608155679) / Math.Sqrt((1.8023112844920022 / (31)) + (46.10575932368977 / (19))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (19) - 2, .025) and -1.2446768917145479 = (63.749959445729196 - 143.0980608155679) / 63.749959445729196 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
Docs
Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
| Architecture | x64 |
|---|---|
| OS | ubuntu 18.04 |
| Baseline | 776a87d9f0c9105c2729d1ab781d74bffa2a182b |
| Compare | 9d7ffb5b0f9c56376460a0a45b90cb3ca90302ea |
| Diff | Diff |
Regressions in System.Tests.Perf_Version
| Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL |
|---|---|---|---|---|---|---|---|---|---|---|
| Ctor4 - Duration of single invocation | 54.92 ns | 95.97 ns | 1.75 | 0.22 | False | |||||
| Ctor3 - Duration of single invocation | 51.64 ns | 73.53 ns | 1.42 | 0.18 | False | |||||
| Ctor2 - Duration of single invocation | 48.49 ns | 63.28 ns | 1.31 | 0.23 | False |
Repro
git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_Version*'Details
Payloads
Histogram
System.Tests.Perf_Version.Ctor4
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 95.9653964201535 > 57.81517049815903.
IsChangePoint: Marked as a change because one of 12/1/2022 7:29:59 PM, 12/6/2022 5:44:36 AM falls between 11/27/2022 4:37:47 PM and 12/6/2022 5:44:36 AM.
IsRegressionStdDev: Marked as regression because -68.53791289679593 (T) = (0 -96.72656882046915) / Math.Sqrt((2.166112228046841 / (31)) + (5.923382867603497 / (19))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (19) - 2, .025) and -0.7785079664400922 = (54.38635679213716 - 96.72656882046915) / 54.38635679213716 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Tests.Perf_Version.Ctor3
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 73.53258813463972 > 54.44255171823497.
IsChangePoint: Marked as a change because one of 12/1/2022 7:29:59 PM, 12/6/2022 5:44:36 AM falls between 11/27/2022 4:37:47 PM and 12/6/2022 5:44:36 AM.
IsRegressionStdDev: Marked as regression because -38.789721575245764 (T) = (0 -75.65424811264478) / Math.Sqrt((2.1752788781519157 / (31)) + (5.739031977705032 / (19))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (19) - 2, .025) and -0.45520962769791584 = (51.98855661251143 - 75.65424811264478) / 51.98855661251143 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Tests.Perf_Version.Ctor2
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 63.27801788496673 > 52.23769541648208.
IsChangePoint: Marked as a change because one of 12/1/2022 7:29:59 PM, 12/6/2022 5:44:36 AM falls between 11/27/2022 4:37:47 PM and 12/6/2022 5:44:36 AM.
IsRegressionStdDev: Marked as regression because -27.21061402026845 (T) = (0 -65.16909851640705) / Math.Sqrt((1.518428932024682 / (31)) + (5.053331740985409 / (19))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (19) - 2, .025) and -0.3060337808413552 = (49.898478486846415 - 65.16909851640705) / 49.898478486846415 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
Docs
Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
Run Information
| Architecture | x64 |
|---|---|
| OS | ubuntu 18.04 |
| Baseline | 776a87d9f0c9105c2729d1ab781d74bffa2a182b |
| Compare | 9d7ffb5b0f9c56376460a0a45b90cb3ca90302ea |
| Diff | Diff |
Regressions in PerfLabTests.BlockCopyPerf
| Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL |
|---|---|---|---|---|---|---|---|---|---|---|
| CallBlockCopy - Duration of single invocation | 86.43 ns | 99.44 ns | 1.15 | 0.15 | False |
Repro
git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'PerfLabTests.BlockCopyPerf*'Details
Payloads
Histogram
PerfLabTests.BlockCopyPerf.CallBlockCopy(numElements: 10)
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 99.43981666225224 > 87.0023639795907.
IsChangePoint: Marked as a change because one of 11/15/2022 8:48:03 PM, 12/1/2022 7:29:59 PM, 12/6/2022 5:44:36 AM falls between 11/27/2022 4:37:47 PM and 12/6/2022 5:44:36 AM.
IsRegressionStdDev: Marked as regression because -23.47984704244632 (T) = (0 -101.44930096610167) / Math.Sqrt((5.209829064913729 / (31)) + (8.017543922096761 / (19))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (19) - 2, .025) and -0.21622082875210036 = (83.41355333487704 - 101.44930096610167) / 83.41355333487704 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
Docs
Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository