Performance regressions in System.Numerics.Tests.Perf_BigInteger.Parse #74158
Description
Run Information
| Architecture | x64 |
|---|---|
| OS | ubuntu 18.04 |
| Baseline | 94e0c70eee98d00a410ac2ac6dcd28c3abc2a4e2 |
| Compare | a5f3676cc71e176084f0f7f1f6beeecd86fbeafc |
| Diff | Diff |
Regressions in System.Numerics.Tests.Perf_BigInteger
| Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL |
|---|---|---|---|---|---|---|---|---|---|---|
| ToByteArray - Duration of single invocation | 58.34 ns | 63.10 ns | 1.08 | 0.01 | False | |||||
| Parse - Duration of single invocation | 1.41 μs | 1.65 μs | 1.17 | 0.02 | False | |||||
| Parse - Duration of single invocation | 133.39 ns | 142.75 ns | 1.07 | 0.04 | False |
Repro
git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Numerics.Tests.Perf_BigInteger*'Details
Payloads
Histogram
System.Numerics.Tests.Perf_BigInteger.ToByteArray(numberString: 12345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890)
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 63.09514035342691 > 61.31842219522817.
IsChangePoint: Marked as a change because one of 7/10/2022 2:23:14 PM, 7/13/2022 7:22:00 AM, 7/18/2022 2:17:16 AM falls between 7/8/2022 10:31:34 AM and 7/18/2022 2:17:16 AM.
IsRegressionStdDev: Marked as regression because -19.57606754469047 (T) = (0 -63.06821020437885) / Math.Sqrt((4.0461528903312995 / (56)) + (0.1429998919536005 / (31))) is less than -1.988267907476882 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (56) + (31) - 2, .025) and -0.09415891535503756 = (57.64081370567108 - 63.06821020437885) / 57.64081370567108 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Numerics.Tests.Perf_BigInteger.Parse(numberString: 12345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890)
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.64638091481311 > 1.4789204100383715.
IsChangePoint: Marked as a change because one of 5/23/2022 3:43:39 PM, 7/2/2022 7:35:06 PM, 7/6/2022 3:55:24 AM, 7/12/2022 12:16:14 PM, 7/18/2022 2:17:16 AM falls between 7/8/2022 10:31:34 AM and 7/18/2022 2:17:16 AM.
IsRegressionStdDev: Marked as regression because -13.450997279262245 (T) = (0 -1622.803624081618) / Math.Sqrt((10269.32144450607 / (51)) + (553.0820727629088 / (36))) is less than -1.988267907476882 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (51) + (36) - 2, .025) and -0.1389816193702231 = (1424.7847344357624 - 1622.803624081618) / 1424.7847344357624 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Numerics.Tests.Perf_BigInteger.Parse(numberString: -2147483648)
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 142.7472876905376 > 138.36476110994943.
IsChangePoint: Marked as a change because one of 7/2/2022 7:35:06 PM, 7/6/2022 3:55:24 AM, 7/12/2022 12:16:14 PM, 7/18/2022 2:17:16 AM falls between 7/8/2022 10:31:34 AM and 7/18/2022 2:17:16 AM.
IsRegressionStdDev: Marked as regression because -4.669140180936088 (T) = (0 -143.62725712142048) / Math.Sqrt((116.92965141088445 / (50)) + (4.935059494685537 / (36))) is less than -1.9886096669751727 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (50) + (36) - 2, .025) and -0.05390755620131213 = (136.28069774839486 - 143.62725712142048) / 136.28069774839486 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
Docs
Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
EDIT: Removed remainder of the original report, as the rest were noisy benchmarks