Regressions in System.IO.Compression.Brotli #86335
Description
Run Information
| Name | Value |
|---|---|
| Architecture | x64 |
| OS | ubuntu 18.04 |
| Queue | TigerUbuntu |
| Baseline | eead6a328cf3a7426e62d1441343b4b0e1f57d19 |
| Compare | 0b5f137ab0eea82f59d22da94a0b21f0bc598f16 |
| Diff | Diff |
| Configs | CompilationMode:tiered, RunKind:micro |
Regressions in System.IO.Compression.Brotli
| Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL |
|---|---|---|---|---|---|---|---|---|---|---|
| Decompress_WithState - Duration of single invocation | 275.33 μs | 290.84 μs | 1.06 | 0.01 | False | |||||
| Decompress_WithoutState - Duration of single invocation | 246.86 μs | 260.12 μs | 1.05 | 0.01 | False | |||||
| Decompress_WithoutState - Duration of single invocation | 275.01 μs | 290.06 μs | 1.05 | 0.01 | False | |||||
| Decompress - Duration of single invocation | 258.04 μs | 271.42 μs | 1.05 | 0.01 | False | |||||
| Decompress - Duration of single invocation | 282.37 μs | 297.75 μs | 1.05 | 0.01 | False | |||||
| Decompress_WithState - Duration of single invocation | 247.79 μs | 262.28 μs | 1.06 | 0.01 | False |
Repro
General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md
Payloads
git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.IO.Compression.Brotli*'Details
Payloads
Histogram
System.IO.Compression.Brotli.Decompress_WithState(level: Optimal, file: "TestDocument.pdf")
Description of detection logic
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 290.84272255291006 > 289.21183704769743.
IsChangePoint: Marked as a change because one of 4/12/2023 4:44:56 PM, 4/21/2023 4:02:09 PM, 5/10/2023 2:00:36 PM, 5/16/2023 7:34:13 AM falls between 5/7/2023 5:11:33 AM and 5/16/2023 7:34:13 AM.
IsRegressionStdDev: Marked as regression because -84.94069941552576 (T) = (0 -290639.580362342) / Math.Sqrt((107646.4212471317 / (20)) + (506031.5362730523 / (19))) is less than -2.026192463026769 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (19) - 2, .025) and -0.05517818741065231 = (275441.2324192894 - 290639.580362342) / 275441.2324192894 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
JIT Disasms
System.IO.Compression.Brotli.Decompress_WithoutState(level: Fastest, file: "TestDocument.pdf")
Description of detection logic
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 260.1242030555556 > 259.227968737793.
IsChangePoint: Marked as a change because one of 4/12/2023 4:44:56 PM, 4/21/2023 4:02:09 PM, 5/10/2023 2:00:36 PM, 5/16/2023 7:34:13 AM falls between 5/7/2023 5:11:33 AM and 5/16/2023 7:34:13 AM.
IsRegressionStdDev: Marked as regression because -62.00641912288732 (T) = (0 -260669.61401916188) / Math.Sqrt((37447.13882312735 / (20)) + (965191.1463435948 / (20))) is less than -2.0243941639098457 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (20) - 2, .025) and -0.0562565098872357 = (246786.2792599409 - 260669.61401916188) / 246786.2792599409 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
JIT Disasms
System.IO.Compression.Brotli.Decompress_WithoutState(level: Optimal, file: "TestDocument.pdf")
Description of detection logic
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 290.0647317129629 > 288.7385472341009.
IsChangePoint: Marked as a change because one of 4/12/2023 4:44:56 PM, 4/21/2023 4:02:09 PM, 5/10/2023 2:00:36 PM, 5/16/2023 7:34:13 AM falls between 5/7/2023 5:11:33 AM and 5/16/2023 7:34:13 AM.
IsRegressionStdDev: Marked as regression because -109.10396387586594 (T) = (0 -289757.7017100669) / Math.Sqrt((64328.792429948866 / (20)) + (301071.5115852465 / (20))) is less than -2.0243941639098457 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (20) - 2, .025) and -0.05362418781284854 = (275010.48766881146 - 289757.7017100669) / 275010.48766881146 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
JIT Disasms
System.IO.Compression.Brotli.Decompress(level: Fastest, file: "TestDocument.pdf")
Description of detection logic
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 271.4164824270557 > 270.9308018344105.
IsChangePoint: Marked as a change because one of 4/12/2023 4:44:56 PM, 4/21/2023 4:02:09 PM, 5/10/2023 2:00:36 PM, 5/16/2023 7:34:13 AM falls between 5/7/2023 5:11:33 AM and 5/16/2023 7:34:13 AM.
IsRegressionStdDev: Marked as regression because -84.4011638108043 (T) = (0 -272065.7148001677) / Math.Sqrt((101261.5224972446 / (20)) + (441966.70041789534 / (20))) is less than -2.0243941639098457 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (20) - 2, .025) and -0.05388187749429565 = (258155.79583456717 - 272065.7148001677) / 258155.79583456717 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
JIT Disasms
System.IO.Compression.Brotli.Decompress(level: Optimal, file: "TestDocument.pdf")
Description of detection logic
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 297.7458011287062 > 296.70168234375.
IsChangePoint: Marked as a change because one of 4/12/2023 4:44:56 PM, 4/21/2023 4:02:09 PM, 5/10/2023 2:00:36 PM, 5/16/2023 7:34:13 AM falls between 5/7/2023 5:11:33 AM and 5/16/2023 7:34:13 AM.
IsRegressionStdDev: Marked as regression because -148.62331465251418 (T) = (0 -297489.1681221059) / Math.Sqrt((103374.3582741498 / (20)) + (96150.73892757313 / (20))) is less than -2.0243941639098457 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (20) - 2, .025) and -0.05252066097941168 = (282644.4925511301 - 297489.1681221059) / 282644.4925511301 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
JIT Disasms
System.IO.Compression.Brotli.Decompress_WithState(level: Fastest, file: "TestDocument.pdf")
Description of detection logic
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 262.27939534722225 > 259.47930234375.
IsChangePoint: Marked as a change because one of 4/12/2023 4:44:56 PM, 4/21/2023 4:02:09 PM, 5/10/2023 2:00:36 PM, 5/16/2023 7:34:13 AM falls between 5/7/2023 5:11:33 AM and 5/16/2023 7:34:13 AM.
IsRegressionStdDev: Marked as regression because -76.42147528289804 (T) = (0 -260977.19148595305) / Math.Sqrt((73102.83454388827 / (20)) + (570478.22836798 / (20))) is less than -2.0243941639098457 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (20) - 2, .025) and -0.05544132342591581 = (247268.30918352964 - 260977.19148595305) / 247268.30918352964 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
JIT Disasms
Docs
Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository