Regressions in Exceptions.Handling #71069
Description
Run Information
| Architecture | x64 |
|---|---|
| OS | Windows 10.0.19042 |
| Baseline | 53e52f28cdc9b800a4b04fee2f200e4087fcfd58 |
| Compare | 3fc61ebb562afc327a8fc6de5c82d76e86bf6f5d |
| Diff | Diff |
Regressions in Exceptions.Handling
Repro
git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'Exceptions.Handling*'Details
Payloads
Histogram
Exceptions.Handling.ThrowAndCatchWhenFinally(kind: Hardware)
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 10.631084321380373 > 9.707719407200473.
IsChangePoint: Marked as a change because one of 5/9/2022 5:23:46 PM, 6/15/2022 3:42:50 PM, 6/21/2022 6:07:15 AM falls between 6/12/2022 5:56:07 PM and 6/21/2022 6:07:15 AM.
IsRegressionStdDev: Marked as regression because -66.897391127366 (T) = (0 -10572.76689171382) / Math.Sqrt((8157.079428463892 / (39)) + (5592.9347822453165 / (30))) is less than -1.9960083540247138 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (39) + (30) - 2, .025) and -0.14396400223531025 = (9242.219922178137 - 10572.76689171382) / 9242.219922178137 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### Exceptions.Handling.MultipleNestedTryCatch_FirstCatches(kind: Hardware)
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 10.416770922440831 > 9.627266102453488.
IsChangePoint: Marked as a change because one of 5/9/2022 6:32:22 AM, 6/15/2022 3:42:50 PM, 6/21/2022 6:07:15 AM falls between 6/12/2022 5:56:07 PM and 6/21/2022 6:07:15 AM.
IsRegressionStdDev: Marked as regression because -55.02461913865396 (T) = (0 -10455.637082992163) / Math.Sqrt((13527.506175182813 / (39)) + (5897.334081995886 / (30))) is less than -1.9960083540247138 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (39) + (30) - 2, .025) and -0.13983765079759564 = (9172.917806036574 - 10455.637082992163) / 9172.917806036574 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### Exceptions.Handling.ThrowAndCatchDeep(kind: Hardware)
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 22.542855210265923 > 21.816747631819343.
IsChangePoint: Marked as a change because one of 5/9/2022 11:56:12 AM, 6/15/2022 3:42:50 PM, 6/21/2022 6:07:15 AM falls between 6/12/2022 5:56:07 PM and 6/21/2022 6:07:15 AM.
IsRegressionStdDev: Marked as regression because -29.29467010348384 (T) = (0 -22455.320507215223) / Math.Sqrt((34371.85874714472 / (39)) + (55530.08190816655 / (30))) is less than -1.9960083540247138 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (39) + (30) - 2, .025) and -0.07318294551296978 = (20924.03778973754 - 22455.320507215223) / 20924.03778973754 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### Exceptions.Handling.MultipleNestedTryCatch_LastCatches(kind: Hardware)
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 10.865562918687917 > 9.95582348269825.
IsChangePoint: Marked as a change because one of 5/25/2022 3:58:08 AM, 6/15/2022 3:42:50 PM, 6/21/2022 6:07:15 AM falls between 6/12/2022 5:56:07 PM and 6/21/2022 6:07:15 AM.
IsRegressionStdDev: Marked as regression because -50.739502176697414 (T) = (0 -10676.76399097377) / Math.Sqrt((14771.900394532195 / (39)) + (8442.262379284302 / (30))) is less than -1.9960083540247138 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (39) + (30) - 2, .025) and -0.13908940438201717 = (9373.069356892285 - 10676.76399097377) / 9373.069356892285 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### Exceptions.Handling.ThrowAndCatchFinally(kind: Hardware)
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 10.504249720357944 > 9.68166809508551.
IsChangePoint: Marked as a change because one of 5/9/2022 7:27:05 AM, 6/15/2022 3:42:50 PM, 6/21/2022 6:07:15 AM falls between 6/12/2022 5:56:07 PM and 6/21/2022 6:07:15 AM.
IsRegressionStdDev: Marked as regression because -44.07090372120486 (T) = (0 -10485.088476715544) / Math.Sqrt((19475.836028772857 / (39)) + (11306.205598673312 / (30))) is less than -1.9960083540247138 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (39) + (30) - 2, .025) and -0.14210183109433208 = (9180.519802397135 - 10485.088476715544) / 9180.519802397135 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### Exceptions.Handling.ThrowAndCatch_ManyCatchBlocks(kind: Hardware)
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 10.832801149425288 > 9.970097115079255.
IsChangePoint: Marked as a change because one of 6/15/2022 3:42:50 PM, 6/21/2022 6:07:15 AM falls between 6/12/2022 5:56:07 PM and 6/21/2022 6:07:15 AM.
IsRegressionStdDev: Marked as regression because -56.32967291557672 (T) = (0 -10745.776543662498) / Math.Sqrt((11694.90279727697 / (39)) + (6418.637792994445 / (30))) is less than -1.9960083540247138 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (39) + (30) - 2, .025) and -0.1348479154134637 = (9468.913321083597 - 10745.776543662498) / 9468.913321083597 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### Exceptions.Handling.ThrowAndCatch_ManyCatchBlocks(kind: ReflectionHardware)
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 24.980106970324364 > 24.23344574602191.
IsChangePoint: Marked as a change because one of 5/9/2022 11:56:12 AM, 6/15/2022 10:21:53 PM, 6/21/2022 6:07:15 AM falls between 6/12/2022 5:56:07 PM and 6/21/2022 6:07:15 AM.
IsRegressionStdDev: Marked as regression because -32.31904356180284 (T) = (0 -24879.29898810313) / Math.Sqrt((78306.19838140166 / (40)) + (35320.83162906436 / (29))) is less than -1.9960083540247138 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (40) + (29) - 2, .025) and -0.07898599004535108 = (23058.037099311572 - 24879.29898810313) / 23058.037099311572 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### Exceptions.Handling.ThrowAndCatchWhen(kind: Hardware)
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 10.591045504967685 > 9.77971976920238.
IsChangePoint: Marked as a change because one of 5/9/2022 5:23:46 PM, 6/15/2022 3:42:50 PM, 6/21/2022 6:07:15 AM falls between 6/12/2022 5:56:07 PM and 6/21/2022 6:07:15 AM.
IsRegressionStdDev: Marked as regression because -50.931387360796705 (T) = (0 -10558.16716957456) / Math.Sqrt((13465.21212598458 / (39)) + (8843.591231293593 / (30))) is less than -1.9960083540247138 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (39) + (30) - 2, .025) and -0.13900446382153012 = (9269.645119871026 - 10558.16716957456) / 9269.645119871026 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### Exceptions.Handling.MultipleNestedTryFinally(kind: Hardware)
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 10.76901643598616 > 10.031316493224164.
IsChangePoint: Marked as a change because one of 6/15/2022 3:42:50 PM, 6/21/2022 6:07:15 AM falls between 6/12/2022 5:56:07 PM and 6/21/2022 6:07:15 AM.
IsRegressionStdDev: Marked as regression because -46.09077460106207 (T) = (0 -10668.942583318307) / Math.Sqrt((19576.50371765474 / (39)) + (6905.685295337485 / (30))) is less than -1.9960083540247138 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (39) + (30) - 2, .025) and -0.13236716566756504 = (9421.804964672072 - 10668.942583318307) / 9421.804964672072 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### Exceptions.Handling.TryAndFinallyDeep(kind: Hardware)
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 24.059202872744535 > 23.701262313237617.
IsChangePoint: Marked as a change because one of 5/9/2022 5:23:46 PM, 6/15/2022 10:21:53 PM, 6/21/2022 6:07:15 AM falls between 6/12/2022 5:56:07 PM and 6/21/2022 6:07:15 AM.
IsRegressionStdDev: Marked as regression because -23.06154536982453 (T) = (0 -24090.350131211093) / Math.Sqrt((105297.76437581454 / (40)) + (41951.22530578471 / (29))) is less than -1.9960083540247138 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (40) + (29) - 2, .025) and -0.06512132911915162 = (22617.470397605928 - 24090.350131211093) / 22617.470397605928 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### Exceptions.Handling.ThrowAndCatchDeep(kind: ReflectionHardware)
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 36.84394253554502 > 36.493989600411865.
IsChangePoint: Marked as a change because one of 5/9/2022 7:27:05 AM, 6/15/2022 10:21:53 PM, 6/21/2022 6:07:15 AM falls between 6/12/2022 5:56:07 PM and 6/21/2022 6:07:15 AM.
IsRegressionStdDev: Marked as regression because -24.123018682862472 (T) = (0 -36846.237351023956) / Math.Sqrt((144377.03195286612 / (40)) + (107536.73252920977 / (29))) is less than -1.9960083540247138 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (40) + (29) - 2, .025) and -0.059327024307226243 = (34782.68420001886 - 36846.237351023956) / 34782.68420001886 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### Exceptions.Handling.TryAndCatchDeep_CaugtAtTheTop(kind: Hardware)
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 24.163793127204585 > 23.31609527812198.
IsChangePoint: Marked as a change because one of 5/9/2022 6:32:22 AM, 6/15/2022 3:42:50 PM, 6/21/2022 6:07:15 AM falls between 6/12/2022 5:56:07 PM and 6/21/2022 6:07:15 AM.
IsRegressionStdDev: Marked as regression because -29.5580790510889 (T) = (0 -23883.544386341473) / Math.Sqrt((39808.69838332002 / (39)) + (62223.092350807456 / (30))) is less than -1.9960083540247138 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (39) + (30) - 2, .025) and -0.07393945634185245 = (22239.19071536464 - 23883.544386341473) / 22239.19071536464 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### Exceptions.Handling.ThrowAndCatch(kind: Hardware)
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 10.622996870771312 > 9.571910399460464.
IsChangePoint: Marked as a change because one of 6/15/2022 3:42:50 PM, 6/21/2022 6:07:15 AM falls between 6/12/2022 5:56:07 PM and 6/21/2022 6:07:15 AM.
IsRegressionStdDev: Marked as regression because -50.49209700358451 (T) = (0 -10460.898931270007) / Math.Sqrt((15499.476225187873 / (39)) + (7319.7593417571425 / (30))) is less than -1.9960083540247138 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (39) + (30) - 2, .025) and -0.13926738995177781 = (9182.127939001914 - 10460.898931270007) / 9182.127939001914 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### Exceptions.Handling.ThrowAndCatch(kind: ReflectionHardware)
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 24.490700333131432 > 23.71146913835988.
IsChangePoint: Marked as a change because one of 5/9/2022 11:56:12 AM, 6/15/2022 10:21:53 PM, 6/21/2022 6:07:15 AM falls between 6/12/2022 5:56:07 PM and 6/21/2022 6:07:15 AM.
IsRegressionStdDev: Marked as regression because -30.59301287270793 (T) = (0 -24546.376362997726) / Math.Sqrt((76541.42502291457 / (40)) + (47174.25147351853 / (29))) is less than -1.9960083540247138 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (40) + (29) - 2, .025) and -0.08009647246323905 = (22726.096222699365 - 24546.376362997726) / 22726.096222699365 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
Docs
Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository