Skip to content

Perf measurement changed due to BDN update  #66664

@performanceautofiler

Description

@performanceautofiler

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 3e2d483153adcab27033340fa40ad0bcdc3acc2a
Compare 6e26872d0a282aa71ea792c3550a3cb0e8bf4e71
Diff Diff

Regressions in PerfLabTests.LowLevelPerf

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
GenericClassGenericInstanceMethod - Duration of single invocation 108.79 μs 152.27 μs 1.40 0.09 False

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'PerfLabTests.LowLevelPerf*'
Details

Payloads

Baseline
Compare

Histogram

PerfLabTests.LowLevelPerf.GenericClassGenericInstanceMethod


Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 152.26848763069455 > 114.1891593163228.
IsChangePoint: Marked as a change because one of 2/1/2022 8:13:55 AM, 2/7/2022 5:30:00 PM, 2/19/2022 11:51:48 PM, 3/11/2022 8:42:43 PM, 3/15/2022 12:53:39 AM falls between 3/6/2022 12:59:26 AM and 3/15/2022 12:53:39 AM.
IsRegressionStdDev: Marked as regression because -18.411230076485225 (T) = (0 -152339.69703271738) / Math.Sqrt((215043981.29227346 / (50)) + (30962.15348726156 / (13))) is less than -1.9996235849941724 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (50) + (13) - 2, .025) and -0.33459358592612964 = (114146.88234621071 - 152339.69703271738) / 114146.88234621071 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

### Run Information
Architecture x64
OS ubuntu 18.04
Baseline 3e2d483153adcab27033340fa40ad0bcdc3acc2a
Compare 6e26872d0a282aa71ea792c3550a3cb0e8bf4e71
Diff Diff

Regressions in System.Numerics.Tests.Perf_VectorOf<UInt32>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
DivisionOperatorBenchmark - Duration of single invocation 19.33 ns 21.34 ns 1.10 0.10 False

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Numerics.Tests.Perf_VectorOf&lt;UInt32&gt;*'
Details

Payloads

Baseline
Compare

Histogram

System.Numerics.Tests.Perf_VectorOf<UInt32>.DivisionOperatorBenchmark


Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 21.343080974054562 > 20.609751481219558.
IsChangePoint: Marked as a change because one of 2/4/2022 6:00:25 PM, 3/3/2022 10:59:13 PM, 3/11/2022 8:42:43 PM, 3/15/2022 12:53:39 AM falls between 3/6/2022 12:59:26 AM and 3/15/2022 12:53:39 AM.
IsRegressionStdDev: Marked as regression because -2.5818159065022552 (T) = (0 -21.613150255722033) / Math.Sqrt((4.7361616956054595 / (50)) + (3.3162067500162973 / (13))) is less than -1.9996235849941724 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (50) + (13) - 2, .025) and -0.0760236823856467 = (20.08612878092388 - 21.613150255722033) / 20.08612878092388 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 3e2d483153adcab27033340fa40ad0bcdc3acc2a
Compare 6e26872d0a282aa71ea792c3550a3cb0e8bf4e71
Diff Diff

Regressions in System.Diagnostics.Perf_Activity

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ActivityAllocations - Duration of single invocation 209.43 ns 233.36 ns 1.11 0.07 False
EnumerateActivityLinkTagsSmall - Duration of single invocation 52.71 ns 56.77 ns 1.08 0.02 False

graph
graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Diagnostics.Perf_Activity*'
Details

Payloads

Baseline
Compare

Histogram

System.Diagnostics.Perf_Activity.ActivityAllocations(idFormat: W3C)


Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 233.36223175128367 > 216.16422674283905.
IsChangePoint: Marked as a change because one of 3/11/2022 8:42:43 PM, 3/15/2022 12:53:39 AM falls between 3/6/2022 12:59:26 AM and 3/15/2022 12:53:39 AM.
IsRegressionStdDev: Marked as regression because -8.621022864458252 (T) = (0 -228.94307270766475) / Math.Sqrt((32.078697121602765 / (50)) + (79.54763117655831 / (12))) is less than -2.0002978220134566 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (50) + (12) - 2, .025) and -0.11300910532417982 = (205.6973942194137 - 228.94307270766475) / 205.6973942194137 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Diagnostics.Perf_Activity.EnumerateActivityLinkTagsSmall

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 56.765787305067256 > 55.25313766355957.
IsChangePoint: Marked as a change because one of 1/19/2022 3:47:09 AM, 3/11/2022 8:42:43 PM, 3/15/2022 12:53:39 AM falls between 3/6/2022 12:59:26 AM and 3/15/2022 12:53:39 AM.
IsRegressionStdDev: Marked as regression because -24.40328217355822 (T) = (0 -56.59224771724442) / Math.Sqrt((0.7261720972172016 / (50)) + (0.05103723337957388 / (13))) is less than -1.9996235849941724 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (50) + (13) - 2, .025) and -0.062214930877315756 = (53.27758636428049 - 56.59224771724442) / 53.27758636428049 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 3e2d483153adcab27033340fa40ad0bcdc3acc2a
Compare 6e26872d0a282aa71ea792c3550a3cb0e8bf4e71
Diff Diff

Regressions in System.Collections.CtorGivenSize<String>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Stack - Duration of single invocation 133.59 ns 147.84 ns 1.11 0.18 False

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Collections.CtorGivenSize&lt;String&gt;*'
Details

Payloads

Baseline
Compare

Histogram

System.Collections.CtorGivenSize<String>.Stack(Size: 512)


Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 147.84130488908252 > 142.5028238654394.
IsChangePoint: Marked as a change because one of 1/11/2022 2:18:34 PM, 1/18/2022 7:04:48 PM, 3/11/2022 8:42:43 PM, 3/15/2022 12:53:39 AM falls between 3/6/2022 12:59:26 AM and 3/15/2022 12:53:39 AM.
IsRegressionStdDev: Marked as regression because -14.02701433603456 (T) = (0 -149.61656383692844) / Math.Sqrt((10.55722883064997 / (50)) + (7.107666420398533 / (12))) is less than -2.0002978220134566 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (50) + (12) - 2, .025) and -0.09174588306105283 = (137.04339641513587 - 149.61656383692844) / 137.04339641513587 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions