[Perf] Regressions in System.Buffers.Text.Tests.Utf8FormatterTests

[DrewScoggins]

Run Information

Architecture	x86
OS	Windows 10.0.18362
Baseline	23697bbc6b365fac14d986af7b23b2a908f3f6e0
Compare	a6d396046ab327281e5a436b8d525f0b51d2ce0c
Diff	Diff

Regressions in System.Buffers.Text.Tests.Utf8FormatterTests

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio	Baseline ETL	Compare ETL
FormatterUInt32 - Duration of single invocation	14.84 ns	16.88 ns	1.14	0.02	False
FormatterInt64 - Duration of single invocation	63.64 ns	67.60 ns	1.06	0.00	False
FormatterInt32 - Duration of single invocation	15.88 ns	18.46 ns	1.16	0.01	False
FormatterUInt64 - Duration of single invocation	7.85 ns	9.25 ns	1.18	0.02	False
FormatterUInt32 - Duration of single invocation	7.49 ns	9.03 ns	1.21	0.01	False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Buffers.Text.Tests.Utf8FormatterTests*'

Related Issues

Regressions

Improvements

• [Perf] Linux/x64: 18 Improvements on 10/3/2022 11:31:18 PM

Details

Payloads

Baseline
Compare

Histogram

Edge Detector Info

Collection Data

System.Buffers.Text.Tests.Utf8FormatterTests.FormatterUInt32(value: 4294967295)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 16.87672409355907 > 15.551577581679089.
IsChangePoint: Marked as a change because one of 10/3/2022 8:45:37 PM, 10/11/2022 2:16:14 AM falls between 10/2/2022 5:38:34 AM and 10/11/2022 2:16:14 AM.
IsRegressionStdDev: Marked as regression because -86.48783555453538 (T) = (0 -16.910185275233378) / Math.Sqrt((0.006843647038777639 / (28)) + (0.009887396348732362 / (27))) is less than -2.005745995316835 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (28) + (27) - 2, .025) and -0.14466714467413372 = (14.773015329314275 - 16.910185275233378) / 14.773015329314275 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Buffers.Text.Tests.Utf8FormatterTests.FormatterInt64(value: -9223372036854775808)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 67.60444567402155 > 66.8818020437595.
IsChangePoint: Marked as a change because one of 10/3/2022 8:45:37 PM, 10/11/2022 2:16:14 AM falls between 10/2/2022 5:38:34 AM and 10/11/2022 2:16:14 AM.
IsRegressionStdDev: Marked as regression because -42.990433655786944 (T) = (0 -67.51615292052068) / Math.Sqrt((0.1347532172176893 / (28)) + (0.07740645620759257 / (26))) is less than -2.0066468050606243 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (28) + (26) - 2, .025) and -0.059545147633526266 = (63.72182730610084 - 67.51615292052068) / 63.72182730610084 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Buffers.Text.Tests.Utf8FormatterTests.FormatterInt32(value: -2147483648)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 18.461227817158658 > 16.66057170563827.
IsChangePoint: Marked as a change because one of 10/3/2022 8:45:37 PM, 10/11/2022 2:16:14 AM falls between 10/2/2022 5:38:34 AM and 10/11/2022 2:16:14 AM.
IsRegressionStdDev: Marked as regression because -109.85241292444329 (T) = (0 -18.388490048367373) / Math.Sqrt((0.003564344671480882 / (28)) + (0.010617424974014515 / (27))) is less than -2.005745995316835 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (28) + (27) - 2, .025) and -0.15780650282893546 = (15.88217893355903 - 18.388490048367373) / 15.88217893355903 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Buffers.Text.Tests.Utf8FormatterTests.FormatterUInt64(value: 12345)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 9.251042835964487 > 8.225190655805443.
IsChangePoint: Marked as a change because one of 10/3/2022 8:45:37 PM, 10/11/2022 2:16:14 AM falls between 10/2/2022 5:38:34 AM and 10/11/2022 2:16:14 AM.
IsRegressionStdDev: Marked as regression because -62.661495870188205 (T) = (0 -9.386376726057215) / Math.Sqrt((0.008733433571281845 / (28)) + (0.008717348374538119 / (27))) is less than -2.005745995316835 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (28) + (27) - 2, .025) and -0.20220425714838552 = (7.807638901829869 - 9.386376726057215) / 7.807638901829869 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Buffers.Text.Tests.Utf8FormatterTests.FormatterUInt32(value: 12345)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 9.03197273345428 > 7.8652182018663686.
IsChangePoint: Marked as a change because one of 9/9/2022 3:12:19 AM, 10/3/2022 8:45:37 PM, 10/11/2022 2:16:14 AM falls between 10/2/2022 5:38:34 AM and 10/11/2022 2:16:14 AM.
IsRegressionStdDev: Marked as regression because -48.232528138947536 (T) = (0 -9.028392764236965) / Math.Sqrt((0.02335899510165956 / (28)) + (0.0014686103444313918 / (27))) is less than -2.005745995316835 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (28) + (27) - 2, .025) and -0.18942134897872537 = (7.590575679500816 - 9.028392764236965) / 7.590575679500816 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

### Run Information

Architecture	x86
OS	Windows 10.0.18362
Baseline	23697bbc6b365fac14d986af7b23b2a908f3f6e0
Compare	a6d396046ab327281e5a436b8d525f0b51d2ce0c
Diff	Diff

Regressions in System.Tests.Perf_Int32

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio	Baseline ETL	Compare ETL
TryFormat - Duration of single invocation	22.68 ns	25.25 ns	1.11	0.14	False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Tests.Perf_Int32*'

Related Issues

Regressions

• [Perf] Windows/x64: 4 Regressions on 10/3/2022 11:31:18 PM
• [Perf] Windows/x86: 6 Regressions on 10/3/2022 11:31:18 PM
• [Perf] Alpine/x64: 1 Regression on 10/3/2022 11:31:18 PM

Improvements

• [Perf] Linux/x64: 18 Improvements on 10/3/2022 11:31:18 PM

Details

Payloads

Baseline
Compare

Histogram

Edge Detector Info

Collection Data

System.Tests.Perf_Int32.TryFormat(value: -2147483648)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 25.253209676768027 > 23.492413317307207.
IsChangePoint: Marked as a change because one of 10/3/2022 8:45:37 PM, 10/11/2022 2:16:14 AM falls between 10/2/2022 5:38:34 AM and 10/11/2022 2:16:14 AM.
IsRegressionStdDev: Marked as regression because -12.5185950978316 (T) = (0 -25.473979481900933) / Math.Sqrt((0.9786087640748207 / (28)) + (0.5832369829044808 / (27))) is less than -2.005745995316835 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (28) + (27) - 2, .025) and -0.13232871150164197 = (22.496982742863175 - 25.473979481900933) / 22.496982742863175 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

[DrewScoggins]

Seems related to #76519. Also see related issues link for improvements from this change as well.

Tagging subscribers to this area: @dotnet/area-system-runtime
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Run Information

Architecture	x86
OS	Windows 10.0.18362
Baseline	23697bbc6b365fac14d986af7b23b2a908f3f6e0
Compare	a6d396046ab327281e5a436b8d525f0b51d2ce0c
Diff	Diff

Regressions in System.Buffers.Text.Tests.Utf8FormatterTests

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio	Baseline ETL	Compare ETL
FormatterUInt32 - Duration of single invocation	14.84 ns	16.88 ns	1.14	0.02	False
FormatterInt64 - Duration of single invocation	63.64 ns	67.60 ns	1.06	0.00	False
FormatterInt32 - Duration of single invocation	15.88 ns	18.46 ns	1.16	0.01	False
FormatterUInt64 - Duration of single invocation	7.85 ns	9.25 ns	1.18	0.02	False
FormatterUInt32 - Duration of single invocation	7.49 ns	9.03 ns	1.21	0.01	False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Buffers.Text.Tests.Utf8FormatterTests*'

Related Issues

Regressions

Improvements

• [Perf] Linux/x64: 18 Improvements on 10/3/2022 11:31:18 PM

Details

Payloads

Baseline
Compare

Histogram

Edge Detector Info

Collection Data

System.Buffers.Text.Tests.Utf8FormatterTests.FormatterUInt32(value: 4294967295)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 16.87672409355907 > 15.551577581679089.
IsChangePoint: Marked as a change because one of 10/3/2022 8:45:37 PM, 10/11/2022 2:16:14 AM falls between 10/2/2022 5:38:34 AM and 10/11/2022 2:16:14 AM.
IsRegressionStdDev: Marked as regression because -86.48783555453538 (T) = (0 -16.910185275233378) / Math.Sqrt((0.006843647038777639 / (28)) + (0.009887396348732362 / (27))) is less than -2.005745995316835 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (28) + (27) - 2, .025) and -0.14466714467413372 = (14.773015329314275 - 16.910185275233378) / 14.773015329314275 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Buffers.Text.Tests.Utf8FormatterTests.FormatterInt64(value: -9223372036854775808)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 67.60444567402155 > 66.8818020437595.
IsChangePoint: Marked as a change because one of 10/3/2022 8:45:37 PM, 10/11/2022 2:16:14 AM falls between 10/2/2022 5:38:34 AM and 10/11/2022 2:16:14 AM.
IsRegressionStdDev: Marked as regression because -42.990433655786944 (T) = (0 -67.51615292052068) / Math.Sqrt((0.1347532172176893 / (28)) + (0.07740645620759257 / (26))) is less than -2.0066468050606243 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (28) + (26) - 2, .025) and -0.059545147633526266 = (63.72182730610084 - 67.51615292052068) / 63.72182730610084 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Buffers.Text.Tests.Utf8FormatterTests.FormatterInt32(value: -2147483648)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 18.461227817158658 > 16.66057170563827.
IsChangePoint: Marked as a change because one of 10/3/2022 8:45:37 PM, 10/11/2022 2:16:14 AM falls between 10/2/2022 5:38:34 AM and 10/11/2022 2:16:14 AM.
IsRegressionStdDev: Marked as regression because -109.85241292444329 (T) = (0 -18.388490048367373) / Math.Sqrt((0.003564344671480882 / (28)) + (0.010617424974014515 / (27))) is less than -2.005745995316835 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (28) + (27) - 2, .025) and -0.15780650282893546 = (15.88217893355903 - 18.388490048367373) / 15.88217893355903 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Buffers.Text.Tests.Utf8FormatterTests.FormatterUInt64(value: 12345)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 9.251042835964487 > 8.225190655805443.
IsChangePoint: Marked as a change because one of 10/3/2022 8:45:37 PM, 10/11/2022 2:16:14 AM falls between 10/2/2022 5:38:34 AM and 10/11/2022 2:16:14 AM.
IsRegressionStdDev: Marked as regression because -62.661495870188205 (T) = (0 -9.386376726057215) / Math.Sqrt((0.008733433571281845 / (28)) + (0.008717348374538119 / (27))) is less than -2.005745995316835 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (28) + (27) - 2, .025) and -0.20220425714838552 = (7.807638901829869 - 9.386376726057215) / 7.807638901829869 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Buffers.Text.Tests.Utf8FormatterTests.FormatterUInt32(value: 12345)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 9.03197273345428 > 7.8652182018663686.
IsChangePoint: Marked as a change because one of 9/9/2022 3:12:19 AM, 10/3/2022 8:45:37 PM, 10/11/2022 2:16:14 AM falls between 10/2/2022 5:38:34 AM and 10/11/2022 2:16:14 AM.
IsRegressionStdDev: Marked as regression because -48.232528138947536 (T) = (0 -9.028392764236965) / Math.Sqrt((0.02335899510165956 / (28)) + (0.0014686103444313918 / (27))) is less than -2.005745995316835 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (28) + (27) - 2, .025) and -0.18942134897872537 = (7.590575679500816 - 9.028392764236965) / 7.590575679500816 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

### Run Information

Architecture	x86
OS	Windows 10.0.18362
Baseline	23697bbc6b365fac14d986af7b23b2a908f3f6e0
Compare	a6d396046ab327281e5a436b8d525f0b51d2ce0c
Diff	Diff

Regressions in System.Tests.Perf_Int32

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio	Baseline ETL	Compare ETL
TryFormat - Duration of single invocation	22.68 ns	25.25 ns	1.11	0.14	False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Tests.Perf_Int32*'

Related Issues

Regressions

• [Perf] Windows/x64: 4 Regressions on 10/3/2022 11:31:18 PM
• [Perf] Windows/x86: 6 Regressions on 10/3/2022 11:31:18 PM
• [Perf] Alpine/x64: 1 Regression on 10/3/2022 11:31:18 PM

Improvements

• [Perf] Linux/x64: 18 Improvements on 10/3/2022 11:31:18 PM

Details

Payloads

Baseline
Compare

Histogram

Edge Detector Info

Collection Data

System.Tests.Perf_Int32.TryFormat(value: -2147483648)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 25.253209676768027 > 23.492413317307207.
IsChangePoint: Marked as a change because one of 10/3/2022 8:45:37 PM, 10/11/2022 2:16:14 AM falls between 10/2/2022 5:38:34 AM and 10/11/2022 2:16:14 AM.
IsRegressionStdDev: Marked as regression because -12.5185950978316 (T) = (0 -25.473979481900933) / Math.Sqrt((0.9786087640748207 / (28)) + (0.5832369829044808 / (27))) is less than -2.005745995316835 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (28) + (27) - 2, .025) and -0.13232871150164197 = (22.496982742863175 - 25.473979481900933) / 22.496982742863175 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Author:	DrewScoggins
Assignees:	stephentoub
Labels:	arch-x86, area-System.Runtime, os-linux, os-windows, tenet-performance, tenet-performance-benchmarks, arch-x64, untriaged, os-linux-musl
Milestone:	-