[Perf -138%] System.Buffers.Tests.RentReturnArrayPoolTests<Byte>.ProducerConsumer

[DrewScoggins]

Run Information

Architecture	x64
OS	Windows 10.0.18362
Changes	diff

Regressions in System.Buffers.Tests.RentReturnArrayPoolTests

Benchmark	Baseline	Test	Test/Base	Modality	Baseline Outlier
ProducerConsumer	227.31 ns	567.92 ns	2.50		False

Historical Data in Reporting System

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f netcoreapp5.0 --filter 'System.Buffers.Tests.RentReturnArrayPoolTests<Byte>*';

Details

Histogram

System.Buffers.Tests.RentReturnArrayPoolTests.ProducerConsumer(RentalSize: 4096, ManipulateArray: False, Async: True, UseSharedPool: True)

[158.326 ; 189.781) | @
[189.781 ; 253.830) | @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
[253.830 ; 311.112) | @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
[311.112 ; 370.078) | @@@@@@@@@@@@@@@
[370.078 ; 431.574) | @@@@@@@@@@@@@
[431.574 ; 489.706) | @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
[489.706 ; 546.987) | @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
[546.987 ; 593.916) | @@@@@@@@@
[593.916 ; 651.198) | 
[651.198 ; 705.890) | 
[705.890 ; 763.172) | @
[763.172 ; 783.237) | 
[783.237 ; 840.519) | @

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Tagging subscribers to this area: @tannergooding, @pgovind
Notify danmosemsft if you want to be subscribed.

[stephentoub]

For reference, this is what I get on my machine currently:
dotnet run -c Release -f netcoreapp3.1 --runtimes netcoreapp3.1 netcoreapp5.0 --filter="System.Buffers.Tests.RentReturnArrayPoolTests.ProducerConsumer"

Method	Job	Runtime	Toolchain	RentalSize	ManipulateArray	Async	UseSharedPool	Mean	Error	StdDev	Median	Min	Max	Ratio
ProducerConsumer	Job-VPWHSS	.NET Core 3.1	netcoreapp3.1	4096	False	False	False	994.7 ns	6.55 ns	6.13 ns	993.4 ns	982.3 ns	1,007.0 ns	1.00
ProducerConsumer	Job-HIDTAI	.NET Core 5.0	netcoreapp5.0	4096	False	False	False	984.5 ns	23.23 ns	26.75 ns	973.9 ns	943.6 ns	1,027.3 ns	1.00
ProducerConsumer	Job-VPWHSS	.NET Core 3.1	netcoreapp3.1	4096	False	False	True	900.9 ns	72.32 ns	83.29 ns	915.1 ns	751.2 ns	993.2 ns	1.00
ProducerConsumer	Job-HIDTAI	.NET Core 5.0	netcoreapp5.0	4096	False	False	True	973.5 ns	32.02 ns	35.59 ns	975.9 ns	897.1 ns	1,033.0 ns	1.08
ProducerConsumer	Job-VPWHSS	.NET Core 3.1	netcoreapp3.1	4096	False	True	False	787.2 ns	14.46 ns	12.82 ns	784.0 ns	773.0 ns	818.0 ns	1.00
ProducerConsumer	Job-HIDTAI	.NET Core 5.0	netcoreapp5.0	4096	False	True	False	805.4 ns	24.08 ns	27.73 ns	790.8 ns	774.4 ns	868.0 ns	1.01
ProducerConsumer	Job-VPWHSS	.NET Core 3.1	netcoreapp3.1	4096	False	True	True	212.8 ns	33.19 ns	38.22 ns	211.7 ns	157.5 ns	276.1 ns	1.00
ProducerConsumer	Job-HIDTAI	.NET Core 5.0	netcoreapp5.0	4096	False	True	True	329.0 ns	39.39 ns	43.78 ns	328.4 ns	265.7 ns	414.3 ns	1.58
ProducerConsumer	Job-VPWHSS	.NET Core 3.1	netcoreapp3.1	4096	True	False	False	2,025.1 ns	37.84 ns	37.16 ns	2,011.9 ns	1,981.1 ns	2,098.5 ns	1.00
ProducerConsumer	Job-HIDTAI	.NET Core 5.0	netcoreapp5.0	4096	True	False	False	2,104.4 ns	50.89 ns	54.45 ns	2,111.0 ns	1,999.8 ns	2,219.6 ns	1.04
ProducerConsumer	Job-VPWHSS	.NET Core 3.1	netcoreapp3.1	4096	True	False	True	2,120.9 ns	115.41 ns	132.90 ns	2,096.0 ns	1,939.9 ns	2,355.9 ns	1.00
ProducerConsumer	Job-HIDTAI	.NET Core 5.0	netcoreapp5.0	4096	True	False	True	2,112.0 ns	94.69 ns	109.04 ns	2,098.3 ns	1,948.9 ns	2,338.6 ns	1.00
ProducerConsumer	Job-VPWHSS	.NET Core 3.1	netcoreapp3.1	4096	True	True	False	2,038.7 ns	9.23 ns	8.18 ns	2,036.6 ns	2,022.9 ns	2,055.0 ns	1.00
ProducerConsumer	Job-HIDTAI	.NET Core 5.0	netcoreapp5.0	4096	True	True	False	2,004.0 ns	13.64 ns	12.09 ns	2,003.6 ns	1,978.2 ns	2,022.3 ns	0.98
ProducerConsumer	Job-VPWHSS	.NET Core 3.1	netcoreapp3.1	4096	True	True	True	2,188.2 ns	36.04 ns	33.71 ns	2,189.6 ns	2,132.2 ns	2,253.4 ns	1.00
ProducerConsumer	Job-HIDTAI	.NET Core 5.0	netcoreapp5.0	4096	True	True	True	2,178.3 ns	43.93 ns	50.59 ns	2,168.9 ns	2,068.6 ns	2,259.2 ns	1.00

I'm assuming the one that stands out is the same as the one called out in this issue, but I can't see which inputs to the theory are being cited in the issue (other than "4096"... everything else is cut off in the image):

Method	Job	Runtime	Toolchain	RentalSize	ManipulateArray	Async	UseSharedPool	Mean	Error	StdDev	Median	Min	Max	Ratio
ProducerConsumer	Job-VPWHSS	.NET Core 3.1	netcoreapp3.1	4096	False	True	True	212.8 ns	33.19 ns	38.22 ns	211.7 ns	157.5 ns	276.1 ns	1.00
ProducerConsumer	Job-HIDTAI	.NET Core 5.0	netcoreapp5.0	4096	False	True	True	329.0 ns	39.39 ns	43.78 ns	328.4 ns	265.7 ns	414.3 ns	1.58

[DrewScoggins]

This is the one where we are seeing a regression and it does match up with what you are seeing in your local testing. I found this by clicking on the Historical Data in Reporting System link in the issue, but I will also file a bug for fixing title font size on long tests names.

System.Buffers.Tests.RentReturnArrayPoolTests<Byte>.ProducerConsumer(RentalSize: 4096, ManipulateArray: False, Async: True, UseSharedPool: True)

[DrewScoggins]

Run Information

Architecture	x64
OS	ubuntu 18.04
Changes	diff

Regressions in System.Buffers.Tests.RentReturnArrayPoolTests

Benchmark	Baseline	Test	Test/Base	Modality	Baseline Outlier
ProducerConsumer	443.11 ns	634.82 ns	1.43	Bimodal	False
ProducerConsumer	615.21 ns	751.29 ns	1.22		True

Related Issue on x64 Windows

[Perf -138%] System.Buffers.Tests.RentReturnArrayPoolTests.ProducerConsumer

Related Issue on x86 Windows

[Perf -359%] System.Buffers.Tests.RentReturnArrayPoolTests (2)

Historical Data in Reporting System

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f netcoreapp5.0 --filter 'System.Buffers.Tests.RentReturnArrayPoolTests<Byte>*';

Details

Histogram

System.Buffers.Tests.RentReturnArrayPoolTests.ProducerConsumer(RentalSize: 4096, ManipulateArray: False, Async: True, UseSharedPool: True)

[362.360 ; 402.651) | @@
[402.651 ; 444.857) | @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
[444.857 ; 461.748) | @@@
[461.748 ; 513.730) | @@@@@@@@@@@@@
[513.730 ; 563.598) | @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
[563.598 ; 611.912) | @@@@@@@@@@@@@@@@@@@@
[611.912 ; 654.119) | @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
[654.119 ; 693.997) | @@@@@@@@@@@@@
[693.997 ; 714.683) | @
[714.683 ; 756.890) | @@@@@@@@@@@@
[756.890 ; 786.963) | @

System.Buffers.Tests.RentReturnArrayPoolTests.ProducerConsumer(RentalSize: 4096, ManipulateArray: False, Async: True, UseSharedPool: False)

[464.052 ; 492.206) | @
[492.206 ; 528.709) | @@@@@@
[528.709 ; 562.585) | @@@@@@@@@@@@@@@@@@
[562.585 ; 585.725) | @@@@@
[585.725 ; 619.601) | @@@@@@@@@@@@@@@@@@@
[619.601 ; 657.384) | @@@@@@@@@
[657.384 ; 671.155) | @
[671.155 ; 705.031) | @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
[705.031 ; 739.166) | @@@@@@@@@@@@@@@@@@@@@@@@@@@
[739.166 ; 773.042) | @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
[773.042 ; 809.388) | @@@@@@@@@@@@@@@

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

[DrewScoggins]

We are also seeing this behavior on Ubuntu x64 and Windows x86