[Perf -26%] System.Net.Http.Tests.SocketsHttpHandlerPerfTest.Get

[DrewScoggins]

Run Information

Architecture	x64
OS	ubuntu 18.04
Changes	diff

Regressions in System.Net.Http.Tests.SocketsHttpHandlerPerfTest

Benchmark	Baseline	Test	Test/Base	Modality	Baseline Outlier
Get	39.50 μs	51.12 μs	1.29		True

Historical Data in Reporting System

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f netcoreapp5.0 --filter 'System.Net.Http.Tests.SocketsHttpHandlerPerfTest*';

Details

Histogram

System.Net.Http.Tests.SocketsHttpHandlerPerfTest.Get(ssl: False, chunkedResponse: False, responseLength: 100000)

[37006.102 ; 38194.341) | @@@
[38194.341 ; 39360.392) | @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
[39360.392 ; 40607.480) | @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
[40607.480 ; 41837.718) | @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
[41837.718 ; 42774.964) | @@@@@@@@@@@@@@@@
[42774.964 ; 43657.232) | 
[43657.232 ; 44904.320) | @
[44904.320 ; 46151.408) | 
[46151.408 ; 47398.496) | 
[47398.496 ; 48645.584) | 
[48645.584 ; 50445.380) | @@@@@@@
[50445.380 ; 52126.865) | @@@@
[52126.865 ; 53544.893) | @@@@

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Tagging subscribers to this area: @dotnet/ncl
Notify danmosemsft if you want to be subscribed.

[danmoseley]

Diff link broken. Need a commit range in this repo.

[DrewScoggins]

#39122 (comment)

This should help you track down the diff.

[ManickaP]

@DrewScoggins Please correct if I'm reading the graphs wrong, but the regression happened between 2.5.-4.5. on runtime master?

[stephentoub]

I can't reproduce this locally, on either Linux or Windows. Every run I've done on both shows .NET 5.0 measurably faster on all variants of the benchmark.

e.g.

dotnet run -c Release -f netcoreapp3.1 --runtimes netcoreapp3.1 netcoreapp5.0 --join --filter *System.Net.Http.Tests.SocketsHttpHandlerPerfTest.Get*

Method	Job	Runtime	Toolchain	ssl	chunkedResponse	responseLength	Mean	Error	StdDev	Median	Min	Max	Ratio	RatioSD	Gen 0	Gen 1	Gen 2	Allocated
Get	Job-SZZVVV	.NET Core 3.1	netcoreapp3.1	False	False	1	131.6 us	2.52 us	2.48 us	131.9 us	126.4 us	135.4 us	1.00	0.00	-	-	-	2.19 KB
Get	Job-XFDOKA	.NET Core 5.0	netcoreapp5.0	False	False	1	124.7 us	2.13 us	1.99 us	125.0 us	120.3 us	128.5 us	0.95	0.02	-	-	-	2.07 KB
Get	Job-SZZVVV	.NET Core 3.1	netcoreapp3.1	False	False	100000	158.2 us	2.41 us	2.25 us	158.2 us	154.7 us	161.2 us	1.00	0.00	-	-	-	2.2 KB
Get	Job-XFDOKA	.NET Core 5.0	netcoreapp5.0	False	False	100000	146.5 us	2.00 us	1.87 us	146.1 us	143.7 us	149.6 us	0.93	0.01	-	-	-	2.08 KB
Get	Job-SZZVVV	.NET Core 3.1	netcoreapp3.1	False	True	1	133.6 us	2.25 us	2.11 us	133.2 us	130.1 us	137.7 us	1.00	0.00	-	-	-	2.19 KB
Get	Job-XFDOKA	.NET Core 5.0	netcoreapp5.0	False	True	1	124.7 us	2.06 us	1.93 us	124.3 us	121.1 us	128.1 us	0.93	0.02	-	-	-	2.07 KB
Get	Job-SZZVVV	.NET Core 3.1	netcoreapp3.1	False	True	100000	184.4 us	4.46 us	5.13 us	182.5 us	178.1 us	194.6 us	1.00	0.00	-	-	-	2.19 KB
Get	Job-XFDOKA	.NET Core 5.0	netcoreapp5.0	False	True	100000	160.2 us	1.94 us	1.82 us	159.5 us	157.6 us	162.9 us	0.87	0.02	-	-	-	2.06 KB
Get	Job-SZZVVV	.NET Core 3.1	netcoreapp3.1	True	False	1	164.3 us	3.61 us	4.16 us	164.0 us	156.3 us	173.1 us	1.00	0.00	-	-	-	2.86 KB
Get	Job-XFDOKA	.NET Core 5.0	netcoreapp5.0	True	False	1	145.2 us	1.66 us	1.47 us	145.6 us	142.1 us	147.3 us	0.88	0.02	-	-	-	2.46 KB
Get	Job-SZZVVV	.NET Core 3.1	netcoreapp3.1	True	False	100000	7,863.0 us	1,186.23 us	1,366.07 us	7,996.4 us	6,359.7 us	9,813.2 us	1.00	0.00	-	-	-	3.03 KB
Get	Job-XFDOKA	.NET Core 5.0	netcoreapp5.0	True	False	100000	7,514.0 us	1,734.49 us	1,997.44 us	7,146.3 us	4,286.0 us	11,377.4 us	0.99	0.32	-	-	-	2.72 KB
Get	Job-SZZVVV	.NET Core 3.1	netcoreapp3.1	True	True	1	167.3 us	3.33 us	3.27 us	166.8 us	162.9 us	175.4 us	1.00	0.00	-	-	-	2.86 KB
Get	Job-XFDOKA	.NET Core 5.0	netcoreapp5.0	True	True	1	150.4 us	2.31 us	2.04 us	150.2 us	147.0 us	153.7 us	0.90	0.02	-	-	-	2.46 KB
Get	Job-SZZVVV	.NET Core 3.1	netcoreapp3.1	True	True	100000	8,239.2 us	1,144.77 us	1,272.40 us	8,149.6 us	5,434.7 us	10,879.9 us	1.00	0.00	-	-	-	3.02 KB
Get	Job-XFDOKA	.NET Core 5.0	netcoreapp5.0	True	True	100000	7,424.2 us	1,081.84 us	1,202.47 us	8,013.7 us	4,040.3 us	9,393.5 us	0.92	0.21	-	-	-	2.58 KB

[karelz]

@DrewScoggins any thoughts on @stephentoub's reply? Looks like we can't reproduce it outside of perf lab :(
cc @adamsitnik