Performance regression in preview 4 to 7

[Rodrigo-Andrade]

I usually test previews on a reverse proxy app that has a decent load (10-20 million requests per minute).

The current one is using .net5 preview3, and it was a bit faster then 3.1, but preview4-7 its consistently slower.

This regression is consistent, 4 to 7 has the same performance.

The machines are running on aws, they have the same machine type and the same linux version (Amazon Linux) and the same version of docker. The only thing that i change is literally the number of the preview on the DockerFile.

Doing some profiling on windows (i'm still struggling a bit to profile on linux) the only thing that seems to change is the scheduling on the Thread Pool and the sockets performance that seems a bit slower, but that's on windows.

My apologies for an issue that is not very actionable, if you need more details ill do my best to provide.
If there is another channel for this kind of feedback, or this is undesired feedback please let me know.

[Pilchie]

@sebastienros can you follow up to get a trace and take a look at what's going on?

[Rodrigo-Andrade]

Got a collect on the machine outside docker (its slower inside docker), but this here is weird:

System.Threading.ThreadPoolWorkQueue.EnsureThreadRequested() takes a long while some times.

[sebastienros]

I have asked another dev if there are known regressions onthe TP for preview7.
Have you tried with the nightly build of .NET ? Check for the "master" column on this page: https://github.com/dotnet/installer#installers-and-binaries

Something to note is that if the latest, or even preview7 in this case, is still faster than 3.1, it will probably not be looked at for 5.0 as we are getting closer to shipping RCs.

In the meantime you might want to give me some sample .NET app that you think shows the regression so I can measure it myself. Including the kind of payload you use, the methodology (machines, tools), and the numbers you got.

[sebastienros]

Maybe I id read the issue too quickly, are you saying that anything version between preview4 up to preview7 is slower than 3.1 on your tests?

[Rodrigo-Andrade]

@sebastienros yes

[sebastienros]

Then please provide ways to repro the issue, because we are only seeing improvements in perf from 3.1 to 5.0 preview.
And would it be possible for you to share traces on 3.1 and preview 7 so we can see what code paths induce the long EnsureThreadRequested calls? I can provide you with a shared driver to upload these

[Rodrigo-Andrade]

@sebastienros I managed to get traces from preview3 and preview7, you can message me the link.

[sebastienros]

Not sure we really care about preview 3 in this case but more about .NET 3.1 since you say the regression is from this version. Also can you share your benchmarking strategy, and if possible the kind of code is running? Is we can't report the issue we won't be able to fix it.

[Rodrigo-Andrade]

@sebastienros I don't think i get it, preview3 is the last version that don't have the regression, its closer to the code on preview7 (so should be easier to see what changed)?

The app is a reverse proxy , so a Middleware that forwards requests with HttpClient, so most of the time is spent on kestrel and HttpClient.

For tracing i'm using dotnet-trace on a aws ec2 machine with 8 virtual cores. Both traces were made on the same machine.

Ill try to make a small repro, if you still want the traces let me know.

[davidfowl]

cc @kouvel

[Rodrigo-Andrade]

Made a repro:

https://gist.github.com/Rodrigo-Andrade/dabd77a2df360b75ace7e9391fabcddc

Tests on the same machine (aws ec2 c5.2xlarge).

.NET Core SDK (reflecting any global.json):
Version: 3.1.302
Commit: 41faccf259

Runtime Environment:
OS Name: amzn
OS Version: 2018.03
OS Platform: Linux
RID: linux-x64
Base Path: /root/dotnet3.1/sdk/3.1.302/

Host (useful for support):
Version: 3.1.6
Commit: 3acd9b0cd1

.NET Core SDKs installed:
3.1.302 [/root/dotnet3.1/sdk]

.NET Core runtimes installed:
Microsoft.AspNetCore.App 3.1.6 [/root/dotnet3.1/shared/Microsoft.AspNetCore.App]
Microsoft.NETCore.App 3.1.6 [/root/dotnet3.1/shared/Microsoft.NETCore.App]

Timmings:

4385.93
3045.00
3042.74
3069.86
3074.63

.NET Core SDK (reflecting any global.json):
Version: 5.0.100-preview.3.20216.6
Commit: 9f62a32109

Runtime Environment:
OS Name: amzn
OS Version: 2018.03
OS Platform: Linux
RID: linux-x64
Base Path: /root/dotnet3/sdk/5.0.100-preview.3.20216.6/

Host (useful for support):
Version: 5.0.0-preview.3.20214.6
Commit: b037784658

.NET SDKs installed:
5.0.100-preview.3.20216.6 [/root/dotnet3/sdk]

.NET runtimes installed:
Microsoft.AspNetCore.App 5.0.0-preview.3.20215.14 [/root/dotnet3/shared/Microsoft.AspNetCore.App]
Microsoft.NETCore.App 5.0.0-preview.3.20214.6 [/root/dotnet3/shared/Microsoft.NETCore.App]

Timmings:

4059.56
2951.67
2906.33
2962.40
2993.35

.NET SDK (reflecting any global.json):
Version: 5.0.100-preview.7.20366.6
Commit: 0684df3a5b

Runtime Environment:
OS Name: amzn
OS Version: 2018.03
OS Platform: Linux
RID: linux-x64
Base Path: /root/dotnet7/sdk/5.0.100-preview.7.20366.6/

Host (useful for support):
Version: 5.0.0-preview.7.20364.11
Commit: 53976d38b1

.NET SDKs installed:
5.0.100-preview.7.20366.6 [/root/dotnet7/sdk]

.NET runtimes installed:
Microsoft.AspNetCore.App 5.0.0-preview.7.20365.19 [/root/dotnet7/shared/Microsoft.AspNetCore.App]
Microsoft.NETCore.App 5.0.0-preview.7.20364.11 [/root/dotnet7/shared/Microsoft.NETCore.App]

Timings:

3463.12
2827.01
2796.74
2765.77
2749.04

[sebastienros]

Thanks. We already have a proxy scenario similar to what you did, but using different machine (3). And in our tests it's much faster with 5.0.

Before I try to run your app, or simulate the same payload on our benchmarks, it looks like the numbers are better in preview 3 and even better in preview 7. Why are you saying there is a regression? Do I read it wrong?