SocketSender/Receiver: also Inline continuations on Windows

[tmds]

Some benchmarks we ran on Linux indicate this may be interesting on Windows.

cc @halter73 @davidfowl @sebastienros @adamsitnik

[analogrelay]

Triage: we do have some hunch that there's a reason we didn't inline continuations on Windows. @davidfowl @halter73 does that ring a bell?

[davidfowl]

@halter73 wrote up the reason we use IOQueues here dotnet/runtime#2346 (comment).

[tmds]

In theory, Inline should be faster than IOQueue, and IOQueue should be faster than ThreadPool.
In practice, it doesn't seem to always work that way.
So let's measure and see the effect.

[tmds]

Can we run a benchmark to see the effect?

[shirhatti]

Triage: We've tried this before, and it didn't end well aspnet/KestrelHttpServer#2573

[tmds]

Triage: We've tried this before, and it didn't end well aspnet/KestrelHttpServer#2573

@shirhatti that issue says it didn't end well on Linux.

[shirhatti]

Ah shoot @halter73?

[tmds]

@halter73 if you know it is faster to use IoQueue here on Windows, I'm fine if the issue gets closed. Otherwise we can run a benchmark and find out.

[halter73]

For SocketReceiver at least, dispatching receive continuations on Windows, even to the threadpool significantly improved perf in the plaintext and json benchmarks. @davidfowl first proved that with aspnet/KestrelHttpServer#2366.

Dispatching to the IOQueue which was introduced by aspnet/KestrelHttpServer#2368 showed even further improvement compared to dispatching to the threadpool. You can look at both PRs for the benchmark results we got at the time on Windows.

For the SocketSender, the difference was within the noise level IIRC, but that makes sense since sends should always complete immediately when there's no backpressure like in the plaintext and json benchmarks.