Support inline application scheduling mode

[tmds]

Using the inline scheduling mode, Pipelines can be configured to not dispatch to ThreadPool(/IoQueue).
This mode is specifically for non-blocking applications.
It's meant to be use in tandem with the inline socket continuation mode of runtime added in dotnet/runtime#34945.

cc @davidfowl @halter73

[adamsitnik]

@halter73 is there any dirty way I could check how this would affect the platform benchmarks using reflection or some other hacks?

[davidfowl]

@adamsitnik reflection, no, you have to change the code here

aspnetcore/src/Servers/Kestrel/Transport.Sockets/src/Internal/SocketConnection.cs

Lines 71 to 72 in 02e4d1a

	var inputOptions = new PipeOptions(MemoryPool, PipeScheduler.ThreadPool, scheduler, maxReadBufferSize.Value, maxReadBufferSize.Value / 2, useSynchronizationContext: false);
	var outputOptions = new PipeOptions(MemoryPool, scheduler, PipeScheduler.ThreadPool, maxWriteBufferSize.Value, maxWriteBufferSize.Value / 2, useSynchronizationContext: false);

[adamsitnik]

@davidfowl thank you!

[halter73]

For now you could just copy the SocketTransport, then change those two lines and the name and register it. Or you could use rhtx in its inline mode if you don't need to use the Socket transport exactly.

You could also look at modifying KestrelServer.ServiceContext.Scheduler to be Inline instead of ThreadPool. You should be able to grab the KestrelServer instance out of DI as an IServer. I don't think this should really matter though unless you're testing with HTTP/2-3 requests and/or chunked HTTP/1 request bodies.

[adamsitnik]

@tmds I've run all the benchmarks for 2 configurations of aspnet:

• 4x inline scheduler

var inputOptions = new PipeOptions(MemoryPool, PipeScheduler.Inline, PipeScheduler.Inline, maxReadBufferSize.Value, maxReadBufferSize.Value / 2, useSynchronizationContext: false);
var outputOptions = new PipeOptions(MemoryPool, PipeScheduler.Inline, PipeScheduler.Inline, maxWriteBufferSize.Value, maxWriteBufferSize.Value / 2, useSynchronizationContext: false);

• 2x inline 2x TP

var inputOptions = new PipeOptions(MemoryPool, PipeScheduler.ThreadPool, PipeScheduler.Inline, maxReadBufferSize.Value, maxReadBufferSize.Value / 2, useSynchronizationContext: false);
var outputOptions = new PipeOptions(MemoryPool, PipeScheduler.Inline, PipeScheduler.ThreadPool, maxWriteBufferSize.Value, maxWriteBufferSize.Value / 2, useSynchronizationContext: false);