Doing synchronous writes while chunking can seems to block the thread indefinitely

[rynowak]

Doing some testing on aspnet/Mvc#3516 I ran into this issue.

It's super easy to overwhelm the server when a large Razor page does synchronous writes. Doing 50 requests per second quickly starves the thread pool.

I ended up with about 140 threads blocked on this callstack:

>   mscorlib.dll!System.Threading.Monitor.Wait(object obj, int millisecondsTimeout, bool exitContext) Line 203  C#
    mscorlib.dll!System.Threading.Monitor.Wait(object obj, int millisecondsTimeout) Line 213    C#
    mscorlib.dll!System.Threading.ManualResetEventSlim.Wait(int millisecondsTimeout, System.Threading.CancellationToken cancellationToken) Line 688 C#
    mscorlib.dll!System.Threading.Tasks.Task.SpinThenBlockingWait(int millisecondsTimeout, System.Threading.CancellationToken cancellationToken) Line 3320  C#
    mscorlib.dll!System.Threading.Tasks.Task.InternalWait(int millisecondsTimeout, System.Threading.CancellationToken cancellationToken) Line 3252  C#
    mscorlib.dll!System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(System.Threading.Tasks.Task task) Line 148 C#
    Microsoft.AspNet.Server.Kestrel.dll!Microsoft.AspNet.Server.Kestrel.Http.SocketOutput.Microsoft.AspNet.Server.Kestrel.Http.ISocketOutput.Write(System.ArraySegment<byte> buffer, bool immediate)    Unknown
    Microsoft.AspNet.Server.Kestrel.dll!Microsoft.AspNet.Server.Kestrel.Http.Frame.WriteChunked(System.ArraySegment<byte> data) Unknown
    Microsoft.AspNet.Server.Kestrel.dll!Microsoft.AspNet.Server.Kestrel.Http.Frame.Write(System.ArraySegment<byte> data)    Unknown
    Microsoft.AspNet.Server.Kestrel.dll!Microsoft.AspNet.Server.Kestrel.Http.FrameResponseStream.Write(byte[] buffer, int offset, int count)    Unknown
    Microsoft.AspNet.Mvc.Core!Microsoft.AspNet.Mvc.HttpResponseStreamWriter.FlushInternal() Line 334    C#
    Microsoft.AspNet.Mvc.Core!Microsoft.AspNet.Mvc.HttpResponseStreamWriter.Write(string value) Line 194    C#
    Microsoft.AspNet.Mvc.ViewFeatures!Microsoft.AspNet.Mvc.ViewFeatures.Buffer.ViewBufferValue.WriteTo(System.IO.TextWriter writer, System.Text.Encodings.Web.HtmlEncoder encoder) Line 55  C#
    Microsoft.AspNet.Mvc.ViewFeatures!Microsoft.AspNet.Mvc.ViewFeatures.Buffer.ViewBuffer.WriteTo(System.IO.TextWriter writer, System.Text.Encodings.Web.HtmlEncoder encoder) Line 145  C#
    Microsoft.AspNet.Mvc.Razor!Microsoft.AspNet.Mvc.Razor.RazorView.RenderLayoutAsync(Microsoft.AspNet.Mvc.Rendering.ViewContext context, Microsoft.AspNet.Mvc.Razor.RazorTextWriter bodyWriter) Line 236   C#
    Microsoft.AspNet.Mvc.Razor!Microsoft.AspNet.Mvc.Razor.RazorView.RenderAsync(Microsoft.AspNet.Mvc.Rendering.ViewContext context) Line 102    C#
    Microsoft.AspNet.Mvc.ViewFeatures!Microsoft.AspNet.Mvc.ViewFeatures.ViewExecutor.ExecuteAsync(Microsoft.AspNet.Mvc.ActionContext actionContext, Microsoft.AspNet.Mvc.ViewEngines.IView view, Microsoft.AspNet.Mvc.ViewFeatures.ViewDataDictionary viewData, Microsoft.AspNet.Mvc.ViewFeatures.ITempDataDictionary tempData, Microsoft.Net.Http.Headers.MediaTypeHeaderValue contentType, int? statusCode) Line 180  C#
    Microsoft.AspNet.Mvc.ViewFeatures!Microsoft.AspNet.Mvc.ViewFeatures.ViewResultExecutor.ExecuteAsync(Microsoft.AspNet.Mvc.ActionContext actionContext, Microsoft.AspNet.Mvc.ViewEngines.IView view, Microsoft.AspNet.Mvc.ViewResult viewResult) Line 165 C#
    Microsoft.AspNet.Mvc.ViewFeatures!Microsoft.AspNet.Mvc.ViewResult.ExecuteResultAsync(Microsoft.AspNet.Mvc.ActionContext context) Line 75    C#
    Microsoft.AspNet.Mvc.Core!Microsoft.AspNet.Mvc.Controllers.FilterActionInvoker.InvokeResultAsync(Microsoft.AspNet.Mvc.IActionResult result) Line 884    C#
    Microsoft.AspNet.Mvc.Core!Microsoft.AspNet.Mvc.Controllers.FilterActionInvoker.InvokeResultFilterAsync() Line 853   C#
    Microsoft.AspNet.Mvc.Core!Microsoft.AspNet.Mvc.Controllers.FilterActionInvoker.InvokeResultFilterAsync() Line 835   C#
    Microsoft.AspNet.Mvc.Core!Microsoft.AspNet.Mvc.Controllers.FilterActionInvoker.InvokeAllResultFiltersAsync(Microsoft.AspNet.Mvc.IActionResult result) Line 738  C#
    Microsoft.AspNet.Mvc.Core!Microsoft.AspNet.Mvc.Controllers.FilterActionInvoker.InvokeResourceFilterAsync() Line 466 C#
    Microsoft.AspNet.Mvc.Core!Microsoft.AspNet.Mvc.Controllers.FilterActionInvoker.InvokeResourceFilterAsync() Line 399 C#
    Microsoft.AspNet.Mvc.Core!Microsoft.AspNet.Mvc.Controllers.FilterActionInvoker.InvokeAllResourceFiltersAsync() Line 313 C#
    Microsoft.AspNet.Mvc.Core!Microsoft.AspNet.Mvc.Controllers.FilterActionInvoker.InvokeAsync() Line 156   C#
    Microsoft.AspNet.Mvc.Core!Microsoft.AspNet.Mvc.Infrastructure.MvcRouteHandler.InvokeActionAsync(Microsoft.AspNet.Http.HttpContext httpContext, Microsoft.AspNet.Mvc.Abstractions.ActionDescriptor actionDescriptor) Line 102    C#
    Microsoft.AspNet.Mvc.Core!Microsoft.AspNet.Mvc.Infrastructure.MvcRouteHandler.RouteAsync.AnonymousMethod__0(Microsoft.AspNet.Http.HttpContext c) Line 71    C#
    Microsoft.AspNet.Routing.dll!Microsoft.AspNet.Builder.RouterMiddleware.Invoke(Microsoft.AspNet.Http.HttpContext httpContext) Line 48    C#
    Microsoft.AspNet.Diagnostics.dll!Microsoft.AspNet.Diagnostics.ExceptionHandlerMiddleware.Invoke(Microsoft.AspNet.Http.HttpContext context) Line 44  C#
    Microsoft.AspNet.Hosting.dll!Microsoft.AspNet.Hosting.Internal.RequestServicesContainerMiddleware.Invoke(Microsoft.AspNet.Http.HttpContext httpContext) Line 55 C#
    Microsoft.AspNet.Hosting.dll!Microsoft.AspNet.Hosting.Internal.HostingApplication.ProcessRequestAsync(Microsoft.AspNet.Hosting.Internal.HostingApplication.Context context) Line 81 C#
    Microsoft.AspNet.Server.Kestrel.dll!Microsoft.AspNet.Server.Kestrel.Http.Frame<Microsoft.AspNet.Hosting.Internal.HostingApplication.Context>.RequestProcessingAsync()   Unknown
    mscorlib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder.Start<Microsoft.AspNet.Server.Kestrel.Http.Frame<Microsoft.AspNet.Hosting.Internal.HostingApplication.Context>.<RequestProcessingAsync>d__3>(ref Microsoft.AspNet.Server.Kestrel.Http.Frame<Microsoft.AspNet.Hosting.Internal.HostingApplication.Context>.<RequestProcessingAsync>d__3 stateMachine) Line 316   C#
    Microsoft.AspNet.Server.Kestrel.dll!Microsoft.AspNet.Server.Kestrel.Http.Frame<Microsoft.AspNet.Hosting.Internal.HostingApplication.Context>.RequestProcessingAsync()   Unknown
    Microsoft.AspNet.Server.Kestrel.dll!Microsoft.AspNet.Server.Kestrel.Http.Frame.Start.AnonymousMethod__111_0(object o)   Unknown
    mscorlib.dll!System.Threading.Tasks.Task<System.Threading.Tasks.Task>.InnerInvoke() Line 686    C#
    mscorlib.dll!System.Threading.Tasks.Task.Execute() Line 2498    C#
    mscorlib.dll!System.Threading.Tasks.Task.ExecutionContextCallback(object obj) Line 2861 C#
    mscorlib.dll!System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state, bool preserveSyncCtx) Line 954  C#
    mscorlib.dll!System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state, bool preserveSyncCtx) Line 902  C#
    mscorlib.dll!System.Threading.Tasks.Task.ExecuteWithThreadLocal(ref System.Threading.Tasks.Task currentTaskSlot) Line 2827  C#
    mscorlib.dll!System.Threading.Tasks.Task.ExecuteEntry(bool bPreventDoubleExecution) Line 2767   C#
    mscorlib.dll!System.Threading.Tasks.Task.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem() Line 2704   C#
    mscorlib.dll!System.Threading.ThreadPoolWorkQueue.Dispatch() Line 820   C#
    mscorlib.dll!System.Threading._ThreadPoolWaitCallback.PerformWaitCallback() Line 1161   C#

It seems like these threads don't make any progress once they get into this state, and the server is unresponsive. More threads are slowly spawned to handle the queued requests until the server OOMs.

I suspect the tasks created by Write aren't completed once the client is gone.

[rynowak]

We'll be changing Razor to avoid synchronous I/O to play better with libuv. There looks like there's still a real bug here.

[benaadams]

I might have bumped into something similar; though a different route (last statement in issue makes me think this), which am investigating.

Same issue; using WriteAsync over Ssl does sync writes :(

https://github.com/aspnet/KestrelHttpServer/blob/dev/src/Microsoft.AspNet.Server.Kestrel/Filter/StreamSocketOutput.cs#L35

[benaadams]

K, I think I've got this one... #567

[benaadams]

However; still good to go async as you will be paying all the async costs anyway but also blocking threads with the sync route.

If you go async you might be able to do overlapped generation+send for an additional benefit? e.g.

var lastTask = Task.CompletedTask;
var dataBuffer = bytePool.Rent(bufferLength);
try
{
    while (stuffToDo)
    {
        GenerateByteData(dataBuffer);
        await lastTask;
        lastTask = Response.WriteAsync(dataBuffer);
    }
    await lastTask;
}
finally
{
    bytePool.Return(dataBuffer);
}

[benaadams]

Beyond "SocketOutput exception completes all waiting tasks" #567
and "Write chunks async" #586

The next step would be the threadpool thread count/create speed as the threads are waiting https://github.com/dotnet/coreclr/issues/1606#issuecomment-144007622

http://referencesource.microsoft.com/#System.Web/HttpRuntime.cs,1207 was something like 20 per core in aspnet4 as max by default? (maxWorkerThreads) with maxConcurrentRequestsPerCPU at 5000. Which even then would suggest a max of 20 sync writes per core.

Which apparently the "hosting" controls the threadpool min threads? /cc @davidfowl
(no api for user code in coreclr)

[benaadams]

Hmmm... but... aspnet4 did use 2 threadpools; on for IO completion and one for user code; whereas here blocking usercode can block IO completion notifications.

[benaadams]

Raised issue https://github.com/dotnet/coreclr/issues/2712

[benaadams]

Future solution dotnet/coreclr#2718
(When using RunContinuationsAsynchronously and not queuing to Threadpool as Task will handle it)

Meanwhile fix should be in #586