Kestrel: Deadlocked in SocketOutput?

[physhi]

There seems to be a dead lock in SocketOutput (and kestrel stops accepting new connections). My server does lock up randomly. See below the two sets of threads that may be causing deadlock. I looked at the code and these two sets of stack does seem to have possibility of deadlocking.

I'd taken 2 other dumps and I saw same pair of stacks getting stuck.

Since I can't repro the deadlock, I can't really tell what's causing these threads to get in deadlock. This issue could be related to #1267, but big difference is that in my case CPU goes down to zero.

[davidfowl]

@physhi do you pass a cancellation token to any WriteAsync calls? What does your application look like? Also what version of kestrel are you using.

edit: Actually that stack looks like the request aborted token is being passed into WriteAsync.

[davidfowl]

Yep there's a deadlock...

[physhi]

I'm using 1.1.0 version of dotnet core running on windows. My application is really a file store with a lot of big files, and I have bandwidth throttling and content range and bunch of content transformations implemented in code. What I do is I try to pass cancellation token as far as possible so that I can terminate request if user has disconnected from the server.

Currently, because of this deadlock issue, I've to implement a watchdog process with very aggressive ping times and recycling kestrel as soon as the server stops processing a request.

[davidfowl]

The only way to avoid it would be to stop passing a cancellation token. It's something we need to fix for 1.1.1.

The problem is that we're disposing the cancellation registration in the write callback under the context lock

KestrelHttpServer/src/Microsoft.AspNetCore.Server.Kestrel/Internal/Http/SocketOutput.cs

Line 429 in b46e48f

waitingTask.CancellationRegistration?.Dispose();

on the IO thread and this waits on any pending cancellation callbacks to execute. Turns out we have a cancellation callback waiting on the same lock

KestrelHttpServer/src/Microsoft.AspNetCore.Server.Kestrel/Internal/Http/SocketOutput.cs

Line 299 in b46e48f

private void CancellationTriggered()

.

/cc @muratg @halter73 @CesarBS

[davidfowl]

@physhi Are you manually calling context.Abort() anywhere?

[physhi]

Yep, I have code path that calls abort, but looking at my logs, I don't see it being called.

[muratg]

Putting this in 1.1.1 so that it's triaged with the rest of 1.1.1 items.

[davidfowl]

@physhi have you tried not passing in the token?

[physhi]

@davidfowl I'd tried not passing the token but still the problem persists.

[davidfowl]

@physhi can you take a snapshot of the threads when it hangs without the token?

[physhi]

It's difficult to take snapshot, as I see hangs in production service and it get's recycled as soon as hang is detected. I just know that the service was restarted and that's it.