Exception at System.Net.Security.SslStreamInternal.EndRead when proxy request from Nginx to Kestrel through HTTPS

[yuwaMSFT]

I was trying out the SSL feature on Kestrel. When putting Nginx in front to proxy request to Kestrel server via https, I always got the following exception for each client request:

\funfile\Scratch\yuwa\Failures\041216\kestrelhttps\kestrellog.log

fail: Microsoft.AspNetCore.Server.Kestrel[0]
      FilteredStreamAdapter.CopyToAsync
System.AggregateException: One or more errors occurred. (The read operation failed, see inner exception.) ---> System.IO.IOException: The read operation failed, see inner exception. ---> System.Threading.Tasks.TaskCanceledException: The request was aborted
   at Microsoft.AspNetCore.Server.Kestrel.Http.SocketInput.GetResult()
   at Microsoft.AspNetCore.Server.Kestrel.Http.SocketInputExtensions.<ReadAsyncAwaited>d__1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Threading.Tasks.ValueTask`1.get_Result()
   at Microsoft.AspNetCore.Server.Kestrel.Filter.LibuvStream.Read(Byte[] buffer, Int32 offset, Int32 count)
   at System.Threading.Tasks.Task`1.InnerInvoke()
   at System.Threading.Tasks.Task.Execute()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.IO.StreamAsyncHelper.EndRead(IAsyncResult asyncResult)
   at System.Net.FixedSizeReader.ReadCallback(IAsyncResult transportResult)
   --- End of inner exception stack trace ---
   at System.Net.Security.SslStreamInternal.EndRead(IAsyncResult asyncResult)
   at System.Net.Security.SslStream.<>c.<ReadAsync>b__85_1(IAsyncResult iar)
   at System.Threading.Tasks.TaskFactory`1.FromAsyncCoreLogic(IAsyncResult iar, Func`2 endFunction, Action`1 endAction, Task`1 promise, Boolean requiresSynchronization)
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.AspNetCore.Server.Kestrel.Filter.StreamExtensions.<CopyToAsync>d__0.MoveNext()
   --- End of inner exception stack trace ---
---> (Inner Exception #0) System.IO.IOException: The read operation failed, see inner exception. ---> System.Threading.Tasks.TaskCanceledException: The request was aborted
   at Microsoft.AspNetCore.Server.Kestrel.Http.SocketInput.GetResult()
   at Microsoft.AspNetCore.Server.Kestrel.Http.SocketInputExtensions.<ReadAsyncAwaited>d__1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Threading.Tasks.ValueTask`1.get_Result()
   at Microsoft.AspNetCore.Server.Kestrel.Filter.LibuvStream.Read(Byte[] buffer, Int32 offset, Int32 count)
   at System.Threading.Tasks.Task`1.InnerInvoke()
   at System.Threading.Tasks.Task.Execute()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.IO.StreamAsyncHelper.EndRead(IAsyncResult asyncResult)
   at System.Net.FixedSizeReader.ReadCallback(IAsyncResult transportResult)
   --- End of inner exception stack trace ---
   at System.Net.Security.SslStreamInternal.EndRead(IAsyncResult asyncResult)
   at System.Net.Security.SslStream.<>c.<ReadAsync>b__85_1(IAsyncResult iar)
   at System.Threading.Tasks.TaskFactory`1.FromAsyncCoreLogic(IAsyncResult iar, Func`2 endFunction, Action`1 endAction, Task`1 promise, Boolean requiresSynchronization)
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.AspNetCore.Server.Kestrel.Filter.StreamExtensions.<CopyToAsync>d__0.MoveNext()<---

When request from wget directly on local machine, I didn’t see such error. It seems something related to how Nginx terminates the https proxy connection. Is this a benign exception? If so, can we suppress it since it would fire for each https request proxied by Nginx.

BTW, I used the self-signed test cert (but I don’t think that’s the problem here).

[yuwaMSFT]

@halter73 @muratg @sivagms @sajayantony

[muratg]

@yuwaMSFT Looks like a server side error, right... Was the response fine on the client side?

[yuwaMSFT]

@muratg Yes it's server side error. Response to client seemed to be fine.

[cesarblum]

@yuwaMSFT What do you have in nginx.conf?

[yuwaMSFT]

This is the Nginx configuration:

            server {
                    listen 8081 ssl;
                    ssl_certificate     /etc/ssl/nginx/new.cert.cert;
                    ssl_certificate_key /etc/ssl/nginx/new.cert.key;
                    ssl_protocols       TLSv1 TLSv1.1 TLSv1.2;

                    location / {
                            proxy_set_header    Host $host;
                            proxy_set_header    X-Real-IP   $remote_addr;
                            proxy_set_header    X-Forwarded-For $proxy_add_x_forwarded_for;
                            access_log  off;
                            error_log on;
                            proxy_pass  https://localhost:5001;
                            proxy_set_header Upgrade $http_upgrade;
                            proxy_set_header Connection "upgrade";
                   }
            }

[yuwaMSFT]

BTW, this is the configuration for the site serving as the reverse proxy. nginx.conf itself was the default one after installing nginx.

[cesarblum]

What version of nginx are you using?

[yuwaMSFT]

nginx version: nginx/1.4.6 (Ubuntu)

[benaadams]

What happens if you add keep alive? Might be ngnix doing a close and the filter getting confused by it

[cesarblum]

@benaadams That's what I was about to try 😄

[benaadams]

There is also some oddness with ngnix, though not likely to be effecting this https://blog.cloudflare.com/the-curious-case-of-slow-downloads/

[yuwaMSFT]

client should already send out request with keep-alive.

Added the header for proxy in the site config:
proxy_set_header Connection "Keep-Alive";
proxy_set_header Proxy-Connection "Keep-Alive";

also set the ssl session cache in nginx.conf
ssl_session_cache shared:SSL:10m;
ssl_session_timeout 10m;

Even setting so, it seems nginx is still doing new connection for each https request from client. Probably it's because nginx needs to decrypt and re-encrypt for each request coming from client. Not sure if there is way to configure nginx to "cache" the proxy https connection...

If client hits Kestrel server directly without nginx in front, it seems the first request would hit this exception, but not subsequent requests, unless browser does refresh.

So this seems to be happening for each new HTTPS connection. Probably it's related to the HTTPS handshaking process?