SslStream tests: timeout

[stephentoub]

http://dotnet-ci.cloudapp.net/job/dotnet_corefx_windows_release_prtest/6135/console

System.Net.Security.Tests.SslStreamStreamToStreamTest.SslStream_StreamToStream_Authentication_Success [FAIL]
        Handshake completed in the allotted time
        Expected: True
        Actual:   False
        Stack Trace:
           d:\j\workspace\dotnet_corefx_windows_release_prtest\src\System.Net.Security\tests\FunctionalTests\SslStreamStreamToStreamTest.cs(36,0): at System.Net.Security.Tests.SslStreamStreamToStreamTest.SslStream_StreamToStream_Authentication_Success()
     System.Net.Security.Tests.CertificateValidationClientServer.CertificateValidationClientServer_EndToEnd_Ok [FAIL]
        Client/Server Authentication timed out.
        Expected: True
        Actual:   False
        Stack Trace:
           d:\j\workspace\dotnet_corefx_windows_release_prtest\src\System.Net.Security\tests\FunctionalTests\CertificateValidationClientServer.cs(80,0): at System.Net.Security.Tests.CertificateValidationClientServer.<CertificateValidationClientServer_EndToEnd_Ok>d__0.MoveNext()
           --- End of stack trace from previous location where exception was thrown ---
              at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
              at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
           --- End of stack trace from previous location where exception was thrown ---
              at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
              at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
           --- End of stack trace from previous location where exception was thrown ---
              at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
              at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
     System.Net.Security.Tests.ClientAsyncAuthenticateTest.ClientAsyncAuthenticate_EachProtocol_Success [FAIL]
        Timed Out
        Expected: True
        Actual:   False
        Stack Trace:
           d:\j\workspace\dotnet_corefx_windows_release_prtest\src\System.Net.Security\tests\FunctionalTests\ClientAsyncAuthenticateTest.cs(252,0): at System.Net.Security.Tests.ClientAsyncAuthenticateTest.<ClientAsyncSslHelper>d__5c.MoveNext()
           --- End of stack trace from previous location where exception was thrown ---
              at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
              at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
           d:\j\workspace\dotnet_corefx_windows_release_prtest\src\System.Net.Security\tests\FunctionalTests\ClientAsyncAuthenticateTest.cs(108,0): at System.Net.Security.Tests.ClientAsyncAuthenticateTest.<ClientAsyncAuthenticate_EachProtocol_Success>d__16.MoveNext()
           --- End of stack trace from previous location where exception was thrown ---
              at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
              at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
           --- End of stack trace from previous location where exception was thrown ---
              at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
              at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
           --- End of stack trace from previous location where exception was thrown ---
              at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
              at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

[CIPop]

/cc @markwilkie
We're already at 10s for these tests. I'll increase to 15s but, unless this is a product bug, this is simply showing that the CI system boxes are not scaling for the number of concurrent tests that they need to run.

[CIPop]

For reference: on my system, the entire System.Net.Security test suite executes in about 1 second.

[stephentoub]

@CIPop, I expect you're right, but should we also consider the possibility that there's a race condition / deadlock somewhere and these timeouts actually represent a real product issue? Or is that not possible?

[CIPop]

consider the possibility that there's a race condition

I'm not disregarding that possibility. If this happens after I increase the timeouts, I'll see if I can repro in my own environment or if I can add memory-dump generation code to the tests (i.e. a call to MiniDumpWriteDump ).

[stephentoub]

Same three failed here: dotnet/corefx#4478 (comment)

[stephentoub]

And again here: http://dotnet-ci.cloudapp.net/job/dotnet_corefx_windows_release_prtest/6163/

[stephentoub]

Again here:
http://dotnet-ci.cloudapp.net/job/dotnet_corefx_windows_debug/1777/console

[stephentoub]

Again here:
http://dotnet-ci.cloudapp.net/job/dotnet_corefx_windows_release_prtest/6224/

[CIPop]

Increased the timeouts to 15s.

If this is still noticed we'll need to try to repro on a system that can be debugged and ETW can be collected. #15570 is preventing us from using the CI failures for investigation.

[CIPop]

Please reopen this issue if the errors still show up in CI.

It would be interesting to understand if the timeouts decreased the frequency of this failure.

[stephentoub]

Increased the timeouts to 15s.

When? This has been reproing fairly regularly for the last few days, in fact I'd say the regularity has actually increased.

Please reopen this issue if the errors still show up in CI.

They're not going to as I disabled them yesterday in dotnet/corefx#4526 due to how many PRs they were taking out.