Many test timeouts -- external server outage #57134
Description
Looks like correlated problem for multiple tests at the same time on the same machines -- likely external problem.
Methods monitored:
or Method == 'ConnectAsync_DnsEndPoint_Success'
or Method == 'SendAsync_CorrectVersionSelected_ExternalServer'
or Method == 'GetStreamAsync_ReadToEnd_Success'
or Method == 'GetAsync_CredentialIsCredentialCacheUriRedirect_StatusCodeOK'
or Method == 'DefaultHeaders_SetCredentials_ClearedOnRedirect'
or Method == 'UseCallback_ValidCertificate_ExpectedValuesDuringCallback'
or Method == 'AuthenticatedProxiedRequest_GetAsyncWithCreds_Success'
or Method == 'GetAsync_UseResponseHeadersReadAndCallLoadIntoBuffer_Success'
or Method == 'SendReceive_PartialMessageBeforeCompleteMessageArrives_Success'
or Method == 'GetAsync_MaxLimited_ConcurrentCallsStillSucceed'
or Method == 'PostLargeContentUsingContentLengthSemantics_Success'
or Method == 'PostUsingChunkedEncoding_Success'
or Method == 'CloseOutputAsync_ExpectedStates'
or Method == 'CloseAsync_CloseOutputAsync_Throws'
or Method == 'CloseAsync_CloseDescriptionHasUnicode_Success'
or Method == 'ReceiveAsync_MultipleOutstandingReceiveOperations_Throws'
or Method == 'PostUsingNoSpecifiedSemantics_UsesChunkedSemantics'
or Method == 'PostSyncBlockingContentUsingChunkedEncoding_Success'
or Method == 'PostRepeatedFlushContentUsingChunkedEncoding_Success'
or Method == 'PostRewindableStreamContentMultipleTimes_StreamContentFullySent'
or Method == 'SendReceive_PartialMessageDueToSmallReceiveBuffer_Success'
or Method == 'SendReceive_Concurrent_Success'
or Method == 'SendAsync_SendCloseMessageType_ThrowsArgumentExceptionWithMessage'
or Method == 'EchoTextMessage_Success'
or Method == 'EchoBinaryMessage_Success'
or Method == 'ConnectAsync_PassNoSubProtocol_ServerRequires_ThrowsWebSocketException'
or Method == 'ConnectAsync_AddCustomHeaders_Success'
or Method == 'ConnectAndCloseAsync_UseProxyServer_ExpectedClosedState'
or Method == 'ReceiveAsync_AfterCancellationDoReceiveAsync_ThrowsWebSocketException'
or Method == 'PostUsingContentLengthSemantics_Success'
or Method == 'PostAsync_EmptyContent_ContentTypeHeaderNotSent'
or Method == 'GetAsync_UseResponseHeadersReadAndCopyToMemoryStream_Success'
or Method == 'PostEmptyContentUsingChunkedEncoding_Success'
or Method == 'GetAsync_ServerNeedsBasicAuthAndSetDefaultCredentials_StatusCodeUnauthorized'
or Method == 'GetAsync_RequestHeadersAddCustomHeaders_HeaderAndValueSent'
or Method == 'ConnectAsync_PassMultipleSubProtocols_ServerRequires_ConnectionUsesAgreedSubProtocol'
or Method == 'AuthenticatedProxyTunnelRequest_PostAsyncWithCreds_Success'
or Method == 'SendAsync_SendRequestUsingMethodToEchoServerWithNoContent_MethodCorrectlySent'
or Method == 'UseCallback_BadCertificate_ExpectedPolicyErrors'
or Method == 'CloseAsync_CloseDescriptionIsNull_Success'
or Method == 'CloseAsync_CloseDescriptionIsMaxLength_Success'
or Method == 'SendAsync_SendCloseMessageType_ThrowsArgumentExceptionWithMessage'
or Method == 'CloseAsync_DuringConcurrentReceiveAsync_ExpectedStates'
or Method == 'GetAsync_ResponseHeadersRead_ReadFromEachIterativelyDoesntDeadlock'
or Method == 'GetAsync_AllowAutoRedirectFalse_RedirectFromHttpToHttp_StatusCodeRedirect'
or Method == 'SendAsync_SendRequestUsingMethodToEchoServerWithContent_Success'
or Method == 'GetAsync_AllowAutoRedirectTrue_RedirectFromHttpToHttp_StatusCodeOK'
or Method == 'GetAsync_AllowAutoRedirectTrue_RedirectFromHttpToHttps_StatusCodeOK'
or Method == 'SendReceive_VaryingLengthBuffers_Success'
or Method == 'ClientWebSocket_Abort_CloseOutputAsync'
or Method == 'Connect_DnsEndPoint_Success'
or Method == 'SendAsync_SendZeroLengthPayloadAsEndOfMessage_Success'
or Method == 'ReadAsStreamAsync_Cancel_TaskIsCanceled'
or Method == 'PostNoContentUsingContentLengthSemantics_Success'
or Method == 'PostAsync_ExpectContinue_Success'
or Method == 'Abort_CloseAndAbort_Success'
or Method == 'Abort_SendAndAbort_Success'
or Method == 'CloseOutputAsync_Cancel_Success'
or Method == 'CloseOutputAsync_ClientInitiated_CanReceive_CanClose'
or Method == 'CloseOutputAsync_CloseDescriptionIsNull_Success'
or Method == 'GetAsync_AllowAutoRedirectTrue_RedirectFromHttpsToHttp_StatusCodeRedirect'
or Method == 'GetAsync_AllowAutoRedirectTrue_RedirectToUriWithParams_RequestMsgUriSet'
or Method == 'GetAsync_AllowAutoRedirectTrue_RedirectWithRelativeLocation'
or Method == 'GetAsync_SetAutomaticDecompression_HeadersRemoved'
or Method == 'PostAsync_CallMethod_StreamContent'
or Method == 'PostEmptyContentUsingConflictingSemantics_Success'
or Method == 'PostEmptyContentUsingContentLengthSemantics_Success'
or Method == 'PostNonRewindableContentUsingAuth_NoPreAuthenticate_ThrowsHttpRequestException'
or Method == 'PostNonRewindableContentUsingAuth_PreAuthenticate_Success'
or Method == 'PostRewindableContentUsingAuth_NoPreAuthenticate_Success'
or Method == 'PostUsingUsingConflictingSemantics_UsesChunkedSemantics'
or Method == 'ReceiveAsync_ReceiveThenCancel_ThrowsOperationCanceledException'
or Method == 'SendAsync_MultipleOutstandingSendOperations_Throws'
or Method == 'SendAsync_RequestVersion20_ResponseVersion20IfHttp2Supported'
or Method == 'SendAsync_SendRequestUsingNoBodyMethodToEchoServerWithContent_NoBodySent'
or Method == 'SslStream_AllowRenegotiation_True_Succeeds'
or Method == 'UploadData_Success'
or Method == 'UploadString_Success'
or Method == 'UploadValues_Success'
or Method == 'UseCallback_NotSecureConnection_CallbackNotCalled'
or Method == 'UseDefaultCredentials_SetGetResponse_ExpectSuccess'
or Method == 'UseDefaultCredentials_SetToFalseAndServerNeedsAuth_StatusCodeUnauthorized'
or Method == 'ZeroByteReceive_CompletesWhenDataAvailable'
or Method == 'UploadFile_Success'
or Method == 'SendAsync_Cancel_Success'
or Method == 'CloseAsync_Cancel_Success'
or Method == 'EventSource_SocketAcceptFailure_LogsAcceptFailed'
or Method == 'BeginReceiveMessageFromV4BoundToAnyV6_Success'
or Method == 'Socket_StaticConnectAsync_ConnectionRefused'
or Method == 'GetAsync_EmptyResponseHeader_Success'Failures 7/1-9/6 (incl. PRs):
| Day and Time range | Notes |
|---|---|
| 7/1-7/31 | Nothing over 5 failures |
| 8/1 (0:22) ... 11:51:08-12:12:49 | 5x failures (5x main=6.0) |
| 8/2 (5 mins) ... 14:36:19-14:41:47 | 18x failures - PR #56723 2 test runs hang (1x Debian.10.Amd64.Open, 1x Centos.8.Amd64.Open) |
| 8/7 (0:52) ... 11:51:06-12:42:59 | 68x failures (68x main=6.0) 6 test runs hang (2x Debian.10.Amd64.Open, 1x Ubuntu.1910.Amd64.Open, 1x Centos.8.Amd64.Open, 1x Fedora.34.Amd64.Open, 1x Windows.Server.Core.1909.Amd64.Open - WinHttpHandler) |
| 8/8 (0:44) ... 11:49:55-12:34:16 | 81x failures (79x main=6.0, 2x 6.0-preview7) 4 test runs hang (2x Centos.8.Amd64.Open, 1x Ubuntu.1910.Amd64.Open, 1x Fedora.34.Amd64.Open) |
| 8/12 (0:31) ... 11:51:35-12:22:48 | 6x failures (6x 6.0-preview7) |
| 8/15 (5 min) ... 12:36:25-12:41:50 | 4 test runs hang (2x Alpine.312.Amd64.Open, Fedora.34.Amd64.Open, Debian.10.Amd64.Open) |
| 8/16 (TBD) ... 10:37:46 | 1 test run hang (Debian.10.Amd64.Open) |
| 8/18 (7 min) ... 13:04:23-13:11:14 | 5 test runs hang (4x 6.0-rc1, 1x 6.0) (Fedora.34.Amd64.Open, Debian.10.Amd64.Open, Alpine.312.Amd64.Open, Mariner.1.0.Amd64.Open, Centos.8.Amd64.Open) |
| 8/22 (6 min) ... 12:37:28-12:43:56 | 5 test runs hang (5x 6.0) (Centos.8.Amd64.Open, Fedora.34.Amd64.Open, Ubuntu.1910.Amd64.Open, Debian.10.Amd64.Open, Mariner.1.0.Amd64.Open) |
| 8/24 (4 min) ... 10:31:46-10:35:31 | PR #58003: 3 test runs hang (3x main=7.0) (1x Alpine.312.Amd64.Open, 1x Fedora.34.Amd64.Open, 1x Centos.8.Amd64.Open) |
| 8/24 (0:32) ... 11:50:40-12:22:28 | 19x failures (7x main=7.0, 6x 6.0, 6x 6.0-rc1) |
| 8/24 (12 min) ... 12:35:08-12:47:10 | 15 test runs hang (8x 6.0, 7x 6.0-rc1) |
| 8/26 (0:47) ... 12:03:42-12:50:28 | 33x failures (20x 6.0-rc1, 13x 6.0) |
| 8/27 (0:33) ... 13:34:056-14:07:51 | 1 test run hang (1x main=7.0) (1x WinHttpHandler) |
| 8/29 (1:02) ... 11:58:58-13:01:18 | 57x failures (35x 6.0, 22x 6.0-rc1) 17 test runs hang (11x 6.0, 6x 6.0-rc1) (2x WinHttpHandler) |
| 8/31 (0:54) ... 11:46:51-12:51:16 | 162x failures (130x 6.0, 31x 6.0-rc1, 1x main=7.0) |
| 9/4 (0:38) ... 11:56:27-12:34:07 | 48x failures (37x 6.0, 11x 6.0-rc1) 6 test runs hang (4x 6.0, 2x 6.0-rc1) |
| 9/5 (0:45) ... 11:47:05-12:31:54 | 29x failures (17x 6.0, 12x 6.0-rc1) 4 test runs hang (2x 6.0, 2x 6.0-rc1) |
| 9/6 (0:52) ... 11:52:55-12:45:37 | 49x failures (33x main=7.0, 10x 6.0, 6x 6.0-rc1) 6 test runs hang (6x main=7.0) |
Original report - only 1 test ConnectAsync_DnsEndPoint_Success
Test: System.Net.Sockets.Tests.TcpClientTest.ConnectAsync_DnsEndPoint_Success.ConnectAsync_DnsEndPoint_Success
Failure:
System.Net.Sockets.SocketException : Connection timed out
at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Sockets/src/System/Net/Sockets/Socket.Tasks.cs:line 1370
at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token) in /_/src/libraries/System.Net.Sockets/src/System/Net/Sockets/Socket.Tasks.cs:line 1286
at System.Net.Sockets.TcpClient.CompleteConnectAsync(ValueTask task) in /_/src/libraries/System.Net.Sockets/src/System/Net/Sockets/TCPClient.cs:line 312
at System.Net.Sockets.Tests.TcpClientTest.ConnectAsync_DnsEndPoint_Success(Int32 mode) in /_/src/libraries/System.Net.Sockets/tests/FunctionalTests/TcpClientTest.cs:line 178
--- End of stack trace from previous location ---