Test: System.Net.Sockets.Tests.SendReceiveEap.SendToRecvFrom_Datagram_UDP(loopbackAddress: ::1) failed in CI

[KristinXie1]

Failed test: System.Net.Sockets.Tests.SendReceiveEap.SendToRecvFrom_Datagram_UDP(loopbackAddress: ::1)
Configuration: OuterLoop_CentOS7.1_debug (build#123)

Message:

System.Net.Sockets.Tests.SendReceiveEap.SendToRecvFrom_Datagram_UDP(loopbackAddress: ::1) [FAIL]
        Assert.True() Failure
        Expected: True
        Actual:   False

Stack Trace:

/mnt/resource/j/workspace/dotnet_corefx/master/outerloop_centos7.1_debug/src/System.Net.Sockets/tests/FunctionalTests/SendReceive.cs(148,0): at System.Net.Sockets.Tests.SendReceive.<SendToRecvFrom_Datagram_UDP>d__17.MoveNext()
           --- End of stack trace from previous location where exception was thrown ---
              at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
              at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
           --- End of stack trace from previous location where exception was thrown ---
              at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
              at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
           --- End of stack trace from previous location where exception was thrown ---
              at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
              at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

Detail: https://ci.dot.net/job/dotnet_corefx/job/master/job/outerloop_centos7.1_debug/123/consoleText

[KristinXie1]

This issue is repro on OuterLoop_CentOS7.1_release (build#123), detail: https://ci.dot.net/job/dotnet_corefx/job/master/job/outerloop_centos7.1_release/123/testReport/System.Net.Sockets.Tests/SendReceiveEap/SendToRecvFrom_Datagram_UDP_loopbackAddress____1_/

[karelz]

cc @Priya91 @ianhays @steveharter

[KristinXie1]

Failed again here: https://ci.dot.net/job/dotnet_corefx/job/master/job/outerloop_centos7.1_debug/134/testReport/System.Net.Sockets.Tests/SendReceiveEap/SendToRecvFrom_Datagram_UDP_loopbackAddress____1_/

[steveharter]

@stephentoub this test was added about three weeks ago. Any thoughts? We should probably disable it.

  int sent = await SendToAsync(right, new ArraySegment<byte>(sendBuffer), leftEndpoint);
>>Assert.True(await receiverAck.WaitAsync(AckTimeout));
  senderAck.Release();

Also there are two Assert.True's here without any userMessage param so that should be added.

[stephentoub]

this test was added about three weeks ago

The test itself has actually been in the repo since 2015:
dotnet/corefx@09932a4#diff-5d77ee23eb2ea596e218f9c1ef09d793R22

What changed a few weeks ago was allowing the test to work with the various Async APIs on Socket, e.g. allowing it to work with the EAP, Task, and APM methods (and the sync ones), rather than just the APM ones. It's possible an issue was introduced as part of that conversion.

[KristinXie1]

Failed again on build 20170322.02: https://mc.dot.net/#/product/netcore/master/source/official~2Fcorefx~2Fmaster~2F/type/test~2Ffunctional~2Fcli~2F/build/20170322.02/workItem/System.Net.Sockets.Tests/analysis/xunit/System.Net.Sockets.Tests.SendReceiveApm~2FSendToRecvFrom_Datagram_UDP(loopbackAddress:%20::1)

[KristinXie1]

Failed again on build 20170324.01

[KristinXie1]

Failed again here: https://ci.dot.net/job/dotnet_corefx/job/master/job/outerloop_centos7.1_release/138/testReport/System.Net.Sockets.Tests/SendReceiveEap/SendToRecvFrom_Datagram_UDP_loopbackAddress____1_/

[stephentoub]

What changed a few weeks ago was allowing the test to work with the various Async APIs on Socket

@steveharter, actually, FYI, it looks like the test has failed not just on EAP but also on APM, which was the one that previously existed.

[stephentoub]

@steveharter, this looks like the same problem as was fixed for some other tests in https://github.com/dotnet/corefx/issues/5185. The test is sending 10 packets over UDP and expecting all 10 to be received, which isn't guaranteed. There's even a TODO in the test stating that it needs to be hardened against such loss:
https://github.com/dotnet/corefx/blob/ed823ad9470f2ecf412d5089fe36cd6958fc5834/src/System.Net.Sockets/tests/FunctionalTests/SendReceive.cs#L71

[steveharter]

What changed a few weeks ago was allowing the test to work with the various Async APIs on Socket

@steveharter, actually, FYI, it looks like the test has failed not just on EAP but also on APM, which was the one that previously existed.

FWIW according to jdash, the tests started failing on 3/9/2017. The refactoring work was on 2/26/2017 - dotnet/corefx@ca392ca#diff-5d77ee23eb2ea596e218f9c1ef09d793

Test Case: System.Net.Sockets.Tests.SendReceiveEap.SendToRecvFrom_Datagram_UDP(loopbackAddress: ::1)
Failed Jenkins Jobs
Build Number 	Machine Name 	Date
dotnet_corefx/master/outerloop_centos7.1_debug 123 	centos71-20170216-outer1fbe70 	03/09 07:52 AM
dotnet_corefx/master/outerloop_centos7.1_debug 125 	centos71-20170216-outer125790 	03/11 07:52 AM
dotnet_corefx/master/outerloop_centos7.1_release 123 	centos71-20170216-outer6ffb00 	03/12 03:32 PM

[steveharter]

I believe many timeouts or proposed UDP packet loss are interference from other tests that happen to listen on another tests port, where tests do a receivefrom on a port that they did not bind to (IPV4\IPV4 dual mode tests), so the 'bad' test receives the data causing the 'good' test to timeout or miss some data. This is mostly a Linux issue due to the randomness in port assignment (vs Windows which is incremental).

So if the refactoring isn't to blame, perhaps a new or other modified test is (that was added\modified on or a few days before 3/9/2017).

[KristinXie1]

Failed again here: https://ci.dot.net/job/dotnet_corefx/job/master/job/outerloop_netcoreapp_centos7.1_release/7/testReport/System.Net.Sockets.Tests/SendReceiveEap/SendToRecvFrom_Datagram_UDP_loopbackAddress____1_/

[KristinXie1]

This issue is repro on portable core tests: https://mc.dot.net/#/product/netcore/master/source/official~2Fcorefx~2Fmaster~2F/type/test~2Ffunctional~2Fportable~2Fcli~2F/build/20170407.01/workItem/System.Net.Sockets.Tests/analysis/xunit/System.Net.Sockets.Tests.SendReceiveApm~2FSendToRecvFrom_Datagram_UDP(loopbackAddress:%20::1)