[arm32/Ubuntu] corefx test failures: System.Net.NetworkInformation.Functional.Tests, System.Net.Sockets.Tests

[RussKeldorph]

From @BruceForstall on April 24, 2018 17:30

These tests:

System.Net.NetworkInformation.Functional.Tests
System.Net.Sockets.Tests

have segmentation faults in the corefx test run, e.g.:

https://ci.dot.net/job/dotnet_coreclr/job/master/job/jitstress/job/arm_cross_checked_ubuntu_corefx_baseline_tst_prtest/1/consoleText

Copied from original issue: dotnet/coreclr#17753

[RussKeldorph]

From @BruceForstall on April 24, 2018 19:2

When fixing this, update tests\arm\corefx_linux_test_exclusions.txt

[RussKeldorph]

From @echesakovMSFT on April 27, 2018 17:57

System.Net.NetworkInformation.Functional.Tests receives SIGSEGV when running test method System.Net.NetworkInformation.Tests.NetworkInterfaceBasicTest.NetworkInterface_LoopbackInterfaceIndex_MatchesReceivedPackets:

Starting:    System.Net.NetworkInformation.Functional.Tests
   System.Net.NetworkInformation.Tests.NetworkInterfaceBasicTest.NetworkInterface_LoopbackInterfaceIndex_MatchesReceivedPackets(ipv6: False) [STARTING]

Program received signal SIGSEGV, Segmentation fault.
0xafa776e8 in __cmsg_nxthdr (__mhdr=<optimized out>, __cmsg=<optimized out>) at /crossrootfs/arm/usr/include/arm-linux-gnueabihf/bits/socket.h:289
289     /crossrootfs/arm/usr/include/arm-linux-gnueabihf/bits/socket.h: No such file or directory.
(gdb) bt
#0  0xafa776e8 in __cmsg_nxthdr (__mhdr=<optimized out>, __cmsg=<optimized out>) at /crossrootfs/arm/usr/include/arm-linux-gnueabihf/bits/socket.h:289
dotnet/corefx#1  GET_CMSG_NXTHDR (mhdr=<optimized out>, cmsg=<optimized out>) at /opt/code/_/fx/src/Native/Unix/System.Native/pal_networking.c:840
dotnet/corefx#2  SystemNative_TryGetIPPacketInformation (messageHeader=0xbeffb580, isIPv4=1, packetInfo=0xbeffb4a4) at /opt/code/_/fx/src/Native/Unix/System.Native/pal_networking.c:861
dotnet/corefx#3  0xad7e3976 in ?? ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

[RussKeldorph]

From @echesakovMSFT on April 27, 2018 18:11

System.Net.Sockets.Tests receives SIGSEGV when running test method System.Net.Sockets.Tests.IPPacketInformationTest.GetHashCode_NonDefaultValue_Succes:

Starting:    System.Net.Sockets.Tests
   System.Net.Sockets.Tests.IPPacketInformationTest.GetHashCode_NonDefaultValue_Succes [STARTING]

Program received signal SIGSEGV, Segmentation fault.
0xaf9866e8 in __cmsg_nxthdr (__mhdr=<optimized out>, __cmsg=<optimized out>) at /crossrootfs/arm/usr/include/arm-linux-gnueabihf/bits/socket.h:289
289     /crossrootfs/arm/usr/include/arm-linux-gnueabihf/bits/socket.h: No such file or directory.
(gdb) bt
#0  0xaf9866e8 in __cmsg_nxthdr (__mhdr=<optimized out>, __cmsg=<optimized out>) at /crossrootfs/arm/usr/include/arm-linux-gnueabihf/bits/socket.h:289
dotnet/corefx#1  GET_CMSG_NXTHDR (mhdr=<optimized out>, cmsg=<optimized out>) at /opt/code/_/fx/src/Native/Unix/System.Native/pal_networking.c:840
dotnet/corefx#2  SystemNative_TryGetIPPacketInformation (messageHeader=0xbeffb740, isIPv4=1, packetInfo=0xbeffb684) at /opt/code/_/fx/src/Native/Unix/System.Native/pal_networking.c:861
dotnet/corefx#3  0xad660632 in ?? ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

[RussKeldorph]

From @echesakovMSFT on April 27, 2018 18:13

Also System.Net.Sockets.Tests hangs when running test method System.Net.Sockets.Tests.NetworkStreamTest.ReadWrite_Memory_LargeWrite_Success

Update: it turned out this test takes 10m to complete:

System.Net.Sockets.Tests.NetworkStreamTest.ReadWrite_Memory_LargeWrite_Success [FINISHED] Time: 603.4356554s

[RussKeldorph]

@karelz We think there's a good chance this isn't a runtime issue. Could you guys take a look?

[karelz]

@RussKeldorph is it 100% repro? Is the problem that it takes too long? Does it take too long only on arm32, or is it combo of arm32 + some JIT stress settings?

[RussKeldorph]

@karelz I believe the crashes are more of a concern than the timeouts. I think they are 100%, but @echesakovMSFT can confirm. There's no JitStress or GCStress involved. These are running on ARM ubuntu14.04, so there could be an incompatibility between corefx and that OS version, but I don't have much to base that on.

[karelz]

We didn't see the crashes in our arm32 runs - @safern did we run on Ubuntu14.04?

[safern]

Did you mean @joperezr? I don't know where we ran them on, don't know anything about the arm32 runs, just that Jose was working on them.

[echesakov]

@karelz Yes, it is 100% repro without any stress mode. Could be reproduced on Ubuntu/arm with a command similar to

ubuntu@tegra-ubuntu:~/echesakov/coreclr/_/fx/bin/tests/System.Net.Sockets.Tests/netcoreapp-Linux-Release-arm$ /home/ubuntu/echesakov/coreclr/_/fx/bin/testhost/netcoreapp-Linux-Release-arm/dotnet xunit.console.netcore.exe System.Net.Sockets.Tests.dll -xml testResults.xml -notrait Benchmark=true -notrait category=nonnetcoreapptests -notrait category=nonlinuxtests -notrait category=IgnoreForCI -notrait category=OuterLoop -notrait category=failing -parallel none -showprogress -method System.Net.Sockets.Tests.IPPacketInformationTest.GetHashCode_NonDefaultValue_Succes
xUnit.net console test runner (32-bit .NET Core)
Copyright (C) 2014 Outercurve Foundation.

Discovering: System.Net.Sockets.Tests
Discovered:  System.Net.Sockets.Tests
Starting:    System.Net.Sockets.Tests
   System.Net.Sockets.Tests.IPPacketInformationTest.GetHashCode_NonDefaultValue_Succes [STARTING]
Segmentation fault

This was done under a combination of Checked coreclr and Release corefx.
To build the testing environment I followed the way what our CI does this using Docker on Ubuntu/x64 and then scp-ed result over my tegra-ubuntu device.

[karelz]

I wonder how it is different from our setup we did - the tests passed there AFAIK.
@joperezr any insights?

[joperezr]

@karelz we ran on Ubuntu16.04 using qemu in order to try emulating the arm32 kernel. The results we had from those specific assemblies where:

System.Net.NetworkInformation.Functional.Tests   Total=55  Failed=4
System.Net.Sockets.Tests  Total=1291  Failed=509