Test failure System.Net.Sockets.Tests.SendFile_CancellableTask.GreaterThan2GBFile_SendsAllBytes

[VincentBu]

Failures 5/25-8/25 (incl. PRs):

Day	Failures	Branch notes
7/14	2x	2x Preview6
7/15	3x	3x Preview6
7/18	4x	4x Preview6
7/20	1x	1x Preview6
7/23	3x	3x Preview6
7/25	2x	2x Preview6
7/26	2x	2x Preview6
7/27	3x	3x Preview7
7/29	3x	3x main
7/31	1x	1x Preview6
8/3	3x	3x Preview7
8/5	3x	3x Preview7
8/8	6x	4x main + 2x Preview7
8/10	6x	6x Preview7
8/12	2x	2x Preview7
8/14	2x	2x main
8/15	3x	3x Preview7
8/16	1x	1x main
8/17	3x	3x Preview7
8/20	4x	3x RC1 + 1x main
8/21	1x	1x main
8/22	3x	2x RC1 + 1x main
8/23	7x	3x RC1 + 2x release/6.0 + 2x main
8/24	3x	3x RC1

Run: runtime-libraries-coreclr outerloop 20210822.2

Failed test:

net6.0-OSX-Release-x64-CoreCLR_release-OSX.1013.Amd64.Open

- System.Net.Sockets.Tests.SendFile_CancellableTask.GreaterThan2GBFile_SendsAllBytes

*Error message:

System.IO.IOException : No space left on device : '/var/folders/89/2s0r6lhd4jj73phl7ckw3vm40000gy/T/d4thqcw1.yle_GreaterThan2GBFile_SendsAllBytes_245'


Stack trace
   at System.IO.Strategies.FileStreamHelpers.CheckFileCall(Int64 result, String path, Boolean ignoreNotSupported) in /_/src/libraries/System.Private.CoreLib/src/System/IO/Strategies/FileStreamHelpers.Unix.cs:line 24
   at System.IO.Strategies.FileStreamHelpers.SetFileLength(SafeFileHandle handle, Int64 length) in /_/src/libraries/System.Private.CoreLib/src/System/IO/Strategies/FileStreamHelpers.Unix.cs:line 38
   at System.IO.Strategies.OSFileStreamStrategy.SetLengthCore(Int64 value) in /_/src/libraries/System.Private.CoreLib/src/System/IO/Strategies/OSFileStreamStrategy.cs:line 226
   at System.IO.Strategies.OSFileStreamStrategy.SetLength(Int64 value) in /_/src/libraries/System.Private.CoreLib/src/System/IO/Strategies/OSFileStreamStrategy.cs:line 219
   at System.IO.FileStream.SetLength(Int64 value) in /_/src/libraries/System.Private.CoreLib/src/System/IO/FileStream.cs:line 408
   at System.Net.Sockets.Tests.SendFile`1.GreaterThan2GBFile_SendsAllBytes() in /_/src/libraries/System.Net.Sockets/tests/FunctionalTests/SendFile.cs:line 248
--- End of stack trace from previous location ---

Tagging subscribers to this area: @dotnet/ncl
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Run: runtime-libraries-coreclr outerloop 20210822.2

Failed test:

net6.0-OSX-Release-x64-CoreCLR_release-OSX.1013.Amd64.Open

- System.Net.Sockets.Tests.SendFile_CancellableTask.GreaterThan2GBFile_SendsAllBytes

*Error message:

System.IO.IOException : No space left on device : '/var/folders/89/2s0r6lhd4jj73phl7ckw3vm40000gy/T/d4thqcw1.yle_GreaterThan2GBFile_SendsAllBytes_245'


Stack trace
   at System.IO.Strategies.FileStreamHelpers.CheckFileCall(Int64 result, String path, Boolean ignoreNotSupported) in /_/src/libraries/System.Private.CoreLib/src/System/IO/Strategies/FileStreamHelpers.Unix.cs:line 24
   at System.IO.Strategies.FileStreamHelpers.SetFileLength(SafeFileHandle handle, Int64 length) in /_/src/libraries/System.Private.CoreLib/src/System/IO/Strategies/FileStreamHelpers.Unix.cs:line 38
   at System.IO.Strategies.OSFileStreamStrategy.SetLengthCore(Int64 value) in /_/src/libraries/System.Private.CoreLib/src/System/IO/Strategies/OSFileStreamStrategy.cs:line 226
   at System.IO.Strategies.OSFileStreamStrategy.SetLength(Int64 value) in /_/src/libraries/System.Private.CoreLib/src/System/IO/Strategies/OSFileStreamStrategy.cs:line 219
   at System.IO.FileStream.SetLength(Int64 value) in /_/src/libraries/System.Private.CoreLib/src/System/IO/FileStream.cs:line 408
   at System.Net.Sockets.Tests.SendFile`1.GreaterThan2GBFile_SendsAllBytes() in /_/src/libraries/System.Net.Sockets/tests/FunctionalTests/SendFile.cs:line 248
--- End of stack trace from previous location ---

Author:	VincentBu
Assignees:	-
Labels:	area-System.Net.Sockets, os-mac-os-x, arch-x64
Milestone:	-

[karelz]

@dotnet/dncenghot looks like OSX.1013 machine(s) don't have enough disk space -- appears across 6.0 & 7.0 branches 7/14-8/22 (total 61 times)

Kusto query:

let failedTests = (methodName : string, includePR : bool, messageSubstr: string) {
cluster('engsrvprod.kusto.windows.net').database('engineeringdata').TestResults
| where Type startswith "System.Net" 
    and Result == 'Fail'
| where Method == methodName
| where Message contains messageSubstr
| distinct JobId, WorkItemId, Message, Exception, StackTrace, Method, Type
| join kind=inner (cluster('engsrvprod.kusto.windows.net').database('engineeringdata').Jobs
    | where ((Branch == 'refs/heads/main') or (Branch == 'refs/heads/master') or (includePR and (Source startswith "pr/")) or (Branch startswith 'refs/heads/release/6.0'))
    | where Type startswith "test/functional/cli/"
        and not(Properties contains "runtime-staging")
    | summarize arg_max(Finished, Properties, Type, Branch, Source, Started, QueueName) by JobId
| project-rename JobType = Type) on JobId
};
failedTests('GreaterThan2GBFile_SendsAllBytes', true, '');

[karelz]

Related to #56597, dotnet/arcade#7668 and https://github.com/dotnet/core-eng/issues/13933
cc @ilyas1974 @MattGal

[lpatalas]

@karelz Thanks for letting us know. I'll take a look at that machine.

[lpatalas]

At the moment all OSX.1013 machines have at least 6GB of free space. I don't see anything unusual about any of them. We automatically offline machines than fall below 3GB so I suspect this may be the same situation as described by @MattGal here: https://github.com/dotnet/core-eng/issues/13933#issuecomment-895338577.

[antonfirsov]

Although this test is cleaning up space when done (succeeded or failed), there are 7 instances of the test case in separate classes. Running them in parallel may attempt to allocate 14 GB of disk space in extreme cases.

Since this is more a special case, than a rule, I would rather serialize the runs of the test than argue against https://github.com/dotnet/core-eng/issues/13933#issuecomment-895338577.

[karelz]

@antonfirsov serialization sounds like a good plan. Can you please put up a PR?