Test failure System.Net.Quic.Tests.QuicStreamTests.WriteCanceled_NextWriteThrows

[v-wenyuxu]

Failed in: runtime-coreclr libraries-pgo 20230528.1

Failed tests:

net8.0-linux-Release-x64-CoreCLR_checked-defaultpgo-Ubuntu.1804.Amd64.Open
    - System.Net.Quic.Tests.QuicStreamTests.WriteCanceled_NextWriteThrows

Error message:

System.AggregateException : One or more errors occurred. (Assert.Throws() Failure
Expected: typeof(System.Net.Quic.QuicException)
Actual:   (No exception was thrown)) (Assert.Throws() Failure
Expected: typeof(System.Net.Quic.QuicException)
Actual:   (No exception was thrown))
---- Assert.Throws() Failure
Expected: typeof(System.Net.Quic.QuicException)
Actual:   (No exception was thrown)
---- Assert.Throws() Failure
Expected: typeof(System.Net.Quic.QuicException)
Actual:   (No exception was thrown)

Stack trace:

   at System.Threading.Tasks.TaskTimeoutExtensions.WhenAllOrAnyFailed(Task[] tasks) in /_/src/libraries/Common/tests/System/Threading/Tasks/TaskTimeoutExtensions.cs:line 88
   at System.Threading.Tasks.TaskTimeoutExtensions.WhenAllOrAnyFailed(Task[] tasks, Int32 millisecondsTimeout) in /_/src/libraries/Common/tests/System/Threading/Tasks/TaskTimeoutExtensions.cs:line 55
   at System.Net.Quic.Tests.QuicTestBase.RunClientServer(Func`2 clientFunction, Func`2 serverFunction, Int32 iterations, Int32 millisecondsTimeout, QuicListenerOptions listenerOptions) in /_/src/libraries/System.Net.Quic/tests/FunctionalTests/QuicTestBase.cs:line 294
   at System.Net.Quic.Tests.QuicTestBase.RunClientServer(Func`2 clientFunction, Func`2 serverFunction, Int32 iterations, Int32 millisecondsTimeout, QuicListenerOptions listenerOptions) in /_/src/libraries/System.Net.Quic/tests/FunctionalTests/QuicTestBase.cs:line 324
   at System.Net.Quic.Tests.QuicTestBase.RunClientServer(Func`2 clientFunction, Func`2 serverFunction, Int32 iterations, Int32 millisecondsTimeout, QuicListenerOptions listenerOptions) in /_/src/libraries/System.Net.Quic/tests/FunctionalTests/QuicTestBase.cs:line 324
   at System.Net.Quic.Tests.QuicTestBase.RunClientServer(Func`2 clientFunction, Func`2 serverFunction, Int32 iterations, Int32 millisecondsTimeout, QuicListenerOptions listenerOptions) in /_/src/libraries/System.Net.Quic/tests/FunctionalTests/QuicTestBase.cs:line 288
   at System.Net.Quic.Tests.QuicStreamTests.WriteCanceled_NextWriteThrows() in /_/src/libraries/System.Net.Quic/tests/FunctionalTests/QuicStreamTests.cs:line 586
--- End of stack trace from previous location ---
----- Inner Stack Trace #1 (Xunit.Sdk.ThrowsException) -----
   at System.Net.Quic.Tests.QuicTestBase.AssertThrowsQuicExceptionAsync(QuicError expectedError, Func`1 testCode) in /_/src/libraries/System.Net.Quic/tests/FunctionalTests/QuicTestBase.cs:line 86
   at System.Net.Quic.Tests.QuicStreamTests.<WriteCanceled_NextWriteThrows>b__24_1(QuicConnection connection) in /_/src/libraries/System.Net.Quic/tests/FunctionalTests/QuicStreamTests.cs:line 625
   at System.Net.Quic.Tests.QuicStreamTests.<WriteCanceled_NextWriteThrows>b__24_1(QuicConnection connection) in /_/src/libraries/System.Net.Quic/tests/FunctionalTests/QuicStreamTests.cs:line 626
   at System.Net.Quic.Tests.QuicTestBase.<>c__DisplayClass39_1.<<RunClientServer>b__0>d.MoveNext() in /_/src/libraries/System.Net.Quic/tests/FunctionalTests/QuicTestBase.cs:line 298
--- End of stack trace from previous location ---
   at System.Threading.Tasks.TaskTimeoutExtensions.GetRealException(Task task) in /_/src/libraries/Common/tests/System/Threading/Tasks/TaskTimeoutExtensions.cs:line 120
----- Inner Stack Trace #2 (Xunit.Sdk.ThrowsException) -----
   at System.Net.Quic.Tests.QuicTestBase.AssertThrowsQuicExceptionAsync(QuicError expectedError, Func`1 testCode) in /_/src/libraries/System.Net.Quic/tests/FunctionalTests/QuicTestBase.cs:line 86
   at System.Net.Quic.Tests.QuicStreamTests.<WriteCanceled_NextWriteThrows>b__24_0(QuicConnection connection) in /_/src/libraries/System.Net.Quic/tests/FunctionalTests/QuicStreamTests.cs:line 606
   at System.Net.Quic.Tests.QuicStreamTests.<WriteCanceled_NextWriteThrows>b__24_0(QuicConnection connection) in /_/src/libraries/System.Net.Quic/tests/FunctionalTests/QuicStreamTests.cs:line 606
   at System.Net.Quic.Tests.QuicTestBase.<>c__DisplayClass39_1.<<RunClientServer>b__1>d.MoveNext() in /_/src/libraries/System.Net.Quic/tests/FunctionalTests/QuicTestBase.cs:line 304
--- End of stack trace from previous location ---
   at System.Threading.Tasks.TaskTimeoutExtensions.GetRealException(Task task) in /_/src/libraries/Common/tests/System/Threading/Tasks/TaskTimeoutExtensions.cs:line 120

Queued	Pipeline	Pipeline_Configuration	OS	Arch	Test	Outcome	Properties	BuildReason
2023-03-30T08:34:47.268Z	runtime	Release	ubuntu.1804.armarch.open	arm64	System.Net.Quic.Tests.QuicStreamTests.WriteCanceled_NextWriteThrows	Failed	{"System.PhaseName":"libraries_test_run_checked_coreclr_linux_arm64_Release","Project":"public","System.StageAttempt":"1","System.JobName":"__default","BuildNumber":"20230330.4","System.StageName":"Build","System.PhaseAttempt":"1","System.JobId":"90b83595-294f-5012-aa73-2e7af0be6394","System.JobAttempt":"1","Reason":"Schedule","AzurePipelinesTestRunId":"4172404","runtimeFlavor":"coreclr","CollectionUri":"https://dev.azure.com/dnceng-public/","operatingSystem":"(Ubuntu.1804.Arm64.Open)Ubuntu.1804.Armarch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm64v8","configuration":"Release","DefinitionId":"129","DefinitionName":"runtime","BuildId":"222367","architecture":"arm64"}	Schedule

Tagging subscribers to this area: @dotnet/ncl
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Failed in: runtime-coreclr libraries-pgo 20230528.1

Failed tests:

net8.0-linux-Release-x64-CoreCLR_checked-defaultpgo-Ubuntu.1804.Amd64.Open
    - System.Net.Quic.Tests.QuicStreamTests.WriteCanceled_NextWriteThrows

Error message:

System.AggregateException : One or more errors occurred. (Assert.Throws() Failure
Expected: typeof(System.Net.Quic.QuicException)
Actual:   (No exception was thrown)) (Assert.Throws() Failure
Expected: typeof(System.Net.Quic.QuicException)
Actual:   (No exception was thrown))
---- Assert.Throws() Failure
Expected: typeof(System.Net.Quic.QuicException)
Actual:   (No exception was thrown)
---- Assert.Throws() Failure
Expected: typeof(System.Net.Quic.QuicException)
Actual:   (No exception was thrown)

Stack trace:

   at System.Threading.Tasks.TaskTimeoutExtensions.WhenAllOrAnyFailed(Task[] tasks) in /_/src/libraries/Common/tests/System/Threading/Tasks/TaskTimeoutExtensions.cs:line 88
   at System.Threading.Tasks.TaskTimeoutExtensions.WhenAllOrAnyFailed(Task[] tasks, Int32 millisecondsTimeout) in /_/src/libraries/Common/tests/System/Threading/Tasks/TaskTimeoutExtensions.cs:line 55
   at System.Net.Quic.Tests.QuicTestBase.RunClientServer(Func`2 clientFunction, Func`2 serverFunction, Int32 iterations, Int32 millisecondsTimeout, QuicListenerOptions listenerOptions) in /_/src/libraries/System.Net.Quic/tests/FunctionalTests/QuicTestBase.cs:line 294
   at System.Net.Quic.Tests.QuicTestBase.RunClientServer(Func`2 clientFunction, Func`2 serverFunction, Int32 iterations, Int32 millisecondsTimeout, QuicListenerOptions listenerOptions) in /_/src/libraries/System.Net.Quic/tests/FunctionalTests/QuicTestBase.cs:line 324
   at System.Net.Quic.Tests.QuicTestBase.RunClientServer(Func`2 clientFunction, Func`2 serverFunction, Int32 iterations, Int32 millisecondsTimeout, QuicListenerOptions listenerOptions) in /_/src/libraries/System.Net.Quic/tests/FunctionalTests/QuicTestBase.cs:line 324
   at System.Net.Quic.Tests.QuicTestBase.RunClientServer(Func`2 clientFunction, Func`2 serverFunction, Int32 iterations, Int32 millisecondsTimeout, QuicListenerOptions listenerOptions) in /_/src/libraries/System.Net.Quic/tests/FunctionalTests/QuicTestBase.cs:line 288
   at System.Net.Quic.Tests.QuicStreamTests.WriteCanceled_NextWriteThrows() in /_/src/libraries/System.Net.Quic/tests/FunctionalTests/QuicStreamTests.cs:line 586
--- End of stack trace from previous location ---
----- Inner Stack Trace #1 (Xunit.Sdk.ThrowsException) -----
   at System.Net.Quic.Tests.QuicTestBase.AssertThrowsQuicExceptionAsync(QuicError expectedError, Func`1 testCode) in /_/src/libraries/System.Net.Quic/tests/FunctionalTests/QuicTestBase.cs:line 86
   at System.Net.Quic.Tests.QuicStreamTests.<WriteCanceled_NextWriteThrows>b__24_1(QuicConnection connection) in /_/src/libraries/System.Net.Quic/tests/FunctionalTests/QuicStreamTests.cs:line 625
   at System.Net.Quic.Tests.QuicStreamTests.<WriteCanceled_NextWriteThrows>b__24_1(QuicConnection connection) in /_/src/libraries/System.Net.Quic/tests/FunctionalTests/QuicStreamTests.cs:line 626
   at System.Net.Quic.Tests.QuicTestBase.<>c__DisplayClass39_1.<<RunClientServer>b__0>d.MoveNext() in /_/src/libraries/System.Net.Quic/tests/FunctionalTests/QuicTestBase.cs:line 298
--- End of stack trace from previous location ---
   at System.Threading.Tasks.TaskTimeoutExtensions.GetRealException(Task task) in /_/src/libraries/Common/tests/System/Threading/Tasks/TaskTimeoutExtensions.cs:line 120
----- Inner Stack Trace #2 (Xunit.Sdk.ThrowsException) -----
   at System.Net.Quic.Tests.QuicTestBase.AssertThrowsQuicExceptionAsync(QuicError expectedError, Func`1 testCode) in /_/src/libraries/System.Net.Quic/tests/FunctionalTests/QuicTestBase.cs:line 86
   at System.Net.Quic.Tests.QuicStreamTests.<WriteCanceled_NextWriteThrows>b__24_0(QuicConnection connection) in /_/src/libraries/System.Net.Quic/tests/FunctionalTests/QuicStreamTests.cs:line 606
   at System.Net.Quic.Tests.QuicStreamTests.<WriteCanceled_NextWriteThrows>b__24_0(QuicConnection connection) in /_/src/libraries/System.Net.Quic/tests/FunctionalTests/QuicStreamTests.cs:line 606
   at System.Net.Quic.Tests.QuicTestBase.<>c__DisplayClass39_1.<<RunClientServer>b__1>d.MoveNext() in /_/src/libraries/System.Net.Quic/tests/FunctionalTests/QuicTestBase.cs:line 304
--- End of stack trace from previous location ---
   at System.Threading.Tasks.TaskTimeoutExtensions.GetRealException(Task task) in /_/src/libraries/Common/tests/System/Threading/Tasks/TaskTimeoutExtensions.cs:line 120

Queued	Pipeline	Pipeline_Configuration	OS	Arch	Test	Outcome	Properties	BuildReason
2023-03-30T08:34:47.268Z	runtime	Release	ubuntu.1804.armarch.open	arm64	System.Net.Quic.Tests.QuicStreamTests.WriteCanceled_NextWriteThrows	Failed	{"System.PhaseName":"libraries_test_run_checked_coreclr_linux_arm64_Release","Project":"public","System.StageAttempt":"1","System.JobName":"__default","BuildNumber":"20230330.4","System.StageName":"Build","System.PhaseAttempt":"1","System.JobId":"90b83595-294f-5012-aa73-2e7af0be6394","System.JobAttempt":"1","Reason":"Schedule","AzurePipelinesTestRunId":"4172404","runtimeFlavor":"coreclr","CollectionUri":"https://dev.azure.com/dnceng-public/","operatingSystem":"(Ubuntu.1804.Arm64.Open)Ubuntu.1804.Armarch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm64v8","configuration":"Release","DefinitionId":"129","DefinitionName":"runtime","BuildId":"222367","architecture":"arm64"}	Schedule

Author:	v-wenyuxu
Assignees:	-
Labels:	os-linux, arch-x64, blocking-clean-ci-optional, area-System.Net.Quic
Milestone:	-

[rzikm]

The test is flaky and the failure happens on regular runs as well (but usually succeeds on reruns)

The case seems to be when the cancellation happens to occur before the call to WriteAsync (and we call it with pre-cancelled token).

runtime/src/libraries/System.Net.Quic/tests/FunctionalTests/QuicStreamTests.cs

Lines 591 to 606 in 844702d

	CancellationTokenSource cts = new CancellationTokenSource(500);
	async Task WriteUntilCanceled()
	{
	var buffer = new byte[64 * 1024];
	while (true)
	{
	await stream.WriteAsync(buffer, cancellationToken: cts.Token);
	}
	}
	// a write would eventually be canceled
	await Assert.ThrowsAsync<OperationCanceledException>(() => WriteUntilCanceled().WaitAsync(TimeSpan.FromSeconds(3)));
	// next write would also throw
	await AssertThrowsQuicExceptionAsync(QuicError.OperationAborted, () => stream.WriteAsync(new byte[1]).AsTask());

We should have a look someday.

[rzikm]

Closing in favor of #76831