[linux-arm64] Random and rare runtime crash System.ArgumentOutOfRangeException (System.Net.Sockets)

[NQ-Merinos]

Description

Random and rare crashes with this exception:

Unhandled exception. System.ArgumentOutOfRangeException: Specified argument was out of the range of valid values. (Parameter 'state')
  at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.InvokeContinuation(Action`1 continuation, Object state, Boolean forceAsync, Boolean requiresExecutionContextFlow)
  at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.OnCompleted(SocketAsyncEventArgs _)
  at System.Net.Sockets.SocketAsyncEngine.System.Threading.IThreadPoolWorkItem.Execute()
  at System.Threading.ThreadPoolWorkQueue.Dispatch()
  at System.Threading.PortableThreadPool.WorkerThread.WorkerThreadStart()

It seems to append only on loaded applications.

Exit signal: Abort (6)

Reproduction Steps

We don't have any reproduction yet. We probably need to heavily stress network! It seems to be a race condition.

Expected behavior

don't crash the runtime when we are using sockets...

Actual behavior

random and rare crashes of the runtime

Regression?

No response

Known Workarounds

No response

Configuration

Dotnet runtime version: 6.0.5 (70ae3df)
OS : GNU/Linux Debian 11 Bullseye
CPU: ARM64 Graviton 2 (AWS)
We are using Orleans with this application

Other information

No response

Tagging subscribers to this area: @dotnet/ncl
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Description

Random and rare crashes with this exception:

Unhandled exception. System.ArgumentOutOfRangeException: Specified argument was out of the range of valid values. (Parameter 'state')
  at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.InvokeContinuation(Action`1 continuation, Object state, Boolean forceAsync, Boolean requiresExecutionContextFlow)
  at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.OnCompleted(SocketAsyncEventArgs _)
  at System.Net.Sockets.SocketAsyncEngine.System.Threading.IThreadPoolWorkItem.Execute()
  at System.Threading.ThreadPoolWorkQueue.Dispatch()
  at System.Threading.PortableThreadPool.WorkerThread.WorkerThreadStart()

It seems to append only on loaded applications.

Exit signal: Abort (6)

Reproduction Steps

We don't have any reproduction yet. We probably need to heavily stress network! It seems to be a race condition.

Expected behavior

don't crash the runtime when we are using sockets...

Actual behavior

random and rare crashes of the runtime

Regression?

No response

Known Workarounds

No response

Configuration

Dotnet runtime version: 6.0.5 (70ae3df)
OS : GNU/Linux Debian 11 Bullseye
CPU: ARM64 Graviton 2 (AWS)
We are using Orleans with this application

Other information

No response

Author:	NQ-Merinos
Assignees:	-
Labels:	area-System.Net.Sockets
Milestone:	-

[stephentoub]

Is it possible you're awaiting the same ValueTask instance multiple times, using a ValueTask concurrently, etc. (all of which is considered erroneous usage)? This is most likely coming from:

runtime/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/ValueTask.cs

Lines 236 to 244 in 0c2e0ec

	if (!(state is ValueTaskSourceAsTask vtst) ||
	!(vtst._source is IValueTaskSource source))
	{
	// This could only happen if the IValueTaskSource passed the wrong state
	// or if this callback were invoked multiple times such that the state
	// was previously nulled out.
	ThrowHelper.ThrowArgumentOutOfRangeException(ExceptionArgument.state);
	return;
	}

[NQ-Brewir]

It is possible. I just spotted a point where a ValueTask is not awaited (but still stored in a _ variable). Could it lead to this error?

[stephentoub]

It is possible. I just spotted a point where a ValueTask is not awaited (but still stored in a _ variable). Could it lead to this error?

It shouldn't. Not consuming a ValueTask can still be problematic if the instance was relying on being returned to a pool as part of it, but it shouldn't lead to this error.

[karelz]

Triage: We believe this is bug in the user code -- awaiting ValueTask multiple times, most likely concurrently. Closing.
If there is evidence it is not the case, please reopen with additional info and we will take further look. Thanks!

[karelz]

Problem in .NET identified after all - duplicate of #84407

Fixed in 7.0.7 in PR #84641 and in 6.0.18 in PR #84432.
Main (8.0) is not affected - see description in #84432.