[release/6.0] Fix corner-case handling of cancellation exception in ForEachAsync

[github-actions]

Backport of #59065 to release/6.0

/cc @stephentoub

Customer Impact

If the body of a Parallel.ForEachAsync loop throws an OperationCanceledException using the supplied CancellationToken and it hasn't had cancellation requested, the Task returned from ForEachAsync will never complete and the calling code will effectively hang, e.g.

await Parallel.ForEachAsync(source, async (item, cancellationToken) =>
{
    ...
    throw new OperationCanceledException(cancellationToken); // when !cancellationToken.IsCancellationRequested
});

This is not a common thing to do. However, if you do happen to do it, your app will hang. The code is filtering out such exceptions, but if they're the only ones, it ends up filtering out all of them, causing the call that would have completed the returned Task to throw an exception which is then inadvertently eaten.

The fix is to change the design to simply not do such filtering: all exceptions thrown in such a case will be stored in the task.

Testing

Additional unit tests.

Risk

Relatively low.

• This is a new API in .NET 6.
• The 99.9% use case is to just await the returned task, which this won't affect, as await only throws the first exception stored in the task.
• The potential risk here is if someone is accessing the Task afterwards, they may see additional exceptions in Task.Exception.InnerExceptions that they didn't previously see.

[ghost]

I couldn't figure out the best area label to add to this PR. If you have write-permissions please help me learn by adding exactly one area label.

[danmoseley]

Didn't it rebase correctly?

[stephentoub]

Didn't it rebase correctly?

No:
https://github.com/dotnet/runtime/pull/59236/files#diff-1ea18ff65faa2ae6fed570b83747086d0317f5e4bc325064f6c14319a9c4ff67R10-R11