[dotnet test for MTP - Retry] Consider only re-tried test nodes when subtracting failures

[Youssef1313]

It occurred to mind that we may (or a third-party retry extension could) have a feature to consider only specific tests for retry, or exclude specific tests. In that case, the current logic won't be correct and may consider the test run as "passed" when this may not be the reality.

In the subsequent re-tries, we should subtract the results of the retries we receive only, and not the full failures.

Related to #48075

[Youssef1313]

@nohwnd Does that make sense?

I was thinking of the possibility for us, or a third-party retry extension, to implement something like --retry-filter that would limit what tests should be considered for retry.

Let me know if you want to take this up, or want me to do it.

[nohwnd]

Does not seem like a bug to me. There are more assumptions about how retry works, and it need to be specified in the protocol between mtp and dotnet test first, for us to avoid having those assumptions.

[Youssef1313]

IMO we should be at least reducing assumptions as much as possible. The current implementation will be blocking extension authors, or us, from adding some sort of --retry-filter. Fixing it in the future will basically mean that such extension will not be compatible with shipped SDKs, and we will in future need to change SDK. I'd personally prefer to do it before we ship .NET 10

[nohwnd]

Then let's specify the retry into the protocol so we can just act on the metadata in the dotnet test client. Rather than dotnet test "guessing" what is going on.

[Youssef1313]

This was partially addressed in #48797, but pending this comment:

sdk/src/Cli/dotnet/Commands/Test/TestApplicationEventHandlers.cs

Lines 67 to 71 in d93c427

	// TODO: If we got some results for ExecutionId1 and InstanceId1
	// Then we started getting ExecutionId1 and InstanceId2,
	// Then we started getting ExecutionId1 and InstanceId1 again.
	// Should we discard the last result from ExecutionId1 and InstanceId1 completely?
	// Or is it considered a violation of the protocol and should never happen? (in that case maybe we should throw?)

I'm not sure if the case I described in this comment can ever happen or not, and whether SDK should address it or not. @nohwnd @mariam-abdulla do you know please?