X509Certificates.Tests: Non-serializable data ('System.Object[]')

[vcsjones]

Running tests in X509Certificates.Tests gives a few test warnings, I think uncovered when #35285 was merged:

[xUnit.net 00:00:00.72] System.Security.Cryptography.X509Certificates.Tests: Non-serializable data ('System.Object[]') found for 'System.Security.Cryptography.X509Certificates.Test
s.X500DistinguishedNameEncodingTests.EncodeSingleRdn'; falling back to single test case.
[xUnit.net 00:00:00.72] System.Security.Cryptography.X509Certificates.Tests: Non-serializable data ('System.Object[]') found for 'System.Security.Cryptography.X509Certificates.Test
s.X500DistinguishedNameEncodingTests.EncodeWithFlags'; falling back to single test case.
[xUnit.net 00:00:00.72] System.Security.Cryptography.X509Certificates.Tests: Non-serializable data ('System.Object[]') found for 'System.Security.Cryptography.X509Certificates.Test
s.X500DistinguishedNameEncodingTests.CheckParserBoundaryCases'; falling back to single test case.
[xUnit.net 00:00:00.80] System.Security.Cryptography.X509Certificates.Tests: Non-serializable data ('System.Object[]') found for 'System.Security.Cryptography.X509Certificates.Test
s.CertificateCreation.ECDsaX509SignatureGeneratorTests.PublicKeyEncoding'; falling back to single test case.

This leads me to believe that there are theories which, when MemberData returns nonserializable data.

What is the impact of this? Does this mean only a single test case is being run and coverage isn't happening, or does it mean that it "looks" like a single test but is actually running all of the cases?

/cc @ViktorHofer

Tagging subscribers to this area: @bartonjs, @vcsjones, @krwq
Notify danmosemsft if you want to be subscribed.

[ViktorHofer]

See #30324 for context. No easy way to disable these messages besides disabling diagnostic messages or changing the log verbosity to quiet.

Tagging subscribers to this area: @safern, @ViktorHofer
Notify danmosemsft if you want to be subscribed.

[vcsjones]

@ViktorHofer thanks; I missed that issue. If you'd prefer to close this as a dupe of that one; I think that is a reasonable thing to do.

Should we be fixing tests so that member data uses serializable types or waiting on a change to the build infrastructure?

[ViktorHofer]

If my memory serves me right, @stephentoub was against fixing all these cases as there would be thousands of such and it would bloat the test code. Actually I attempted doing that when I saw the warnings: dotnet/arcade#3334. I'm not sure what the best option going forward is...

[bartonjs]

I'm not sure what it's complaining about. The cases print correctly in the XML output and the CLI output, and definitely work (or, at least all of this was true when the tests were written...).

Maybe the VS runner serializes the data and therefore it needs to be serializable in order to print nicely in the VS test explorer? Dunno if that means recent infrastructure changes dropped these tests, or not. Easiest answer is to break one of the expected values and see if it fails 😄.

[ViktorHofer]

Yes this is purely for VS Test Explorer support to run/debug single theory test cases. It doesn't affect execution itself.

[vcsjones]

@bartonjs it seems to work. In X509Certificates there are only a handful of these messages. In S.S.C.Algorithms there is approximately 100 of them.

There are other messages in S.S.C.Algorithms that might be worth looking in to, like:

Skipping test case with duplicate ID '658dd5afec16f3397ac5ef6b298eb8b71ae5a86f' ('System.Security.Cryptograph
y.Hashing.Algorithms.Tests.BlockSizeValueTests.BlockSizeValueTest(hmacBlockSizeValue: 64, expectedBlockSizeValue: 64)' and 'System.Security.Cryptography.Hashing.Algorithms.Tests.BlockSizeValueTests.BlockSizeValueTest(hmacBlockSizeValue: 64, expectedBlockSizeValue: 64)')

Perhaps we aren't permuting something correctly.

[stephentoub]

After #35837, these warnings should go away.

[vcsjones]

Okay, thanks @stephentoub. I think then it makes sense to close this.