build.cmd -testFSharpCore for .NET Framework throws: System.InvalidCastException : Unable to cast object of type 'Arrow@249-1[Int32,FSharpFunc'2[FsCheck.Property]]' to type 'Testable'1[FSharpFunc'2......

[abelbraaksma]

Running build.cmd -testFSharpCore gives a bunch of the following exceptions:

System.InvalidCastException : Unable to cast object of type 'Arrow@249-1[System.Int32,Microsoft.FSharp.Core.FSharpFunc2[System.Int32,FsCheck.Property]]' to type 'Testable1[Microsoft.FSharp.Core.FSharpFunc2[System.Int32,Microsoft.FSharp.Core.FSharpFunc2[System.Int32,FsCheck.Property]]]'.

Repro steps

On a Windows machine with a fresh checkout of the current main branch of FSharp:

• run git clean -xdf
• run build.cmd -testFSharpCore

Expected behavior

No errors expected

Actual behavior

A few dozen of the follow errors appear (they are all InvalidCastException and all have something like Arrow@999 in the name, suggesting it's failing a cast of a compiler-generated function type).

It turns out that every test that uses Check.QuickThrowOnFailure exhibits this behavior. These errors muddled the analysis of #13557. After a long while I finally realized this only happened on .NET Framework 4.72 tests. The normal net6.0 run didn't raise this exception.

Why CI doesn't also expose these issues, I don't know. I can only imagine that there are slight differences in the systems leading to a different JIT, leading to a (slightly) different type resolution. But I'm just theorizing.

 FSharp.Core.UnitTests.DiscriminatedUnionTypes+UnionsFSCheckTests.struct unions are comparable
   Source: DiscriminatedUnionType.fs line 141
   Duration: 96 ms

  Message: 
    System.InvalidCastException : Unable to cast object of type 'Arrow@249-1[System.Int32,Microsoft.FSharp.Core.FSharpFunc`2[System.Int32,FsCheck.Property]]' to type 'Testable`1[Microsoft.FSharp.Core.FSharpFunc`2[System.Int32,Microsoft.FSharp.Core.FSharpFunc`2[System.Int32,FsCheck.Property]]]'.

  Stack Trace: 
    IntrinsicFunctions.UnboxGeneric[T](Object source)
    Runner.check[a](Config config, a p)
    Check.QuickThrowOnFailure[Testable](Testable property)

Example of a test that fails:

[<Fact>]
member _.``struct unions are comparable`` () =
    Check.QuickThrowOnFailure <|
    fun (i1:int) (i2:int) ->
        i1 <> i2 ==>
        let sr1 = SU (i1, i2)
        let sr2 = SU (i1, i2)
        let sr3 = SU (i2, i1)
        (sr1 = sr2)                    |@ "sr1 = sr2" .&.
        (sr1 <> sr3)                    |@ "sr1 <> sr3" .&.
        (sr1.Equals sr2)               |@ "sr1.Equals sr2"

Known workarounds

Don't run tests. Just commit and wait a few hours for CI, which isn't really a workaround, but was my go-to method for a while.

Cause

After many sessions on Slack with @vzarytovskii (thanks!) and today writing a new analysis to him about what I had found so far, I noticed that the FsCheck library, while a 3.0 alpha release, is 4 years old!.

I already saw a bunch of these errors, for instance: microsoft/fsharplu#14. But I dismissed them as I figured "hey, we have a 3.0x version, must be hot of the press!".

But noooooo 😮

Still, I have no idea why these tests pass on other people's machines, or why CI passes them. Anyway, testing one by one showed:

Version	Release	Used by FSharp	Exhibits this problem
3.0.0-beta2	2022-01		Probably not*
3.0.0-beta1	2021-09		Probably not*
3.0.0-alpha5	2020-12		No
3.0.0-alpha4	2018-06	YES	YES
3.0.0-alpha3	2017-12		YES
3.0.0-alpha2	2017-10		No
3.0.0-alpha1	2017-09		No
2.16.5	2022-06		No**

* I couldn't use this because these are not in the NuGet sources used by FSharp, they seem to be quite far behind with only having the 2020 version
** I only tested this stable version with the FSharp.Core tests for .NET Framework, it works

Proposal

• Create some mechanism that we can clearly see what platform (Framework/Core) a test is failing for. Unless you suspect this, it is very hard to find. Just knowing that all tests are run twice would already be helpful (and I should've known!), I guess.
• Upgrade FsCheck to at least 3.0.0-alpha5, or downgrade to 2.16.5 until the 3.x versions become stable. A lot of 3.x features have been backported into 2.x anyway.