Running fsharpqa tests gives "parse_oracle.exe has stopped working" !?!?!

[abelbraaksma]

When I run build -testsFSharpQA on the F# repo, I get the following error:

parse_oracle.exe has stopped working

This seems to originate here (this line is weird, there's a looooot of whitespace and then some more commands):

fsharp/tests/fsharpqa/Source/Libraries/Portable/env.lst

Line 4 in fead0aa

NoMT SOURCE=parse_tests.fs SCFLAGS="--standalone -g -a" POSTCMD="\$CSC_PIPE /debug+ /r:parse_tests.dll /r:System.Numerics.dll parse_oracle.cs && parse_oracle.exe" # parse_tests.fs (Desktop)

In the output, I see this after I click Close:

<snip />
Warnings (FS0988AtEndOfFile.fs) -- passed
Warnings (Repro1548.fs) -- passed
Libraries\Portable (consumer.fs (Desktop)) -- passed
Libraries\Portable (parse_tests.fs (Desktop)) -- failed
<snip />

This is on a Windows box, and considering that it is in the "Portable" directory, I don't know if this test should run at all on my machine. There's an underlying assertion error as well:

---------------------------
Assertion Failed: Abort=Quit, Retry=Debug, Ignore=Continue
---------------------------
Unexpected exception seen in compiler: parse error: unexpected end of file
System.Exception: parse error: unexpected end of file
   at Internal.Utilities.Text.Parsing.Implementation.interpret[tok](Tables`1 tables, FSharpFunc`2 lexer, LexBuffer`1 lexbuf, Int32 initialState)
   at Internal.Utilities.Text.Parsing.Tables`1.Interpret(FSharpFunc`2 lexer, LexBuffer`1 lexbuf, Int32 startState)
   at FSharp.Compiler.Parser.implementationFile(FSharpFunc`2 lexer, LexBuffer`1 lexbuf)
   at FSharp.Compiler.CompileOps.ParseInput(FSharpFunc`2 lexer, ErrorLogger errorLogger, LexBuffer`1 lexbuf, FSharpOption`1 defaultNamespace, String filename, Tuple`2 isLastCompiland)
   at FSharp.Compiler.CompileOps.input@3545-3.Invoke(LexBuffer`1 lexbuf)
   at FSharp.Compiler.Lexhelp.reusingLexbufForParsing[a](LexBuffer`1 lexbuf, FSharpFunc`2 f)

   at FSharp.Compiler.CompileOps.OutputExceptionR@620.Invoke(StringBuilder os, Exception error)
   at FSharp.Compiler.CompileOps.OutputPhasedErrorR(StringBuilder os, PhasedDiagnostic err, Boolean canSuggestNames)
   at FSharp.Compiler.CompileOps.OutputPhasedDiagnostic(StringBuilder os, PhasedDiagnostic err, Boolean flattenErrors, Boolean suggestNames)
   at FSharp.Compiler.CompileOps.report@1781-1.Invoke(PhasedDiagnostic err)
   at FSharp.Compiler.CompileOps.CollectDiagnostic(String implicitIncludeDir, Boolean showFullPaths, Boolean flattenErrors, ErrorStyle errorStyle, Boolean warning, PhasedDiagnostic err, Boolean suggestNames)
   at FSharp.Compiler.Driver.get_Provider@142-1.HandleIssue(TcConfigBuilder tcConfigBuilder, PhasedDiagnostic err, Boolean isError)
   at FSharp.Compiler.Driver.ErrorLoggerUpToMaxErrors.DiagnosticSink(PhasedDiagnostic err, Boolean isError)
   at FSharp.Compiler.ErrorLogger.ErrorLoggerExtensions.ErrorLogger.ErrorRecovery(ErrorLogger x, Exception exn, range m)
   at FSharp.Compiler.ErrorLogger.errorRecovery(Exception exn, range m)
   at FSharp.Compiler.CompileOps.ParseOneInputLexbuf(TcConfig tcConfig, LexResourceManager lexResourceManager, FSharpList`1 conditionalCompilationDefines, LexBuffer`1 lexbuf, String filename, Tuple`2 isLastCompiland, ErrorLogger errorLogger)
   at FSharp.Compiler.CompileOps.ParseOneInputFile(TcConfig tcConfig, LexResourceManager lexResourceManager, FSharpList`1 conditionalCompilationDefines, String filename, Tuple`2 isLastCompiland, ErrorLogger errorLogger, Boolean retryLocked)
   at FSharp.Compiler.Driver.inputs@1845.Invoke(Tuple`2 tupledArg)
   at Microsoft.FSharp.Primitives.Basics.List.choose[T,TResult](FSharpFunc`2 f, FSharpList`1 xs)
   at Microsoft.FSharp.Collections.ListModule.Choose[T,TResult](FSharpFunc`2 chooser, FSharpList`1 list)
   at FSharp.Compiler.Driver.main0[a](a ctok, String[] argv, Resolver legacyReferenceResolver, Boolean bannerAlreadyPrinted, ReduceMemoryFlag reduceMemoryUsage, CopyFSharpCoreFlag defaultCopyFSharpCore, Exiter exiter, ErrorLoggerProvider errorLoggerProvider, DisposablesTracker disposables)
   at FSharp.Compiler.Driver.typecheckAndCompile(CompilationThreadToken ctok, String[] argv, Resolver legacyReferenceResolver, Boolean bannerAlreadyPrinted, ReduceMemoryFlag reduceMemoryUsage, CopyFSharpCoreFlag defaultCopyFSharpCore, Exiter exiter, ErrorLoggerProvider loggerProvider, FSharpOption`1 tcImportsCapture, FSharpOption`1 dynamicAssemblyCreator)
   at Legacy.FSharp.Compiler.Hosted.InProcCompiler.Compile(String[] argv)
   at Legacy.FSharp.Compiler.Hosted.FscCompiler.Compile(String[] args)
   at Legacy.FSharp.Compiler.Hosted.CompilerHelpers.fscCompile(Resolver legacyReferenceResolver, String directory, String[] args)
   at <StartupCode$HostedCompilerServer>.$Program.clo@34.Invoke(Unit unitVar0)
   at MLang.Test.HostedCompilerServer.processMessage(String message)
   at <StartupCode$HostedCompilerServer>.$Program.loop@83T.Invoke(Unit _arg1)
   at MLang.Test.HostedCompilerServer.Run()
   at MLang.Test.Program.main(String[] argv)

Repro steps

Just run build -testsFSharpQA.

Expected behavior

I expect some tests to fail here, but didn't expect this ;).

Actual behavior

See above.

Known workarounds

n/a

Related information

I'm trying to fix certain errors in my PR #9549, and I believe they are in what are colloquially called the "baseline tests", I figured I can run them in isolation by running with -testsFSharpQA. Input is welcome as to whether I'm just doing something stupid here ("don't run these tests on Windows!!!") and/or how to run the failing tests from that PR.

[abelbraaksma]

Btw, debugging the error (by clicking Retry) first gives this (expected):

And then, when opening in new instance of VS 2019, gives:

And the same happens if I try to debug parse_oracle.exe.

If I click Debug again, it crashes Visual Studio again... Eternal debugging loop :P

I seem to be able to break the loop if I open the next debugger not in RTM VS 2019, but in VS 2019 Preview, which gives:

Not sure what to make of all this...

[cartermp]

I'm not aware of any way to run this old test suite outside of the CI environment. This is one of the reasons why we want to migrate them.

[abelbraaksma]

@cartermp, thanks, I didn't know they couldn't run locally, it's part of my limited understanding of certain tests and test sets (the contributor page just says that you should be able to run the tests from the command line).

If this is part of the ongoing larger effort of the test migration, than we can disregard this.

Though I found it certainly interesting that, when you try to debug, this crashes VS in a loop, from crash to crash to crash... ;)

[abelbraaksma]

Let's cross this one off the list, no need to make this one working.

[abelbraaksma]

I'm not aware of any way to run this old test suite outside of the CI environment.

@cartermp, as I've found out today, there is a way to test run the fsharpqa tests locally, and this may help others as well when they run into the same issues. The following command, from the root repo dir, works. Note that -c Release is required and -ci -bl as well:

build -testFSharpQA -ci -bl -c Release

If all goes well, this ends with:

3040 tests passed (100.00%)
0 tests failed (0.00%)
0 tests were cascaded failures (0.00%)
0 tests returned "no result" (0.00%)
0 tests timed out (0.00%)
0 tests had test errors (0.00%)
========
3040 Total executed tests

16 tests were skipped
========
3056 Total tests (executed & skipped)

15m 56s Total time

Or, in case of error, you can find the error in artifacts\TestResults\Release\test-net40-fsharpqa-errors.log.