Test framework fails to report errors #19542
Description
Compiler version
3.4.1-RC1
Minimized example
Given in #19538.
Output
scala3-boostrapped/test passes.
Expectation
The tests should fail since there are two errors in neg tests that fail. They get reported, but don't show in the test summary.
[info] Test dotty.tools.dotc.CompilationTests.negAll started
Wrong number of errors encountered when compiling tests/neg/i19334.scala
expected: 2, actual: 1
Unfulfilled expectations:
tests/neg/i19334.scala:5
Unexpected errors:
-> following the errors:
at 6: Missing parameter type
I could not infer the type of the parameter _$1
in expanded function:
_$1 => f(_$1)
Expected type for the whole anonymous function:
T
where: T is a type variable
Wrong number of errors encountered when compiling tests/neg/i19506.scala
expected: 2, actual: 1
Unfulfilled expectations:
tests/neg/i19506.scala:8
tests/neg/i19506.scala:7
Unexpected errors:
-> following the errors:
at 0: 3.4-migration, is not a valid choice for -source
Output from 'tests/neg/i19470.scala' did not match check file. Actual output:
-- [E007] Type Mismatch Error: tests/neg/i19470.scala:9:12 -------------------------------------------------------------
9 | List(foo(f())) // error
^^^^^^^^
Found: Inv[? >: IO <: box IO^{f}]
Required: box Inv[box IO^?]^?
longer explanation available when compiling with `-explain`
Test output dumped in: tests/neg/i19470.check.out
See diff of the checkfile (`--color=always` for colored diff)
> git diff --no-index -- tests/neg/i19470.check tests/neg/i19470.check.out
Replace checkfile with current output
> mv tests/neg/i19470.check.out tests/neg/i19470.check
I also noted that in the output on the terminal error counts oscillate for other kinds of tests as well. I.e. pickling would jump from 0 to 1 and back. But in the end nothing gets reported.
It looks like a bunch of race conditions that were introduced recently to the testing framework. I am very surprised that I am the first to notice this.
This is a serious problem that needs to be treated at the highest priority. Without a reliable testing framework we cannot make progress anywhere.