Test framework fails to report errors

[odersky]

Compiler version

3.4.1-RC1

Minimized example

Given in #19538.

Output

scala3-boostrapped/test passes.

Expectation

The tests should fail since there are two errors in neg tests that fail. They get reported, but don't show in the test summary.

[info] Test dotty.tools.dotc.CompilationTests.negAll started

Wrong number of errors encountered when compiling tests/neg/i19334.scala
expected: 2, actual: 1
Unfulfilled expectations:
tests/neg/i19334.scala:5
Unexpected errors:

-> following the errors:
 at 6: Missing parameter type

I could not infer the type of the parameter _$1
in expanded function:
  _$1 => f(_$1)
Expected type for the whole anonymous function:
  T

where:    T is a type variable

Wrong number of errors encountered when compiling tests/neg/i19506.scala
expected: 2, actual: 1
Unfulfilled expectations:
tests/neg/i19506.scala:8
tests/neg/i19506.scala:7
Unexpected errors:

-> following the errors:
 at 0: 3.4-migration, is not a valid choice for -source
Output from 'tests/neg/i19470.scala' did not match check file. Actual output:
-- [E007] Type Mismatch Error: tests/neg/i19470.scala:9:12 -------------------------------------------------------------
9 |    List(foo(f())) // error
         ^^^^^^^^
         Found:    Inv[? >: IO <: box IO^{f}]
         Required: box Inv[box IO^?]^?

 longer explanation available when compiling with `-explain`


Test output dumped in: tests/neg/i19470.check.out
  See diff of the checkfile (`--color=always` for colored diff)
    > git diff --no-index -- tests/neg/i19470.check tests/neg/i19470.check.out
  Replace checkfile with current output
    > mv tests/neg/i19470.check.out tests/neg/i19470.check

I also noted that in the output on the terminal error counts oscillate for other kinds of tests as well. I.e. pickling would jump from 0 to 1 and back. But in the end nothing gets reported.

It looks like a bunch of race conditions that were introduced recently to the testing framework. I am very surprised that I am the first to notice this.

This is a serious problem that needs to be treated at the highest priority. Without a reliable testing framework we cannot make progress anywhere.

[nicolasstucki]

I could reproduce it with the sligtly faster command

sbt:scala3> testOnly dotty.tools.dotc.CompilationTests -- *negAll

...

[info] Test run dotty.tools.dotc.CompilationTests finished: 2 failed, 0 ignored, 1 total, 50.822s
[info] Passed: Total 1, Failed 0, Errors 0, Passed 1
[success] Total time: 51 s, completed 29 Jan 2024, 10:49:54

Also with testCompilation tests/neg/i

[nicolasstucki]

It seem that this issue started showing up with 756ae34. This change is probably making it easier to trigger another issue.

[nicolasstucki]

Minimal reproduction

sbt:scala3> testCompilation tests/neg/i8984.scala
[info] Test run dotty.tools.dotc.CompilationTests started
[info] Test dotty.tools.dotc.CompilationTests.parallelBackend ignored
[info] Test dotty.tools.dotc.CompilationTests.negAll started
[                                        ] completed (0/1, 0 failed, 0s)[info] Test dotty.tools.dotc.CompilationTests.runAll started
No files matched "tests/neg/i8984.scala" in test                                
[info] Test dotty.tools.dotc.CompilationTests.pickling started
[                                        ] completed (0/1, 1 failed, 1s)No files matched "tests/neg/i8984.scala" in test                                
[                                        ] completed (0/1, 1 failed, 1s)[info] Test dotty.tools.dotc.CompilationTests.rewrites started
No files matched "tests/neg/i8984.scala" in test                                
[info] Test dotty.tools.dotc.CompilationTests.pos started
[                                        ] completed (0/1, 1 failed, 1s)No files matched "tests/neg/i8984.scala" in test                                
[info] Test dotty.tools.dotc.CompilationTests.warn started
No files matched "tests/neg/i8984.scala" in test                                
[info] Test dotty.tools.dotc.CompilationTests.checkInit started
No files matched "tests/neg/i8984.scala" in test                                
No files matched "tests/neg/i8984.scala" in test                                
No files matched "tests/neg/i8984.scala" in test                                
No files matched "tests/neg/i8984.scala" in test                                
No files matched "tests/neg/i8984.scala" in test                                
No files matched "tests/neg/i8984.scala" in test                                
No files matched "tests/neg/i8984.scala" in test                                
[info] Test dotty.tools.dotc.CompilationTests.posTwice started
No files matched "tests/neg/i8984.scala" in test                                
[info] Test dotty.tools.dotc.CompilationTests.explicitNullsNeg started
No files matched "tests/neg/i8984.scala" in test                                
[info] Test dotty.tools.dotc.CompilationTests.explicitNullsPos started
No files matched "tests/neg/i8984.scala" in test                                
[info] Test dotty.tools.dotc.CompilationTests.explicitNullsRun started
[                                        ] completed (0/1, 1 failed, 1s)No files matched "tests/neg/i8984.scala" in test                                
[info] Test dotty.tools.dotc.CompilationTests.checkInitGlobal started
No files matched "tests/neg/i8984.scala" in test                                
No files matched "tests/neg/i8984.scala" in test                                
No files matched "tests/neg/i8984.scala" in test                                
[info] Test dotty.tools.dotc.CompilationTests.fuzzyAll started
No files matched "tests/neg/i8984.scala" in test                                
[info] Test dotty.tools.dotc.CompilationTests.genericJavaSignatures started
No files matched "tests/neg/i8984.scala" in test                                

================================================================================
Test Report
================================================================================

0 suites passed, 0 failed, 0 total
[info] Test run dotty.tools.dotc.CompilationTests finished: 2 failed, 1 ignored, 14 total, 2.115s
[info] Test run dotty.tools.dotc.BootstrappedOnlyCompilationTests started
[info] Test run dotty.tools.dotc.BootstrappedOnlyCompilationTests finished: 0 failed, 0 ignored, 0 total, 0.0s
[info] Test run dotty.tools.dotc.coverage.CoverageTests started
[info] Test run dotty.tools.dotc.coverage.CoverageTests finished: 0 failed, 0 ignored, 0 total, 0.0s
[info] Passed: Total 14, Failed 0, Errors 0, Passed 14, Ignored 1
[success] Total time: 3 s, completed 29 Jan 2024, 13:31:23

Note the line

[info] Test run dotty.tools.dotc.CompilationTests finished: 2 failed, 1 ignored, 14 total, 2.115s

[nicolasstucki]

I have no idea what is causing it yet. I suspect that the issue is in the reporter.

We can patch the current failure on main temporarely if we delete tests/neg/i8984.scala.

[nicolasstucki]

As far as I can tell, this only affects CompilationTests and BootstrappedOnlyCompilationTests

[som-snytt]

I noticed "-source:3", is taken shell-wise as -source:3, (an error) but -Wunused:all, is -Wunused:all because

"all,".split(",").length == 1

there is no trailing empty string in the result. Yes, I'm a total noob.

[nicolasstucki]

The issue is that one of the reporters throws an exception. Furthermore, that exception throws an exception on getMessage.

[nicolasstucki]

To close this issue we need to

• Make sure that the test frameworks catches exceptions while reporting: Make tests resilient to exceptions thrown while reporting exceptions #19566
• Reenable tests/neg/i8984.scala/tests/neg/i8984.check (disabled in Workaround #19542 #19561) and fix the exception that they throw while reporting the error (see Make tests resilient to exceptions thrown while reporting exceptions #19566 (comment))

[som-snytt]

Relieved it was not InterruptedException handling.