tag filtering ignores "failed to init" test classes

[shrey-ghost]

Describe the bug
If a scala test class fails to be constructed (for e.g., a NullPointerException), it is ignored when tests are filtered with junit tags, specifically, the flag includeTags. This should instead be reported as a failure (similar behavior to what happens without a tag filter)

To Reproduce
Steps to reproduce the behavior:

1. I created a sample project with the exact same failure https://github.com/shrey-ghost/scala-tag-filter-failure-demo
2. Checkout the above project and run ./gradlew test. This will fail because 2/4 test classes have an initialization error (NullPointerException)
3. Run the tests with the tag included. 2/4 test classes are annotated with @MyNewTag. The command to run the filtered tests is ./gradlew test -DincludeTags=tags.MyNewTag. This command will pass. However, this should have failed because the class foo.bar.InvalidTestWithTag has a NullPointerException during construction and also has the tag @MyNewTag annotated.

Expected behavior
Ideally, the last step above should have failed. In the current implementation, the "failed to init" test class is just skipped. However, the failed to init class should also have run and reported a failure.

I looked at the code and I think I know where the bug is. The original scala test class can be wrapped in one of these two descriptors

1. ScalatestTestDescriptor - which includes the tags from the original test class. This is used when the construction of the test class succeeds.
2. ScalatestFailedInitDescriptor - which does NOT include the tags from the original test class. This is used when the construction of the test class fails.

Since the ScalatestFailedInitDescriptor does not have tags (returns empty tags from the super class), any classes that failed to construct, will never pass the includeTags filtering done by JUnit. I think the ScalatestFailedInitDescriptor should also have the tags from the actual scala test class just like ScalatestTestDescriptor

Screenshots

1. Failures when I run all the tests

➜ scala-tag-filter-failure-demo git:(main) ./gradlew test
Task :test

UnknownClass.Failed to load test from foo.bar.InvalidTestWithTag: 'null' FAILED
java.lang.NullPointerException
at foo.bar.InvalidTestWithTag.(InvalidTestWithTag.scala:16)
at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
........

UnknownClass.Failed to load test from foo.bar.InvalidTestWithoutTag: 'null' FAILED
java.lang.NullPointerException
at foo.bar.InvalidTestWithoutTag.(InvalidTestWithoutTag.scala:14)
at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
........

results: FAILURE (4 tests, 2 successes, 2 failures, 0 skipped) Report file: ***/scala-tag-filter-failure-demo/build/reports/tests/test/index.html

4 tests completed, 2 failed
There were failing tests. See the report at: file://***/scala-tag-filter-failure-demo/build/reports/tests/test/index.html

2. The task passed when I run with includeTags

➜ scala-tag-filter-failure-demo git:(main) ./gradlew test -DincludeTags=tags.MyNewTag
Task :test

Gradle Test Executor 25 STANDARD_ERROR
Apr 14, 2022 12:38:57 PM org.junit.platform.launcher.core.EngineDiscoveryOrchestrator lambda$logTestDescriptorExclusionReasons$7
INFO: 0 containers and 3 tests were excluded because tags do not match tag expression(s): [tags.MyNewTag]
results: SUCCESS (1 tests, 1 successes, 0 failures, 0 skipped) Report file: /Users/shrey/github/scala-tag-filter-failure-demo/build/reports/tests/test/index.html

BUILD SUCCESSFUL in 1s
3 actionable tasks: 1 executed, 2 up-to-date

We can clearly see (INFO line) that 3 test classes were skipped because they do not match tag expression. Ideally, only two should have been skipped and we should have run two test classes (out of which one will pass and one will fail)

Build/Test environment:

• Gradle 7.1
• Scala 2.12.x
• Scalatest 2.12:3.0.8
• junit 1.7.1
• full details in the build.gradle file of the repro project

Let me know if you guys have the time to work on this or else I am happy to provide a patch for this (using the bug description I put above)

[giurim]

Hi, thank you very much for the detailed report, I will release a fixed version in about a week.

[shrey-ghost]

Thanks a lot!

[shrey-ghost]

Hey @giurim, any update on this ?

[giurim]

Hi @shrey-ghost,

I apologise for the delay I was traveling and the fix also needed some research (if the class is not initialized I have no Suit object to get the tags from) but it is out as version 0.1.11 The OSS repository/Maven Central needs some time to index new releases tho.

Thank you again for the report! In my opinion class initialization errors are the weakest part of the Junit 5 systems so far.

[shrey-ghost]

Thanks a lot for the fix!